Internet Assisted Review

![](media/image1.jpeg){width="6.496527777777778in"
height="0.7423611111111111in"}

Scope Document

Version 1.03 06/20/02 5:00 PM

Revision History

  ---------- ------------- ------------------------------------- ------------------
  **Date**   **Version**   **Description of Change**             **Author**

  2/24/02    1.0           Initial Draft                         Tracy Soto

  3/7/02     1.01          Modified per IAR Focus Group comments Tracy Soto
                           (3/4/02)                              

  3/18/02    1.02          Modified per IAR Focus Group comments Tracy Soto
                           (3/14/02)                             

  3/22/02    1.03          Modified per IAR Focus Group comments Tracy Soto
                           (3/18/02)                             

  4/1/02     1.04          Modified per IAR Focus Group comments Tracy Soto
                           (3/25/02), security recommendations   
                           for registration and account creation 
                           (3/28/02)                             

  4/4/02     1.05          Modified per IAR Focus Group comments Tracy Soto
                           (4/1/02) and security recommendations 

  4/10/02    1.06          Modified per IAR Focus Group comments Tracy Soto
                           (4/8/02)                              

  4/11/02    1.061         Modified per clarification of         Tracy Soto
                           security recommendations for          
                           registration and login process        
                           (4/11/02)                             

  4/17/02    1.07          Modified per IAR Focus Group comments Tracy Soto
                           (4/15/02)                             

  4/26/02    1.08          Modified per IAR Focus Group comments Tracy Soto
                           (4/15/02)                             

  5/10/02    1.09          Modified per IAR Focus Group minutes  Tracy Soto
                           (5/6/02)                              

  5/16/02    1.09          Modified per new streamlining         Tracy Soto
                           requirements.                         

  6/5/02     1.10          Modified per IAR Focus Group comments Tracy Soto, Daniel
                           (6/3/02)                              Fox

  6/6/02     1.10          Added Dr. Sinnett's comments on       Tracy Soto
                           Version 2 streamlining and reports.   

  6/12/02    1.2           Modified per IAR Focus Group meeting  Tracy Soto
                           comments (6/10/02) and policy         
                           decision to only accept critiques in  
                           Word or text format.                  

  6/17/02    1.21          Modified to remove action items and   Tracy Soto, Daniel
                           update person address and profile     Fox
                           issues based on meeting with Sara     
                           Silver and Dan Hall. Modified based   
                           on discussion with Daniel Fox on      
                           purge and closure items related to    
                           meeting minutes.                      

  6/19/02    1.3           Modified to reflect architecture      Tracy Soto
                           review and newly proposed procedure   
                           for registration, authentication and  
                           account creation per Kalpesh Patel,   
                           Dan Hall.                             
  ---------- ------------- ------------------------------------- ------------------

Table of Contents

[1. Introduction 5](#introduction)

# [1.1 Background 5](#background) {#background-5 .TOC-2}

# [1.2 Purpose 5](#purpose) {#purpose-5 .TOC-2}

# [1.3 Definitions, Acronyms, and Abbreviations 5](#definitions-acronyms-and-abbreviations) {#definitions-acronyms-and-abbreviations-5 .TOC-2}

# [1.4 References 5](#references) {#references-5 .TOC-2}

[2. Positioning 6](#positioning)

# [2.1 Business Opportunity/Scope 6](#business-opportunityscope) {#business-opportunityscope-6 .TOC-2}

[3. Stakeholder and User Descriptions
7](#stakeholder-and-user-descriptions)

# [3.1 User Environment 7](#user-environment) {#user-environment-7 .TOC-2}

# [3.2 Stakeholder Profiles 7](#stakeholder-profiles) {#stakeholder-profiles-7 .TOC-2}

# [3.3 User Profiles 9](#user-profiles) {#user-profiles-9 .TOC-2}

[4. Product Overview 11](#product-overview)

# [4.1 Product Perspective 11](#product-perspective) {#product-perspective-11 .TOC-2}

# [4.2 Summary of Capabilities 13](#summary-of-capabilities) {#summary-of-capabilities-13 .TOC-2}

# [4.3 Assumptions and Dependencies 13](#assumptions-and-dependencies) {#assumptions-and-dependencies-13 .TOC-2}

# [4.4 Cost and Pricing 13](#cost-and-pricing) {#cost-and-pricing-13 .TOC-2}

# [4.5 Licensing and Installation 14](#licensing-and-installation) {#licensing-and-installation-14 .TOC-2}

[5. Product Features 14](#product-features)

# [5.1 System (SYS) Feature 15](#system-sys-feature) {#system-sys-feature-15 .TOC-2}

## [5.1.1 Error Handling 15](#error-handling) {#error-handling-15 .TOC-3}

## [5.1.2 Usage Reporting 15](#usage-reporting) {#usage-reporting-15 .TOC-3}

## [5.1.3 System Control 15](#system-control) {#system-control-15 .TOC-3}

## [5.1.4 Browser Interface 15](#browser-interface) {#browser-interface-15 .TOC-3}

## [5.1.5 Interface Conventions 15](#interface-conventions) {#interface-conventions-15 .TOC-3}

## [5.1.6 On-line help 15](#on-line-help) {#on-line-help-15 .TOC-3}

## [5.1.7 Release Notes 15](#release-notes) {#release-notes-15 .TOC-3}

## [5.1.8 Bug Status 15](#bug-status) {#bug-status-15 .TOC-3}

## [5.1.9 Availability 15](#availability) {#availability-15 .TOC-3}

## [5.1.10 Performance 15](#performance) {#performance-15 .TOC-3}

## [5.1.11 Auditing 15](#auditing) {#auditing-15 .TOC-3}

## [5.1.12 Exception Reporting 15](#exception-reporting) {#exception-reporting-15 .TOC-3}

## [5.1.13 External Interfaces 15](#external-interfaces) {#external-interfaces-15 .TOC-3}

# [5.2 Release Meeting to IAR / IAR Control Center 15](#release-meeting-to-iar-iar-control-center) {#release-meeting-to-iar-iar-control-center-15 .TOC-2}

# [5.3 Pre-Registration System Process 18](#pre-registration-system-process) {#pre-registration-system-process-18 .TOC-2}

# [5.4 User Registration 19](#user-registration) {#user-registration-19 .TOC-2}

# [5.5 Log on to IAR 20](#log-on-to-iar) {#log-on-to-iar-20 .TOC-2}

# [5.6 Password Replace 21](#password-replace) {#password-replace-21 .TOC-2}

# [5.7 Account Access and Expiration 22](#account-access-and-expiration) {#account-access-and-expiration-22 .TOC-2}

# [5.8 Changing Passwords 22](#changing-passwords) {#changing-passwords-22 .TOC-2}

# [5.9 Submit Phase 22](#submit-phase) {#submit-phase-22 .TOC-2}

# [5.10 Read Phase 28](#read-phase) {#read-phase-28 .TOC-2}

# [5.11 Streamlining 30](#streamlining) {#streamlining-30 .TOC-2}

# [5.12 Edit Phase 33](#edit-phase) {#edit-phase-33 .TOC-2}

# [5.13 Summary Statement Assembly 34](#summary-statement-assembly) {#summary-statement-assembly-34 .TOC-2}

# [5.14 Reports 37](#reports) {#reports-37 .TOC-2}

# [5.15 IC Program Officer Access 38](#ic-program-officer-access) {#ic-program-officer-access-38 .TOC-2}

# [5.16 Purging Assignments 39](#purging-assignments) {#purging-assignments-39 .TOC-2}

# [5.17 Meeting Closure 39](#meeting-closure) {#meeting-closure-39 .TOC-2}

# [5.18 Other General Features 40](#other-general-features) {#other-general-features-40 .TOC-2}

[6. Constraints 40](#constraints)

[7. Quality Ranges 40](#quality-ranges)

[8. Precedence and Priority 40](#precedence-and-priority)

[9. Other Product Requirements 41](#other-product-requirements)

# [9.1 Applicable Standards 41](#applicable-standards) {#applicable-standards-41 .TOC-2}

# [9.2 System Requirements 41](#system-requirements) {#system-requirements-41 .TOC-2}

# [9.3 Performance Requirements 41](#performance-requirements) {#performance-requirements-41 .TOC-2}

# [9.4 Environmental Requirements 41](#environmental-requirements) {#environmental-requirements-41 .TOC-2}

# [9.5 Security Recommendations 41](#security-recommendations) {#security-recommendations-41 .TOC-2}

[10. Documentation Requirements 41](#documentation-requirements)

# [10.1 User Manual 41](#user-manual) {#user-manual-41 .TOC-2}

# [10.2 On-line Help 42](#on-line-help-1) {#on-line-help-42 .TOC-2}

# [10.3 Installation Guides, Configuration, Read Me File 42](#installation-guides-configuration-read-me-file) {#installation-guides-configuration-read-me-file-42 .TOC-2}

# [10.4 Labeling and Packaging 42](#labeling-and-packaging) {#labeling-and-packaging-42 .TOC-2}

# [Appendix A: Business Requirements for an IMPAC II Internet Assisted Peer Review System from a Sub-Committee of the CSR Information Resources Advisory Committee 43](#appendix-a-business-requirements-for-an-impac-ii-internet-assisted-peer-review-system-from-a-sub-committee-of-the-csr-information-resources-advisory-committee) {#appendix-a-business-requirements-for-an-impac-ii-internet-assisted-peer-review-system-from-a-sub-committee-of-the-csr-information-resources-advisory-committee-43 .TOC-2}

# [Appendix B: Dr. Everett Sinnett's Design Issues for Electronic Critique System 50](#appendix-b-dr.-everett-sinnetts-design-issues-for-electronic-critique-system) {#appendix-b-dr.-everett-sinnetts-design-issues-for-electronic-critique-system-50 .TOC-2}

# [Appendix C: Dr. Thomas Tatham's Request for ER Enhancements 54](#appendix-c-dr.-thomas-tathams-request-for-er-enhancements) {#appendix-c-dr.-thomas-tathams-request-for-er-enhancements-54 .TOC-2}

# [Appendix D: Registration Email to Reviewer 55](#appendix-d-registration-email-to-reviewer) {#appendix-d-registration-email-to-reviewer-55 .TOC-2}

# [Appendix E: IAR User Agreement 56](#appendix-e-iar-user-agreement) {#appendix-e-iar-user-agreement-56 .TOC-2}

####   {#section .TOC-4}

# Introduction

## Background

A Web-based system to manage the process of electronic submission of
critiques by Reviewers was developed by the National Institute of
Allergy and Infectious Diseases (NIAID). This system, Electronic Review
(ER), has been successfully implemented at several ICs and has provided
proof of concept for this electronic process. The NIAID system will be
used as a model for development of an eRA system. An eRA
Internet-Assisted Review system will expedite the scientific review of
grant applications by standardizing the current process of critique and
initial priority scores submissions by reviewers via the Internet.
Currently, Reviewers usually do not submit their critiques before the
actual meeting and they do not have the opportunity to see others'
critiques before the meeting. When critiques are finally submitted, they
may not all be in the same format. Since critiques are used to build the
summary statement body text, this method poses problems for staff. An
Internet Assisted Review system would improve this process. Review
meetings would contain more informed discussions because reviewers would
be able to read the evaluations entered by others prior to the review
meeting (except where there is a conflict of interest). The system will
also serve to facilitate the generation of summary statements since all
critiques would be submitted in the same electronic format and be stored
centrally.

## Purpose

The purpose of this document is to define the scope and high-level
business requirements of the Internet Assisted Review System. The
structure and content of this document is based on the Rational Unified
Process (RUP). It focuses on the capabilities and features needed by the
stakeholders and the target users. The detailed requirements that are
derived from these features are specified in the Software Requirements
document (which will include Use Cases) and the Supplementary
Specifications document.

The Internet Assisted Review System will be developed in multiple
releases. This document will be a living document. Initially it will
focus on functionality to be delivered in releases 1 and 2. As
additional upgrades are planned in the future, this document will evolve
to capture the capabilities and features for those future releases.

## Definitions, Acronyms, and Abbreviations

> **SRA** Scientific Review Administrator
>
> **GTA** Grants Technical Assistant
>
> **PI** Principal Investigator
>
> **ER** Electronic Review (NIAID system)
>
> **RUG** Review Users Group
>
> **IAR** Internet Assisted Review
>
> **COI** Conflict of Interest

## References

-   *IMPAC II Peer Review User Guide*, Version 2.1.1.0

-   Information provided at NIAID's ER Info site
    <http://grants.niaid.nih.gov/review/staff/SRA_Erhelp_cvrpg-grants.htm>

Documents Submitted 2/11/02:

-   Appendix A: *Business Requirements for an IMPAC II Internet Assisted
    Peer Review System* From a Sub-Committee of the CSR Information
    Resources Advisory Committee (contact: Richard Panniers)

-   Appendix B: Dr. Everett Sinnett's *Design Issues For Electronic
    Critique System (ECS)*

-   Appendix C: Dr. Thomas Tatham\'s request for ER enhancements

Focus Group meetings were held:

March 4, 2002; March 14, 2002; March 18, 2002; March 25, 2002; April 1,
2002; April 8, 2002; April 15, 2002; April 25, 2002; May 6, 2002; May
13, 2002; May 20, 2002; June 3, 2002; June 10, 2002; June 21, 2002.

# Positioning

## Business Opportunity/Scope

An eRA-developed Internet-Assisted Review system will help expedite the
scientific review of grant applications by providing a standard process
for Reviewers to submit their critiques and initial priority scores via
the Internet. Currently, for staff not using the NIAID ER system,
reviewers might submit their critiques and initial priority score using
several methods including paper copies, diskettes, or email attachments,
but usually not before the actual meeting. Since critiques are used to
build the summary statement body text, each of these current methods
pose problems for staff. Data provided in paper copies must be manually
entered or scanned, electronic documents provided by email or diskette
may have been written in different, incompatible word processing
formats, and all must be combined into a single document. Electronic
documents submitted have the potential of containing a computer virus.
Another major flaw in the current business model is that reviewers do
not have the opportunity to review other critiques before the meeting.

An eRA-developed Internet-Assisted Review system would eliminate many
problems evident in the current method. All critiques would be submitted
in the same electronic format (Word \*.doc or plain text \*.txt). After
the deadline for submission has passed, reviewers would be able to read
the evaluations entered by others prior to the review meeting (except
where there is a conflict of interest). This pre-meeting review of
critiques would provide for more informed discussions at the review
meeting. The SRA/GTA would also have the ability to generate a
preliminary report of upper and lower scores. Subsequent to the meeting,
reviewers would be permitted to update their critiques; the system could
also serve to facilitate the generation of summary statements.

Providing this system to the eRA user community would offer several
major benefits. First, critiques would be available immediately after
the review meeting. This greatly expedites the creation of summary
statements. Staff who currently use the NIAID system report that this
method has saved them 1--3 weeks time. Second, this method greatly
reduces human errors associated with manipulation of the documents and
problems encountered using computer diskettes. Third, the system would
expedite the approval/funding process via easier, more efficient
administration of reviews (e.g., summary statement preparation by NIH
staff). Fourth, it would improve the overall quality of reviews and
permit more efficient and effective use of reviewers' time at meetings.

# Stakeholder and User Descriptions

## User Environment

Since IAR essentially facilitates the exchange of information between
NIH Scientific Review Administrators (SRAs), Grants Technical Assistants
(GTAs) and Reviewers, these are the major users groups and stakeholders
of the system. While each IC may hold their own Reviews, a majority of
Reviews are held by the Center for Scientific Review (CSR). SRAs may be
considered primary users; GTAs also work closely with the SRAs and may
perform similar functions in the system.

## Stakeholder Profiles

This section describes the stakeholders' profiles, in terms of their
roles, responsibilities, success criteria, and involvement in this
development effort. The following terms will help define a stakeholders'
profile:

> **Representative**---the stakeholders' envoy to the project, this will
> either be the name (or names) of individual(s), or a specific body of
> people
>
> **Description**---a brief explanation of the stakeholder type
>
> **Type**---the stakeholders' expertise, technical background, and
> degree of sophistication
>
> **Responsibilities**---The stakeholders' key tasks in this effort;
> that is, their interest as a stakeholder. Examples might be "captures
> details," "produces reports," or "coordinates work."
>
> **Success Criteria**---the stakeholders' definition of accomplishment
> of this project
>
> **Involvement**---the stakeholders' role in this project, if any
>
> **Deliverables**---any documents or resultant products the stakeholder
> produces, and for whom
>
> **Comments/Issues**---problems that interfere with success and any
> other relevant information (these would include trends that make the
> stakeholders' job easier or harder)

+---------+------------+------------+--------------------------------+
| Stak    | Repre      | Profile    |                                |
| eholder | sentatives |            |                                |
+=========+============+============+================================+
| NIH IT  | J. J.      | D          | IT Project Management          |
| Man     | McGowan,   | escription |                                |
| agement |            |            |                                |
|         | James Cain |            |                                |
+---------+------------+------------+--------------------------------+
|         |            | Type       | Manages IT finances and        |
|         |            |            | priorities                     |
+---------+------------+------------+--------------------------------+
|         |            | Respon     | Responsible for all aspects of |
|         |            | sibilities | the eRA project in general,    |
|         |            |            | and all aspects of IMPAC II    |
|         |            |            | project in particular. Formal  |
|         |            |            | management reviews, as defined |
|         |            |            | in the "eRA J2EE Project       |
|         |            |            | Management Plan."              |
+---------+------------+------------+--------------------------------+
|         |            | Success    | Success is completion of the   |
|         |            | Criteria   | project within approved        |
|         |            |            | budget, and fulfillment of     |
|         |            |            | user needs in a timely manner. |
+---------+------------+------------+--------------------------------+
|         |            | I          | Project guidance and review    |
|         |            | nvolvement |                                |
+---------+------------+------------+--------------------------------+
|         |            | De         | Responsible for delivering the |
|         |            | liverables | Internet Assisted Review       |
|         |            |            | system to the user community.  |
+---------+------------+------------+--------------------------------+
|         |            | Comments/  | None.                          |
|         |            | Issues     |                                |
+---------+------------+------------+--------------------------------+
| Group   | Eileen     | D          | Communicates needs of NIH      |
|         | Bradley    | escription | community to the development   |
| A       |            |            | team.                          |
| dvocate |            |            |                                |
+---------+------------+------------+--------------------------------+
|         |            | Type       | Possesses strong communication |
|         |            |            | and facilitation skills, along |
|         |            |            | with good domain expertise.    |
+---------+------------+------------+--------------------------------+
|         |            | Respon     | Manages expectations of the    |
|         |            | sibilities | users. Approves requirements   |
|         |            |            | documents. Works with the      |
|         |            |            | analyst and development team   |
|         |            |            | to set priorities.             |
+---------+------------+------------+--------------------------------+
|         |            | Success    | Project meets the              |
|         |            | Criteria   | specifications that have been  |
|         |            |            | agreed upon with the NIH       |
|         |            |            | community in a timely manner.  |
+---------+------------+------------+--------------------------------+
|         |            | I          | Project guidance and review    |
|         |            | nvolvement |                                |
+---------+------------+------------+--------------------------------+
|         |            | De         | Responsible for delivering the |
|         |            | liverables | system to the user community.  |
+---------+------------+------------+--------------------------------+
|         |            | Comments/  | None                           |
|         |            | Issues     |                                |
+---------+------------+------------+--------------------------------+

## User Profiles

This section describes in detail the profile of each system user, in
terms of their roles, responsibilities, success criteria, and
involvement in the development effort. The same aspects will be used to
define a user's profile as were used to define the stakeholders'
profiles, above.

+---------+------------+------------+--------------------------------+
| User    | Repre      | Profile    |                                |
|         | sentatives |            |                                |
+=========+============+============+================================+
| SRA     | IAR Focus  | D          | Scientific Review              |
|         | Group      | escription | Administrator                  |
+---------+------------+------------+--------------------------------+
|         |            | Type       | Possesses strong communication |
|         |            |            | skills, along with good domain |
|         |            |            | expertise.                     |
+---------+------------+------------+--------------------------------+
|         |            | Respon     | 1.  Assigns Reviewers          |
|         |            | sibilities |                                |
|         |            |            | 2.  Identifies any COI         |
|         |            |            |                                |
|         |            |            | 3.  Determine deadlines for    |
|         |            |            |     critique submission,       |
|         |            |            |     read-only phase,           |
|         |            |            |     post-meeting edit phase.   |
|         |            |            |                                |
|         |            |            | 4.  Reviews critiques and      |
|         |            |            |     scores                     |
|         |            |            |                                |
|         |            |            | 5.  May submit critiques for   |
|         |            |            |     Reviewers                  |
|         |            |            |                                |
|         |            |            | 6.  Attends review meeting     |
|         |            |            |                                |
|         |            |            | 7.  Writes summary statements  |
+---------+------------+------------+--------------------------------+
|         |            | Success    | Project meets the              |
|         |            | Criteria   | specifications that have been  |
|         |            |            | agreed upon with the NIH       |
|         |            |            | community.                     |
+---------+------------+------------+--------------------------------+
|         |            | I          | Product reviewer, beta tester. |
|         |            | nvolvement | Attends and participates in    |
|         |            |            | meetings. Validates business   |
|         |            |            | rules.                         |
+---------+------------+------------+--------------------------------+
|         |            | De         | Any documents necessary to     |
|         |            | liverables | collect requirements. Examples |
|         |            |            | of reports or sample screen    |
|         |            |            | shots as requested by Analyst  |
|         |            |            | and Development team members.  |
+---------+------------+------------+--------------------------------+
|         |            | Comments/  |                                |
|         |            | Issues     |                                |
+---------+------------+------------+--------------------------------+
| GTA     | IAR Focus  | D          | Grants Technical Assistant     |
|         | Group      | escription |                                |
+---------+------------+------------+--------------------------------+
|         |            | Type       | Possesses strong communication |
|         |            |            | skills, along with good domain |
|         |            |            | expertise.                     |
+---------+------------+------------+--------------------------------+
|         |            | Respon     | 1.  Assists SRA                |
|         |            | sibilities |                                |
|         |            |            | 2.  May perform some tasks     |
|         |            |            |     listed above in SRA        |
|         |            |            |     Responsibilities           |
+---------+------------+------------+--------------------------------+
|         |            | Success    | Project meets the              |
|         |            | Criteria   | specifications that have been  |
|         |            |            | agreed upon with the NIH       |
|         |            |            | community.                     |
+---------+------------+------------+--------------------------------+
|         |            | I          | Product reviewer, beta tester. |
|         |            | nvolvement | Attends and participates in    |
|         |            |            | meetings. Validates business   |
|         |            |            | rules.                         |
+---------+------------+------------+--------------------------------+
|         |            | De         | Any documents necessary to     |
|         |            | liverables | collect requirements. Examples |
|         |            |            | of reports or sample screen    |
|         |            |            | shots as requested by Analyst  |
|         |            |            | and Development team members.  |
+---------+------------+------------+--------------------------------+
|         |            | Comments/  |                                |
|         |            | Issues     |                                |
+---------+------------+------------+--------------------------------+
| R       | IAR Focus  | D          | Reviewer (Non-NIH)             |
| eviewer | Group      | escription |                                |
+---------+------------+------------+--------------------------------+
|         |            | Type       | For the purpose of gathering   |
|         |            |            | requirements, this user will   |
|         |            |            | be represented by the SRAs and |
|         |            |            | GTAs.                          |
+---------+------------+------------+--------------------------------+
|         |            | Respon     | 1.  Reviews grant applications |
|         |            | sibilities |     for scientific merit.      |
|         |            |            |                                |
|         |            |            | 2.  Writes and submits         |
|         |            |            |     critiques and scores for   |
|         |            |            |     assigned applications      |
|         |            |            |                                |
|         |            |            | 3.  Reviews critiques          |
|         |            |            |     submitted by others in     |
|         |            |            |     preparation for meeting.   |
|         |            |            |                                |
|         |            |            | 4.  Attends and participates   |
|         |            |            |     in review meeting.         |
+---------+------------+------------+--------------------------------+
|         |            | Success    | Project meets the              |
|         |            | Criteria   | specifications that have been  |
|         |            |            | agreed upon with the NIH       |
|         |            |            | community.                     |
+---------+------------+------------+--------------------------------+
|         |            | I          | Beta tester.                   |
|         |            | nvolvement |                                |
+---------+------------+------------+--------------------------------+
|         |            | De         | None.                          |
|         |            | liverables |                                |
+---------+------------+------------+--------------------------------+
|         |            | Comments/  |                                |
|         |            | Issues     |                                |
+---------+------------+------------+--------------------------------+

#  Product Overview

This section provides a high-level view of the system capabilities,
interfaces, etc.

## Product Perspective

The system will be Web-based and developed using J2EE technology. The
basic logic and functionality of the NIAID ER system will be used as an
example to build this system. Other current NIH business practices for
electronic critique submission may also be reviewed to aid in collecting
requirements. Processes already integrated into the Peer Review module
such as Assigning Reviewers and defining COI will not be duplicated in
IAR. Since Reviewers are key system users, IMPAC II data items relevant
to IAR will be replicated to the Commons database to ensure data
security and prevent unauthorized access to the IMPAC II database. IAR
will be an external interface to the Commons database. The following
diagram shows how key processes of Internet Assisted Review fit into the
existing Peer Review business process.

![](media/image2.png){width="7.95in" height="5.447916666666667in"}

## Summary of Capabilities

The system will comprise several primary functions: user registration
and management; critique/priority score submission and modification;
streamline voting; reports; and critique combination/merging to create
summary statement draft. Within these functions, the following
capabilities will be included:

-   The IMPAC II Peer Review module will allow the SRA/GTA to finalize
    assignments to be created and made accessible in the Internet
    Assisted Review (IAR) system via the Commons database.

-   The system will allow the SRA/GTA to provide deadlines for critique
    submission and other events.

-   The system will manage accounts, passwords and registration for
    Reviewers.

-   The system will accept critiques of a Word (\*.doc) or plain text
    (\*.txt) format (version TBD).

-   The system will allow the SRA/GTA to modify the critique submission
    deadline and block certain reviewers from accessing the critiques of
    others.

-   The system should allow reviewers to post streamlining votes and
    allow for the production of related reports.

-   The system should provide reports that show the status of reviewer's
    critique submissions.

-   The system will facilitate summary statement creation by merging the
    critiques into one document to be used as the draft summary
    statement body text and allowing the SRA/GTA to export this
    document.

## Assumptions and Dependencies

The system will have dependencies with the IMPAC II Peer Review module
and will use data entered into this module. It will be mandatory for
Scientific Review Administrators (SRA/GTAs) to assign reviewers to
applications, check conflict of interest, and organize the meeting,
etc., in IMPAC II Peer Review to collect critiques in this system. When
the SRA/GTA has "finalized" their assignments, the Peer Review module
should error check for simultaneous conflicts and assignments. It is
assumed that IAR will use data in the Commons database. Reviewers will
not connect to the IMPAC II Production Database (IMPP).

## Cost and Pricing

IAR development efforts will meet the cost guidelines established by the
NIH eRA oversight boards and committees. The following preliminary
budget for Version 1 of IAR was submitted with the FY2002 Business Plan
and may be baselined after the Critical Design Review.

Estimated Cost: Non-Government Staff

<table>
<colgroup>
<col style="width: 14%" />
<col style="width: 6%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 6%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 7%" />
<col style="width: 10%" />
</colgroup>
<tbody>
<tr class="odd">
<td><p>Remaining</p>
<p>Work</p>
<p>To be done/ hrs by role &amp;</p>
<p>SDLC</p></td>
<td><blockquote>
<p>req.anal/ JADs</p>
</blockquote></td>
<td><blockquote>
<p>prelim. design &amp; prototype</p>
</blockquote></td>
<td><blockquote>
<p>Detailed design &amp; CDR</p>
</blockquote></td>
<td><blockquote>
<p>Hrs. build</p>
</blockquote></td>
<td><blockquote>
<p>Hrs. test</p>
</blockquote></td>
<td><blockquote>
<p>pilot</p>
</blockquote></td>
<td><blockquote>
<p>doc/ train.</p>
</blockquote></td>
<td><blockquote>
<p>Maint.. and enhancements</p>
</blockquote></td>
<td><blockquote>
<p>Tot hrs.</p>
</blockquote></td>
<td><blockquote>
<p>$/hr</p>
</blockquote></td>
<td><blockquote>
<p>Tot$</p>
</blockquote></td>
</tr>
<tr class="even">
<td>Total hrs non gov’t</td>
<td>300</td>
<td>550</td>
<td>150</td>
<td>1200</td>
<td>160</td>
<td>240</td>
<td>200</td>
<td>420</td>
<td>3220</td>
<td></td>
<td></td>
</tr>
<tr class="odd">
<td>--Designer</td>
<td>300</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>300</td>
<td>$95</td>
<td>$28,500</td>
</tr>
<tr class="even">
<td>--Pgmr.</td>
<td></td>
<td>550</td>
<td>150</td>
<td>1200</td>
<td>80</td>
<td>160</td>
<td></td>
<td>300</td>
<td>2440</td>
<td>$83</td>
<td>$202,520</td>
</tr>
<tr class="odd">
<td>--Test/QA</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>80</td>
<td>80</td>
<td></td>
<td>80</td>
<td>240</td>
<td>$83</td>
<td>$19,920</td>
</tr>
<tr class="even">
<td>--Writer</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td>200</td>
<td>40</td>
<td>240</td>
<td>$41</td>
<td>$9,840</td>
</tr>
<tr class="odd">
<td colspan="11"><p>Hrs. Hrs. Hrs. Hrs. Hrs. Hrs. Hrs. Hrs.</p>
<p>Total $ required to complete</p></td>
<td>$260,780</td>
</tr>
</tbody>
</table>

**Government Staff**

Tracy Soto (OD) 30% of total time

Sherry Zucker, (OD) 10% of total time

Scarlett Gibb (OD) 10% of total time

## Licensing and Installation

There are no licensing requirements for IAR. Since the IAR product is
Web-based, installation of the product is not necessary. Only a Web
browser will be required for users.

# Product Features

This section defines and describes the proposed features of this
application. Where applicable, information regarding Version and
Priority are provided.

Key to "Version" in tables:

-   Ver.1: Must be in the initial version

-   Ver.2: Must be in the second version

-   Ver.3: Can be added at any point

Proposed features are prioritized using MuSCoW: "Must", "Should",
"Could", or "Won't".

-   Must---Version can't be deployed without function; requirement is
    basic functionality.

-   Should---Requirement should be deployed in version. Version can be
    deployed before function is implemented.

-   Could---'Nice to have' enhancement.

-   Won't---Requirement should not be included in functionality.

## System (SYS) Feature

### Error Handling

The system shall graciously handle and log all errors encountered.

### Usage Reporting

The system shall provide reports on general system use and exceptional
behavior.

### System Control

The system shall provide convenient mechanisms for startup, shutdown,
and recovery of individual subsystems. 

### Browser Interface

The system shall provide a user interface through a thin, browser-based
client.

### Interface Conventions

The user interface shall follow standard interface conventions based on
acceptable industry standards.

### On-line help

The user interface shall include on-line help features.

### Release Notes

The user interface shall include links to the release notes.

### Bug Status

The user interface shall include links to the status of reported bugs.

### Availability

The system shall be generally available for use on a 24x7 basis with
limited downtime acceptable for system upgrades and unexpected
conditions.

### Performance

The system shall provide performance and response times generally
consistent with industry standards for Internet applications.

### Auditing

The system shall provide configurable auditing capabilities.

### Exception Reporting

The system shall report exceptional conditions to an administrator via
e-mail.

### External Interfaces

The system shall provide an interface to the Review community.

## Release Meeting to IAR / IAR Control Center

Before Reviewers can access a meeting in IAR, the SRA/GTA must enable
the meeting to be used by IAR. There are several functions that the
SRA/GTA will need to perform. Below are the requirements for the "IAR
Control Center" which will be available in Peer Review module.

  --------------------------------------------------------------------------
      Requirement                                         Version   MuSCoW
  --- --------------------------------------------------- --------- --------
      The Peer Review module should allow SRA/GTA to      1         M
      navigate from the Peer Review banner screen to the            
      "IAR Control Center" module with default meeting as           
      selected on the banner screen.                                

      The IAR Control Center must have an ability to      1         M
      easily enable the entire list of reviewers for the            
      meeting for IAR.                                              

      The IAR Control Center should have the ability to   2         S
      easily enable selected Reviewers for IAR.                     

      To Enable the Reviewers for IAR, Reviewers in the   1         M
      meeting must be on the Committee Management Meeting           
      Roster. SRA/GTA will be alerted to any                        
      discrepancies (if Reviewer isn't on Roster).                  

      The IAR Control Center must prevent selection of    1         M
      Reviewers for IAR if it detects the absence of the            
      MLG role e-mail address or person_id.                         

      Provide a "Roster Reconciliation" screen to allow   2         C
      the SRA/GTA to link people from the CM roster to              
      people listed on the Reviewer List. No person                 
      search would be required, and the process would be            
      straightforward and quick for the user. IAR could             
      then \"transfer\" the assignments to the CM                   
      identified reviewers.                                         

      The IAR Control Center must allow SRA/GTA to view   1         M
      the MLG email address (on role) for each reviewer.            

      The IAR Control Center must allow SRA/GTA to edit   2         C
      the MLG email address (on role) for each reviewer.            
      Per 6/17/02 meeting with Sara Silver, Dan Hall this           
      requirement is on hold pending future                         
      enterprise-wide person and address issues.                    

      When SRA/GTA edits the MLG role email address, the  2         C
      IAR Control Center must give SRA/GTA the option of            
      copying that email address to the profile record.             
      Per 6/17/02 meeting with Sara Silver, Dan Hall this           
      requirement is on hold pending future                         
      enterprise-wide person and address issues.                    

      To Enable at least one reviewer for IAR, the Submit 1         M
      Phase End (Critique Submission Due) Date and Time             
      (ET) must be entered.                                         

      To Enable at least one reviewer for IAR, the Read   1         M
      Phase End Date and Time (ET) must be entered.                 

      To Enable the meeting for IAR, the End Date and     1         M
      Time (ET) for the EDIT Phase must be an optional              
      field. If Edit Phase End date is not specified,               
      there will be no Edit Phase. If Edit Phase End has            
      been entered, the Edit Phase Start date coincides             
      with the Read Phase End Date.                                 

      The IAR Control Center must verify that dates and   1         M
      time entered for the phases are sequential (1.                
      Critique Submission Due Date, 2. Read Phase End               
      Date, 3. Edit Phase End Date)                                 

      The IAR Control Center must display the Meeting     1         M
      Closure Date. This date will be blank until meeting           
      is released and then calculated to be 6 months from           
      meeting release date.                                         

      The IAR Control Center must allow SRA/GTA to        1         M
      disable (block) [individual]{.underline} Reviewers            
      from reading critiques on applications where they             
      are assigned but did not post their own critique.             

      The IAR Control Center should allow SRA/GTA to      1         S
      easily disable (block) [all]{.underline} Reviewers            
      from reading critiques on applications where they             
      are assigned but did not post their own critique.             

      The IAR Control Center should allow SRA/GTA to      1         M
      toggle show/hide preliminary scores from all                  
      Reviewers in IAR. If Scores are hidden, Reviewer              
      would only see scores they've entered.                        

      The IAR Control Center should allow SRA/GTA to      W         
      toggle the ability to show only average scores or             
      raw scores but show the entire matrix for                     
      Reviewers. Per 5/20/02 group meeting, group decided           
      this feature was not useful.                                  

      The IAR Control Center should allow SRA/GTA to      1         S
      toggle ability for reviewers to enter critiques or            
      comments for applications where they are not                  
      assigned.                                                     

      The IAR Control Center must allow SRA/GTA to run a  2         M
      printable report, which will list all reviewers in            
      the meeting, their MLG role email address,                    
      username, answer and account active status.                   

      The IAR Control Center must allow SRA/GTA to extend 1         M
      read permissions on critiques to Telephone                    
      Reviewers for entire meeting instead of the default           
      view of only their assigned applications.                     

      IAR Control Center must allow SRA/GTA to send       2         S
      custom batch emails to ALL reviewers in the meeting           
      (Reviewers' email addresses should be BCC so                  
      Reviewers cannot see other recipients).                       

      IAR Control Center must allow SRA/GTA to specify    2         S
      the "From" addressee in the custom batch email to             
      all Reviewers.                                                

      IAR Control Center must send the Carbon Copy emails 2         S
      to SRA/GTA when batch emails are sent to all                  
      Reviewers.                                                    

      IAR Control Center must return undeliverable emails 2         S
      to the SRA/GTA when batch emails are sent to                  
      Reviewers.                                                    

      The IAR Control Center must allow SRA/GTA to        1         M
      trigger an email with a customized registration URL           
      (embedded with unique reviewer identifier/system              
      generated random number) to individual reviewers.             
      The focus group drafted and agreed on standard                
      language for this email, see Appendix                         
      D(Registration Email to Reviewer).                            

      The IAR Control Center should include the ability   1         S
      for the SRA/GTA to identify hyperlinks to documents           
      for display within their meeting in IAR: Reviewer             
      Guidelines (for specific mechanisms, human                    
      subjects, etc.), grant application, prior summary             
      statements, program announcements, meeting roster,            
      blank COI form, master list of applications/order             
      of review report, travel instructions (hotel,                 
      airline, etc.), meeting agenda. Specific document             
      types to be determined.                                       

      For documents not available on the Web, the IAR     2         S
      Control Center should allow the SRA/GTA to upload             
      documents for display within IAR: cover letter                
      (specific to each SRA), travel instructions (hotel,           
      airline, etc). Specific document types to be                  
      determined.                                                   

      The IAR Control Center should allow SRA/GTA to      1         S
      choose which document or document hyperlinks to               
      include in their meeting.                                     

      Changes in assignments, COI, Roster in Peer Review  1         M
      must be immediately available in IAR.                         

      When changes occur (assignments, COI, application   2         S
      added or withdrawn), affected Reviewers should                
      receive email notification of the changes. This               
      requirement may be met by another upcoming eRA                
      system -eNotification.                                        

      The IAR Control Center must display a Reviewer's    1         M
      IAR username and answer to their personal question            
      to the SRA/GTA.                                               

      When action is taken by the SRA/GTA, the system     1         M
      should provide confirmation screens for Reviewer              
      options (enable/disable for IAR, block from                   
      reading)                                                      

      The IAR Control Center must have an ability to      1         M
      Disable the meeting for IAR (this does not purge              
      the meeting).                                                 
  --------------------------------------------------------------------------

## Pre-Registration System Process

Once the meeting is released to IAR via "IAR Control Center," the
process of inviting Reviewers to access IAR and enabling the system to
allow Reviewer accounts to be created for IAR is activated. Reviewers
with Commons account will use those accounts. Reviewers who do not have
Commons accounts will be sent an email containing an URL with a unique
identifier. The requirement for this process described below:

  --------------------------------------------------------------------------
      Requirement                                         Version   MuSCoW
  --- --------------------------------------------------- --------- --------
      If Reviewer has a Commons account, the meeting      1         M
      release to IAR will enable the Commons account to             
      be used in IAR.                                               

      If Reviewer has a Commons account, system will      1         M
      email the reviewer an invitation to the IAR                   
      website, requesting him/her to use existing Commons           
      Account log on information to enter IAR (exact text           
      to be determined).                                            

      If Reviewer does not have a Commons account, email  1         M
      the reviewer invitation to the IAR website, and               
      include detailed instructions on how to register to           
      obtain an IAR account. The email will provide a               
      hyperlink containing an embedded token identifier             
      for the reviewer.                                             

      Undeliverable registration invitation emails should 1         M
      be sent to SRA/GTA. (GTA must be on roster to get             
      the email.)                                                   
  --------------------------------------------------------------------------

## User Registration

Once the registration invitation email is sent, Reviewers that do not
have Commons accounts are asked to register. To access the registration
form, Reviewers will click on the customized URL in the email. The form
requires the Reviewer to enter a username and password that they would
like to use for IAR as well as mandatory information such as full name,
phone and optional information such as institution, social security
number and date of birth. After successfully entering this information,
the system sends the information to the OER Data Quality staff for
verification. Upon verification, the account is activated. The Reviewer
will receive an email when their account is activated, it is expected
that this process will take less than 24 hours. Below are requirements
for the registration form.

+---+---------------------------------------------------+------+--------+
|   | Requirement                                       | Ver  | MuSCoW |
|   |                                                   | sion |        |
+===+===================================================+======+========+
|   | The IAR Registration module must require Reviewer | 1    | M      |
|   | to enter a User Name of their choice (must adhere |      |        |
|   | to reasonable security rules).                    |      |        |
+---+---------------------------------------------------+------+--------+
|   | The IAR Registration module must require Reviewer | 1    | M      |
|   | to enter a password of their choice (must adhere  |      |        |
|   | to reasonable security rules).                    |      |        |
+---+---------------------------------------------------+------+--------+
|   | The IAR Registration module must require Reviewer | 1    | M      |
|   | to retype the password (since the password field  |      |        |
|   | is masked with (\*) asterisks.                    |      |        |
+---+---------------------------------------------------+------+--------+
|   | The IAR Registration module must require Reviewer | 1    | M      |
|   | to enter their full name.                         |      |        |
+---+---------------------------------------------------+------+--------+
|   | The IAR Registration module must require Reviewer | 1    | M      |
|   | to enter their phone number.                      |      |        |
+---+---------------------------------------------------+------+--------+
|   | The IAR Registration module must allow Reviewer   | 1    | M      |
|   | to enter of these optional fields: Institution    |      |        |
|   | name, social security number, and date of birth.  |      |        |
+---+---------------------------------------------------+------+--------+
|   | The IAR Registration module must require Reviewer | 1    | M      |
|   | to accept a User Agreement before creating        |      |        |
|   | account. (See Appendix E for User Agreement       |      |        |
|   | language.)                                        |      |        |
+---+---------------------------------------------------+------+--------+
|   | The IAR Registration module must allow Reviewer   | 1    | M      |
|   | to choose a question and enter a corresponding    |      |        |
|   | answer to be used in the future if they forget    |      |        |
|   | their password. These questions should be         |      |        |
|   | displayed:                                        |      |        |
|   |                                                   |      |        |
|   | What is your city of birth?                       |      |        |
|   |                                                   |      |        |
|   | What is your mother's maiden name?                |      |        |
|   |                                                   |      |        |
|   | What is your pet's name?                          |      |        |
|   |                                                   |      |        |
|   | What is your spouse's name?                       |      |        |
+---+---------------------------------------------------+------+--------+
|   | The system should allow Reviewer to enter         | 2    | C      |
|   | additional email addresses where they may be      |      |        |
|   | reached. Per 6/17/02 meeting with Sara Silver,    |      |        |
|   | Dan Hall this requirement is on hold pending      |      |        |
|   | future enterprise-wide person and address issues. |      |        |
+---+---------------------------------------------------+------+--------+
|   | The IAR Registration module must notify the       | 1    | M      |
|   | Reviewer that registration has been submitted     |      |        |
|   | successfully and instruct them to wait for an     |      |        |
|   | email to verify that their account has been       |      |        |
|   | activated.                                        |      |        |
+---+---------------------------------------------------+------+--------+
|   | When registration has been submitted              | 1    | M      |
|   | successfully, the system must expire the original |      |        |
|   | registration URL.                                 |      |        |
+---+---------------------------------------------------+------+--------+
|   | The IAR Registration module must provide the      | 1    | M      |
|   | following information to the OER Data Quality     |      |        |
|   | staff immediately upon submission of registration |      |        |
|   | form by Reviewer:                                 |      |        |
|   |                                                   |      |        |
|   | Information provided by IAR, IMPP database:       |      |        |
|   |                                                   |      |        |
|   | SRA Name                                          |      |        |
|   |                                                   |      |        |
|   | SRA Phone number                                  |      |        |
|   |                                                   |      |        |
|   | Reviewer Person_id                                |      |        |
|   |                                                   |      |        |
|   | Reviewer Name associated with Person_id in IMPP   |      |        |
|   |                                                   |      |        |
|   | Institution associated with Person_id in IMPP     |      |        |
|   |                                                   |      |        |
|   | Information provided by Reviewer on Registration  |      |        |
|   | form:                                             |      |        |
|   |                                                   |      |        |
|   | Full name                                         |      |        |
|   |                                                   |      |        |
|   | Phone number                                      |      |        |
|   |                                                   |      |        |
|   | Institution (if entered)                          |      |        |
|   |                                                   |      |        |
|   | Social security number (if entered)               |      |        |
|   |                                                   |      |        |
|   | Date of birth (if entered)                        |      |        |
+---+---------------------------------------------------+------+--------+
|   | All system generated email communication with the | 1    | S      |
|   | Reviewer should include the IAR Banner screen     |      |        |
|   | URL.                                              |      |        |
+---+---------------------------------------------------+------+--------+

## Log on to IAR

Once users have a valid account, they can log into the IAR via the Log
on screen. Below are the requirements for a log on screen.

  --- --------------------------------------------------- --------- --------
      Requirement                                         Version   MuSCoW

      The IAR Log on module must accept Reviewer user     1         M
      name and password to log in.                                  

      The IAR Log on module should accept existing IMPAC  1         M
      II user name and password for SRA/GTA users.                  

      The IAR Log on module must display an error message 1         M
      if log in fails.                                              

      The IAR Log on module must provide a method for     1         M
      Reviewers to replace their forgotten password.                

      When user enters correct User Name and Password,    1         M
      User must be logged in to IAR.                                
  --- --------------------------------------------------- --------- --------

## Password Replace

*If Reviewers don't know their User Name, they must contact the
SRA/GTA*. If Reviewers have forgotten their password, they can create a
new one via the Password Replace Form. This feature will be accessible
from the Log on screen.

  --------------------------------------------------------------------------
      Requirement                                         Version   MuSCoW
  --- --------------------------------------------------- --------- --------
      The IAR Password Replace module must allow          1         M
      Reviewers to enter their User name.                           

      The IAR Password Replace module must validate user  1         M
      name.                                                         

      If user name is not valid, the IAR Password Replace 1         M
      module must display a message indicating invalid              
      username and prompt Reviewer to try again or                  
      contact SRA/GTA.                                              

      The IAR Password Replace module must allow Reviewer 1         M
      3 attempts to provide a valid user name.                      

      If user name is valid, the IAR Password Replace     1         M
      module must display the question entered by the               
      Reviewer at time of account creation.                         

      The IAR Password Replace module must allow Reviewer 1         M
      to enter the answer to the question.                          

      The IAR Password Replace module must validate the   1         M
      answer to the question.                                       

      If answer is not valid, the IAR Password module     1         M
      must display a message indicating invalid answer              
      and prompt Reviewer to try again or contact                   
      SRA/GTA.                                                      

      The IAR Password replace module must allow Reviewer 1         M
      3 attempts to provide a valid answer.                         

      If all 3 attempts to provide valid username fail,   1         M
      Reviewer will be alerted to contact their SRA/GTA             
      or eRA Helpdesk.                                              

      If all 3 attempts to provide valid answer fail,     1         M
      Reviewer will be alerted to contact their SRA/GTA             
      or eRA Helpdesk.                                              

      If answer is valid, system must require Reviewer to 1         M
      create a new password.                                        

      The module must accept a password of Reviewer's     1         M
      choice (must adhere to reasonable security rules)             

      The module must allow Reviewer to retype the        1         M
      password (since the password field is masked with             
      (\*) asterisks.                                               

      After password is successfully reset, system will   1         M
      present Reviewer with IAR log on screen.                      
  --------------------------------------------------------------------------

## Account Access and Expiration

  --------------------------------------------------------------------------
      Requirement                                         Version   MuSCoW
  --- --------------------------------------------------- --------- --------
      Once in the IAR system, Reviewers will be able to   1         M
      access all active meetings that they participate in           
      as Reviewers.                                                 

      SRA/GTA must have access to all their active (prior 1         M
      to meeting closure date) IAR meetings.                        

      The list of Meetings shall display the meeting      1         M
      identifiers, SRA of the meeting, meeting start                
      date, meeting phase (SUBMIT, READ, EDIT).                     

      Reviewers' accounts will become inactive after 1    1         M
      year of IAR inactivity. Enabling the Reviewer for a           
      new meeting will reactivate account.                          
  --------------------------------------------------------------------------

## Changing Passwords

Changing of passwords is a standard eRA website utility available to all
eRA web development initiatives. This system will adhere to the
standards set forth by Commons 2.

## Submit Phase

The Submit Phase is the first phase of the IAR workflow. During the
Submit phase, Reviewers post and edit their critiques. They will only
have access to their assigned applications and can only view critiques
they post. They cannot view critiques posted by other Reviewers. All
functions needed in the Submit phase will be accessible from the
Reviewer list of applications screen. From Reviewer List of Applications
screen, Reviewers may:

-   See all applications assigned to them for one meeting.

-   Submit the Critique and/or Score.

-   Re-Submit Critique and/or Score.

-   See critiques and scores they have submitted.

-   SRA/GTAs may:

```{=html}
<!-- -->
```
-   See all applications in the meeting.

-   See critiques and scores for the applications one at a time.

-   See critiques and scores batched by application into one file.

-   See critiques and scores batched by reviewer into one file.

-   See ALL critiques and scores for the meeting batched into one file.

-   Submit the Critique and/or score for the Reviewer.

+---+--------------------------------------------------+------+--------+
|   | Requirement                                      | Ver  | MuSCoW |
|   |                                                  | sion |        |
+===+==================================================+======+========+
|   | **REVIEWER'S VIEW**                              |      |        |
+---+--------------------------------------------------+------+--------+
|   | After successful login, the system will display  | 1    | M      |
|   | a List of Meetings where Reviewer has            |      |        |
|   | assignments.                                     |      |        |
+---+--------------------------------------------------+------+--------+
|   | The list of Meeting screen should display        | 1    | M      |
|   | meeting identifier, meeting title, meeting       |      |        |
|   | dates, phase (Submit, Read, Edit), SRA name and  |      |        |
|   | contact information, Critique due date and time, |      |        |
|   | edit phase start and end date and time.          |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system must allow Reviewer to easily         | 1    | M      |
|   | navigate from List of Meetings to their assigned |      |        |
|   | List of Applications for a specified meeting.    |      |        |
+---+--------------------------------------------------+------+--------+
|   | If meeting is telephone conference, system       | 1    | S      |
|   | should display "Teleconference" with meeting     |      |        |
|   | title.                                           |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen must allow       | 1    | M      |
|   | Reviewer to easily navigate to screen for        |      |        |
|   | submitting and viewing their critiques and       |      |        |
|   | scores.                                          |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen for Reviewers    | 1    | M      |
|   | should provide a way to easily identify New      |      |        |
|   | Investigators.                                   |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen for the Reviewer | 1    | M      |
|   | should display the Reviewer's name (Last, First  |      |        |
|   | and M.I.), meeting identifier, meeting title,    |      |        |
|   | meeting phase, meeting dates, critique due date  |      |        |
|   | and time and a list of their assigned            |      |        |
|   | applications.                                    |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen for the Reviewer | 1    | M      |
|   | should display a message instructing them to     |      |        |
|   | contact their SRA/GTA if they identify conflicts |      |        |
|   | or assignment discrepancies in IAR.              |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen for the Reviewer | 1    | S      |
|   | should provide easy access to SRA contact        |      |        |
|   | information (name, email and phone from WRK      |      |        |
|   | addr)                                            |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen for the Reviewer | 1    | M      |
|   | must display the following for each application: |      |        |
|   | application number, PI name, new PI indicator,   |      |        |
|   | project title, assignment role, score and        |      |        |
|   | critique submitted date and time.                |      |        |
+---+--------------------------------------------------+------+--------+
|   | If a critique has been submitted for an          | 1    | M      |
|   | application, the List of Applications should     |      |        |
|   | display these action items: View, Submit,        |      |        |
|   | Delete.                                          |      |        |
+---+--------------------------------------------------+------+--------+
|   | If a critique has not been submitted for an      | 1    | M      |
|   | application, the List of Applications should     |      |        |
|   | display the Submit action item.                  |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen for Reviewer's   | 1    | M      |
|   | should have a default sort order of PI name with |      |        |
|   | a secondary sort on Activity Code/IC/Serial      |      |        |
|   | Number.                                          |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen for Reviewer's   | 1    | S      |
|   | should allow reviewer to sort their list of      |      |        |
|   | applications by these column headings:           |      |        |
|   | application number (activity code/IC/serial), PI |      |        |
|   | name (secondary sort on application number),     |      |        |
|   | assignment role (secondary sort on PI name),     |      |        |
|   | score, and critique submitted date (show blanks  |      |        |
|   | on top and do secondary sort on application      |      |        |
|   | number).                                         |      |        |
+---+--------------------------------------------------+------+--------+
|   | The default List of Applications screen for the  | 1    | S      |
|   | Reviewer should show only applications assigned  |      |        |
|   | to the Reviewer but provide access to show "All  |      |        |
|   | Applications," if the SRA has opened the meeting |      |        |
|   | for unassigned critiques or comments to be       |      |        |
|   | posted                                           |      |        |
+---+--------------------------------------------------+------+--------+
|   | The All Applications list should show all        | 1    | M      |
|   | applications for the meeting, including those    |      |        |
|   | with conflicts.                                  |      |        |
+---+--------------------------------------------------+------+--------+
|   | On the All Applications list, if a Reviewer is   | 1    | M      |
|   | in conflict with an application, the assignment  |      |        |
|   | role should show "COI" and the Reviewer should   |      |        |
|   | not be able to perform any action (Submit, View, |      |        |
|   | Delete).                                         |      |        |
+---+--------------------------------------------------+------+--------+
|   | At the time of posting of each critique, the     | 1    | W      |
|   | system should remind the reviewer whether there  |      |        |
|   | are any special considerations involved. These   |      |        |
|   | special considerations should be flagged:        |      |        |
|   |                                                  |      |        |
|   | a\. The involvement of human subjects can be     |      |        |
|   | flagged (this could be tied to a optional text   |      |        |
|   | entry screen where the text can be auto-appended |      |        |
|   | to the reviewer's posted critique or the         |      |        |
|   | reviewer may wish to complete their critique and |      |        |
|   | post it later)                                   |      |        |
|   |                                                  |      |        |
|   | b\. Vertebrate animal subjects involved.         |      |        |
|   |                                                  |      |        |
|   | c\. New R01 investigator reminder with a simple  |      |        |
|   | flag.                                            |      |        |
|   |                                                  |      |        |
|   | d\. Reminder with a simple flag that an extra    |      |        |
|   | section is required when the application is      |      |        |
|   | submitted by a foreign organization. (Again, an  |      |        |
|   | optional text box might be tied to the flag.)    |      |        |
|   |                                                  |      |        |
|   | 3/25/02 Focus Group felt this was not necessary  |      |        |
|   | and instead added the next requirement.          |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen should display a      | 1    | S      |
|   | reminder about special considerations. 4/8/02    |      |        |
|   | Group agreed on this language:                   |      |        |
|   |                                                  |      |        |
|   | IMPORTANT REMINDER---Human Subjects are part of  |      |        |
|   | the review criteria and need to be assessed by   |      |        |
|   | the assigned reviewers. This includes:           |      |        |
|   |                                                  |      |        |
|   | protection of human subjects from research       |      |        |
|   | risks; data and safety monitoring; inclusion of  |      |        |
|   | women; inclusion of minorities; inclusion of     |      |        |
|   | children. NIH policy also requires the review    |      |        |
|   | panel to consider the following items:           |      |        |
|   |                                                  |      |        |
|   | -   animal welfare                               |      |        |
|   |                                                  |      |        |
|   | -   biohazards; budgetary overlap                |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen for Reviewers should  | 1    | M      |
|   | show the application number, project title, pi   |      |        |
|   | name, reviewer type and score (if already        |      |        |
|   | entered).                                        |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen must allow Reviewers  | 1    | M      |
|   | to browse their computer to find and select a    |      |        |
|   | file to upload to IAR.                           |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen for Reviewers should  | 1    | M      |
|   | accept critiques in Word or Plain text format.   |      |        |
+---+--------------------------------------------------+------+--------+
|   | Critiques should be stored in their native       | 1    | M      |
|   | format (Word or Plain text).                     |      |        |
+---+--------------------------------------------------+------+--------+
|   | Critiques should be displayed in Adobe PDF       | 1    | M      |
|   | format (to be word-processor independent).       |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen must allow Reviewers  | 1    | M      |
|   | to submit numeric preliminary scores or choose   |      |        |
|   | either DF (deferred) or NR (not recommended) or  |      |        |
|   | UN/NC (unscored/not competitive). Only one is    |      |        |
|   | permitted.                                       |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen should only accept a  | 1    | M      |
|   | score of one or two digits or two digits with    |      |        |
|   | decimal point (acceptable range is 1.0--5.0)     |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen should convert a      | 1    | M      |
|   | 2-digit score to 2-digit with decimal (example   |      |        |
|   | 20 would be converted to 2.0) and convert        |      |        |
|   | 1-digit to 2-digit with decimal (example 3 would |      |        |
|   | be converted to 3.0).                            |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen should allow          | 2    | C      |
|   | Reviewers to submit user-defined alphanumeric    |      |        |
|   | preliminary scores.                              |      |        |
+---+--------------------------------------------------+------+--------+
|   | If a Reviewer submits an alphanumeric score, the | 2    | C      |
|   | Critique Upload screen should limit the entry to |      |        |
|   | 3 characters.                                    |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload Screen should verify that    | 2    | C      |
|   | the alphanumeric score submitted by the Reviewer |      |        |
|   | exists on the score list of values (acceptable   |      |        |
|   | values need to be determined by group).          |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen must allow Reviewers  | 1    | M      |
|   | to resubmit (edit) their critiques and scores.   |      |        |
+---+--------------------------------------------------+------+--------+
|   | After successful upload of Critique and/or       | 1    | M      |
|   | Score, system must provide a confirmation        |      |        |
|   | screen.                                          |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system might allow the Reviewer to input of  |      | W      |
|   | a score mark as conditional with an explanation. |      |        |
|   | Often reviewers indicate their score is          |      |        |
|   | conditional on outcome of discussion. 3/25/02    |      |        |
|   | Group felt this was not necessary since scores   |      |        |
|   | are preliminary and therefore conditional.       |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system must allow the Reviewer to view       | 1    | M      |
|   | critiques and preliminary scores they have       |      |        |
|   | posted for individual applications.              |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system must allow the Reviewer to generate a | 1    | S      |
|   | printable report of critiques and preliminary    |      |        |
|   | scores they have posted for all applications.    |      |        |
|   | Critiques will be assembled by PI name.          |      |        |
+---+--------------------------------------------------+------+--------+
|   | The printable report of critiques and            | 1    | S      |
|   | preliminary scores posted by a Reviewer should   |      |        |
|   | be assembled by PI name.                         |      |        |
+---+--------------------------------------------------+------+--------+
|   | The printable report of critiques and            | 1    | S      |
|   | preliminary scores posted by a Reviewer must     |      |        |
|   | have a page break between each critique.         |      |        |
+---+--------------------------------------------------+------+--------+
|   | The printable report of critiques and            | 1    | S      |
|   | preliminary scores posted by a Reviewer must     |      |        |
|   | show the following: PI Name, Grant Number,       |      |        |
|   | Critique Submitted Date, Score (if entered),     |      |        |
|   | Assignment Role, Critique Text.                  |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system must track date and time the critique | 1    | M      |
|   | was posted.                                      |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system could allow Reviewers to view         | 3    |        |
|   | abstracts.                                       |      |        |
+---+--------------------------------------------------+------+--------+
|   | If specified by the SRA/GTA within the IAR       | 1    | S      |
|   | Control Center, the system should allow          |      |        |
|   | Reviewers to enter critiques/comments for        |      |        |
|   | applications where they are not assigned.        |      |        |
+---+--------------------------------------------------+------+--------+
|   | The score entry for the unassigned application   | 1    | M      |
|   | is NOT allowed.                                  |      |        |
+---+--------------------------------------------------+------+--------+
|   | **SRA/GTA'S VIEW**                               |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen must provide     | 1    | M      |
|   | SRA/GTAs easy access to corresponding screens    |      |        |
|   | for posting and viewing critiques and scores.    |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen should provide   | 2    | S      |
|   | SRA/GTAs with the ability to toggle the ability  |      |        |
|   | to show/hide Discussants, Mail Reviewers and     |      |        |
|   | Readers.                                         |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen for SRA/GTAs     | 1    | M      |
|   | should provide a way to easily identify New      |      |        |
|   | Investigators.                                   |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen for the SRA/GTA  | 1    | M      |
|   | should display the SRA's name, meeting           |      |        |
|   | identifier, meeting phase, critique due date and |      |        |
|   | all applications for the meeting.                |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen for the SRA/GTA  | 1    | M      |
|   | must display the following for each application: |      |        |
|   | application number, pi name, project title,      |      |        |
|   | reviewer name, assignment role, score and        |      |        |
|   | critique submitted date.                         |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen for SRA/GTAs     | 1    | S      |
|   | should have a default sort order of PI name with |      |        |
|   | a secondary sort on Activity Code/IC/Serial      |      |        |
|   | Number.                                          |      |        |
+---+--------------------------------------------------+------+--------+
|   | The List of Applications screen should allow     | 1    | S      |
|   | SRA/GTA to sort their list of applications by    |      |        |
|   | these column headings: application number        |      |        |
|   | (Activity Code/IC/Serial Number), PI name,       |      |        |
|   | reviewer name, assignment role, score and        |      |        |
|   | critique submitted date.                         |      |        |
+---+--------------------------------------------------+------+--------+
|   | On the list of Applications screen for SRA/GTAs  | 1    | S      |
|   | a sort on critique submitted date should show    |      |        |
|   | blank dates at the top of the list.              |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen for SRA/GTAs should   | 1    | M      |
|   | show the application number, project title, PI   |      |        |
|   | name, and score (if already entered).            |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen for SRA/GTAs must     | 1    | M      |
|   | allow SRA/GTA to choose the Reviewer Name that   |      |        |
|   | should be associated with the critique they need |      |        |
|   | to upload.                                       |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen must allow SRA/GTAs   | 1    | M      |
|   | to browse their computer to find and select a    |      |        |
|   | file to upload to IAR.                           |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen for the SRA/GTAs      | 1    | M      |
|   | should accept critiques in Word or plain text    |      |        |
|   | format.                                          |      |        |
+---+--------------------------------------------------+------+--------+
|   | Critiques should be stored in their native       | 1    | M      |
|   | format (Word or Plain text).                     |      |        |
+---+--------------------------------------------------+------+--------+
|   | Critiques should be displayed in Adobe PDF       | 1    | M      |
|   | format (to be word-processor independent).       |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen must allow SRA/GTAs   | 1    | M      |
|   | to submit numeric preliminary scores or choose   |      |        |
|   | either DF (deferred) or NR (not recommended) or  |      |        |
|   | UN/NC (unscored/not competitive). Only one is    |      |        |
|   | permitted.                                       |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen should only accept a  | 1    | S      |
|   | score of two digits or two digits with decimal   |      |        |
|   | point (acceptable range is 1.0--5.0)             |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen should convert a      | 1    | S      |
|   | 2-digit score to 2-digit with decimal (example   |      |        |
|   | 20 would be converted to 2.0) and convert        |      |        |
|   | 1-digit to 2-digit with decimal (example 3 would |      |        |
|   | be converted to 3.0).                            |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen should allow          | 2    | C      |
|   | Reviewers to submit user-defined alphanumeric    |      |        |
|   | preliminary scores.                              |      |        |
+---+--------------------------------------------------+------+--------+
|   | If an SRA/GTA submits an alphanumeric score, the | 2    | C      |
|   | Critique Upload screen should limit the entry to |      |        |
|   | 3 characters.                                    |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload Screen should verify that    | 2    | C      |
|   | the alphanumeric score submitted by the SRA/GTA  |      |        |
|   | exists on the score list of values (acceptable   |      |        |
|   | values need to be determined by group).          |      |        |
+---+--------------------------------------------------+------+--------+
|   | The Critique Upload screen must allow the        | 1    | M      |
|   | SRA/GTAs to resubmit (edit) critiques and scores |      |        |
|   | for Reviewers.                                   |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system must allow the SRA/GTA to view        | 1    | M      |
|   | critiques and preliminary scores.                |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system must allow the SRA/GTA to generate a  | 1    | M      |
|   | printable report of all meeting critiques and    |      |        |
|   | preliminary scores grouped by application.       |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system must allow the SRA/GTA to generate a  | 2    | S      |
|   | printable report of all meeting critiques and    |      |        |
|   | preliminary scores grouped by reviewer.          |      |        |
+---+--------------------------------------------------+------+--------+
|   | The printable reports of all meeting critiques   | 1    | M      |
|   | should separate each critique with a page break. |      |        |
+---+--------------------------------------------------+------+--------+
|   | The printable reports should be available in two | 1    | S      |
|   | formats: MS Word documents, Adobe PDF documents. |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system must track date and time of critique  | 1    | M      |
|   | posting.                                         |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system could allow the SRAs/GTAs to post the | 3    |        |
|   | application abstract themselves (for reviewers   |      |        |
|   | to read or that can be used to auto assemble the |      |        |
|   | summary statement).                              |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system could include online, completely      | 3    |        |
|   | electronic, conflict of interest forms.          |      |        |
+---+--------------------------------------------------+------+--------+
|   | Another option for deadline dates for posting    |      | W      |
|   | critiques can be considered. Once a reviewer has |      |        |
|   | posted their finalized critique, they can be     |      |        |
|   | blocked from editing it and will then            |      |        |
|   | immediately be able to see other posted          |      |        |
|   | critiques for that application. 3/25/02 After    |      |        |
|   | discussion, Group decided not to include this    |      |        |
|   | feature in the system. Most critiques aren't     |      |        |
|   | submitted until 48 hours before the Read phase   |      |        |
|   | so this would have little benefit and may        |      |        |
|   | introduce problems since Reviewers often need to |      |        |
|   | modify a critique during the Post phase.         |      |        |
+---+--------------------------------------------------+------+--------+

## Read Phase

During the Read Phase, IAR is opened for Reviewers to read critiques
posted by other Reviewers. This enables Reviewers to be prepared for
Review meeting discussions. If a Reviewer failed to submit their
critique during the Submit Phase, the SRA/GTA may specify via the
Control Center that the Reviewer cannot read other critiques until they
submit their own. If a Reviewer is blocked from reading, their only
option is to submit their late critique. Once submitted, the system
automatically unblocks the Reviewer from reading on that application.
Critiques cannot be modified during the Read Phase.

  -------------------------------------------------------------------------
      Requirement                                        Version   MuSCoW
  --- -------------------------------------------------- --------- --------
      The system should not allow critiques to be        1         M
      modified by Reviewers during the read phase.                 

      The system must never allow Reviewers to see       1         M
      critiques or scores for applications on which they           
      are in conflict.                                             

      Unless blocked by the SRA/GTA or in conflict, the  1         M
      system should allow all [Regular]{.underline}                
      Reviewers to view critiques and scores for all               
      applications.                                                

      For blocking, subprojects should be treated as     1         M
      other applications. If a Reviewer is assigned to 2           
      subprojects and didn't submit critique for one,              
      he/she will only be blocked from reading that                
      application's critiques, not the entire project.             

      Unless blocked by the SRA/GTA for not submitting   1         M
      or in conflict, the system should allow all                  
      [Telephone]{.underline} Reviewers to view all                
      critiques and scores for their assigned                      
      applications (this is the default view).                     

      If specified by the SRA/GTA in the IAR control     1         M
      center, the Telephone reviewers may view all                 
      meeting critiques (unless in conflict or blocked             
      by the SRA/GTA for not submitting)                           

      The system should not allow Mail-in Reviewers to   1         M
      view critiques submitted by others.                          

      For the Reviewer, the application number, pi name, 1         M
      assignment type, average score, score and the date           
      and time of posting need to be displayed in the              
      critique header. Critiques will be viewed in Adobe           
      pdf format.                                                  

      For the SRA/GTA, the application number, PI name,  1         M
      Reviewer name, assignment type, average score,               
      score and the date and time of posting need to be            
      displayed in the critique header. Critiques will             
      be viewed in Adobe pdf format.                               

      If the SRA/GTA designates in IAR Control Center to 1         M
      hide scores, they should not display on Critiques.           

      Critiques or comments from unassigned Reviewers    1         M
      should be marked as "Unassigned" when viewing the            
      list of critiques.                                           

      The system should allow a blocked (by the SRA/GTA) 1         M
      Reviewer to post their critique.                             

      After a blocked Reviewer posts a late critique     1         M
      during the read only phase his block must be                 
      lifted for that application and he will be able to           
      read the critiques submitted by other Reviewers.             

      Default sort for the SRA/GTA "View by Reviewer"    1         M
      should be first by Reviewer Name, second by Role             
      and third by PI name.                                        

      A sort on Application number should have a         1         M
      secondary sort on PI Name.                                   

      A sort on PI name should have a secondary sort on  1         M
      Application number.                                          

      A sort by Average Score should have a secondary    1         M
      sort on PI name.                                             

      Some SRA/GTAs read critiques as they are added to  2         S
      the ER Web site allowing them to be better                   
      prepared for meeting and to spot potential                   
      problems. A useful feature would be the ability to           
      mark an application as read and approved by the              
      SRA/GTA to help streamline the assembly of triaged           
      summary statements in particular. If a critique is           
      updated then the check mark will be removed                  
      automatically.                                               

      Allow reviewers to submit comments after the                 W
      submission deadline \[these comments would be                
      marked as after the deadline and changes to the              
      pre-deadline critiques should not be allowed until           
      after the meeting\]. This might be limited to the            
      unassigned or might also include assigned                    
      reviewers, or comments could be mailed to SRA/GTA            
      who will have ability to post for the unassigned             
      or assigned. Development of such a function will             
      require more policy input. Clearly, the timing the           
      peer review process would have to be substantially           
      changed for reviewers to utilize it optimally.               
      Careful thought has to be given to such a step and           
      clear policies developed. 4/25/02 discussion,                
      group decided that during the read phase, no one             
      should be permitted to submit critiques for                  
      unassigned applications. This feature was not                
      recommended.                                                 
  -------------------------------------------------------------------------

## Streamlining

Streamlining is the process for identifying applications that will not
be reviewed at the meeting. To accomplish this task, both SRAs and
Reviewers must agree on the list of applications (usually with high
scores or NR) that will not be reviewed. Any member can object to the
streamlining of an application. To assist in identifying the
applications, there is a need for score matrix screen that will list all
applications in the meeting with scores. The screen will allow SRAs to
identify streamlined applications. Streamlining allows more time for
discussion of applications and shortens meetings.

+---+--------------------------------------------------+------+--------+
|   | Requirement                                      | Ver  | MuSCoW |
|   |                                                  | sion |        |
+===+==================================================+======+========+
|   | The system shall allow users to navigate to the  | 1    | M      |
|   | score matrix screen from the list of             |      |        |
|   | applications screen.                             |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system shall allow users to navigate to the  | 1    | M      |
|   | list of applications from the score matrix       |      |        |
|   | screen.                                          |      |        |
+---+--------------------------------------------------+------+--------+
|   | System shall provide a score matrix view of      | 1    | M      |
|   | applications in the meeting for both SRA/GTAs    |      |        |
|   | and Reviewers.                                   |      |        |
+---+--------------------------------------------------+------+--------+
|   | System shall allow reviewers to access score     | 1    | M      |
|   | matrix in Read phase.                            |      |        |
+---+--------------------------------------------------+------+--------+
|   | System shall allow SRA/GTAs to access score      | 1    | M      |
|   | matrix in both Submit and Read phases.           |      |        |
+---+--------------------------------------------------+------+--------+
|   | Score matrix screen should show application      | 1    | M      |
|   | number, PI name, average score, individual       |      |        |
|   | scores, and lower half indicator.                |      |        |
+---+--------------------------------------------------+------+--------+
|   | Average score is computed only if every score    | 1    | M      |
|   | for the application is numeric.                  |      |        |
+---+--------------------------------------------------+------+--------+
|   | SRA/GTA must be able to edit the lower half      | 1    | M      |
|   | indicator to identify an application as          |      |        |
|   | streamlined.                                     |      |        |
+---+--------------------------------------------------+------+--------+
|   | SRA/GTA and Reviewers must be able to toggle     | 1    | M      |
|   | between the complete list of applications and    |      |        |
|   | the streamlined only applications list.          |      |        |
+---+--------------------------------------------------+------+--------+
|   | Reviewers cannot edit anything on score matrix   | 1    | M      |
|   | screen.                                          |      |        |
+---+--------------------------------------------------+------+--------+
|   | System shall filter out applications where       | 1    | M      |
|   | Reviewers are in conflict.                       |      |        |
+---+--------------------------------------------------+------+--------+
|   | If Reviewer is blocked from reading critiques    | 1    | M      |
|   | (if they didn't submit) they cannot see the      |      |        |
|   | score matrix for that application.               |      |        |
+---+--------------------------------------------------+------+--------+
|   | Score matrix should allow sort by                | 1    | M      |
|   | activity/IC/serial number, IC/Serial number,     |      |        |
|   | activity/PI name, PI name, Lower Half, Average.  |      |        |
+---+--------------------------------------------------+------+--------+
|   | The default sort order for applications on the   | 1    | M      |
|   | score matrix screen should be by PI name.        |      |        |
+---+--------------------------------------------------+------+--------+
|   | Subprojects should be sorted under the parent    | 1    | M      |
|   | application (applications grouped by Parent PI). |      |        |
|   | The subproject should show the Core Leader       |      |        |
|   | instead of Parent PI.                            |      |        |
+---+--------------------------------------------------+------+--------+
|   | The lower half sort shall be sorted by:          | 1    | M      |
|   |                                                  |      |        |
|   | Lower Half, Activity Code, PI Name, Average,     |      |        |
|   | where applications without lower half            |      |        |
|   | designation and without average are sorted to    |      |        |
|   | the top, then the lower half applications, then  |      |        |
|   | average descending (from worst to best).         |      |        |
|   |                                                  |      |        |
|   | Lower Half, PI Name, Average, where applications |      |        |
|   | without lower half designation and without       |      |        |
|   | average are sorted to the top, then the lower    |      |        |
|   | half applications, then average descending (from |      |        |
|   | worst to best).                                  |      |        |
+---+--------------------------------------------------+------+--------+
|   | The sort for average should sort the mixed       | 1    | M      |
|   | scores (no average) AND no Lower half            |      |        |
|   | designation (x) to the top, then average         |      |        |
|   | ascending (best to worst), then all the          |      |        |
|   | application designated as a Lower Half (x).      |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system shall prevent Telephone reviewers     | 1    | M      |
|   | from seeing applications for which they are not  |      |        |
|   | assigned unless designated by SRA/GTA in Control |      |        |
|   | Center.                                          |      |        |
+---+--------------------------------------------------+------+--------+
|   | If scores are not visible (as designated by      | 1    | M      |
|   | SRA/GTA in Control Center) Reviewer will not see |      |        |
|   | score portion of score matrix---they will only   |      |        |
|   | see lower half.                                  |      |        |
+---+--------------------------------------------------+------+--------+
|   | The system shall prevent Mail-in reviewers from  | 1    | M      |
|   | seeing the score matrix.                         |      |        |
+---+--------------------------------------------------+------+--------+
|   | Reviewers should not see scores (score matrix or | 1    | M      |
|   | on list of applications screen) after end date   |      |        |
|   | of meeting.                                      |      |        |
+---+--------------------------------------------------+------+--------+
|   | The ability to enter or modify scores on the     | 2    | S      |
|   | Critique Upload screen should be disabled after  |      |        |
|   | the end date of the meeting.                     |      |        |
+---+--------------------------------------------------+------+--------+
|   | It would be useful for SRA/GTAs to control the   | 2    | C      |
|   | numeric score assigned to applications that the  |      |        |
|   | reviewers have designated as "UN" or "LH." ER    |      |        |
|   | assigns a score of 0 to unscored applications    |      |        |
|   | when computing averages. Thus, an application    |      |        |
|   | with the following scores: LH, LH, 2.0 is        |      |        |
|   | assigned an average of 2.0, whereas an           |      |        |
|   | application with scores of 2.1, 2.2, 2.0 is      |      |        |
|   | given an average of 2.1. This reduces the        |      |        |
|   | utility of using the score matrix to monitor     |      |        |
|   | spreading of scores and could lead to            |      |        |
|   |                                                  |      |        |
|   | Confusion on the part of reviewers. If SRA/GTAs  |      |        |
|   | are not given control over the handling of LHs,  |      |        |
|   | then it might be reasonable to assign a 4.0 to   |      |        |
|   | all LH nominations.                              |      |        |
+---+--------------------------------------------------+------+--------+
|   | UN/LH voting. Reviewers could have the ability   | 2    | C      |
|   | to post streamlining votes. The Reviewers would  |      |        |
|   | pull up their assigned applications and have the |      |        |
|   | ability to select applications for lower half.   |      |        |
+---+--------------------------------------------------+------+--------+
|   | Streamline voting: The SRA/GTA needs to define   | 2    | C      |
|   | ineligible reviewers---Mail Reviewers are        |      |        |
|   | generally not eligible to vote for streamlining  |      |        |
|   | an application; however, others on the committee |      |        |
|   | may wish to see the opinion of the Mail          |      |        |
|   | Reviewer. Thus, a screen with the list of        |      |        |
|   | reviewers and three columns is needed so as to   |      |        |
|   | exclude access, include but display only (i.e.,  |      |        |
|   | don\'t count toward the criteria of two UN       |      |        |
|   | votes), or include fully. All regular reviewers  |      |        |
|   | should default to "include fully" while Mail     |      |        |
|   | Reviewers should default to "display only."      |      |        |
+---+--------------------------------------------------+------+--------+
|   | The SRA/GTA needs to monitor votes---A display   | 2    | C      |
|   | building on the 1500--50 (Tally) screen would be |      |        |
|   | useful, with the number of UN votes (or scores)  |      |        |
|   | displaying next to that utilizing the same set   |      |        |
|   | of columns headings. This would allow the        |      |        |
|   | SRA/GTA to know who hasn't voted at all, who     |      |        |
|   | might have forgotten to vote on discussant       |      |        |
|   | assignments, or who has such a light load that   |      |        |
|   | the lack of UN votes may not be a concern.       |      |        |
+---+--------------------------------------------------+------+--------+
|   | The SRA/GTA needs to be able to exclude          | 2    | C      |
|   | applications from streamlining based on activity |      |        |
|   | code criteria.                                   |      |        |
+---+--------------------------------------------------+------+--------+
|   | There should be a separate date for streamlining | 2    | C      |
|   | to be set and for display. A bold display of the |      |        |
|   | Deadline for Posting (set by the SRA/GTA)        |      |        |
|   | information should appear when reviewers logon   |      |        |
|   | to the web. Any UN votes submitted after the     |      |        |
|   | deadline would register as "late votes" and      |      |        |
|   | would not count toward preliminary streamlining. |      |        |
|   | They need to be confirmed at the meeting.        |      |        |
+---+--------------------------------------------------+------+--------+
|   | Export to Order of Review---Some SRA/GTAs like   | 2    | C      |
|   | to manipulate the Order of Review so as to push  |      |        |
|   | all the UN applications to the bottom of the     |      |        |
|   | list. Such an Export button would transfer the   |      |        |
|   | existing streamlining information to the Order   |      |        |
|   | of Review screen, causing all UN applications to |      |        |
|   | migrate down (but keeping the same order while   |      |        |
|   | doing so) and then be Resequenced.               |      |        |
+---+--------------------------------------------------+------+--------+
|   | Update & Transfer to Score Entry screen---After  | 2    | C      |
|   | the meeting, the SRA/GTA or GTA could update UN  |      |        |
|   | results (add UN\'s or change to D), then         |      |        |
|   | transfer these results to the Master Sheet for   |      |        |
|   | score entry.                                     |      |        |
+---+--------------------------------------------------+------+--------+
|   | (At the push of a button) The system should      | 2    | C      |
|   | provide the ability for the SRA to easily        |      |        |
|   | determine which applications had two or more     |      |        |
|   | lower half votes ("tentative lower half"). The   |      |        |
|   | results should display for Reviewers and         |      |        |
|   | SRA/GTAs on the list of applications screen.     |      |        |
|   | Reviewers not in conflict should have the        |      |        |
|   | ability to register objections to the lower half |      |        |
|   | designations. This will help SRAs and Reviewers  |      |        |
|   | prepare for the meeting and schedule reviews.    |      |        |
+---+--------------------------------------------------+------+--------+
|   | Using scores, the system should determine which  | 2    | C      |
|   | applications have two votes of 3.0 or worse.     |      |        |
+---+--------------------------------------------------+------+--------+
|   | SRA/GTA needs the ability to establish "Floating | 2    | C      |
|   | Cutoff"---If scores or percentile votes are      |      |        |
|   | registered, pushing the Floating Cutoff button   |      |        |
|   | would perform an iterative procedure whereby a   |      |        |
|   | score or percentile is found for which at least  |      |        |
|   | 50% of the applications have two or more scores  |      |        |
|   | as bad or worse than the cutoff. A window should |      |        |
|   | open indicating, for instance, "A cutoff of      |      |        |
|   | [2.6]{.underline} resulted in [55]{.underline}   |      |        |
|   | percent of the applications falling into the     |      |        |
|   | \"floating lower half\" (two or more votes of    |      |        |
|   | [2.6]{.underline} or worse)." An "Accept" button |      |        |
|   | would establish that as the cutoff, while "Step  |      |        |
|   | Back" and "Step Forward" buttons would move the  |      |        |
|   | floating cutoff to worse or better scores.       |      |        |
|   | SRA/GTA should have Cancel button to abort.      |      |        |
+---+--------------------------------------------------+------+--------+
|   | One additional column should be added to the     | 2    | C      |
|   | Viewing Streamlining Votes screen to allow       |      |        |
|   | reviewers to add a late vote (only assigned      |      |        |
|   | reviewers/discussants). The system should allow  |      |        |
|   | "me too" (late) votes to be registered. This     |      |        |
|   | will help SRAs and Reviewers prepare for the     |      |        |
|   | meeting and schedule reviews.                    |      |        |
+---+--------------------------------------------------+------+--------+

## Edit Phase

The SRA may wish to have an after meeting Edit phase during which time
Reviewers can modify their critiques or post critiques for unassigned
applications (if designated by the SRA). IAR functionality previously
defined for the Read Phase and Submit Phase should be applied to the
Edit Phase. Additional Edit Phase requirements are described below.

  --------------------------------------------------------------------------
       Requirement                                       Version   MuSCoW
  ---- ------------------------------------------------- --------- ---------
       Unassigned Reviewers should be allowed to submit  1         M
       critiques (as specified by SRA/GTA in IAR Control           
       Center).                                                    

       Reviewers should be allowed to edit their earlier 1         M
       critiques (by posting a new critique to replace             
       the earlier one).                                           

       System should easily indicate date and time       1         M
       critiques were submitted so SRA/GTA can locate              
       critiques submitted after a certain date/time.              
       This could be achieved by a sort on critique                
       submission date.                                            
  --------------------------------------------------------------------------

## Summary Statement Assembly

Critiques submitted by Reviewers are used to build the summary statement
text. Often these critiques must be slightly modified by the SRA and
additional items are added to the critiques to create the summary
statement. IAR will create a file of merged critiques for each
application, the SRA/GTA could download/save the file locally, modify as
needed and then import into the Prepare Summary Statement screen in Peer
Review.

+---+---------------------------------------------------+------+--------+
|   | Requirement                                       | Ver  | MuSCoW |
|   |                                                   | sion |        |
+===+===================================================+======+========+
|   | After the submit phase ends, the system should    | 1    | M      |
|   | provide the ability to view a file of merged      |      |        |
|   | critiques for each application (pre-summary       |      |        |
|   | statement body).                                  |      |        |
+---+---------------------------------------------------+------+--------+
|   | The pre-summary statement body should be in MS    | 1    | M      |
|   | Word (\*.doc) format.                             |      |        |
+---+---------------------------------------------------+------+--------+
|   | The SRA/GTA should have the ability to            | 1    | M      |
|   | download/locally save the pre-summary statement   |      |        |
|   | body.                                             |      |        |
+---+---------------------------------------------------+------+--------+
|   | The pre-summary statement body file should        | 1    | M      |
|   | conform to IMPAC II summary statement standards:  |      |        |
|   | Arial 11 pt, 0.75-inch margins all around,        |      |        |
|   | 0.5-inch header and footer margins, no section    |      |        |
|   | breaks.                                           |      |        |
+---+---------------------------------------------------+------+--------+
|   | Critiques should be merged in order of assignment | 1    | M      |
|   | priority (Primary, Secondary, Tertiary, Reader,   |      |        |
|   | Discussant, Unassigned) with one blank line       |      |        |
|   | between critiques.                                |      |        |
+---+---------------------------------------------------+------+--------+
|   | The assembled pre-summary statement body should   | 1    | M      |
|   | have a first line like this: grant_num\<space\>PI |      |        |
|   | last name, PI first name.                         |      |        |
+---+---------------------------------------------------+------+--------+
|   | Each critique should begin with "Critique (#)"    | 1    | M      |
|   | heading with \# representing the number of the    |      |        |
|   | critique (1, 2, 3, 4, etc.).                      |      |        |
+---+---------------------------------------------------+------+--------+
|   | If critiques for Primary, Secondary, or Tertiary  | 1    | M      |
|   | are missing, merged file should still contain     |      |        |
|   | "Critique" heading with 4 blank lines.            |      |        |
+---+---------------------------------------------------+------+--------+
|   | If critiques for Readers, Discussants or          | 1    | M      |
|   | Unassigned are missing, "Critique" heading or 4   |      |        |
|   | blank lines should not be included.               |      |        |
+---+---------------------------------------------------+------+--------+
|   | If it is possible to come up with standard text   | 1    | C      |
|   | and placement inside the pre-Summary Statement    |      |        |
|   | body across all ICs for Human Subject             |      |        |
|   | Concerns---the Text should be included in the     |      |        |
|   | document if there are Human Subject Concerns.     |      |        |
+---+---------------------------------------------------+------+--------+
|   | SRA/GTA needs the ability to print all critiques, | 1    | C      |
|   | sorted by PI, with each PI beginning on a new     |      |        |
|   | page. The requested feature would allow all       |      |        |
|   | SRA/GTAs to bring a collated copy of each \"proto |      |        |
|   | summary statement\" to the meeting and be able to |      |        |
|   | refer to them during the meeting. This allows     |      |        |
|   | SRA/GTAs to ensure that key points made at the    |      |        |
|   | meeting are actually written into the critiques.  |      |        |
|   | The ability to add the line number from the vote  |      |        |
|   | sheet is also needed.                             |      |        |
+---+---------------------------------------------------+------+--------+
|   | Critiques for subprojects should be included in   | 1    | M      |
|   | the parent grant pre-summary statement.           |      |        |
|   | Subprojects should be sorted using the order      |      |        |
|   | specified in the Order of Review (in the IMPAC II |      |        |
|   | Peer Review Module). (SRA/GTAs will need to       |      |        |
|   | renumber their Order of Review before the         |      |        |
|   | pre-summary statement bodies are created.) Within |      |        |
|   | each subproject, critiques should be sorted using |      |        |
|   | the standard order of Primary, Secondary, etc.    |      |        |
+---+---------------------------------------------------+------+--------+
|   | The main post-meeting report is the assembled     | 2    | C      |
|   | critiques in a pre-summary statement draft.       |      |        |
|   | Critiques would begin with the heading "Critique" |      |        |
|   | (a nice touch would allow SRA/GTAs to rearrange   |      |        |
|   | the order of critiques; the default order should  |      |        |
|   | be by role). \[Although many reviewers add the    |      |        |
|   | heading "critique," they can be asked not to do   |      |        |
|   | this.\] The description would be added if         |      |        |
|   | available. A further nice touch would create an   |      |        |
|   | output with as many template headings as          |      |        |
|   | possible. So, for example, if there are human     |      |        |
|   | subjects codes, the appropriate headings can be   |      |        |
|   | created in the output. The bolded statement       |      |        |
|   | proposed by OER for separating reviewer and       |      |        |
|   | SRA/GTA remarks can be added. If biohazard of     |      |        |
|   | foreign are checked, these headings can be added, |      |        |
|   | etc. If such an option is provided, it will be    |      |        |
|   | important to be able to toggle off the template.  |      |        |
+---+---------------------------------------------------+------+--------+
|   | **Export to Summary Statement Module.** This      | 2    | C      |
|   | option would formally associate each file for the |      |        |
|   | designated application to allow access through    |      |        |
|   | the summary statement module. Until the button is |      |        |
|   | pushed, the files should remain in a temporary    |      |        |
|   | file. There would need to be an "update" button   |      |        |
|   | that would bring in the most recent posting, and  |      |        |
|   | there should be a warning when a newer version    |      |        |
|   | has been posted. The advantage of this scheme     |      |        |
|   | would be in knowing which version you are working |      |        |
|   | with so that an update would not be posted        |      |        |
|   | without your knowing.                             |      |        |
+---+---------------------------------------------------+------+--------+
|   | **Direct Storage in the Summary Statement         | 2    | C      |
|   | Module.** Posted critiques would be available to  |      |        |
|   | the SRA/GTA through the IMPAC II Peer Review      |      |        |
|   | Summary Statement Module as soon as posted. The   |      |        |
|   | difficulty would be in keeping track of when a    |      |        |
|   | review has been modified. A log could show the    |      |        |
|   | SRA/GTA when updates have been posted, but it     |      |        |
|   | might be difficult to keep track of those changes |      |        |
|   | when working offline on a draft in Word or Plain  |      |        |
|   | text.                                             |      |        |
+---+---------------------------------------------------+------+--------+
|   | Automated Assembly. The IAR and/or the summary    | 2    | C      |
|   | statement module should have a display of which   |      |        |
|   | reviews are in and which are missing. When all    |      |        |
|   | expected reviews are there, an Export Raw Reviews |      |        |
|   | button should assemble the reviews in a           |      |        |
|   | prescribed order (e.g., Primary, Secondary,       |      |        |
|   | Tertiary, Mail, Discussant) and allow the SRA/GTA |      |        |
|   | to save the assembled document on the c: drive    |      |        |
|   | with the prescribed file name format needed for   |      |        |
|   | later upload.                                     |      |        |
|   |                                                   |      |        |
|   | **PROBLEM**: How to deal with files created in    |      |        |
|   | different word processing programs. As noted      |      |        |
|   | above, we'd like to retain special characters. If |      |        |
|   | the SRA/GTA specifies that the downloaded         |      |        |
|   | document should be in Word, for instance, are     |      |        |
|   | there conversion programs to handle a WordPerfect |      |        |
|   | document on the fly?                              |      |        |
+---+---------------------------------------------------+------+--------+
|   | The summary statement contains a "Description"    | 2    | C      |
|   | submitted on the grant application. Since         |      |        |
|   | applications are scanned and bookmarked, this     |      |        |
|   | "Description" section should be evaluated for     |      |        |
|   | feasibility of automatically incorporating it     |      |        |
|   | into the summary statement during                 |      |        |
|   | generation/combination of critiques.              |      |        |
+---+---------------------------------------------------+------+--------+
|   | Pre-summary statement body report for entire      | 2    | C      |
|   | meeting: If SRAs/GTAs have the capacity to        |      |        |
|   | download all the summary statements (collected    |      |        |
|   | critiques under each grant number) with a         |      |        |
|   | separate file for each summary statement (named   |      |        |
|   | with the grant number or PI name), this could be  |      |        |
|   | incorporated into their post meeting processes as |      |        |
|   | a one-time event and will prevent inadvertent     |      |        |
|   | loss of data. The files might be zipped together  |      |        |
|   | for the download in a procedure similar to that   |      |        |
|   | now used in IMPAC II.                             |      |        |
|   |                                                   |      |        |
|   | The critiques for each application should be      |      |        |
|   | ordered (e.g., Primary, Secondary, Tertiary,      |      |        |
|   | Mail, Discussant) and the SRA/GTA needs control   |      |        |
|   | over the order (PI name vs application number).   |      |        |
|   | The insertion of some special character string    |      |        |
|   | (e.g., page breaks) between applications would    |      |        |
|   | allow efficient separation for storage in         |      |        |
|   | individual files.                                 |      |        |
+---+---------------------------------------------------+------+--------+
|   | A feature can be provided to use the text to      | 2    | C      |
|   | assemble the IMPAC II PDF draft summary statement |      |        |
|   | avoiding an intermediary Word file. Often         |      |        |
|   | streamlined summary statements will need no       |      |        |
|   | editing and they can be rapidly released.         |      |        |
|   | However, such a function should be built to avoid |      |        |
|   | inadvertent release of unread critiques. It could |      |        |
|   | be combined with a check box indicating that the  |      |        |
|   | SRA/GTA has approved the critique. The check box  |      |        |
|   | would only be visible on the SRA/GTA's screen     |      |        |
|   | similar to the private check box on the Review    |      |        |
|   | module 1500 screen.                               |      |        |
+---+---------------------------------------------------+------+--------+

## Reports

SRA/GTAs as well as Reviewers need the ability to print reports of IAR
data.

+---+-------------------------------------------------+------+--------+
|   | Requirement                                     | Ver  | MuSCoW |
|   |                                                 | sion |        |
+===+=================================================+======+========+
|   | All screens in IAR should be printable to       | 1    | M      |
|   | provide a hard copy report of what is on the    |      |        |
|   | screen.                                         |      |        |
+---+-------------------------------------------------+------+--------+
|   | System should allow the ability to create a     | 2    | C      |
|   | streamlining report to include PI name,         |      |        |
|   | application number, LH (lower half, no          |      |        |
|   | objection), D (Discuss-Objection), single       |      |        |
|   | votes, late votes. This report can be           |      |        |
|   | distributed to Reviewers at the start of the    |      |        |
|   | meeting. It can also be adapted as, or used to  |      |        |
|   | guide setting up, the actual order of review.   |      |        |
+---+-------------------------------------------------+------+--------+
|   | System should allow the ability to create a     | 2    | C      |
|   | significant difference report. Identification   |      |        |
|   | of significant difference could occur one of    |      |        |
|   | two ways: SRA scans the list of scores and      |      |        |
|   | checks to indicate applications with major      |      |        |
|   | differences of opinions; or, allow SRA to set   |      |        |
|   | their own definition of what would indicate a   |      |        |
|   | significant difference. Reviewer should have    |      |        |
|   | the ability to sort by Lower Half or            |      |        |
|   | Significant Difference.                         |      |        |
+---+-------------------------------------------------+------+--------+
|   | A useful additional feature would be to allow   | 1    | M      |
|   | the reviewers to print their own copies of      |      |        |
|   | their assignment lists. This requirement can be |      |        |
|   | met by allowing Reviewer to print the Reviewer  |      |        |
|   | View during the Submit Phase.                   |      |        |
+---+-------------------------------------------------+------+--------+
|   | The format would need to be SRA/GTA             | 2    | C      |
|   | controlled---either "Assignment List and        |      |        |
|   | Conflicts by Reviewer" (full assignment         |      |        |
|   | information on only those applications assigned |      |        |
|   | to the reviewer) or "Assignment List and        |      |        |
|   | Conflicts by Reviewer (Restricted Version)" (no |      |        |
|   | information on co-reviewers).                   |      |        |
+---+-------------------------------------------------+------+--------+
|   | SRA/GTA should have a Meeting Report 1---A      | 1    | M      |
|   | numbered report displaying PI name and          |      |        |
|   | application number along with the current       |      |        |
|   | scoring and streamlining information. Another   |      |        |
|   | column would print a C next to those            |      |        |
|   | applications for which there is a conflict in   |      |        |
|   | the system. Numbering should be according to    |      |        |
|   | the Order of Review from the Review Module.     |      |        |
|   | Explanation: The concept here is to have a      |      |        |
|   | \"streamlining results\" sheet that can be      |      |        |
|   | included in the meeting folders. Everyone could |      |        |
|   | pull it out as a guide at the start of the      |      |        |
|   | meeting as the Chair calls out applications for |      |        |
|   | confirmation/ objections/additions to           |      |        |
|   | streamlining. If someone is in conflict, any    |      |        |
|   | discussion or consideration of adding that      |      |        |
|   | application to the streamline list would be     |      |        |
|   | deferred. An example format follows:            |      |        |
|   |                                                 |      |        |
|   | STREAMLINED REVIEW RESULTS                      |      |        |
|   |                                                 |      |        |
|   | 01 X ALBERT, RICHARD 1 R01 HL070853-01 4.0 3.0  |      |        |
|   | 2.8                                             |      |        |
|   |                                                 |      |        |
|   | C 02 X BABB, TONY 1 R01 AG021140-01 3.0 2.9 2.8 |      |        |
|   |                                                 |      |        |
|   | 03 BILLMAN, GEORGE 1 R01 HL068609-01A1 1.9 1.5  |      |        |
|   | 1.5                                             |      |        |
|   |                                                 |      |        |
|   | C 04 BORIEK, ALADIN 2 R01 HL046230-12A1 2.9 2.8 |      |        |
|   | 1.9                                             |      |        |
|   |                                                 |      |        |
|   | The X would print for those applications put in |      |        |
|   | the lower half by the SRA. The C next to Boriek |      |        |
|   | would remind the Chair not to ask about adding  |      |        |
|   | this application to the lower half, even though |      |        |
|   | the preliminary votes suggest it might get      |      |        |
|   | streamlined when it comes up in the schedule.   |      |        |
|   |                                                 |      |        |
|   | This requirement could be met by adding a       |      |        |
|   | column for conflicts to the score matrix        |      |        |
|   | screen. Reviewers could print the score matrix  |      |        |
|   | screen for a copy of the report.                |      |        |
+---+-------------------------------------------------+------+--------+
|   | SRA/GTA should have a Meeting Report 2---For    | 2    | C      |
|   | reference, a copy of the master assignment list |      |        |
|   | with reviewers who voted to streamline a        |      |        |
|   | particular application printing in bold.        |      |        |
|   | Sample:                                         |      |        |
|   |                                                 |      |        |
|   | 1 1 R01 HL072472-01 ANNAPRAGADA, ANANTH V (P1)  |      |        |
|   | Tsuda, A Hsia, C                                |      |        |
|   |                                                 |      |        |
|   | CFD Simulation of the human respiratory system  |      |        |
|   | (S1) Loring, S Mitzner, W                       |      |        |
|   |                                                 |      |        |
|   | CLEVELAND STATE UNIVERSITY                      |      |        |
|   |                                                 |      |        |
|   | 2 1 R01 HL069030-01A1 BISSONNETTE, JOHN M (P1)  |      |        |
|   | Mifflin, S Donnelly, D                          |      |        |
|   |                                                 |      |        |
|   | Calcium-Activated K+ Channels and Respiratory   |      |        |
|   | Control (S1) Gozal, D Bonham, A                 |      |        |
|   |                                                 |      |        |
|   | OREGON HEALTH & SCIENCE UNIVERSITY              |      |        |
|   |                                                 |      |        |
|   | Where Tsuda and Mitzner had voted LH for        |      |        |
|   | Annapragada                                     |      |        |
+---+-------------------------------------------------+------+--------+
|   | SRA/GTA should have the ability to print the    | 1    | M      |
|   | Preliminary Score Matrix View for SRA/GTA. It   |      |        |
|   | must print all rows.                            |      |        |
+---+-------------------------------------------------+------+--------+
|   | SRA/GTA will need a printable report of the     | 2    | C      |
|   | lower half list (to take as the official list   |      |        |
|   | to the meeting) and their associated combined   |      |        |
|   | critiques.                                      |      |        |
+---+-------------------------------------------------+------+--------+
|   | SRA/GTA will need a printable report of a list  | 2    | C      |
|   | of applications that have been nominated by one |      |        |
|   | reviewer for streamlining.                      |      |        |
+---+-------------------------------------------------+------+--------+
|   | SRA/GTA will need a Critique Posting Status     | 1    | M      |
|   | Report showing Application number, PI name,     |      |        |
|   | Reviewer name, assignment priority and last     |      |        |
|   | updated date for the critique. If critique is   |      |        |
|   | missing date field should be blank. Report      |      |        |
|   | should be sortable by Reviewer name and         |      |        |
|   | assignment priority. This report will be added  |      |        |
|   | to Peer Review Reports menu.                    |      |        |
+---+-------------------------------------------------+------+--------+
|   | SRA/GTA will need a printable report of the     | 2    | C      |
|   | significant difference list and their           |      |        |
|   | associated combined critiques.                  |      |        |
+---+-------------------------------------------------+------+--------+

## IC Program Officer Access

6/10/02 The majority of IAR Focus Group members recommend Program
Officers do not have access to IAR. These items will be revisited at a
later meeting but will most likely become "Won't"s.

  -------------------------------------------------------------------------
       Requirement                                       Version   MuSCoW
  ---- ------------------------------------------------- --------- --------
       SRA/GTA should have the option of making the      2         C
       lower half list public to Program Officers. This            
       allows the PO to optimize their travels between             
       SRG meetings.                                               

       System should allow the ability to release/email  2         C
       to program staff a streamlining report to include           
       PI name, application number, LH (lower half, no             
       objection), D (Discuss-Objection), single votes,            
       late votes.                                                 

       The system could place critique text directly     2         C
       into draft summary statements. In IMPAC II, there           
       is already a Preview mode for SRA/GTAs to share             
       summary statements with POs. If critiques can be            
       directly loaded as draft summary statements then            
       the preview feature will allow quick IC access if           
       necessary.                                                  
  -------------------------------------------------------------------------

## Purging Assignments

Currently, Reviewer assignments are purged when the meeting is released
(usually 3 days after the meeting). Many SRA/GTAs have an Edit phase
after the meeting where the Reviewers can modify their critiques. This
Edit phase will extend past the meeting release. However, once
assignments are removed, Reviewers can't resubmit critiques. IAR is
proposing a slight change to the current policy on purging assignments
to allow Reviewers to modify their critiques after the meeting release.

+---+-------------------------------------------------+------+--------+
|   | Requirement                                     | Ver  | MuSCoW |
|   |                                                 | sion |        |
+===+=================================================+======+========+
|   | Delete assignment information (reviewers,       | 1    | M      |
|   | assignments to applications, and conflicts) 15  |      |        |
|   | days (TBD---approval needed by RPC) after the   |      |        |
|   | meeting is released.                            |      |        |
+---+-------------------------------------------------+------+--------+
|   | Keep the existing Peer Review ability for the   | 1    | M      |
|   | SRA/GTA to manually purge the assignments.      |      |        |
+---+-------------------------------------------------+------+--------+
|   | After assignments are purged, critiques and     | 1    | M      |
|   | preliminary scores would still be associated    |      |        |
|   | with corresponding priorities (Primary 1,       |      |        |
|   | Secondary, etc.)                                |      |        |
+---+-------------------------------------------------+------+--------+
|   | When meeting is released or assignments are     | 1    | M      |
|   | purged manually, the Peer Review system should  |      |        |
|   | check that the assignment purge date is on or   |      |        |
|   | later than the Edit Phase End Date. If this is  |      |        |
|   | not the case---the user of the system should    |      |        |
|   | get an error message preventing them from doing |      |        |
|   | the task and instructing them to change the     |      |        |
|   | Edit Phase end date if there is a need to       |      |        |
|   | release a meeting or purge assignments.         |      |        |
|   |                                                 |      |        |
|   | Purge Date cannot be earlier than Edit Phase    |      |        |
|   | End Date.                                       |      |        |
+---+-------------------------------------------------+------+--------+
|   | Reviewers should not have access to their       | 1    | M      |
|   | meeting in IAR after the Purge date.            |      |        |
+---+-------------------------------------------------+------+--------+

## Meeting Closure

  --------------------------------------------------------------------------
       Requirement                                       Version   MuSCoW
  ---- ------------------------------------------------- --------- ---------
       The system should automatically set the Meeting   1         M
       closure date to be 6 months from the meeting                
       release date.                                               

       On the Meeting closure date all data in IAR       1         M
       corresponding to the meeting should be deleted              
       (critiques, preliminary scores, etc.)                       
  --------------------------------------------------------------------------

## Other General Features

  -------------------------------------------------------------------------
       Requirement                                       Version   MuSCoW
  ---- ------------------------------------------------- --------- --------
       Privacy for the IMPAC II Review module is at the  1         M
       level of the IRG (or equivalent). The same level            
       of privacy appears to be appropriate for IAR.               
       However, the private check box on the Review                
       assignment screen limiting access to the SRA/GTA            
       only would have to work in the same way for IAR             
       if this privacy mode is to be kept in the Review            
       module.                                                     

       There should be cluster security on the IAR       1         M
       Control Center in Peer Review.                              

       There may be an opportunity to handle Travel      2         C
       Voucher via IAR.                                            
  -------------------------------------------------------------------------

# Constraints

IAR will be developed using the new eRA development environment, J2EE.
Consequently, the technology used for IAR will differ from the current
technology of the Peer Review module. Due to technology and time
constraints, it is recommended that the current Peer Review module only
be modified where necessary to accommodate data or functions critical to
the IAR process. Data and functions accessible through IAR should not be
replicated in Peer Review. With the future redesign of Peer Review in
J2EE, integration of SRA/GTA IAR functions will be addressed and
integrated where feasible.

# Quality Ranges

This section defines the quality ranges for performance, robustness,
fault tolerance, usability, and similar characteristics for this
application. These characteristics will be discussed in more detail in
the Supplemental Specification document.

**Availability:** The System shall be available 24 hours a day, 7 days a
week.

**Usability**: The System shall allow the users to provide 'human
factors' feedback online.

**Usability:** The System shall include on-line help for the user. Users
should not require the use of a hardcopy Manual to use the System.

**Maintainability:** The system shall not hardcode system parameters.

# Precedence and Priority

This section provides some direction on the relative importance of the
proposed system features. Until the detailed requirements are fully
defined, it is difficult to estimate schedules and establish priorities.
As time progresses, this section will be filled in with a prioritized
list of features per release.

# Other Product Requirements

## Applicable Standards

The desktop user-interface shall run under the Netscape Navigator
Version TBD or greater or Internet Explorer Version TBD or greater.

## System Requirements

-   The system shall interface with the existing IMPAC II System.

-   The server component of the system shall operate on a Sun Solaris
    operating system, located at the NIH CIT.

-   The client component of the system shall operate on any personal
    computer with Netscape Navigator Version TBD or greater or Internet
    Explorer Version TBD or greater.

## Performance Requirements

Detailed performance requirements will be described in the Supplementary
Specification document.

## Environmental Requirements

None.

## Security Recommendations

TBD. Deferred to Architecture Group.

# Documentation Requirements

This section describes the documentation requirements of the Internet
Assisted Review System.

## User Manual

The User Manual shall describe use of the System from users' viewpoint.
The User Manual shall include:

-   Minimum system requirements

-   Logging on

-   Logging off

-   All system features

-   Customer support information

-   System Administrators Manual

The User Manual shall be available as hardcopy and through online help.

## On-line Help

On-line Help shall be available to the user for each system function.
Each topic covered in the User Manual shall also be available through
the on-line help.

## Installation Guides, Configuration, Read Me File

Since this application will be a Web-based application, no specific user
installation will be required.

## Labeling and Packaging

The NIH eRA logo shall be prominent on the user documentation and splash
screens.

## Appendix A: Business Requirements for an IMPAC II Internet Assisted Peer Review System from a Sub-Committee of the CSR Information Resources Advisory Committee {#appendix-a-business-requirements-for-an-impac-ii-internet-assisted-peer-review-system-from-a-sub-committee-of-the-csr-information-resources-advisory-committee .unnumbered}

*(Contact: Richard Panniers)*

\[For the purpose of this document, the current Web-based review
retrieval system will be referred as ER and the new system to be
integrated into IMPAC II will be IAPR\] **Preamble.** The following
document was assembled by a subcommittee of IRAC to generate ideas for
business requirements for the new IMPAC II Internet Assisted Peer Review
module, which will be designed this year. The document was distributed
to all CSR Review staff and their input has been incorporated. The
document assumes some knowledge of ER functions. The integration of IAPR
with IMPAC II provides new opportunities and the group has attempted to
capture such opportunities. The document is not intended as a detailed
analysis of the current ER system but rather points out areas where
there are opportunities to take advantage of lessons learned from ER.
While many ideas presented as enhancements of functions are already
available in ER, there are also new potential functions for IAPR
introduced. Notable amongst these are: discussion of the potential to
add a post-deadline commentary to critiques posted by assigned
reviewers; the opportunities to handle streamlining; and auto-assembly
of summary statements.

**Both integration with IMPAC II to obtain application data and
independence from IMPAC II to achieve reasonable speeds are important
initial considerations.**

While there are clear advantages of integration of IAPR with IMPAC II,
the current independence of the current ER system has a major advantage.
It remains in operation even when IMPAC II is down or very slow. IAPR
should be built so that it acts truly in a modular function with no slow
connection to the database necessary. Once it has data loaded from IMPAC
II, it should work independently and just as efficiently as the current
ER system. It is likely that Reviewer compliance will plummet if IAPR
works as slowly as current IMPAC II client server or Web modules.

A. Application Data Entry

**Integration with current IMPAC II eliminates the need for duplicate
data entry.**

1\. Application data is already part of IMPAC II and does not need
re-entry.

2\. Reviewer Assignment and conflict of interest data is already entered
in REV. Role information is also entered into REV (pri1, rev1 etc).

3\. Roster information---As well as regular member information, the
status of reviewers (mail, telephone) is also entered and this
represents an opportunity to account for their roles in review.

4\. Initially, assignment information will be under preparation in REV
and will need a trigger to transfer it for IAPR use. There is a period
of time when the reviewer assignment list is under construction with
considerable shifting for workload balancing. A trigger will be useful
at a reviewer level allowing each reviewer's load to be made "final" for
early communication of assignments to some reviewers. However, a global
trigger will also be needed to indicate readiness of all reviewer
assignments at one time.

5\. After finalization of meeting arrangements, changes in assignments,
COI and roster are frequently needed. A method to update the information
in IAPR will be required.

6\. It can be foreseen that conflict of interest forms will be dealt
with online eventually. It is not clear whether this is beyond the scope
of IAPR at this time.

B. Passwords

**Password handling needs to be upgraded from current system used for
the ER.**

Presently in ER, passwords are created centrally, delivered to SRAs by
email, who, in turn, deliver them to reviewers by email or regular mail.
The passwords are re-used for all meetings. A more secure method of
delivery and password creation may be possible.

By using the SRAs to deliver passwords, SRAs are given potential access
to all meetings accessible to their reviewers in the current ER system.
Further, loss of one password may allow an unauthorized party access to
the current and future meetings after the deadline. It is unclear what
the best way may be to allow for an optimal security arrangement but
careful consideration is required because of the sensitivity of the
material on the site.

One approach could be to provide access to IAPR through the same
password that will allow access to the Commons and the reviewer's own
profile.

Delivery of a hotlinked, meeting-specific Web address to reviewers\'
email addresses is another possible approach. This Web page might allow
a one-time self-creation of a username/password for each meeting or a
one-time creation, multiple-use password with a time limit. Whatever
method is used, the infrequent use of passwords by reviewers needs to be
taken into account (in ER, SRAs are advised to deliver the same
passwords for each event to overcome the infrequent use).

C. Privacy

**Privacy for the IMPAC II Review module is at the level of the IRG (or
equivalent). The same level of privacy appears to be appropriate for
IAPR.** However, the private check box on the Review assignment screen
limiting access to the SRA only would have to work in the same way for
IAPR if this privacy mode is to be kept in the Review module.

D. Reviewer Critique Posting

**Posting by reviewers of critiques and scores is a major opportunity
for enhancement of the ER system but the overriding requirement is to
keep it intuitive (ER does this intuitively).**

1\. The current cut and paste method in ER is very effective because of
its simplicity. It is important to retain this method but to ensure that
the system is capable of accepting rich text. The cut and paste text
should be easily readable with no truncation at the end of lines in the
pasting box.

2\. While a cut-and-paste feature is still desirable, it is likely that
many reviewers will prefer simply attaching files by dragging and
dropping from their own hard drives to the IAPR server. Such a system
presumably would have to be limited to the two main word processors
(Word and WordPerfect) because of conversion issues. Further, the files
should be transparent to Review staff who only will be interested in the
text and not file handling. There should be a straightforward way for
reviewers to view and print their loaded text and scores.

3\. It can be foreseen that some reviewers will appreciate batch loading
of files. However, such files would need to be named with an accurate
identifier, such as the application number, and there would have to be
error checking to ensure the number exists. This option is seen as a
frill of lower priority but one that might help increase reviewer
compliance.

4\. The current method of ER to locate an application for posting and to
indicate that something has been posted is very efficient but 1--2% of
critiques are misposted each round. There is an opportunity with the
integration into IMPAC II to provide an enhanced set of application
identifiers. Addition of the application title for reviewers to check
that they are posting to the correct application will help.

5\. A useful additional feature would be to allow reviewers to print
their own copies of their assignment lists. The format would need to be
SRA-controlled---either \"Assignment List and Conflicts by Reviewer\"
(full assignment information on only those applications assigned to the
reviewer) or \"Assignment List and Conflicts by Reviewer (Restricted
Version)\" (no information on co-reviewers). The New Investigator
asterisk should print.

6\. At the time of posting of each critique, the opportunity can be
taken (with appropriate labels for each application appearing only when
relevant) to remind the reviewer whether there are any special
considerations involved. Four special considerations stand out as
reasonable to flag:

> a\. The involvement of human subjects can be flagged (this could be
> tied to an optional text-entry screen where the text can be
> auto-appended to the reviewer's posted critique or the reviewer may
> wish to complete their critique and post it later).
>
> b\. Vertebrate animal subjects involved.
>
> c\. New R01 investigator reminder with a simple flag.
>
> d\. Reminder with a simple flag that an extra section is required when
> the application is submitted by a foreign organization (again, an
> optional text box might be tied to the flag).

7\. The score entry box should be as simple as the ER system but should
allow two digitd or two digits with decimal point (prohibiting
three-digit entry with an error flag when attempted may be required to
allow for accurate averaging).

8\. The availability of the application abstract from the scanned
application might be useful. However, it is recognized that extracting
it might pose difficulties. Alternatively, the ability of SRAs/GTAs to
post the abstracts themselves (for reviewers to read or that can be used
to auto-assemble the summary statement---see later) would be found
useful by some, particularly those who do expedited review.

9\. Posting by unassigned study-section members before the deadline
should be allowed. The current limitation in ER only to allow posting by
assigned reviewers does not have a strong rationale. The ability to post
to any application might be made an option for SRAs to turn on or off.
However, this added function will increase complexity for the SRG
members and it should be separated from the function to post to assigned
applications to retain the simplicity and ease of posting to assigned
applications (the primary requirement of IAPR). There will have to be a
feature added for SRAs to be able to track what unassigned members have
posted. In the "reading" phase, it will be important for viewers to see
who has posted and their role (primary etc).

10\. The current simple functioning of the ER system after the deadline
to allow reviewers to intuitively locate applications must be retained
in IAPR.

11\. There is an opportunity to re-engineer how conflict-of-interest
forms are handled. This may be made completely electronic. However, this
may be too ambitious for the first version of IAPR.

E. SRA Functions and Reports

1\. The current method in ER for setting deadlines to within the hour is
well done. However, the post-meeting edit phase deadline will conflict
with the reviewer purge timing in IMPAC II.

2\. In ER, SRAs can block reviewer viewing of another reviewer\'s
critique when they have not posted their own by the deadline. This
important function needs to remain. The ER system does not do this very
efficiently and it needs to be improved in IAPR. As well as a one-by-one
reviewer application block, there needs to be an ability to batch block
a reviewer's access to their complete set of assignments.

3\. Rather than adding the reviewer name to critiques as in ER, the
reviewer role should be displayed along with the recommended score, PI
name and title as a header.

4\. A major departure from the current ER system would be to allow
members to post comments after the deadline \[these comments would be
marked as after the deadline and changes to the pre-deadline critiques
should not be allowed until after the meeting\]. This might be limited
to the unassigned or might also include assigned reviewers, or comments
could be mailed to SRAs who will have ability to post for the unassigned
or assigned.

> Whatever is decided, careful thought has to be given to such a step
> and clear policies developed. Such a system would allow online
> dialogue about applications and enable discussion before the meeting
> with the possibility of eliminating discussion that would have
> occurred at the meeting and reduce participation to the assigned
> reviewers only and lower the quality of peer review. However, such a
> tool might enable the peer review process to proceed even in the face
> of another airline crisis or allow streamlining of SEPs that take
> place by telephone conference. The function could be added but as an
> option to be turned on by the SRA only when needed. There appears to
> be an opportunity for utilizing such a tool to streamline meetings
> with only a small number of applications (one P01 or R01, etc.) where
> all study-section members are likely to follow the discussion.
>
> \[Concerns about the possible negative impact on the NIH peer review
> process of a comment line feature are addressed by the following from
> IRAC members: I\'m VERY concerned about E4 above. I think the write-up
> needs to reflect the possibility that the public interchange that
> takes place at a meeting, with the opportunity for all around the
> table to listen and chime in, could be subverted by a virtually
> private chat room discussion. Reviewers have a hard enough time
> checking into the reviews posted by the deadline before coming to the
> meeting; it is unreasonable to think they could or would look into all
> the threads of interchange that might take place on 70--100
> applications. This has the capacity for allowing two reviewers to
> decide the fate of both scored and unscored applications simply by
> saying, "we worked out our differences and agree on a 1.9" with no one
> the wiser as to what the issues were.\] This concern is a general one
> that most SRAs and management will have. Development of such a
> function will require more policy input. Clearly, the timing of the
> peer review process would have to be substantially changed for
> reviewers to utilize it optimally.

5\. In ER, SRG members can view the same compiled list of scores as the
SRA (except for conflicts). While most find this feature an integral
part of the process, a few would like to be able to turn this feature
off to hide scores from members before the meeting *\[this would also
impact display of scores with critiques\].*

6\. There is an opportunity to enhance the handling of streamlining
through IAPR. The lower half list is now communicated to members by
email. A new feature would enable the SRA to select in IAPR those
applications from the meeting to add to a lower half list to be viewed
by the SRG members. In the reviewers' and SRA "views" of the list, the
applications would be linked to posted critiques. Batch printing of
critiques will allow efficient retrieval of this subset by members or
SRA. *\[A detailed description of features to maximize use of IAPR to
aid streamlining making efficient use of IMPAC II integration is
addressed in the attached document authored by Ev Sinnett. Included are
ideas for a floating lower half cutoff, and a variety of reports to
support lower half and score tracking.\]*

7\. The same type of selection method can be used to create a list of
applications with significant differences in scores to bring to the
attention of members and help focus discussion. Again the SRA would
select these. Again links from the list to critique views and batch
printing will enhance such a function.

8\. The above two functions would work in conjunction with a score
viewing screen. Sorting applications by score would enhance the
selection of lower half applications. *\[LH should be included in the
average as 4.0.\]*

> **Note**: With the addition of lower half and significant difference
> lists, the Reviewers' READ (term used in ER) page becomes a little
> more complex but not overly so. After accessing the correct meeting,
> they will get a choice of "View Critiques by Application," "View the
> Lower Half List," or "View the Significant Difference List." Each
> application in these lists will be linked to a view of the combined
> critiques. An alternate approach to such lists is too simply allow the
> lists to be generated for the SRA to email them to members, but this
> removes the advantage of quick links to the critiques.

9\. The current system in ER---allowing the SRA to post critiques for
reviewers irrespective of deadlines---is an essential feature.

10\. How functions work together needs careful consideration to increase
efficiency for SRAs e.g., critique posting status and reviewer blocking
can be combined on one screen. Screens should be intuitive (particularly
for reviewers) taking advantage of current Web conventions.

11\. The current ability of ER to track assigned reviewer postings is
very useful. If unassigned postings are allowed, tracking will need to
be enhanced. To help with tracking, an option to view new postings since
a particular date will be useful. Color-coding may also help here. A
number of other enhancements can be made to tracking that are addressed
in the attached document authored by Ev Sinnett.

12\. All the current tools for SRAs in ER to view and print score and
critiques are useful. The ability to sort outputs can be made more
flexible with the addition of secondary sorts where applicable. Further,
reports can be enhanced with functions to batch print selected
critiques.

13\. The SRA will also need to print the lower half list (to take as the
official list to the meeting) and significant difference lists and their
associated combined critiques. SRAs also generate for the meeting a list
of applications that have been nominated by one reviewer for
streamlining.

14\. Some SRAs read critiques as they are added to the ER website
allowing them to be better prepared for meeting and to spot potential
problems. However, most critiques are well written. A useful feature
would be the ability to mark an application as read and approved by the
SRA to help streamline the assembly of triaged summary statements in
particular. If a critique is updated, then the check mark will be
removed automatically.

F. Post Meeting Editing

1\. The date stamp in ER that allows SRAs/GTAs to track when a critique
was updated is important. This date can be used as a filter in IAPR to
retrieve the subset of critiques posted since a particular date.
Color-coding could be used to aid tracking. The ability in ER to set an
end date to post-meeting editing is a critical function.

2\. Unassigned members should be allowed to post post-meeting comments
or to edit their earlier comments.

G. Purging

Critiques and reviewer assignments should not be linked together.
Unlinked assignments can be purged leaving critiques in place.
\[Depending on timing, there is a potential clash of reviewer purge and
post-meeting editing.\]

H. Summary Statement Assembly

1\. The main post-meeting report is the assembled critiques in a
pre-summary statement draft. Two versions are required: one similar to
the current ER draft that shows application, reviewer role (the reviewer
names are not wanted by many and will be purged anyway) and score
information, and a second Word output that is closer to a true summary
statement draft. *\[RTF output will provide compatibility with all major
word processors.\]* Production of this second output must take the
opportunity to maximize auto-assembly of draft summary statement text.
Critiques would begin with the heading "Critique" (a nice touch would
allow SRAs to rearrange the order of critiques; the default order should
be by role). *\[Although many reviewers add the heading \"critique,\"
they can be asked not to do this.\]* The description would be added if
available. A further nice touch would create an output with as many
template headings as possible. So, for example, if there are human
subjects codes, the appropriate headings can be created in the output.
The bolded statement proposed by OER for separating reviewer and SRA
remarks can be added. If biohazard of foreign are checked, these
headings can be added, etc. If such an option is provided, it will be
important to be able to toggle off the template.

2\. If a feature to allow reviewers to attach files is created, SRAs
will not want to download these files but rather handle the rich-text
only.

3\. A feature can be provided to use the text to assemble the IMPAC II
PDF draft summary statement, avoiding an intermediary Word file. Often
streamlined summary statements will need no editing and they can be
rapidly released. However, such a function should be built to avoid
inadvertent release of unread critiques. It could be combined with a
check box indicating that the SRA has approved the critique. The check
box would only be visible on the SRA\'s screen, similar to the private
check box on the Review module 1500 screen.

I. Links to IC Program Officers

1\. Program officers are often provided with the lower half list by SRAs
so that they can optimize their travels between SRG meetings. Hence a
feature to make the lower half list public to POs will be useful. This
function must be under the control of the SRA.

2\. Advantage can be taken of the feature described above to place
critique text directly into draft summary statements. In IMPAC II, there
is already a Preview mode for SRAs to share summary statements with POs.
If critiques can be directly loaded as draft summary statements, then
the preview feature will allow quick IC access if necessary.

J. Other Possible Features

1\. It is important to consider all the potential tracking tools that
will be required by SRAs and GTAs to efficiently monitor the system.

2\. One possible security feature is to have CPU-limited access for SRG
members, but this may place too great a limit on reviewers.

3\. Another entirely different option for deadline dates for posting
critiques can be considered. Once reviewers have posted their finalized
critiques, they can be blocked from editing them and will then
immediately be able to see other posted critiques for that application.
One advantage is there might be encouragement to post earlier, allowing
a more protracted review process, allowing greater unassigned
involvement over time and helping the SRA assess critiques before the
meeting. However, there are several disadvantages, including a possible
disparity in how different applicants are \[viewed\] and the increase in
complexity of timing frustrating reviewers---they might tend to keep on
accessing the system to see if the other reviewer has posted yet.
However, such a function might be made optional allowing piloting.

4\. A separate deadline for lower half suggestions might help management
of this process with post-deadline additions or subtractions from the
lower half list allowed \[see attached document by ES\]. Inclusion of a
nomination list might allow reviewers to add a second for lower half \[a
reviewer comment box might help explain changes in initial
recommendation when initial critiques do not support lower half\].

5\. Often reviewers indicate their score is conditional on the outcome
of discussion. The system might allow input of a score mark as
conditional with an explanation.

6\. There may be an opportunity to handle Travel Voucher via IAPR.

CSR IRAC Sub-committee

> Bobbie David
>
> Kathy Dinterman
>
> Karl Malik
>
> Denise McGarrell
>
> Rich McKay
>
> Richard Panniers
>
> Alex Politis
>
> Ev Sinnett
>
> Ranga Srinivas
>
> Linda Thee
>
> Larry Yager

Additional Comments from:

> Anita Sostek
>
> John Bishop
>
> Mike Radtke
>
> Joanne Fujii

## Appendix B: Dr. Everett Sinnett's Design Issues for Electronic Critique System {#appendix-b-dr.-everett-sinnetts-design-issues-for-electronic-critique-system .unnumbered}

Design Issues For Electronic Critique System (ECS)

**Overview:** The ECS will utilize assignment and conflict information
as entered in the current 1500 screen as a starting point. When the SRA
has "finalized" the assignments (error check for simultaneous conflicts
and assignments), the SRA will press a "Release to ECS" button, which
will allow reviewers access to the system. A secure website is required
for the reviewers to post critiques and scores and, later, to view the
critiques posted by others. The SRA needs to be able to control the
timing of these phases and needs to be able to block certain reviewers
from accessing the critiques of others. The module should allow
reviewers to post streamlining votes and allow for the production of
related reports. Finally, the system needs to allow the SRA to export
critiques for the preparation of summary statements.

**Reviewer Access**

**1. Passwords to a secure website will be required.** Ideally, the
system will generate passwords and automatically e-mail the passwords to
the reviewers, although the latter feature may be a \"could.\"

**2. Assignment information:** Several potential features could be
considered.

> a\) The reviewers need to select the application for which a review is
> to be posted. The display needs to include both the PI name and
> application number, sortable on both, as well as an "Assignment Role"
> button to sort according to Primary, Secondary, Tertiary and
> Discussant (with a subsort on PI name). A coordinated display (such as
> on the 1500 screen) providing more details (title, university) could
> be used, or these fields should also be built in, since some
> applicants submit more than one application to the same meeting. New
> Investigators should be designated with a checkbox. Buttons could
> allow the reviewer to select "Your assignments" (the default) or "All
> applications," because there will be times (often after the meeting)
> when an unassigned member will need to post comments raised at the
> meeting. The All Applications feature would be blocked during initial
> posting. The right side of the current 1300 (Workload) screen could
> serve as a base for this screen.
>
> b\) A useful additional feature would be to allow reviewers to print
> their own copies of their assignment lists. The format would need to
> be SRA-controlled, either "Assignment List and Conflicts by Reviewer"
> (full assignment information on only those applications assigned to
> the reviewer) or "Assignment List and Conflicts by Reviewer
> (Restricted Version)" (no information on co-reviewers). The New
> Investigator asterisk should print.

**3. Posting Reviews:** Once an application has been selected, the
reviewer needs to be able to post the critique. Here, the design may be
difficult, since, ideally, the document should be both stored and
displayed in its native format so as to allow the inclusion of special
characters. I don't know if this is feasible, but it would defeat the
efforts made in designing the summary statement module if we can't.
Perhaps Word and WordPerfect documents could be accepted and displayed
as PDF documents, while reviewers using other platforms would need to
convert to text (or have the system do that for them). The reviewer's
name and the date and time of posting need to be attached and displayed.
Reviewers need to be able to view their own posted critiques during this
phase.

**4. Viewing Critiques:** Once the website is open for viewing,
reviewers could go the screen described in 2a and select either
applications from their own list or from the list of all applications
for viewing critiques. Applications on which they are in conflict would
be blocked. Posted reviews should be identified both with the name of
the reviewer and their assignment type (Primary, Secondary, etc).
Scores/UN votes should also display. On selecting a review, a PDF
display of the review would open. No modification of existing critiques
would be allowed during this phase.

**5. Posting Streamlining Votes:** In its simplest form, a display such
as in 2a (the "Your Assignments" list) would appear with a checkbox in a
UN column to allow the reviewer to vote. A system with the added value
of allowing the SRA to pick a "floating" cutoff (see below) would allow
reviewers to vote scores or percentiles in addition to the checkbox
system. That is, the ideal distribution of 50% of the scores being worse
than 3.0 is rarely attained, but recording actual scores for all
applications (the good, the bad, and the ugly) could allow the cutoff
for streamlining to float up to a 2.6, for instance.

**Note:** *SRAs who do not use the ECS could utilize the streamlining
functions described in this document by using a simple screen showing
the list of reviewers for the meeting and clicking on the name of the
reviewer for whom they wish to post votes.*

**6. Viewing Streamlining Votes, Objecting to Streamlining, Late Votes,
and Average Scores:** Once the deadline for voting has passed (see
below), reviewers need to access a Preliminary Streamlining Results
(PSR) screen. The display would be a simple list of all applications
coming to the meeting (sortable by PI (default) or application number)
with a "Results" field to display the following options, with a key to
one side:

\\ = one vote for UN

UN = two or more votes for UN

D = discuss (objection to UN registered or ineligible activity)

(\\) = late vote(s) for streamlining

Two additional columns would allow reviewers to object to streamlining
(any study section member not in conflict) or to add a late vote (only
assigned reviewers/discussants).

Another portion of the screen could display a preliminary score matrix:
average, range, and individual votes. The latter could be arranged from
best to worst (and so noted) so as to preserve reviewer confidentiality.
It might be best to freeze this (?). The average score column should be
sortable.

Another portion of the screen could display the initial streamlining
percentage and the current streamlining percentage (that is, subtracting
those now with a D).

Also, if a \"Floating Cutoff\" is used (see below), there would be a
notation, "A 'Floating Cutoff' of [2.7]{.underline} was used, resulting
in [45]{.underline} percent of the applications getting a UN."

SRA Access

**1. Reviewer Access Control**

> a\) **Review Posting, Viewing, Updating, and Closure:** The SRA needs
> to be able to set and adjust the dates and times when the site will be
> open for these four phases.
>
> b\) **The SRA needs the ability to block specific reviewers from
> viewing.** Those who are blocked would continue to have the ability to
> post, and could later be cleared for viewing.

**2. Monitor Posting:** A screen showing which critiques have been
posted and which have not is needed. One format might be to duplicate
the current 1500 screen and utilize bolding for posted critiques and
gray for those not posted. A less informative but perhaps easier-to-scan
screen would simply display columns with checkboxes; column headings
would be pulled in from 1500 (Pri 1, Dis 1, etc).

**3. Direct Posting by SRA:** The SRA needs to be able to post reviews
that may come in by e-mail, for instance.

**4. View Posted Critiques:** See 5 under Reviewer Access

**5. Streamlining:**

> **a) Defining ineligible activity codes:** A screen is needed to
> display the activity codes of the applications coming to the meeting
> with a checkbox to make all applications of that code ineligible for
> streamlining.
>
> **b) Defining ineligible reviewers:** Mail Reviewers are generally not
> eligible to vote for streamlining an application; however, others on
> the committee may wish to see the opinion of the Mail Reviewer. Thus,
> a screen with the list of reviewers and three columns is needed so as
> to exclude access, include but display only (i.e., don't count toward
> the criteria of two UN votes), or include fully. All regular reviewers
> should default to "include fully" while Mail Reviewers should default
> to "display only."
>
> **c) Deadline for Posting:** The SRA needs to be able to set the date
> and time by which streamlining votes need to be posted. A bold display
> of that information should appear when reviewers log on to the Web.
> Any UN votes submitted after the deadline would register as "late
> votes" and would not count toward preliminary streamlining. They need
> to be confirmed at the meeting.
>
> **d) Monitoring Votes:** A display building on the 1500--50 (Tally)
> screen would be useful, with the number of UN votes (or scores)
> displaying next to that utilizing the same set of columns headings.
> This would allow the SRA to know who hasn\'t voted at all, who might
> have forgotten to vote on discussant assignments, or who has such a
> light load that the lack of UN votes may not be a concern.
>
> **e) Deadline for Objecting:** Generally, SRAs allow two days for
> reviewers to register objections to streamlining. The SRA needs to be
> able to set the date and time.
>
> **f) Release Preliminary Streamlining to Program:** Once the deadline
> for objecting has passed, SRAs would need to permit program officials
> to have access to the PSR screen (see above). While "view only" for
> them, it should not be frozen; late votes from reviewers should appear
> on the program screen as they are entered.
>
> **g) Preliminary Results:** Use PSR screen.
>
> **h) "Floating Cutoff:"** If scores or percentile votes are
> registered, pushing the Floating Cutoff button would perform an
> iterative procedure whereby a score or percentile is found for which
> at least 50% of the applications have two or more scores as bad or
> worse than the cutoff. A window should open indicating, for instance,
> "A cutoff of [2.6]{.underline} resulted in [55]{.underline} percent of
> the applications falling into the 'floating lower half' (two or more
> votes of [2.6]{.underline} or worse)." An "Accept" button would
> establish that as the cutoff, while "Step Back" and "Step Forward"
> buttons would move the floating cutoff to worse or better scores.
> Cancel to abort.
>
> **i) Export to Order of Review:** Some SRAs like to manipulate the
> Order of Review so as to push all the UN applications to the bottom of
> the list. Such an Export button would transfer the existing
> streamlining information to the Order of Review screen, causing all UN
> applications to migrate down (but keeping the same order while doing
> so) and then be Resequenced.
>
> **j) Program Access:** At some point after
>
> **k) Meeting Report 1:** A two column report displaying the PI name
> and application number along with the current information in the
> Results field in the PSR screen. Another column would print a C next
> to those applications for which there is a conflict in the system,
> since those cannot be discussed at all when streamlining is confirmed
> at the beginning of the meeting. Numbering is needed to correspond
> with reviewer vote sheets.
>
> **l) Meeting Report 2:** For reference, a copy of the master
> assignment list with reviewers who voted to streamline a particular
> application printing in bold.
>
> **m) Meeting Report 3:** Preliminary Score Matrix
>
> **n) Update & Transfer to Score Entry screen:** After the meeting, the
> SRA or GTA could revisit the screen described in item 6 of Reviewer
> Access to update UN results (add UN's or change to D), then press a
> button to transfer these results to the Master Sheet for score entry.

**Export Critiques for Summary Statement Production:** Several options
deserve consideration.

**1. Export to Summary Statement Module.** This option would formally
associate each file for the designated application to allow access
through the summary statement module. Until the button is pushed, the
files should remain in a temporary file. There would need to be an
"update" button that would bring in the most recent posting, and there
should be a warning when a newer version has been posted. The advantage
of this scheme would be to know which version you are working with so
that an update would not be posted without your knowing.

**2. Direct Storage in the Summary Statement Module.** Posted reviews
would be available to the SRA through the Summary Statement Module as
soon as posted. The difficulty would be in keeping track of when a
review has been modified. A log could show the SRA when updates have
been posted, but it might be difficult to keep track of those changes
when working offline on a draft in Word or WordPerfect.

**3. Automated Assembly.** The ECS and/or the summary statement module
should have a display showing which reviews are in and which are
missing. When all expected reviews are there, an Export Raw Reviews
button should assemble the reviews in a prescribed order (e.g., Primary,
Secondary, Tertiary, Mail, Discussant) and allow the SRA to save the
assembled document on the c: drive with the prescribed filename format
needed for later upload. **PROBLEM**: How to deal with files created in
different word processing programs. As noted above, we\'d like to retain
special characters. If the SRA specifies that the downloaded document
should be in Word, for instance, are there conversion programs to handle
a WordPerfect document on the fly?

**4. Download of All Critiques.** If all critiques are saved in a single
document, some time can be saved by running certain "clean-up" macros
just once instead of 100--400 times. The critiques for each application
should be ordered as in 3 above, and the SRA needs control over the
order (PI name vs. application number). The insertion of some special
character string (e.g., page breaks) between applications would allow
efficient separation for storage in individual files.

**5. Retention of Paragraph Breaks, Deletion of Superfluous Hard
Returns.** One problem with the current NIAID system is that some types
of output result in the loss of paragraph breaks while others put a hard
return at the end of every line. Macros can often take care of these
problems, but the system should provide clean output if possible.

## Appendix C: Dr. Thomas Tatham's Request for ER Enhancements {#appendix-c-dr.-thomas-tathams-request-for-er-enhancements .unnumbered}

*Dr. Thomas Tatham\'s request for ER enhancements*

Thank you for purging old data and for implementing the new purge
procedure that we requested in such a timely manner. To enable this
procedure to work smoothly, we are hoping that you can add an extra
feature to help prevent any possible loss of reviewer critiques because
of the purge.

If GTAs have the capacity to download all the summary statements
(collected critiques under each grant number) with a separate file for
each summary statement (named with the grant number or PI name), this
could be incorporated into their post meeting processes as a one-time
event and will prevent inadvertent loss of data. The files might be
zipped together for the download in a procedure similar to that now used
in IMPAC II and QVR. Although there is already an ability provided by
the system to obtain a Word file of all compiled summary statements,
such a file does not have very great utility because each summary
statement has to be located within a very large file.

Our users have also reported to us some other enhancements that can help
increase their efficiency. Can you take a look at these to see if they
can be incorporated into the system?

It would be useful for SRAs to control the numeric score assigned to
applications that the reviewers have designated as "UN" or "LH." At
present, the system assigns a score of 0 to unscored applications when
computing averages. Thus, an application with the following scores, LH,
LH, 2.0, is assigned an average of 2.0, whereas an application with
scores of 2.1, 2.2, 2.0 is given an average of 2.1. This reduces the
utility of using the score matrix to monitor spreading of scores and
could lead to confusion on the part of reviewers. If SRAs are not given
control over the handling of LHs, then it might be reasonable to assign
a 4.0 to all LH nominations.

In all screens displaying application numbers, can the mechanism field
be displayed? It is very difficult to evaluate score distributions and
perform other essential tasks if mechanisms are not displayed.

Can some screens be combined? In particular, a single screen can show
the posting status of each reviewer\'s application and also allow
blocking/unblocking of the reviewer, plus SRA posting of the critique
for the reviewer. At present, the first two functions are combined, but
posting requires using a different screen. The present scheme is
cumbersome for late reviews---the SRA must unblock the review in one
screen, and then drill down through several screens to post the review.
This can be done in about 25% of the time if these functions were on the
same screen.

Also, can the position on a screen be maintained after an action? For
example, the screen resets to the top of the Web page after each
reviewer block requiring multiple navigations through the page to block
more than one application for one reviewer.

Can the system provide a command for printing all reviews, sorted by PI,
with each PI beginning on a new page? At present, all reviews can be
printed at once but applications are not separated. Also, they come out
by application number and not all SRAs run meetings by application
number. The requested feature would allow all SRAs to bring a collated
copy of each "proto summary statement" to the meeting and be able to
refer to them during the meeting. This allows SRAs to ensure that key
points made at the meeting are actually written into the critiques.

Can the score matrix display allow multilevel sorts? For example, it
would be very useful to be able to sort by mechanism then PI name.

## Appendix D: Registration Email to Reviewer {#appendix-d-registration-email-to-reviewer .unnumbered}

Subject: NIH IAR Review Registration

Dear Reviewer:

Thank you for agreeing to be a participant on a peer review panel. To
submit your written reviews electronically, you will need to log on to
the NIH eRA Internet Assisted Review (IAR) website with a user name and
password. To establish that user name and password, we have set up a
special URL (address on the Internet) that is unique to you. Open your
Web browser and go to the URL
[https://xxx.xxx.xxx.xxx](https://xxx.xxx.xxx.xxx/). (You can copy and
paste this address into the "Location" window of your browser, and press
*Enter.*) Follow the instructions on the screen to enter information
about yourself and select your user name, password, and password
reminder question.

After submitting your registration request, you should receive an email
within 24 hours indicating that your account is active.

Once your account is active, you may log in to IAR. If you have
questions or problems setting up your account, please call the NIH eRA
Helpdesk at 1-301-402-7469 or 1-866-504-9552.

Thank you for your time and effort.

Sincerely,

SRA, Ph.D./MD

Scientific Review Administrator

(street address)

Bethesda, MD 2089x-xxx

301-xxx-xxxx

## Appendix E: IAR User Agreement {#appendix-e-iar-user-agreement .unnumbered}

User Agreement

By acceptance of this User Name and Password, I agree to safeguard the
security of the Internet Assisted Review data. This information is
protected by the Privacy Act of 1974 (PL93-579). In addition, I agree to
the following:

1.  I will not disclose my User Name and Password to anyone.

2.  My User Name and Password are considered the equivalent of my legal
    signature.

3.  I will not attempt to learn another user's User Name and Password or
    access information in the system by using a User Name and Password
    other than my own.

4.  I will only access information for which I have a demonstrable "need
    to know," based on my official duties.

5.  If I have reason to believe that the confidentiality of my User Name
    and Password has been breached, I will contact the eRA Helpdesk at
    1-866-504-9552 immediately so that the suspect User Name and
    Password can be deleted and a new one assigned to me.

6.  My User Name and Password will be deleted from the IAR system when I
    no longer require access for work-related activities.

7.  I will log off terminals when I am not actively using them to
    protect them against unauthorized access.

If I knowingly fail to comply with any of the above requirements, I may
be subject to disciplinary action. Reissue of a User Name and Password
to me after violation of any of the above statements will be dependent
on review by appropriate IAR officials.