> **Feasibility Study for an Evaluation of the Patient Navigation
> Research Program**
>
> **Contract #QTV40073**
>
> ***Final Report***
>
> March 25, 2005
>
> *Prepared for*
>
> A. Roland Garcia, Ph.D.
>
> Program Director
>
> Disparities Research Branch
>
> Center to Reduce Cancer Health Disparities
>
> National Cancer Institute
>
> 6116 Executive Blvd, Room 602
>
> Bethesda, MD 20852
>
> *Prepared by*

##### Margaret K. Gwaltney, M.B.A.

> Heather Handel, M.H.S.
>
> Alan White, Ph.D.
>
> Abt Associates Inc.
>
> 4550 Montgomery Avenue
>
> Suite 800 North
>
> Bethesda, MD 20814

Contents

1\. The NCI's Patient Navigation Research Program
[1](#__RefHeading___Toc99435709)

1.1. Program Overview [1](#__RefHeading___Toc99435710)

1.2. PNRP Research Objectives [1](#__RefHeading___Toc99435711)

1.3. Purpose of this Feasibility Study [1](#__RefHeading___Toc99435712)

1.4. Organization of this Report [1](#__RefHeading___Toc99435713)

2\. Evaluation of the Patient Navigation Research Program
[1](#__RefHeading___Toc99435714)

2.1. Evaluation Objectives [1](#__RefHeading___Toc99435715)

2.2. Overall Issues for Evaluation Feasibility
[1](#__RefHeading___Toc99435716)

2.3. Evaluation Framework [1](#__RefHeading___Toc99435720)

2.4. Evaluation Measures and Data Sources
[1](#__RefHeading___Toc99435721)

2.5. Analysis Plan [1](#__RefHeading___Toc99435730)

3\. Recommended Role for the Data Coordination Center and National
Evaluator [1](#__RefHeading___Toc99435731)

3.1. The Role of the Data Coordination Center
[1](#__RefHeading___Toc99435732)

3.2. The Role of the National Evaluator [1](#__RefHeading___Toc99435735)

References [1](#__RefHeading___Toc99435738)

1.  []{#__RefHeading___Toc99435709 .anchor}The NCI's Patient Navigation
    Research Program

    1.  []{#__RefHeading___Toc99435710 .anchor}Program Overview

In Fiscal Year 2005, the National Cancer Institute (NCI) plans to award
up to six cooperative agreements for projects under its Patient
Navigation Research Program (PNRP). The PNRP seeks to develop
"operationally effective and cost-effective patient navigation
interventions that reduce the time to delivery of standard cancer care
services . . . after identifying an abnormal finding from a cancer
detection procedure" (NIH, 2004). The PNRP, which will be managed by the
NCI's Center to Reduce Cancer Health Disparities (CRCHD), is aimed at
providing access to timely, standard cancer care for all racial and
ethnic minorities, individuals of low socio-economic status, residents
of rural areas, and members of other underserved populations who have
traditionally encountered a variety of barriers to timely access of
health services (NIH, 2004).

The multiple health system barriers encountered by racial and ethnic
minorities and other disadvantaged groups are documented in several
recent reports -- the Institute of Medicine's (IOM's) *Crossing the
Quality Chasm: A New Health System for the 21^st^ Century* (2001) and
*Unequal Treatment: Confronting Racial and Ethnic Disparities in Health
Care* (2002), the Agency for Healthcare Research and Quality's (AHRQ's)
*National Healthcare Disparities Report* (2003), and the report of the
President's Cancer Panel, *Voices of a Broken System: Real People, Real
Problems* (2001). Barriers include fragmentation of healthcare services;
lack of health insurance or underinsurance; not having a usual source of
care; long distances to diagnostic and treatment facilities; lack of
transportation; lack of childcare; poor access to information and
education about health services and disease management; illiteracy;
language and cultural barriers; and other factors (NCI, 2004; Baquet,
Carter-Pokras, and Bengen-Seltzer, 2004).

Health system barriers are implicated in a variety of health
disparities, including disproportionately higher cancer incidence and
mortality rates (NCI, 2003). Interventions that remove barriers and
improve patient-provider relationships are urgently needed to increase
cancer survival rates for minority populations and eliminate racial and
ethnic health disparities. Patient navigation programs are one such
intervention. They seek to reduce or eliminate disparities in clinical
outcomes by ensuring timely access to quality, standard cancer care, as
well as coordination and continuity of care.

Navigators support and guide patients and their families through the
cancer care continuum, from the time of abnormal finding through cancer
diagnostic tests and, if diagnosed with cancer, through the completion
of cancer treatment (see Exhibit 1). They assist patients in overcoming
a variety of social and economic barriers to expeditious health care,
thereby closing the gap between discovery of an abnormal finding and its
non-cancer resolution or delivery of appropriate cancer treatment. The
PNRP Request for Applications (RFA), which was released by NCI on August
6, 2004, invited applications for five-year cooperative agreements to
develop and evaluate patient navigation programs in diverse settings.
The RFA hypothesizes that patients who receive navigator assistance
will:

-   Receive timelier, definitive diagnosis following screening and
    abnormal finding;

-   Receive timelier cancer treatment following positive diagnosis; and

-   Improve their satisfaction with the healthcare system (Garcia,
    2004).

![](media/image1.wmf){width="5.259722222222222in"
height="4.095138888888889in"}

Each project must focus on one or more of four types of cancers --
breast, cervical, prostate, and colorectal -- for which early detection
has been shown to significantly reduce cancer-related mortality. Patient
navigators may be nurses, social workers, other healthcare
professionals, or community lay persons and may work in primary care
settings, community health centers, hospitals, or large cancer centers.

1.  []{#__RefHeading___Toc99435711 .anchor}PNRP Research Objectives

Patient navigation programs have been implemented in a variety of
settings nationwide.[^1] An important aspect of NCI's PNRP is the
requirement that grantees conduct formal qualitative and quantitative
evaluation research to determine the effectiveness, including
cost-effectiveness, of the program. In addition, PNRP grantees must
collaborate with NCI and a data coordination center on a national
program evaluation. As stated in the RFA, the purpose of the research is
to:

-   Improve NCI's knowledge and understanding of how to best support
    racial and ethnic minorities and other underserved populations with
    abnormal screening findings in accessing and navigating the cancer
    care system;

-   Assess the impact of patient navigators on timely provision of
    quality standard care and patients' adherence to standards of care;
    and

-   Encourage research collaborations and partnerships across cancer
    care delivery systems and organizations (NIH, 2004).

Each PNRP project must enroll a sufficient number of patients to be able
to answer four primary and four secondary research questions (see
Exhibit 2) and test the hypotheses set forth in the grant application.

1.  []{#__RefHeading___Toc99435712 .anchor}Purpose of this Feasibility
    Study

NCI contracted with Abt Associates to conduct a Feasibility Study for an
evaluation of the PNRP. The purpose of this Feasibility Study was to
provide information and recommendations to NCI's CRCHD regarding an
evaluation of the Patient Navigation Research Program.

The Feasibility Study was conducted over a five-month period from
September 2004 through January 2005 and relied primarily on information
contained in the PNRP RFA. The project team gathered background
documents for the Feasibility Study and information from CRCHD staff in
the first weeks of the project. Documents included the RFA and Q&As
prepared in response to questions asked by prospective applicants during
the Bidders' Conference and submitted subsequently to NCI via telephone
or electronic mail. In addition, the project team reviewed a number of
journal articles presenting the results of recent research on patient
navigation programs and other interventions to reduce cancer health
disparities. The project team also conducted interviews with key CRCHD
staff to gain a thorough understanding of the PNRP project requirements
and the desired program outcomes.

**Exhibit 2. PNRP Primary and Secondary Research Questions**

+-----------------------------------------------------------------------+
| **Primary Research Questions**                                        |
|                                                                       |
| 1.  What is the impact of the patient navigator assisting patients in |
|     coordinating services, from point of suspicious cancer finding    |
|     through non-cancer resolution or cancer treatment -- e.g.,        |
|     overcoming access barriers such as financial, lack of             |
|     information, and health system barriers?                          |
|                                                                       |
| 2.  To what extent does type/degree of service result in reduction    |
|     and/or elimination of patient-access barriers, thereby providing  |
|     more timely access to quality standard cancer care for all        |
|     patients?                                                         |
|                                                                       |
| 3.  To what extent does demographic matching of patient and navigator |
|     (e.g., race, ethnicity, gender, etc.) or fluency in primary       |
|     language of the patient affect standard-of-care adherence and     |
|     perceived satisfaction with the health care system?               |
|                                                                       |
| 4.  How effective in terms of cost and meeting research program       |
|     objectives is a patient navigator in providing patient support    |
|     and assistance to eliminate patient access barriers and improve   |
|     timely delivery of quality, standard cancer care?                 |
+-----------------------------------------------------------------------+
| **Secondary Research Questions**                                      |
|                                                                       |
| 1.  Which patient navigation strategies are most effective, i.e.,     |
|     those of an indigenous nonprofessional (cancer survivor,          |
|     community layperson) or those of a professional health care       |
|     provider (nurse, social worker, other allied health               |
|     professional), paid navigator, or volunteer navigator?            |
|                                                                       |
| 2.  Does the primary location (e.g., community-based organization,    |
|     primary care screening/diagnosis clinic/center, or                |
|     hospital/center) of the patient navigator have an impact on       |
|     navigation success?                                               |
|                                                                       |
| 3.  Does a patient navigator assisting patients in coordinating care  |
|     among multiple physicians affect standard-of-care adherence and   |
|     perceived satisfaction with the health care system?               |
|                                                                       |
| 4.  Does a patient navigator assisting patients through the cancer    |
|     care continuum increase patients' and their families'             |
|     identification and use of a usual source of care, for both cancer |
|     follow-up and other medical conditions?                           |
+-----------------------------------------------------------------------+

Source: NIH, 2004.

Unlike other Feasibility Studies, which usually examine the feasibility
of evaluating an existing program (e.g., to determine whether the
program is amenable to full-scale evaluation and to assess the
availability and quality of data), this study was aimed at recommending
an evaluation strategy, not knowing the characteristics of the grantees
or the individual patient navigation projects that would be funded. In
addition, because the RFA offers applicants flexibility in selecting
among four cancers and provides fairly broad guidance on the settings in
which projects may operate and the type of research design and
comparison group that projects should implement, it is difficult at this
time to provide a definitive evaluation design or methodology. Further
refinement of the evaluation strategy and finalization of the evaluation
design will therefore need to occur after PNRP grant awards in
consultation with NCI and the funded PNRP project sites.

NCI has built into its program the formation of a Steering Committee,
which will be composed of the CRCHD Program Scientist, grantee principal
investigators, other grantee staff (e.g., project-level evaluators), as
well as key staff of the data coordination center. An important role of
the Steering Committee will be to agree on a common study protocol for
standardized data collection and project documentation and a core set of
measures for the evaluation. The products of this Feasibility Study are
intended to facilitate the Steering Committee's discussion and arrival
at consensus around these issues.

1.  []{#__RefHeading___Toc99435713 .anchor}Organization of this Report

This report is the final product of the Feasibility Study for an
evaluation of the PNRP. The first section of the report has provided
background information on the PNRP and its research objectives. The next
section of the report, Section 2, discusses the overall evaluation
framework and the types of qualitative and quantitative measures and
data sources that should be considered for the PNRP evaluation. In
addition, this section discusses possible approaches for conducting a
cost-effectiveness analysis. Section 3 of the report discusses the
potential role of the data coordination center.

2.  []{#__RefHeading___Toc99435714 .anchor}Evaluation of the Patient
    Navigation Research Program

    1.  []{#__RefHeading___Toc99435715 .anchor}Evaluation Objectives

The primary purpose of the PNRP evaluation is to document and assess
program implementation, outcomes, and cost-effectiveness in order to
determine the key components of effective patient navigator
interventions. The evaluation will address the four primary and four
secondary research questions provided in the PNRP RFA (see Exhibit 2 in
Section 1 of this report). Individual PNRP projects may propose
additional research questions. However, these research questions will
not be the major focus of the national, cross-site program evaluation.

2.  []{#__RefHeading___Toc99435716 .anchor}Overall Issues for Evaluation
    Feasibility

Several issues guided the evaluation Feasibility Study and development
of a framework and evaluation strategy for the PNRP. The first set of
issues had to do with recommending an appropriate and feasible research
design. Other issues had to do with minimizing burden on PNRP projects
and ensuring uniformity of reporting across sites. These issues are
discussed below.

1.  Research Design

The most rigorous, and preferred, research design is an experimental
design, which compares outcomes for a treatment group (i.e., those who
participate or are enrolled in an intervention) to the outcomes of a
control group (i.e., those who do not have access to the intervention).
Implementation of an experimental research design for the PNRP may occur
in one of two ways. The first way is to assign patients randomly to the
treatment or control group within a particular setting (e.g., a large
out-patient clinic). Patients would first be screened to determine
whether they met the agreed upon selection, or inclusion, criteria and
would then be asked if they wanted to participate in the research
project. Eligible patients would then be randomly assigned to patient
navigation or to a comparison (control) group. The second approach to
randomization would be to assign clinics, or primary care settings,
randomly to either the intervention or control group. Again, patients in
each of the participating clinics would be screened to determine their
eligibility. Eligible patients in one clinic would then be assigned to
the patient navigation program, and patients in the other clinic would
be enrolled in the control group. All patients would be asked to sign a
consent form for participation in the research study.

At this time, however, it is unknown whether the project sites will
agree to implement an experimental research design. Even though
experimental designs are considered the gold standard for evaluation,
this type of design may pose challenges for project sites for several
practical and ethical reasons:

-   *Complex intervention with many variables.* Studies that use
    experimental designs generally examine a limited number of variables
    that can be reasonably controlled and tracked as part of the
    "experiment." For the PNRP, however, the number of variables of
    interest is fairly large and includes community, organizational, and
    individual factors.

-   *Inadequate sample size.* In a single clinic or other healthcare
    setting, the number of patients eligible for and willing to
    participate in the patient navigation program may not be large
    enough to assign patients randomly to the navigation program.

-   *Ethical considerations.* Ethical issues arise when evaluating
    programs that are known to have a positive effect on health outcomes
    including morbidity and mortality.

Given what may be the impracticality of an experimental research design,
PNRP projects may propose to implement non-experimental designs for
their research projects. Two options include:

-   A *historical, comparison group design.* In this type of design,
    patients in the navigation program would be compared to data records
    of patients who were screened, diagnosed, and treated for cancer in
    the same healthcare setting prior to implementation of the
    navigation program.

-   A *quasi-experimental, non-equivalent group design.* This design
    resembles the experimental approach (with treatment and control
    groups), but assignment is not random.

The above nonequivalent group designs include treatment and comparison
groups that are as similar as possible. To discern any selection bias
(i.e., the primary threat to internal validity), research would
carefully assess the degree of similarity between the two groups and any
other potential confounders using multivariate regression models. Of
note here is that project sites will still need to ensure some minimal
sample size to provide the statistical power to determine the effect of
the overall PNRP on program outcomes. The research design for the PNRP
should aim to be as rigorous as possible, to ensure that the research
findings are able to stand up to objective scrutiny and analysis.

1.  Collection of Uniform Data and Minimizing Burden

Assessing the effectiveness of patient navigation programs will require
pooling of grantee data and findings across project sites. This is
possible only if all grantees collect relatively uniform data.
Therefore, very early in the grant period, it will be important for NCI
and the grantees to agree on a set of core measures which all grantees
will then be required to collect and report to NCI via the PNRP data
coordination center using a uniform data collection and submission
protocol.

In agreeing on these measures, there will also be a need to balance
carefully the benefits of a uniform evaluation methodology across sites,
which is important for a national evaluation of the PNRP, with the
burden associated with collecting and reporting these data. One way to
do this is to develop a set of core measures that correspond to and can
be easily cross-walked with the PNRP logic model and the evaluation
questions.

2.  Maximizing the Research Potential of the PNRP

Finally, the national, cross-site evaluation should recognize that
individual projects may propose their own research questions in addition
to those in the PNRP RFA and have research plans that reflect their
unique interventions and project goals. The PNRP evaluation framework
and evaluation strategy should therefore strike a balance between giving
sites the flexibility to pursue their individually defined program
objectives, in addition to NCI's overall research goals, and allowing
for an evaluation framework that maximizes the research potential of the
navigation program.

1.  []{#__RefHeading___Toc99435720 .anchor}Evaluation Framework

As part of the Feasibility Study, Abt Associates developed a logic model
for the PNRP to serve as a framework for the development of evaluation
measures and an overall evaluation strategy (see Exhibit 3). A logic
model is a plausible model of how a program should work to solve
identified problems (Bickman, 1997). It identifies the unique features
of a program and recognizes the outcomes that the program hopes to
achieve. The essential components of a logic model are resources (or
program inputs), activities, outputs, short-, intermediate- and
long-term outcomes, and external, contextual conditions.

A logic model for the PNRP was developed using information from the PNRP
RFA and other documents and input from CRCHD staff. The logic model has
four input components (NCI resources, PNRP resources, community
resources, and community health data) and four program activity
components (infrastructure, patient referral/enrollment, medical support
services, and social support services), which are expected to lead to
the following program outcomes:

-   Changes in patients' knowledge and attitudes;

-   Changes in patient behavior;

-   Changes in healthcare provider practices;

-   Decreased time from abnormal finding to non-cancer resolution or
    cancer diagnosis and treatment;

-   Improved patient and family satisfaction with the cancer care
    system;

-   Reductions in stage at diagnosis;

-   Increased cancer screening rates;

-   Increased cancer prevention, screening and treatment knowledge;

-   Increased cancer preventive behaviors;

-   Reduced health system barriers;

-   Greater completion of cancer treatment;

-   Decreased morbidity and mortality due to cancer; and

-   Reduced disparities in health status.

![](media/image2.wmf){width="7.0in" height="5.25in"}

The logic model also illustrates that contextual factors outside the
program's direct control may influence (either positively or negatively)
the project's implementation and ability to achieve the desired
outcomes. External, contextual conditions may include antecedent
variables -- the external conditions that exist when the program begins
-- and mediating factors -- influences that emerge as the program
operates.

[]{.mark}

1.  []{#__RefHeading___Toc99435721 .anchor}Evaluation Measures and Data
    Sources

    1.  Recommended Process and Outcome Measures

Development of evaluation measures for the PNRP will ensure that project
sites uniformly collect quantitative and qualitative data for a
cross-site evaluation of the PNRP and also have the necessary data to
assess the effectiveness of their individual navigation projects.
Evaluation measures will capture a variety of outcomes and will provide
formative data to help projects make adjustments to their navigation
programs, if needed, to ensure project success.

Using the program logic model as the overall evaluation framework, the
Feasibility Study developed qualitative and quantitative process and
outcome measures for the PNRP. Process measures, which measure the
process of program implementation, correspond to each of the Input and
Activity boxes on the logic model. For the PNRP, process measures focus
on issues such as program staffing, navigator characteristics,
characteristics of the health care setting, program partnerships,
barriers encountered by patients in the navigation program, and
navigation services that are provided.

Outcome measures correspond to the boxes of the logic model falling
under the Short-term and Intermediate Outcome heading. Outcome measures
assess changes in patients' knowledge about their disease, knowledge
about the availability of services, changes in provider behaviors,
changes in the healthcare system, and patient satisfaction with specific
services and the overall healthcare system. Outcomes also include the
time from abnormal finding to non-cancer diagnosis or the time from
abnormal finding to cancer diagnosis and the initiation of treatment.

2.  Measuring Cost-Effectiveness

One of the questions to be addressed in the evaluation of the PNRP is
how cost-effective the funded programs are in providing support and
assistance to cancer patients. Prior to undertaking a cost-effectiveness
study, however, it would be necessary first to determine whether the
patient navigation program was effective (Gold et al., 1996). This
section considers options for measuring cost-effectiveness.

Without knowing the details of the types of interventions and evaluation
design frameworks that will be funded by the NCI as part of the program,
it is not possible to specify definitively how cost-effectiveness will
be measured. Specifically, it is crucial to understand the research
designs proposed and the robustness of the comparison groups used by the
PNRP project sites. Below, we recommend a basic framework be developed
to measure cost-effectiveness.

Types of Economic Analyses

Cost-effectiveness analysis is a tool often used by researchers to gain
insight into which type of treatment or service should be offered to a
patient. Cost-effectiveness compares the cost and effectiveness of at
least two alternatives to determine if the cost of a given service or
treatment is warranted given the health benefits it produces (*Cancer
Medicine*, no date).

A cost-benefit analysis looks at the economic impact of a given
treatment by assigning a dollar value to a health outcome. Cost-utility
analysis is a more specific form of cost-effectiveness analysis and
considers the ratio of dollars spent on a service or treatment to a
change in quality-adjusted life years (QALYs) (*Cancer Medicine*, no
date).

Measuring Cost-Effectiveness of the PNRP: Cost Utility Analysis

Cost-utility analysis is one method that should be used to measure the
cost effectiveness of the PNRP. This method of economic evaluation would
analyze the cost-effectiveness of interventions at each PNRP site by
comparing the benefits of the medical intervention (in this case, the
changes in care facilitated by the navigator program) to the costs of
providing that intervention (which would include both the costs of the
PNRP and the incremental services that patients receive as a result of
the program). Using this framework, cost-effectiveness would be measured
in costs per quality-adjusted life years (QALYs), a unit that expresses
the additional costs required to generate a year of perfect health (one
QALY).

Measuring Benefits and Costs

To measure cost-effectiveness, evaluators should understand how
improvements in access and coordination lead to improvements in patient
outcomes (i.e., increases in life expectancy or quality of life). For
example, if the program reduces the time that it takes to resolve
patients' abnormal finding (compared to either baseline historical data
or to a comparison group), this fact alone would not be sufficient for
an analysis of cost-effectiveness---evaluators would also need to know
how this quicker diagnosis and treatment affects patient outcomes in
order to estimate how this impact affects patient quality-adjusted life
years. It is also important to have a measure of the costs of the
program---e.g., administrative costs, the costs of the navigator, and
the costs associated with any additional services that patients receive
as a result of their services.

Modeling Approaches

The information that will be available on how the PNRP impacts patient
outcomes depends on the framework that grantees use for their program
and the data elements collected by the data coordination center. In
addition, an evaluation of cost-effectiveness would be most
straightforward if the patient navigator projects used experimental
designs where patients were randomly assigned to either an intervention
group or a comparison group. However, given that random assignment may
not occur, this Feasibility Study suggests collecting as much baseline
information as possible, in order to control for differences between the
comparison group and those receiving the patient navigation services. An
alternative approach would be to combine information on intermediate
outcomes that presumably will be available with information obtained
from a review of the literature on, for example, the relationship
between earlier screening and patient outcomes.

Recommendations for PNRP

Based on the previous discussion, the following recommendations are made
for assessing the cost-effectiveness of the PNRP:

-   Collect accurate information on costs associated with the program;

-   Conduct a cost-utility analysis on all PNRP programs; and

-   Consider alternative approaches if expected change in QALYs will be
    small and sample size is inadequate.

    1.  Data Sources and Data Collection Schedule

The national, cross-site evaluation of the PNRP will rely on process and
outcome data collected by the project sites, as well as on data
collected by the national evaluator, or data coordination center. Data
sources for process and outcome measures may include program or
administrative records (e.g., patient intake forms or activity tracking
forms), semi-structured interviews with program staff (e.g., principal
investigators, patient navigators) and program participants, and patient
surveys. These potential data sources are described briefly below.

***PNRP Site Visit Protocol.*** The site visit protocol should consist
of a set of semi-structured interview questions aimed at eliciting
discussion from principal investigators, patient navigators, project
partners, and other project staff. Questions should ask about the
setting in which the patient navigation program is being implemented,
organizational structure and staffing of the navigation project, the
navigator selection process, the role and activities of the community
advisory panel, project evaluation activities, the process of referring
patients to the navigation program, navigator assistance provided to
patients, and project costs.

***Patient Intake Form.*** The Patient Intake Form should be used to
record information about the patient when they first enroll in the
navigation program.

***Program Activity Tracking Form.*** An automated tracking system
should be used by PNRP sites to track each contact between the navigator
and patient through the continuum of care. The Activity Tracking Form
should also contain information about the date of referral to the
navigation project, date of abnormal finding, site of abnormal finding,
and cancer status.

***Patient Satisfaction Survey.*** Patient satisfaction is one element
of determining the quality and effectiveness of the patient navigation
program. Patient satisfaction surveys should be developed by the data
coordination center and administered to a random sample of the patients
in the navigation program and, when possible, the comparison group.

***Written Documents and Archival Records.*** The cross-site evaluation
should also collect and review written documents and archival records
available from program sites. These documents should be collected during
on-site visits to PNRP project sites.

Exhibit 4 recommends a process for collecting data from PNRP sites and
indicates the data collection instrument, frequency of data collection,
and who would be responsible for each data collection activity.

**Exhibit 4. Recommended Data Collection Schedule**

+----------------+---------------+-----------------+------------------+
| **Data         | **Data        | **Data          | **Frequency of   |
| Collection     | Collection    | Collection      | Data             |
| Activity**     | Instrument**  | R               | Collection**     |
|                |               | esponsibility** |                  |
+----------------+---------------+-----------------+------------------+
| Site visits to | Site Visit    | Data            | Semi-annually,   |
| each grantee   | Protocol      | coordination    | Years 1-2        |
|                |               | center          |                  |
| Document       |               |                 | Annually, Years  |
| review         |               |                 | 3-5              |
+----------------+---------------+-----------------+------------------+
| Patient        | Patient       | Data            | Ongoing          |
| tracking       | Intake Form   | coordinating    |                  |
|                |               | center will     |                  |
|                | Program       | develop         |                  |
|                | Activity      | web-based       |                  |
|                | Tracking Form | tracking system |                  |
|                |               | and paper-based |                  |
|                |               | forms in        |                  |
|                |               | collaboration   |                  |
|                |               | with NCI and    |                  |
|                |               | project sites   |                  |
|                |               |                 |                  |
|                |               | PNRP sites      |                  |
|                |               | (sites will     |                  |
|                |               | collect data    |                  |
|                |               | from treatment  |                  |
|                |               | and control     |                  |
|                |               | sites)          |                  |
+----------------+---------------+-----------------+------------------+
| Patient        | Patient       | Data            | Years 2 and 4    |
| satisfaction   | Satisfaction  | coordination    |                  |
|                | Survey        | center          |                  |
+----------------+---------------+-----------------+------------------+

1.  []{#__RefHeading___Toc99435730 .anchor}Analysis Plan

Analysis of the effectiveness of the PNRP should include use of
descriptive and inferential statistics to assess patient and community
outcomes. Descriptive statistics (e.g., frequencies, cross-tabulations,
and means) may be used to present information about the number of
patients enrolled in the PNRP and their demographic characteristics,
navigator characteristics, barriers encountered, and navigation
assistance provided to patients. These data will be available from the
Patient Intake and Program Activity Tracking Forms. A series of summary
tables may be used to display these data across sites. The PNRP logic
model will help guide the data analysis, including preparation of
summary tables.

Qualitative data collected during site visits will also be analyzed. In
particular, answers to questions on the Site Visit Protocol will be
coded and then analyzed to discern patterns across sites. In conducting
the analyses, the national evaluator will assess whether and how
specific program processes had an impact on program outcomes. Analysis
of data collected during site visits will also include review and
analysis of program documents and written materials.

Inferential statistics (e.g., t-tests, correlations, regression
analyses) will also be included in the analysis. These statistics will
be used to test the statistical significance of relationships (e.g.,
between navigation assistance and patient satisfaction and between
navigation assistance and time between abnormal finding and non-cancer
resolution or initiation of treatment). The specific inferential
statistics might include ANOVA (analysis of variance---used to test
differences in means) and ANCOVA (analysis of covariance---used to test
differences in means while controlling for other factors).

3.  []{#__RefHeading___Toc99435731 .anchor}Recommended Role for the Data
    Coordination Center and National Evaluator

This Feasibility Study is the preliminary phase of an ongoing effort to
develop and implement a plan for evaluating the Patient Navigation
Research Program (PNRP). As discussed in the previous section of this
report, the primary activities of the Feasibility Study have been to
develop an evaluation framework (or logic model), propose specific types
of process and outcome measures for evaluating the PNRP, and recommend a
data collection strategy, data collection schedule, and analysis plan.
These products are intended to facilitate development and finalization
of an evaluation plan for the PNRP upon award of the PNRP project sites.

Later in Fiscal Year 2005, NCI intends to award one or more contracts
for (1) a data coordination center, (2) an independent evaluation of the
Patient Navigation Research Program, and (3) the delivery of technical
assistance to PNRP project sites on program development, implementation,
and evaluation. Although it is possible to manage these functions under
separate contracts held by different organizations, the
interrelationship of these functions and the need for their ongoing
coordination argues for a single organization to design and implement
all three activities. Abt Associates therefore recommends that these
functions be performed under a single contract, thus ensuring their
coordination and streamlining communication with PNRP project sites.

At the same time, practical considerations, including resource
constraints, may require NCI to separate the data coordination,
evaluation, and technical assistance activities. The following two
sections outline the core tasks for the data coordination center and the
national evaluation contractor. Within each of these sections is a
discussion of these entities' technical assistance roles.

1.  []{#__RefHeading___Toc99435732 .anchor}The Role of the Data
    Coordination Center

Data Coordination

The Data Coordination Center for the Patient Navigation Research Program
(PNRP) will provide services to ensure collection and reporting of a
common set of data from each PNRP project site in support of NCI's
national program evaluation and monitoring activities. The following are
anticipated to be the core functions of the data coordination center:

-   Collaborate with the PNRP Steering Committee and the CRCHD Program
    Director on the development of a uniform study protocol and data
    collection forms;

-   Develop, pilot test, and maintain data systems for sites to use in
    collecting and reporting research data, based on the agreed-upon
    study protocol and data collection forms;

-   Develop and disseminate an Operations Manual for the automated
    reporting system;

-   Train project sites to use the data collection and reporting
    systems;

-   Develop and maintain a data quality control system;

-   Coordinate the collection of common data across PNRP project sites;

-   Establish and maintain a centralized data management system, serving
    as the repository for PNRP research data;

-   Prepare periodic and ad hoc reports summarizing program data;

-   Produce and deliver program data for federal reporting requirements;

-   Provide data to the national evaluator for use in the evaluation;
    and

-   Develop and maintain electronic procedures for sharing data and
    documents across sites.

Technical Assistance

In addition to serving as the coordinator of data collection, the data
coordination center is also expected to have a technical assistance role
throughout the study period. Technical assistance might include:

-   Telephone consultation and/or site visits to project sites for
    technical assistance on data collection, including use of a uniform
    study protocol;

-   Preparation of manuals, technical assistance bulletins, and other
    products to assist project sites in data reporting and use of the
    automated reporting system; and

-   Assistance to project sites on development of site-specific reports
    using data from the data management system.

Because the full range of technical assistance needs is unknown at this
time, it will be important to build flexibility into the data
coordination center's scope of work which would enable the center to
respond to technical assistance needs as they emerge.

1.  []{#__RefHeading___Toc99435735 .anchor}The Role of the National
    Evaluator

National Evaluation

The role of the national PNRP evaluation will be to "conduct formal
qualitative and quantitative evaluations to address process and outcome
research questions, along with a comprehensive cost-effectiveness
evaluation of the patient navigator process" (NIH, 2004). The national
evaluator should also collaborate with the CRCHD Program Director, PNRP
Steering Committee, and the data coordination center to design and
implement a uniform study protocol across all project sites.

The following are anticipated to be the core functions of the national
program evaluation:

-   Plan the multi-site research study in collaboration with the PNRP
    Steering Committee and providing overall study coordination;

-   Refine the qualitative and quantitative process and outcome measures
    for evaluating the PNRP, in collaboration with the Steering
    Committee;

-   Design and conduct a national evaluation of the PNRP, working with
    the NCI and the project sites;

-   Participate as a member of the PNRP Steering Committee and support
    the activities of the Steering Committee, including preparing
    meeting summaries;

-   Conduct site visits to project sites to collect evaluation data;

-   Respond to NCI requests for data analysis, as requested;

-   Perform data processing and statistical analysis for a process and
    outcome evaluation;

-   Analyze, publish and disseminate the results of the PNRP evaluation.

One of the first activities of the national evaluator will be to review
the funded grant applications and identify the key characteristics of
the proposed intervention (e.g., types and number of settings, type of
navigator) and their proposed research designs. Through this review, the
evaluator will assess the design and methodological approaches sites
propose to use to conduct their research projects and, in collaboration
with the CRCHD Program Director and others, determine whether an
experimental research design for the site-specific and national
evaluations might be feasible.

Another important assessment is the extent to which the interventions
share common characteristics across sites. When there are relatively few
program sites, as in the PNRP, limiting some of the variation across
sites facilitates the evaluation's ability to make conclusions about the
relationships, if any, between process and outcome variables with
reasonable certainty. When a research study includes many independent
variables, larger sample sizes are necessary to enable sufficient power
for statistical analyses.

Technical Assistance

Technical assistance is also a core component of the national
evaluation. Project sites may require consultations, information
dissemination, referrals, or other assistance on specific issues,
topics, or problems in order to participate in the evaluation. Project
sites are likely to be diverse and have varying levels of capacity with
regard to program planning, implementation, and research. Therefore, it
is expected that technical assistance may be required from the national
evaluator on the following topics during the five-year grant period:

-   Selection and implementation of an appropriate comparison group;

-   Sampling and/or randomization procedures;

-   Analyses of project-specific data to answer primary, secondary, and
    other research questions and to respond to ad hoc inquiries from
    NCI;

-   Design of project-level cost-effectiveness studies and collection
    and analysis of cost data;

-   Development of research or project materials, including community
    communication plans, memoranda of agreement with community partners,
    patient consent forms, and other procedures to ensure protection of
    human subjects and confidentiality of research data; and

-   Preparation of reports and manuscripts for publication.

The national evaluation contractor might also plan an initial
consultation with each project site to identify technical assistance
needs and then provide regular follow-up to ensure effective
implementation of the navigation program and the project evaluation. The
initial consultation may result in distribution of materials, referral
to other resources, and review of project materials such as
implementation and evaluation plans. Technical assistance may be
provided via telephone consultation, fax, e-mail, review of documents,
or on-site visits and may be provided by CRCHD staff, staff of the data
coordination center, or by external consultants.

As with the data coordination center, it is not known at this time the
full range of evaluation technical assistance needs. Therefore, it will
be important for NCI to build flexibility into the evaluation
contractor's scope of work so that the evaluator can be responsive to
technical assistance needs as they are identified or become known.

[]{#__RefHeading___Toc99435738 .anchor}References

Baquet, Carter-Pokras, and Bengen-Seltzer, "Healthcare Disparities and
Models for Change," *The American Journal of Managed* Care, Volume 10,
Special Issue, 2004.

Bickman, Leonard, "The Functions of Program Theory," in L. Bickman
(ed.), *Using Program Theory in Evaluation,* New Directions for Program
Evaluation, No. 3, San Francisco: Jossey-Bass, 1987.

\-\-\-\--, *Cancer Medicine*, 6^th^ Edition. Section 23. Principles of
Societal Oncology, no date.

Garcia, Roland, "Patient Navigation Research Program: Pre-Application
Workshop," PowerPoint Slides, National Cancer Institute, Center to
Reduce Cancer Health Disparities, Rockville, MD, September 15, 2004.

Gold, M.R., Siegel, J.E., Russell, L.B., & Weinstein, M.C.,
*[Cost-effectiveness in Health and
Medicine](http://www.oup-usa.org/docs/0195108248.html).* New York:
Oxford University Press, 1996.

Institute of Medicine, *Crossing the Quality Chasm: A New Health System
for the 21^st^ Century*, The National Academies Press, Washington, DC,
2001.

Institute of Medicine, *Unequal Treatment: Confronting Racial and Ethnic
Disparities in Health Care*, The National Academies Press, Washington,
DC, 2002.

National Cancer Institute, "Cancer Health Disparities: Fact Sheet,"
April 23, 2003, <http://www.cancer.gov/newscenter/healthdisparities>.

National Institutes of Health, "Patient Navigation Research Program,"
RFA-CA-05-019, National Cancer Institute, August 6, 2004.

Till, J.E., "Evaluation of Support Groups for Women with Breast Cancer:
Importance of the Navigator Role," *Health and Quality of Life
Outcomes,* 2003, 1(16), <http://www.hqlo.com/content/1/1/16>.

**Appendix A**

**Proposed Process and Outcome Measures**

**for the Patient Navigation Research Program**

**Appendix B**

**Proposed Data Collection Strategy**

[^1]: Different terms have been used to describe the patient navigator
    role. These include case manager, clinical coordinator, patient
    advocate, and support nurse (Till, 2003).