4

NNRA EVALUATION 

From its inception, the development and implementation of the NNRA by the NCNR have been 

guided by an iterative process of evaluation. The NCNR long-range plan, of which the NNRA is one 

component, is designed to use both formative (ongoing) and summative (outcome) evaluation to 

increase planning effectiveness (Patton, 1982). Program evaluation at the NCNR is the responsibility 

of the Office of Planning, Analysis and Evaluation. The evaluation of the NNRA has been carried 

out in collaboration with the entire NCNR staff, as are all evaluations within the NCNR. The 

essential measure of the NNRA "success" is the quantity and quality of the science stimulated by the 

agenda setting and implementation processes. 

Evaluation Framework 

The formative evaluation of the NNRA provides iterative feedback on the structure and processes as 

the agenda is set and implemented and is the basis for periodic modification and identification of 

issues and policy concerns regarding implementation. The summative evaluation of the NNRA 

measures the extent to which the NNRA objectives are being met. 

The NNRA has three objectives: 

z Provide structure for selecting scientific opportunities and initiatives. 

z Promote depth in developing a knowledge base for nursing practice. 

z Provide direction for nursing research within the discipline. 

Phases of the NNRA 

The three objectives are being met in two phases: Phase I, the agenda-setting phase, and Phase II, the 

implementation phase (Figure 1). Phase I occurred in two steps. First, a diverse group of nurse 

scientists representing the breadth of nursing research was invited to identify broad priority areas 

through structured, interactive, analytical processes at a Conference on Research Priorities (CORP) 

(Hinshaw, Heinrich & Bloch, 1988). Additional nurse scientist experts with similar breadth, as well 

as scientists from disciplines related to nursing research, were then asked to serve as members of 

Priority Expert Panels (PEPs) for each of the seven priority areas identified by the CORP. Each PEP 

reviewed the state of nursing-related science and research and elaborated comprehensive 

recommendations for its priority area. The second phase, implementation of recommendations, has 

now been undertaken to build scientific knowledge in the priority areas. There are three objectives 

for the NNRA implementation: 1) to disseminate information about the NNRA through multiple 

channels; 2) to attract quality applications through NNRA-related extramural research initiatives 

supported alone or in collaboration with other NIH institutes, centers, or divisions (ICDs); and 3) to 

develop NNRA-related intramural initiatives. Phase II is followed by dissemination of research 

findings made by investigators in the priority areas. 

Phase I: Agenda Setting 

Conference on Research Priorities 

NNRA EVALUATION, NINR Page1of6

http://ninr.nih.gov/ninr/research/vol1/Chapter4.htm 10/18/2006
 

Organization of Data 

Before analysis of the NNRA could begin, it was necessary to create a framework to guide the 

collection of data. A number of conditions were considered in the construction of the framework: the 

interdependence of the phases; the overlapping dimensions within the phases; the concurrent conduct 

of the formative evaluation and activities of the NNRA; the use of multiple data types and data 

sources; the use of data collection for evaluation and other reporting needs; and the roles and 

perspectives of key audiences. The choice of data for the formative evaluation was guided by the 

following questions, which reflect the concerns of both the NCNR and the nursing research 

community. 

z Were all major areas of nursing science represented by the scientists who participated in the 

CORP? 

z How can the PEP process be changed to increase efficiency? 

z Are NIH peer review groups aware of the NNRA priorities? 

z Do the NNRA priorities stimulate research of national health problems as defined by the 

USPHS, NIH and other public health institutions? 

z Does the NNRA process result in science for nursing practice in the areas of highest priority 

for nursing and critical health care problems? 

z How is the NCNR collaborating with other ICDs in joint sponsorship of NNRA-related 

initiatives? 

z How have perspectives of CORP and PEP participants and NCNR staff been solicited to refine 

processes for future NNRA priority deliberations? 

z How is feedback concerning the NNRA solicited from the nursing research community at 

conferences attended by NCNR staff? 

Data for the formative evaluation are systematically collected from minutes, formal reports, and 

publications. The primary users of the answers from the formative evaluation are the NCNR staff 

and NACNR members. Feedback is both informal and formal, such as informal discussions and 

formal periodic reports. 

The following questions guided choices of data collected for the summative evaluation: 

z Should the program be continued? Expanded? 

z Is this an effective method for priority setting? 

z What conclusions can be made about the effects of the NNRA and its components? 

z Do the priorities reflect key areas of research based on societal and health care needs? 

z To what extent have the three objectives of the NNRA been met? 

The questions identified for the summative evaluation include critical science indicators. These 

indicators reflect the degree to which the science emanating from the NNRA is cutting edge, the 

Priority Expert Panel Recommendations 

Phase II: Implementation--Dissemination of NNRA Information

Extramural initiatives 

Intramural initiatives 

Post-Phase II: Dissemination--Advances in Science 

Dissemination of research findings 

Figure 1. NNRA Phases and Activities

NNRA EVALUATION, NINR Page2of6

http://ninr.nih.gov/ninr/research/vol1/Chapter4.htm 10/18/2006
existence of a critical mass of scientists in similar areas of research, and the application of the 

knowledge to patient outcomes. Data collection and analysis for the summative evaluation focus on 

responses of nurse researchers and the nursing community to dissemination activities, extramural 

initiatives, and outcomes of intramural research. Users of the summative evaluation results are the 

Congress, the NACNR, the NIH, and the NCNR as policy makers, and the research and practice 

community as the consumers of the NNRA implementation. 

Once collected, evaluation data are organized into inputs, processes, or outcomes for meaningful 

analysis (Shadish, Cook & Leviton, 1991) of Phase I and Phase II. Measurability issues are 

considered in setting up these analytical categories (Rossi & Freeman, 1989). Inputs are defined as 

the recorded decisions concerning courses of action and the recorded advice upon which those 

decisions are based, as well as the various types of resources used to develop and act on the 

decisions. Processes are the policies and procedures put into place and the activities carried out. 

Outcomes are the products and consequences of dissemination of the NNRA and enactment of the 

PEP recommendations. Listed below are examples of inputs, processes, and outcomes that will be 

addressed in the two phases. 

Phase I: Agenda Setting 

Inputs -- research priority designations by the CORP; detailed recommendations for each priority 

area by PEPs; time frame; financial resources. 

Processes -- selection of conference participants; selection of PEP members; formulation of 

recommendations; preparation of reports. 

Outcomes -- priorities and related recommendations; formal panel reports; congressional action; 

response from the scientific community. 

Phase II: Implementation 

Inputs -- Human resources allocated to manage implementation; program decisions concerning 

methods of implementation; financial resources each fiscal year; manpower to conduct intramural 

research; USPHS and NIH strategic plans. 

Processes -- selection of communication channels for dissemination; feedback processes; 

development and execution of initiatives; contacts with other ICDs for collaborative intramural and 

extramural initiatives. 

Outcomes -- Congressional action; grant applications, reviews, and awards; scientific investigations; 

research findings; reports of research findings; and utilization of findings in nursing practice. 

Data Sources 

Evaluation data are obtained from a variety of sources. Information can be obtained from federal 

sources, such as records of applications and funding decisions and progress reports submitted by 

principal investigators. Discussions with and records from the PEP members and NCNR staff can 

yield substantive information about the identification and development of research priorities. 

Analysis of NIH peer review outcomes, surveys of the scientific community and health professional 

organizations, queries of bibliographic systems such as citation indexes, evidence of a scientific base 

for practice, and external studies of manpower development are other sources of information. 

Methods of Evaluation 

NNRA EVALUATION, NINR Page3of6

http://ninr.nih.gov/ninr/research/vol1/Chapter4.htm 10/18/2006
Standard methods are used to collect both qualitative and quantitative data and to analyze data for 

both ongoing and summative assessment (Herman, Morris & Fitz-Gibbon, 1987). Strategies for both 

methods are summarized in Table 1. 

Methods for formative evaluation include content analysis of minutes and agendas, interviews with 

participants using an interview schedule, and a timetable. The flow of events and the problems that 

ensue from timing, resources, structuring of events, and intervening events are analyzed. The 

effectiveness of each of these strategies is also assessed in terms of achieving the NNRA objectives. 

For a broader discussion of the NCNR's long-range plan and how the NNRA fits within it, see 

Chapter 1. 

Specific methods for summative evaluation include construction of line and bar graphs to illustrate 

application rates by NNRA priority, application rates at specific intervals following announcement 

of initiatives, or funding rates for NNRA-related versus non-NNRA-related applications. NNRA 

priorities are compared with those of other NIH ICDs and with initiatives of professional nursing 

organizations. Research program development within each NNRA priority will be monitored. 

The evaluation also includes periodic external summative evaluation of implementation outcomes of 

the NNRA. These external examinations take the form of additional priority-setting conferences. 

Such conferences provide fresh perspectives that can enhance current efforts and provide direction 

for future activities. External evaluators for the NNRA may include representatives from other 

components of NIH and other federal agencies, from the nursing research community and other 

scientific disciplines, and from consumers, including both caregivers and care recipients. The 

conferences are convened when the implementation processes are well under way in the previous 

group of priorities. 

Audiences 

Table 1 

NNRA Evaluation Strategies 

Using Formative and Summative Methodologies

Evaluation 

Activity Formative Summative

Timing of data 

collection Ongoing qualitative and quantitative Periodic at stated intervals

Data to be 

collected 

Minutes, personal perspectives, preliminary reports, reports, publications, 

program announcements and requests for 

applications, applications, NCNR 

procedures and policies, planning 

documents 

Reports, NACNR recommendations, funding rates, 

advances in science, joint initiatives, NCNR policies 

Methods and 

analyses

Micro-level analyses of each phase and 

recommendations for input and process 

changes

Macro-level analyses of outcomes of each phase by 

objectives and seren-dipitous findings 

Feedback 

audiences 

NNRA Subcommittee, PEP members, 

extramural staff, intramural staff, NACNR 

members

Congress, USPHS, NIH, NACNR, nursing 

researchers, practicing nurses, public

Feedback 

mechanisms

Meetings, informal conversations, agenda 

discussions, task forces, formal reports and 

recommendations 

Formal reports such as CORP reports, PEP 

publications, reports to NACNR, recurring reports, 

NIH reports and briefings, congressional reports and 

briefings, media information, nursing journals, 

conference proceedings, and evidence of scientific 

advances applied to practice

NNRA EVALUATION, NINR Page4of6

http://ninr.nih.gov/ninr/research/vol1/Chapter4.htm 10/18/2006
 

The evaluation plan was also designed to answer questions likely to be posed by audiences identified 

as having a "stake" in the choices of priorities of the NNRA and the means to implement them 

(Veney & Kaluzny, 1984). These audiences, such as the Congress, the NACNR, and NIH and NCNR 

officials, hold positions of authority that influence the scope of NCNR activities and therefore are 

interested in information about the NNRA that will assist with policy formation and decisionmaking. 

Scientists who are involved in identifying 

knowledge gaps, developing depth in the 

knowledge base for nursing practice, and 

seeking support for studies to address NNRA 

priorities will be interested in assessments of 

implementation strategies and the state of the 

science. Health-care practitioners will be 

concerned with the effectiveness of 

dissemination and the quality of research 

because they must apply the knowledge. Healthcare recipients, whose health status can be 

influenced by the results of nursing research and 

who provide support for federally funded 

research through their tax dollars, also have vested interests in the evaluation results of the NNRA. 

Summary 

The evaluation of the NNRA provides both ongoing guidance and a systematic determination of the 

extent to which the objectives of the NNRA are being met. It addresses the two phases of the NNRA, 

agenda setting and implementation, in terms of inputs, processes, and outcomes. Both formative and 

summative evaluation methods, using a range of data sources, are employed to answer questions of 

interest to several constituencies. Evaluation results assist in decision-making regarding further 

agenda-setting and implementation activities for nursing research, as well as the status, needs, and 

future directions of nursing research and NCNR programs. 

References 

Herman, J., Morris, L.L., & Fitz-Gibbon, C.T. (1987). Evaluators handbook, Volume 1. Beverly 

Hills, CA: Sage. 

Hinshaw, A.S., Heinrich, J., & Bloch, D. (1988). Evolving clinical nursing research priorities: A 

national endeavor. Journal of Professional Nursing, 4(6), 398, 458-459. 

Patton, M.Q. (1982). Practical evaluation. Beverly Hills, CA: Sage. 

Rossi, P. H. & Freeman, H.W. (1989). Evaluation: A systematic approach. Beverly Hills, CA: Sage. 

Shadish, W.R. Jr., Cook, T.D., Leviton, L.C. (1991). Foundations of program evaluation: Theories 

of practice. Beverly Hills, CA: Sage. 

Veney, J.E., & Kaluzny, A.D. (1984). Evaluation and decision making for health services programs.

Englewood Cliffs, NJ: Prentice-Hall, Inc. 

 

NNRA EVALUATION, NINR Page5of6

http://ninr.nih.gov/ninr/research/vol1/Chapter4.htm 10/18/2006
 

TABLE OF CONTENTS

NNRA EVALUATION, NINR Page6of6

http://ninr.nih.gov/ninr/research/vol1/Chapter4.htm 10/18/2006