# Presentation: 210386

## CSREES Reporting Web Conference

- April 14, 2008

## User Support

**User Support**

- (202) 690-2910 *or* C2IT@csrees.usda.gov
- Do not contact Texas A&M support
- FAQs and other information on the CSREES Reporting Web Conference web page at www.csrees.usda.gov/rwc

## Format and Logistics

**Format and Logistics**

- E-mail questions to rwc@csrees.usda.gov
- Also e-mail topic suggestions to rwc@csrees.usda.gov
- Opportunity to vote for topics for next conference
- Conferences are recorded and will be available on the Reporting Web Conference web page at www.csrees.usda.gov/rwc

## Deb Hamernik

**Deb Hamernik**

- Deb is the CSREES National Program Leader for Animal Physiology and NRI Program Director for Bovine Genome Sequencing and Porcine Genome Sequencing. She has represented USDA on the NSTC-COS-Research Business Models Subcommittee to develop the Research Performance Progress Report (RPPR) since 2004.
- (202) 401-4202
- dhamernik@csrees.usda.gov
- www.csrees.usda.gov/onesolution

## Standard Progress Report 

## Standard Reporting Across the Federal Government

**Standard Reporting Across the Federal Government**

- Implementation of the Federal Financial Assistance management Improvement Act of 1999 (Public Law 106-107)
- Facilitate information collection in lieu of numerous agency-specific forms
- Does not change reporting requirements in OMB Circulars A-102 and A-110. Provides a standard format for collecting the information
- Draft Research Performance Progress Report (RPPR) available at: www.nsf.gov/bfa/dias/policy/rppr/draftformat.pdf

## Standard Reporting Across the Federal Government

**Standard Reporting Across the Federal Government**

- Agencies will use the standard categories & instructions developed for each category
- Agencies may provide additional program-specific instructions to clarify program requirements
- Agencies may develop additional agency- or program-specific reporting categories & instructions

## One Solution: CRIS Transition Standard Report

**One Solution: CRIS Transition Standard Report**

- The revised CRIS AD 416 sections include:
- Goals/Objectives/Expected Outputs
- Methods
- Non-Technical Summary
- The revised CRIS AD 421 sections include:
- Outputs
- Outcomes/Impacts
- Publications
- Participants
- Target Audiences
- Project Modifications

**Notes:**
Last time, Greg Crosby described the CSREES One Solution project and provided an overview of the CRIS Transition Standard Report.

Today, I will provide more details on the CRIS Transition Standard Report.

## Outputs

**Outputs**

- Activities, events, services, or products that reach people
- Examples: conferences, field days, videos, curricula, patent applications, germplasm, genetic maps, students graduated, etc.
- Do not report publications in this category

**Notes:**
Outputs is a new field for FY2008 reporting. In the previous AD421, CSREES had no way of collecting this type of information. Congress and OMB are interested in significant outputs generated by CSREES funding.

Do NOT include findings or conclusions that have been reached; these should be reported as changes in knowledge in the Outcomes field (discussed in a later slide).

Outputs are activities, events, services, or products that reach people.

Examples of activities include: conducting or analyzing experiments or surveys, assessments, facilitating, teaching, or mentoring.

Examples of events include: conferences, demonstration sites, field days, symposia, workshops, or trainings.

Examples of services include: consulting, counseling, or tutoring.

Examples of products include: videos, curricula, databases, equipment/instruments, patent applications, germplasm, genetic maps, students graduated, etc.

## Outputs--Dissemination

**Outputs--Dissemination**

- Dissemination refers to outreach activities to reach intended audiences to advance knowledge, encourage positive actions, or change conditions.
- If educational materials and resources were distributed, describe the distribution method and the intended audience.

**Notes:**
Outreach activities to current and potential partners and collaborators should be included.

## Publications

**Publications**

- Publications are outputs.
- For technical reasons, the CRIS system collects publications in a separate box.
- Include paper or electronic publications
- Include status of publication (e.g., pending, accepted, in press)

**Notes:**
No significant change to this field.

## Outcomes/Impacts

**Outcomes/Impacts**

- Changes in knowledge, actions, conditions
- Results of basic research projects should be described as a change in knowledge (rather than experimental/technical details)
- Results of extension activities should be described as a change in actions or conditions

**Notes:**
Describe how findings, results, techniques, or other products that were developed or extended from the project generated or contributed to an outcome/impact. Describe the results of the project evaluation. Indicate how resources and activities helped to produce project outputs and achieve project outcomes and impacts. 

Examples of a change in knowledge include: new fundamental or applied knowledge for publication; methods or techniques; policy knowledge; improved skills; increased knowledge of decision-making, life skills, and positive life choices among youth and adults.

Examples of a change in actions include: adoption of new or improved skills; adoption and use of new methods or improved technologies; use of skills by youth and adults in making informed choices; adoption of practical policy and use of decision-making knowledge.

Examples of a change in condition include: management and behavioral changes and adjustments; better and less expensive animal health; changes in wages or health care benefits of the agricultural workforce; higher productivity in food provision; safer food supply; reduced obesity or improved human health, etc.

## Participants

**Participants**

- Provide information about individuals who worked on the project—their role and how they participated in the project
- If applicable, describe partner organizations, collaborators, and contacts. Include collaborators outside the U.S.
- Describe opportunities for training or professional development (trainees, K-12 teachers, producers, farmers, staff, volunteers, etc.

**Notes:**
Individuals include Project Directors and each person who worked at least one person month per year (approximately 160 hours of effort) on the project during the reporting period and received salary, wages, a stipend, or other support. Include new individuals that have been added to the project since initiation.

Partner organizations are academic institutions, nonprofit organizations, state or local governments, industrial or commercial firms, schools, or school systems. Partner organizations may provide financial or in-kind support, supply facilities or equipment, or collaborate on research projects.

Collaborators and contracts are others within the recipient’s organization (especially interdepartmental or interdisciplinary collaborations), non-formal collaborations with others outside the recipient’s organization, or with others outside the United States. Include any that have been added to the project since initiation.

Training or professional development includes any opportunities for training or professional development provided by the project (such as undergraduate or graduate students, post-doctoral fellows, faculty, K-12 teachers, farmers, producers, managers, staff, or volunteers).

## Target Audiences

**Target Audiences**

- Provide information on target audiences for efforts designed to cause a change in knowledge, actions or conditions.
- Include: individuals, groups, communities served by the project
- Delivery of science-based knowledge to people through formal or informal educational programs

**Notes:**
Target Audiences can also include racial or ethnic minorities and those who are socially, economically, or educationally disadvantaged

Examples of efforts include: formal classroom instruction, laboratory instruction, or practicum experiences; development of curriculum or innovative teaching methodologies; internships; workshops; experiential learning opportunities; extension and outreach.

## Project Modifications

**Project Modifications**

- Describe major changes in approach and reason for change
- Examples: changes in Assurance Statements (animals, humans, or biohazards); major problems or delays that have significant impact on rate of expenditures

**Notes:**
Emphasize “major changes”

If applicable, provide special and/or additional reporting requirements specified in the award Terms and Conditions (for grants).

## Tips

**Tips**

- Do NOT re-enter the objectives and methods (already entered on the AD416)
- Do NOT copy and paste abstracts for scientific meetings into the Standard Report
- Use general terms for a lay audience
- More information is not necessarily better information

**Notes:**
Enter changes to objectives and methods in the field for Project Modifications

Abstracts for scientific meetings contain too many technical details for reporting to Congress, OMB, etc.

The target audience for these progress reports is federal administrators (not scientific societies). Use lay language that is easy to understand. 

More information is NOT necessarily better information. Be concise and provide only the requested information.

## Standard Progress Report: For More Information

**Standard Progress Report: ****For More Information**

- Deb Hamernik (202)401-4202 dhamernik@csrees.usda.gov

## Questions?

**Questions?**

- E-mail questions to rwc@csrees.usda.gov
- For more information, visit the One Solution web page at www.csrees.usda.gov/onesolution

## Djimé Adoum

- Djimé assists the Director, Office of Planning and Accountability, in developing monitoring and evaluation systems to analyze program activities funded by CSREES and implemented by our Land Grant System partners, and provides leadership in strategic planning and the CSREES Portfolio Review process.
- (202) 720-4564
- dadoum@csrees.usda.gov
- www.csrees.usda.gov/opa

## Practical, Realistic Approaches to Measuring Impacts of Basic Research 

## Outline of the Presentation  

- Reasons for Measuring Impact of Basic Science
- Difficulties with Measuring Impact of Basic Science
- Metrics and Efforts to Date
- R&D Criteria as Starting Point
- Experience from CSREES PREP
- Use of the Logic Model
- Summary and Conclusions

## Why Measure the Impact of Basic Research?  

- Pressure due to limited resources
- Problems have become extremely complex and require multi-disciplinary collaboration
- To secure public buy-in
- To demonstrate public value

## Stating the Obvious

- Measuring the impact of Basic Research is difficult
- Attempts have been made to identify ways to measure
- A few approaches have been determined to be of value

**Notes:**
.1.	The difficulty with measuring the impact of science and technology has been fully documented.  As Geisler (2002) stated:  “Science and technology are complex and unstructured phenomena as we address them within the social, managerial, and organizational sciences.  Evaluation of Science and Technology requires the measurements of these phenomena and the constructs that we develop to define them.  Developing conceptual constructs and modeling such unstructured phenomena are tasks that call for adequate measurement for what may be defined as “unquantifiable” dimensions and variables.” (The Metrics of Science and Technology, 200. p. 43)

## A Few Suggested Indicators

| Inputs/Investments in S&T | Outputs | Outcomes |
| --- | --- | --- |
|   Expenditures
  Comparison of expenditures
  Expenditures related to a product line
  Source of funding |   Publications
  Patents
  Training of scientific and technical people
  Special Presentations and Honors
  Development of new testing methodologies
  Intellectual challenges | Measuring things of value to society, to the economy, to the nation and to society.
  Vaccines
  New plant variety
  Sustainable socio-economic and environmental development
  Less dependency on foreign oil |

**Notes:**
These indicators are gleaned from various sources including the Wisconsin Logic Model (www.uwex.edu),  the Metrics of Science and Technology (Eleizer Geisler, 2000)

Generally speaking, indicators for inputs and outputs are straightforward while those measuring long term outcomes are problematic.  

Input indicators include resources in terms of funds, staff time, physical infrastructure, etc.
Output indicators are also mostly reported in terms of number of publications, citations, presentations, workshop held, number of students trained in a new scientific way, etc.
Outcome indicators are more challenging pose the most challenges to measurements, particularly those that cannot be easily documented.  As can be seen in the Outcomes column above, it is easier to document vaccines and new plant varieties but extremely difficult to measure sustainable socio-economic and environmental development.  Several attempts have been undertaken to deal with such complexities.  The use of external expert panel has been powerful means through which benefits of basic research can be assessed.  Additional efforts are under way to develop better tools.

## Metric Defined

- Definition of a metric (Geisler 2000):  It is a system of measurement that includes three elements:
- the *item* being measured
- the *unit* of measurement
- the *value* of the unit

## A Few Suggested Metrics

- Bibliometric Analysis refers to measures of scientific and technical published outputs from science and its disciplines.  It measures both quantity and quality (Geisler 2000).

**Notes:**
Bibliometric Analysis:  “It is a procedure that counts the number of S&T publications and other means of knowledge transfer, and also performs an analysis of citations and cocitations in order to assess the quality of these publications and their trends.”  Geisler 2000.  Researchers use bibliometric analysis routinely to determine “the state of the art” in their own discipline area.  Geisler reported that commercial organizations conduct more comprehensive bibliometric analyses compared to some government agencies and some universities.
Best use:  In combination with other metrics such as peer review, it provides a good yardstick against which academic institutions can assess the productivity of an individual scientist, industries can use them to evaluate their scientific personnel with R&D unit as well as evaluate the R&D function in the firm.  

Shortcomings and a few problems related to coverage, measurement, generalizability, and biases.
Coverage:  only published articles are reported but does not cover other reports, personal communications etc..
Measurements:  most influential articles are cited and lacks cross disciplinary outputs
Generalizability: lack of standard to validate their quality makes it difficult to generalize; citations may not go far enough
Biases:  highly influential small number of prolific publishers mostly cited

## A Few Suggested Metrics

- Economic Analysis:  a process that correlates financial measures for both investments/expenditures and outputs (source:  Geisler 2000).  It is extremely  difficult to predict the outcomes of R&D

- .

## A Few Suggested Metrics

- Peer Review:  A process by which a selective jury of experts in a given scientific field is asked to evaluate the undertaking of scientific activity or its outcomes (e.g., research, projects, or scientific publications.)  Source: Geisler 2000

**Notes:**
Peer Review as a Metric:

Definition of a Metric according to Geisler 2000
A metric must contain three elements:
The item being measured (what we are measuring)
The units of measurements (how we are measuring)
The inherited value associated with the metric (why we measure or what we intend to achieve by this measurement)

Peer Review as a Metric:
Item being measured:  Scientific Outcome
Unit of measurement:  Subjective Assessment
Inherent value:  performance and productivity of scientists

## Efforts to date

- A few models are selected to highlight evaluation of publically funded research:
- United Kingdom’s Research Assessment Exercise (RAE)
- The Japanese model
- The Australian (Australia’s Relative Funding Model) - RFM
- The United States:  GPRA and PART (NIH, NSF, ARS and CSREES, NAS)

**Notes:**
United Kingdom’s Research Assessment Exercise (RAE) 
RAE is a performance-based system of evaluation used to conduct periodic evaluations of the research undertaken by universities to inform allocation of resource funding, giving each university a quality rating and reward excellence based on the RAE.  Formula funds are then distributed to institutions that score high on the RAE criteria.  This was a way to ensure the continuity and protection of the best research in the UK.  The first RAE was conducted in 1986 and repeated every 3 years since then.  The next one is scheduled for 2008.  Research activities within are grouped into an equivalent of a portfolio called Unit of Assessment). 10 to 15 experts are assembled through a nomination process by professional associations and learned societies (for additional details, see American Journal of Evaluation, Volume 28, Number 4, December 2007).  Quality of publications as judged against international standards weighs heavily of the research institution.

The Japanese Model
This model allocates large sums of money to regional governments of ministries which give out in grants as they see it fit.  It is less stringent but has recently recognized the merit of performance-based assessment

The Australian’s Relative Funding Model (RFM)
The RFM was introduced in 1990 and was improved by 2004.  The Australian government moved to a framework for assessing the quality and impact of publically funded research activities.  They developed a more robust system called the Research Qualify Framework (RQF) to make the process more transparent to government and taxpayers and to ensure that all publically funded research agencies and research providers focus on relevance and quality.  This RQF starts in 2008 and will become fully implemented in 2009.

The United States – The Portfolio Expert Panel Review (PREP)
Self assessment document written by NPLs and OPA
NPLS and OPA self score
Panel invited
Document sent to panelists prior to arrival
2 1⁄2 meeting held at CSREES HQ
NPLs present highlights
Panel deliberates for 1 1⁄2 day
Panel scores and provides comments to Agency 
Relevance

## Evaluation based on the R&D Criteria

- R&D Criteria
- Relevance
- R&D investments must have clear plans, must be relevant to national priorities, agency missions,
- relevant fields, and “customer” needs, and must justify their claim on taxpayer resources.
- Quality
- Programs should maximize the quality of the R&D they fund through the use of a clearly stated,
- defensible method for awarding a significant majority of their funding.
- Performance
- R&D programs should maintain a set of high priority, multi-year R&D objectives with annual
- performance outputs and milestones that show how one or more outcomes will be reached.
- To be used as broad guidelines applicable to all Federally funded R&D efforts.

**Notes:**
According to OMB, these criteria are useful for planning, management, perspective assessment as well as retrospective review to determine the extent to which investments were well-directed, efficient and productive.  This is essentially done to validate program design and instilling confidence that future investments will be wisely invested (Source:  OMB)
I. Relevance
R&D investments must have clear plans, must be relevant to national priorities, agency missions,
relevant fields, and “customer” needs, and must justify their claim on taxpayer resources.
The following “must” serve as guidelines for the Relevance Criterion
A. Programs must have complete plans, with clear goals and priorities.
B. Programs must articulate the potential public benefits of the program.
C. Programs must document their relevance to specific Presidential priorities to
receive special consideration.
D. Program relevance to the needs of the Nation, of fields of Science & Technology, and
of program “customers” must be assessed through prospective external review.
E. Program relevance to the needs of the Nation, of fields of S&T, and of program
“customers” must be assessed periodically through retrospective external review.
II. Quality
Programs should maximize the quality of the R&D they fund through the use of a clearly stated,
defensible method for awarding a significant majority of their funding. 
A. Programs allocating funds through means other than a competitive, merit-based
process must justify funding methods and document how quality is maintained.
B.  Program quality must be assessed periodically through retrospective expert review.
III. Performance
R&D programs should maintain a set of high priority, multi-year R&D objectives with annual
performance outputs and milestones that show how one or more outcomes will be reached.
Metrics should be defined not only to encourage individual program performance but also to
promote, as appropriate, broader goals, such as innovation, cooperation, education, and
dissemination of knowledge, applications, or tools.
A. Programs may be required to track and report relevant program inputs annually.
B. Programs must define appropriate output and outcome measures, schedules, and
decision points.
C. Program performance must be retrospectively documented annually.

## CSREES Experience:  Portfolio Review Expert Panel Process

- R&D Criteria and Dimensions
- Relevance
**Scope**
**Focus**
**Contemporary and/Emerging Issues**
**Solicitation and/or receptiveness for Stakeholders Input**
**Utilization of Stakeholder Input**
- Quality
**Significance of Results**
**Usefulness and Utilization of Results**
**Integration**
**Interdisciplinary Balance**
**Alignment with Current State of Science**
- Performance
**Productivity**
**Comprehensiveness of Work Produced**
**Accountability**
**Management**
- To be used as broad guidelines applicable to all Federally funded R&D efforts.

**Notes:**
These R&D criteria (Relevance, Quality and Performance) and their dimensions were scored on a 3-point scale:  3 high to 1 low.  In addition to scores, panelists also provided extremely useful comments that helped the Agency realign and manage better.  Program staff implemented a number of recommendations pertinent to their portfolios.   Annual reviews are conducted internally by program staff to determine the extent to which progress is made consistent with the national priorities.

## The Logic Model

- What is it?  A roadmap, a conceptual framework, a program theory, program theory of action (Weiss, 1998; Patton, 1997, Bickman, 1987)
- It is a concise way to show how a program is designed and will make a difference (Harvard Family Research Project)
- It is the **core** of program planning, evaluation, program management and communications (NAS, Kellogg Foundation, and UNW)

**Notes:**
Source:  The Harvard Family Research Project
The Logic Model provides a clear picture of how a program elements fit together, describing your approach and articulates the intended outcomes and how these outcomes can be measured.  It provides a basis upon which a program designer might be able to establish cause-and-effect relationship between the program and its intended outcomes.  According to the Harvard Family Research Project, the use of logic model provides useful benefits as follows:
Strategic and Program Planning - Developing a logic model is a form of strategic planning.  The process forces you to identify your vision, the rationale behind your program, and how your program will work.  This process is also a good way to get a variety of program stakeholders involved in program planning and to build consensus on the program’s design and operations.
Effective Communications - Logic models allow you to provide a snapshot view of your program and intended outcomes to funders, staff, policymakers, the media, or other colleagues.  They are particularly useful for funding proposals as a way to show that what you are doing is strategic, and that you have a plan for being accountable.
Evaluative Planning - A logic model provides the basic framework for an evaluation.  It identifies the outcomes you are aiming for – based on your program’s design – and puts those outcomes in measurable terms.
Continuing Learning and Improvement - A completed logic model provides a point of reference against which progress towards achievement of desired outcomes can be measured on an ongoing basis.

## Use of logic model to

- Set the context within which research takes place
- Consider the concept of public value
- Provide a conceptual roadmap
- Ascertain the extent to which outputs led to new knowledge, applications, solutions and reasonably consistent with expenditures
- Provide framework for evaluation

**Notes:**
A logic model...
is a simplified picture of a program, initiative, or intervention that is a response to a given situation. shows the logical relationships among the resources that are invested, the activities that take place, and the benefits or changes that result. 
Some call this program theory (Weiss, 1998) or the program's theory of action (Patton, 1997). It is a "plausible, sensible model of how a program is supposed to work." (Bickman, 1987, p. 5). 
It portrays the underlying rationale of the program or initiative. (Chen, Cato & Rainford, 1998-9; Renger & Titcomb, 2002)
is the core of program planning, evaluation, program management and communications. 
Some think the logic model is only used in evaluation. We find it equally helpful for planning and program design, managing programs and communicating. 
Public value is the value of a program to those who do not directly benefit from the program
The public sector provides information that allows consumers and producers to make better choices.
The public sector addresses a crucial concern about fairness. 
Consumers or producers create benefits for others or impose costs on others.

Laura Kalambokidis, University of Minnesota Extension

Public Value:  Fundamental secular values which provides the normative consensus about the rights and prerogatives to which citizens should be entitled and about the principles on which governments and policies should be based.  Presentation to Workshop on “Assessing the Benefits of ARS R&D Within an Economic Framework” (The Social Implications of Public Science Policy:  Public Value Mapping - Barry Bozeman, University of Georgia)

## Generic Logic Model for CSREES Reporting CSREES – Office of Planning & Accountability(This model is intended to be illustrative guide for reporting on CSREES-funded research, education and extension activities.  It is not a comprehensive inventory of our programs.)

- Generic Logic Model for CSREES Reporting CSREES – Office of Planning & Accountability**(****This model is intended to be illustrative guide for reporting on CSREES-funded research, education and extension activities.  It is not a comprehensive inventory of our programs.)**

**Outcomes**

- Actions

**Inputs**

**Situation**

**Activities**

- Knowledge

- Conditions

**Outputs**

- Version 1.2

- - New fundamental or
-   applied knowledge
- - Scientific publications
- - Patents
- - New methods &
-   technology
- - Plant & animal varieties
- - Practical knowledge for
-   policy and
-   decision-makers
- - Information, skills &
-   technology for individuals,
-   communities and programs
- - Participants reached
- - Students graduated in
-   agricultural sciences

## Logic model as a planning and an evaluation tool

| Situation | Inputs | Activities | Outputs | Outcomes |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  | Knowledge | Actions | Conditions |
| A problem of local and national interest
For example:  
Reduce the nation’s dependency on foreign oil
 | $$$$
Expenditures or
Investments
Staff and partners | Genetically engineer a bacterium to convert cellulosic material into ethanol
 | One bacterium engineered
Patents
New applications
 | Advanced knowledge | High efficiency conversation
 | Cost savings
Improved quality
Reduced dependency on foreign oil
 |

- Assumptions

- External factors

## Summary and Conclusions

- It is fully recognized that measuring the impact of basic science is difficult (OMB and NAS)
- Impact might not be realized long after studies are completed
- Debate not over yet but limited public resources have led to scrutiny about return on investments and the need to document effectiveness and efficiency of investment
- The use of logic model as a planning and evaluation tool has gained some ground
- The CSREES PREP Process has been quite helpful

## Questions?

- E-mail questions to rwc@csrees.usda.gov
- For more information, visit the Planning and Accountability web page at www.csrees.usda.gov/opa

## Topics for Next Time...

**Topics for Next Time**...

## See you in June!!!

**See you in June!!!**

- Next CSREES Reporting Web Conference will be on Thursday, June 12 from 2-4 pm (Eastern)
- Visit the conference web site at www.csrees.usda.gov/rwc for:
- The recording of this conference
- The slides from this conference
- Recordings and slides from past conferences
- Announcements