# Presentation: 351258

## NIAMSTraining Grant and Career Development Award Program Evaluation 

**Presented by**
**David Wofsy, M.D.**
**Chairman**
**Evaluation Working Group**
**September 27, 2007**

## Working Group Members

**David Wofsy, M.D., *****Chair ;  *****University of California, San Francisco;  VA Medical Center (IIIR)**
**Paul D. Allen, M.D., Ph.D.; ****Brigham and Women's Hospital**
**Anne Davidson, M.D.; Feinstein Medical Research Institute **
**Milton Hernandez, Ph.D.; National Institute of Allergy and Infectious Diseases **
**Alan Moshell, M.D.; Washington Hospital Center**
**Alice Pentland, M.D.; University of Rochester**
**Sean P. Scully, M.D., Ph.D. ; Cedars Medical Center**
**Elizabeth Shane, M.D. ; Columbia University**

## Overarching Questions

**1.****	****How successful has each component of the training program been (T32, F32, K01, K08) in maintaining the pipeline of researchers in NIAMS mission areas?**
**2.****	****Is the existing structure still appropriate to meet current and projected training needs?**

## Definition of Success - Trainees

_**Research is primary focus of career**_
**   ****- Academia**
**   ****- Industry**
**   ****- Government**
_**Research is a secondary career focus**_
**	****- Educators in research environment**
**	****- Clinicians who contribute to research**

## Definition of Success - Programs

_**Quantitative Measures**_
-    **- Percent of recipients who achieve success**
**             ****[*****Note: we did not set the bar*****]**
_**Qualitative Measures**_
**	****- Importance of the research**
**   ****- Breadth of research activities**
**   ****- Responsiveness to scientific environment**
**   ****- Collaboration**

## Definition of Success - NIAMS

_**Quantitative Measures**_
**   ****- Percent of trainees who achieve success**
**   ****- Percent of programs that achieve success**
_**Qualitative Measures**_
**	****- Impact on public health (translational research?)**
**   ****- Breadth of programs (NIAMS mission areas)**
**   ****- Recruitment of promising investigators**
**   ****- Effective strategic planning for the future**

## Bottom Line

**1.****	****The committee was impressed by the success of each component of the training program (T32, F32, K01, K08), which exceeded the committee members’ expectations in each case.**
**2.****	****Many important questions could not be answered with the available data and methodology.**
**3.****	****NIAMS should design and implement prospective mechanisms to assess the success of individual trainees, institutional training programs, and each component of its own training portfolio.**

## Methodology

**Outside contractor conducted the data collection**
**Career outcomes of trainees**
**Interviews with NIAMS EP staff**
**Discussions within working group and with colleagues**
**Study evaluated  career outcomes for sample of trainees who received:**
**Post-doctoral T32 grants in 1993-1994 ****(n=109)**
**Post-doctoral F32 grants in 1993-1994 ****(n=44)**
**K01 awards in 1995-1996 ****(n=6)**
**K08 awards in 1995-1996 ****(n=58)**

**Notes:**
Outside contractor:  Nancy Potok of McManis & Monsalve Associates
Working group members:
David Wofsy, Chairman
Paul Allen
Anne Davidson
Milton Hernandez
Alan Moshell
Alice Pentland
Sean Scully
Elizabeth Shane

## Findings

**With regard to the working group’s definition of success for individual trainees: **
**	****	**** ****T32****	**** F32****	**** K01****	**** K08**
**	****	****(n=109)****	****(n=44)****	**** (n=6)****	****(n=58)**
**__________________________________________**
**Science-Related Career****	**** 75%****	**** 84%****	****100%****	****100%**
**Current Active Research****	**** 55%****	**** 75%****	****100%****	****  62%**
**Recent Publication ****	**** 50%****	**** 59%****	****  83%****	****  85%**
**__________________________________________**

## Findings (con’d)

**17% of T32s, 34% of F32s, 83% of K01s, and 55% of K08s in sample received R01 grants.**
**Success rates reflected a logical continuum, with lower percentage of T32 trainees establishing independent research careers than F32,  K01, and K08 awardees. **

## Limitations

**No comparison of distinct areas of NIAMS’s mission (e.g., arthritis, skin, musculoskeletal)**
**No way to assess cause-and-effect relationship between training components and eventual success**
**No analysis of NIAMS’s performance relative  to other NIH institutes**

## Limitations (con’d)

**Inadequate data regarding qualitative measures**
**No data regarding retention over the long haul**
**Questionable applicability of a retrospective analysis to determine future needs in a changing landscape**

## Recommendation 1

**Establish a structured data collection mechanism to support ongoing evaluation of training program effectiveness.**

**Notes:**
R1 Notes: 
Design and implement prospective mechanisms that can assess the success of individual trainees over the course of their research careers, as well as the success of institutional training programs and each individual NIAMS training mechanism. 

Implement appropriate follow-up evaluations on an on-going basis to assess the precise nature, quality, and duration of each trainee’s career.  

Design and implement prospective mechanisms that can assess the success of individual trainees over the course of their research careers, as well as the success of institutional training programs and each individual NIAMS training mechanism. 

Implement appropriate follow-up evaluations on an on-going basis to assess the precise nature, quality, and duration of each trainee’s career.  

## Recommendation 2

**Acknowledge the economic aspects of research by providing more flexibility on the percent effort required for K awards to accommodate clinical responsibilities and other personal and professional circumstances, and by lifting restrictions that limit other sources of funding.**

**Notes:**
R2 Notes:
Look at amount of stipend being offered as well as amount of time that must be devoted to research.

## Recommendation 3

**Avoid imposing a time limit from completion of degree on applications.  Maintain flexibility and discretion of the peer review board to reward outstanding candidates.**** **

**Notes:**
R3 Notes:
The process for applying for grants can be too long with many applicants applying multiple times until they are successful.  Some people also slow down their career timeline due to family considerations.  No benefit to time limit.

## Recommendation 4

**Build on current success of the training grant and career development award program - as well as the recent increase of participants in NIAMS mission-related programs - by increasing funding for NIAMS training grant mechanisms.  The pipeline of researchers cannot be expanded unless the number of  awards and the amount of funding is also increased.**

**Notes:**
R4 Notes:
NIAMS should look at selectively increasing the awards including funding to 
	- enhance the T32 grants to encourage collaboration between programs and 
	interdepartmental work
	- additional support for F32 grants
	- support for developing the mentor-mentee relationship
	- increased funding levels for K awards: PRIORITY

NOTE: Working Group aware that budget increases in one area could result in cuts in other areas.  
Not recommending any cuts in R01s.  That would defeat the purpose.

## Recommendation 5

**Consider integrating a new component into NIAMS institutional training grant strategy that would address the related dilemmas of prolonged training followed by multiple application cycles in pursuit of a K award, which were seen as major deterrents to a career in science.**

**Notes:**
R5 Notes:
1) Some committee members favored implementation of a new mechanism in which selected trainees with particularly high potential could be identified early in their training by the institution rather than by NIAMS and provided with K level support (in dollars and duration). 
2) Some members of the committee favored implementation of a bridge-type award that would support trainees at a K level for 2-3 years after fellowship training while they sought a K award. One way of implementing this idea might be by the addition of junior faculty positions to established T32 programs so that selected trainees could be provided with financial support after completing their fellowship training. The support should be equivalent to a K award but perhaps of shorter duration, with the goal to provide a bridge while the individuals sought a K award. 
3) Some members of the committee favored using whatever funds might be devoted to the two proposed mechanisms above to expand funding for the existing K programs, and thus reduce the likelihood that an applicant might have to survive several application cycles before succeeding.

## Recommendation 6

**Increase NIAMS budget for R01 grants so that there are more opportunities for trainees to conduct independent research at the end of the pipeline. **

**Notes:**
R6 Notes:
The most significant impediment to attracting and retaining qualified individuals for careers in NIAMS-related fields is the (accurate) perception that this is a high-risk career path.  

All NIAMS-supported trainees eventually face the increasingly daunting challenge of achieving and maintaining independent R01 support.  This problem is the single most important reason for departure from research careers in favor of other options (e.g., clinical practice) as evidenced by surveys such as those conducted by the American College of Rheumatology.  

Given this reality, the most important measure that can be taken to allow trainees to achieve successful research careers (and, by so doing, validate the success of NIAMS training programs) is to insure that there is a reasonable likelihood of support for them at the other end. 

## Recommendation 7

**	****Centralize training information to make information on different mechanisms more accessible to potential applicants.  Encourage collaborative interaction with professional and constituent organizations to develop a robust complementary portfolio of training funding. **

**Notes:**
R7 Notes:
Applicants don’t necessarily have complete information, especially those eligible for career awards who may need additional income sources.
NIAMS should work with professional associations and constituent organizations with research arms to help strengthen the diversity of offerings.

## Recommendation 8

**Structure the criteria for success in grant review to encourage and reward integrated and interdepartmental approaches, foster innovation, and support interdisciplinary mentorship in applications.  Reinforce the value of grant writing and management in program curriculum.**

**Notes:**
R8 Notes:
Encourage more integration between different departments at institutions in order to reflect the changing research environment, which is becoming more interdisciplinary

Because interdisciplinary teams need more management skills, programs should teach trainees project management and grant writing

NIAMS staff play an important role in giving feedback to applicants on how criteria are being applied

## Recommendation 9

***   *****Reinforce the value of mentorship by providing a range of opportunities (e.g., annual  meeting at NIAMS, web-based modules, etc) that support training of mentors as well as trainees, and that foster an environment of collaboration and support for mentors and those being mentored.  **

**Notes:**
R9 Notes:
Mentors play a key role in developing successful researchers.

NIAMS should invest in the development of good mentors by: 
	- helping trainees find good mentors
	- proactively reaching out to mentors and trainees to help them 
	understand and develop their relationship
	- work with professional associations and organizations to 
	encourage them to be partners in promoting high functioning mentors

## Recommendation 10

**Work with other NIH Institutes and private foundations to insure that there is a comprehensive and complementary portfolio of funding mechanisms for trainees.**** **

**Notes:**
R10 Notes:
It would be in the best interest of trainees if the various agencies at NIH and other potential funding sources coordinated their activities to insure maximum efficiency and appropriate balance.