# Presentation: 517417

## Strategies for Internal Evaluation and Project Review

- City College of New York – Memorial Sloan-Kettering Cancer Center Collaboration
- Karen Hubbard, Ph.D. & Bruce Rapkin, Ph.D.
- November 20, 2003

- Supported by the National Cancer Institute
- Comprehensive Minority Biomedical Branch
- Grants #5U56CA96301 and 5U56CA096299

## Components of Evaluation

- Overall Evaluation of the Partnership
- Products and Processes
- Role of the IAC
- The Pilot Review Process
- A Case Study
- Other Examples
- Building Review Capacity

## ACKNOWLEDGEMENTS

## Program Steering Commitee

- Frank Bayliss, Ph.D. San Francisco State U.
- Stephen Lepore, Ph.D. Brooklyn College/Columbia
- Francesca Gany, M.D., New York University
- David Gershon, Ph.D., Technion
- Harvey Ozer, M.D., UMDNJ – New Jersey Medical School
- Sanya Springfield, Ph.D., NCI

## Internal Advisory Core(Years 1 and 2)

- Patricia Agre
- Robert Alfano
- Peter Bach
- A.J. Franklin
- Jerry Hurwitz
- Eliot Jurist
- Andrew Koff
- Carol Moore
- Sara Olson

- Jamie Ostroff
- Tanya Pagan-Raggio
- Richard Payne
- Margaret Rosario
- Charlotte Russell
- Mark Steinberg
- John Tarbell
- Vicky Tartter
- Roger Wilson

## Administrative Core

- Collette Houston, Director, MSKCC Office of Clinical Research
- Jenny Romero, M.D., Ralph Lauren Cancer Center
- Ellen Smiley, Ph.D., CCNY, Acting Dean of Social Sciences

## Partnership Staff

- Ron Edwards
- Margaret Michel
- Maureen Wallace

## Monitoring Overall Performance

- Anticipated in the Original Application
- Administrative Core: Smiley and Houston
- Relevant Experience with Other Mechanisms
- SCORE Grant
- T32 Training Grant
- Enumerated Markers of Success
- Challenge: Capturing these Data

## Indicators of Products

- Activities Supported by the U56
- Pilots, Trainees and Hires
- Publications and Presentations
- Grants Submitted and Funded
- Fellowships and Assistantships
- Thesis Projects
- Trainees’ Career Trajectories
- Diverse Participation in Clinical Research
- Protocols Addressing Disparities

## Challenges and Solutions

- No Baseline Information
- Systems being developed by partnership
- Disparate Sources of Information
- Involvement of Administrative Core
- Networking
- Elbow Grease
- No Intrinsic Metric for Success
- Role of the IAC and PSC

## Process Evaluation

- Rationale
- Develop Infrastructure as well as Products
- Fostering Communication is Central
- Document the Basis for Decisions
- Efficiency of Procedures
- Strategies
- Goal Setting and Goal Attainment
- Meeting Agenda and Minutes
- Narrative of the Annual Report

## Observations and Recommendations from Year One Evaluation

- Easier than anticipated to attract interest from both CCNY and MSK groups
- Many teams need unanticipated assistance and/or mentorship from co-PIs
- More defined mentoring plans and on-going review needed to encourage rigorous pilot studies
- Greater support needed for co-PIs to sustain their level of involvement partnership
- Greater definition of scientific areas needed, to reflect areas of mutual strength and to help set priorities

## Changing Role of the Internal Advisory Core

- Initially, emphasized networking
- To evaluate potential
- To foster awareness of and participation in partnership
- Many individuals with diffuse responsibilities
- Presently, emphasize academic leadership
- To provide guidance and support to PIs
- To evaluate new and proposed pilots
- Fewer individuals with more defined roles

## Original Organization of IAC

- Basic Science
- Clinical Research
- Behavioral Research
- Populations Research
- Ethics

## Amended Organization of IAC

- Cell Biology and Aging
- Molecular Biology
- Immunology
- Bioinformatics
- Health Disparities
- Communication, education and outreach
- Coping and adaptation
- Translational Research
- Imaging and Image Processing
- Medical Physics
- Clinical Applications
- Ethics and Training:  Addressed in All Areas

## Preparation of Pilot ApplicationsSpecial Review Criteria

- Scientific Plan in Context of a Longer Range Research Strategy
- Infrastructure/Partnership Development
- Description of the Collaboration
- Shared Scientific Activities
- Appropriate Plan for Joint Products
- Delineation of Roles at Each Institution
- Mentoring Plan for Junior Investigators
- Mentor need not be same as collaborator
- Plan for Involvement and Support of Students
- Integrated Budget Justification & Timeline

## Pilot Review Process

- PIs Determine Completeness and Readiness of Application
- Identification of Qualified External Reviewers for Each Grant
- Use Internal MSK Review Committee When Possible, Otherwise External Reviewers
- Reviews Submitted to IAC for Recommendations to Pilot Teams
- Revised Applications Prioritized for Submission to EPSC

## Case Study of a Pilot ReviewReligious Strains Study

- Original 5-Page Pilot Application Approved
- Greater detail required for MSK Protocol
- Departmental review at MSK included IAC members and investigators
- Major recommendations from this review led to revised aims, modified procedures and clearer long range plans

## Reviewers’ Suggestions and Investigators’ Responses: Religious Strains Study

- Better justification for sample
- Made use of existing data for purposive sampling
- Clearer instrument development strategy
- Devised three-part plan to gain needed data on survivors, new, and late-stage patients
- Spawned SCORE application and new research ties
- More clinical support for research team
- Initiated on-going Spiritual Beliefs Seminar series including MSK clinicians and chaplains, CCNY faculty and graduate students

## Other Examples of Pilot Study Changes in Response to Reviews

**Colorectal screening team **pushed out resilience theory to explain paradoxical results from studies based on rational, cognitive models
**DNA Damage team **clarified how complementary data from three different labs would address novel questions & required a certain sample size
**Bioinformatics team **spent an additional year developing relationships among programs, came in with revised pilot
**CCNY Medical professor **requested MSK collaborators to help think through a new research area

## Building Capacity for Review

- Human Subjects Research Enhancement Grant - NIH (Wilson, Houston, Smiley, PIs)
- Enhance CCNY’s capacity to monitor and review independent and joint cancer research
- Areas of Focus
- IRB Infrastructure and Consent Procedures
- Data Safety and Monitoring Procedures
- Third Party Rights and Risks
- Education for Faculty and Students

## Proposed Education Modules

- Patient recruitment in clinical research
- Interview studies in hospital settings
- Linkage of clinical data and self report
- Ethical issues in the specification of control conditions in treatment trials
- Acquisition and use of human tissue
- Research with low health literacy populations
- Standards for translation of research materials
- Distress & unmet needs identified in research
- Ethical issues in cancer screening studies
- Ethical issues in community-based research

## Conclusions

- On-going evaluation has led us to change pilot studies and even structure of IAC
- When possible, we have used **existing mechanisms**
- Resources have been used to **increase capacity** for effective evaluation and oversight – U56 and Institutional
- We have built in strategies to help investigators **anticipate criteria **for evaluation
- Evaluation and program development are **two sides of the same coin**
- Rigorous review is necessary to **support teams and build capacity** of the partnership