# Presentation: 162597

## CTWG Evaluation Plan

**CTWG Evaluation Plan**
**Results of Baseline Feasibility Analysis**

- DRAFT CONFIDENTIAL

## Rationale for Systematic Evaluation of NCI Clinical Trials System 

- Past evaluations based predominantly on opinions of expert panels
- NCI never previously performed a systematic evaluation that  integrates qualitative/perceptual and quantitative information about its clinical trials activities
- Establishes a structured framework for continuous monitoring and feedback for mid-course corrections
***Initial results of a feasibility analysis for an ongoing evaluation  process requiring regular CTAC input***** **

- DRAFT CONFIDENTIAL

## CTWG Overall Evaluation Plan

- Establish structured evaluation system
- Designed by experienced evaluation specialists
- Blend of quantitative/qualitative measures
- Perceptions of clinical trial experts and structured empirical data
- Perform baseline feasibility analysis
- Perform periodic evaluations as CTWG implementation proceeds

- DRAFT CONFIDENTIAL

## Two Categories of Measures:Comparison of Baseline to Future

**System Outcome Measures**
- Is the overall output of the NCI clinical trials system improving?
**System Performance Measures**
- Are the individual CTWG initiatives having the desired effect on the performance of the NCI clinical trials system?

- DRAFT CONFIDENTIAL

## System Outcome Measures

**Quality of Trials**
- Publications
- Strength of trial designs
**Impact of Trials**
- Guide new therapeutics or diagnostics development
- Lead to changes in patient management
**Efficiency of Trial Development and Initiation**
- Time to first patient on study
**Efficiency of Trial Conduct**
- Rate of accrual, cost-effectiveness

- DRAFT CONFIDENTIAL

## Interlocking Data Collection Methods

- Interviews
- Qualitative: Perception of current system and practices
- Database analysis
- Quantitative: How long, how fast, how many
- Document review
- Factual information
- Use multiple data sources to triangulate analysis

- DRAFT CONFIDENTIAL

## Baseline Interviews

**Discussions with 81 individuals in 2007**
- NCI staff (25 interviews across CTEP/IDB, CTEP/CIB, OCTR, DCP, OC)
- Phase I/II trialists (25 interviews with N01 holders, U01 holders, CG trialists, R01/R21/P50 trialists)
- Phase III trialists (17 interviews with all nine Cooperative Group PIs and eight CG disease committee chairs)
- CCOP/CCOP Research Base PIs (9)
- Industry trialists (5)
**Primarily open-ended questions, some designed to elicit perceptions of specific facts/events**

- DRAFT CONFIDENTIAL

## Baseline Database Analysis

**Databases analyzed**
- CTEP Clinical Data Update System (CDUS)
- DCP Enterprise System Knowledgebase (DESK)
**All trials active 1/1/2000-12/31/2005**
- Patient registration by institution, by trial
- Rate of accrual
- Publication of trials
**All LOIs/concepts active 1/1/2000-12/31/2005**
- Time from LOI/concept submission to decision point
- Time from LOI/concept submission to first patient on study
**No current database captures all clinical trials performed at Cancer Centers**

- DRAFT CONFIDENTIAL

## Baseline Document Review

- NCI Program Guidelines
- Cancer Treatment Guidelines (e.g., ASCO, ACS, NCCN, US Preventive Services Task Force)
- Academic medical center tenure and promotion guidelines

- DRAFT CONFIDENTIAL

## Baseline Feasibility Analysis for Evaluation:  Expert Panel 

- Participated in development of measures
- 	and interview guides at beginning
- Membership
- 9 NCI-funded trialists
- 1 industry trialist
- 1 patient advocate
- Reviewed key findings at end

- DRAFT CONFIDENTIAL

## Quality of Trials: Publications

**Data Source:**
- CTEP database; Cooperative Group publication lists
**Feasibility:**
- Cooperative Group trials can be linked to publications
- CTEP database useful for future but not baseline
- No easy linkage for non-CTEP trials
**Baseline Findings:**
- 50% of closed Cooperative Group Phase II and Phase III trials resulted in publications (4 Groups)
**Recommendation:**
- Include field for reporting publications in clinical trials databases

- DRAFT CONFIDENTIAL

## Impact of Trials: Patient Management 

**Data Source:**
- Cancer Treatment Guidelines (e.g., ASCO, ACS, NCCN, US Preventive Services Task Force)
- JCO “Clinical Cancer Advances 2006: Major Research Advances in Cancer Treatment, Prevention, and Screening”
**Feasibility:**** **
- Feasible but time-intensive to link Guidelines to trials
- Feasible to use JCO Clinical Cancer Advances Series
**Baseline Findings:**
- 4 of 9 “major advances” supported by NCI clinical trial
- 9 of 15 “other notable advances” supported by NCI clinical trial
**Recommendations: **
- Use annual JCO article to assess impact
- Assess including ASCO plenary session presentations

- DRAFT CONFIDENTIAL

## CTWG Coordination InitiativesIncentives for Collaboration in NCI Guidelines

**Data Source:**
- NCI Award Guidelines (Cancer Center, Cooperative Group, SPORE, P01)
**Feasibility:**
- Guidelines clear concerning whether and in what way collaboration rewarded
**Baseline Findings:**
- Cancer Center: Weak incentives for collaboration across Centers
- Cooperative Group: Strong incentives for collaboration across Groups
- SPORE: Strong incentives for collaboration
- P01: No incentives for collaboration
**Recommendations:**
- Repeat analysis at regular intervals during CTWG implementation

- DRAFT CONFIDENTIAL

## CTWG Prioritization InitiativesPhase I/II Investigational Drug Trials

**Data Source:**
- Phase I/II trialist interviews
**Feasibility:**
-  Perceptions can be determined by interviews
**Baseline Findings:**
- Perceptions concerning CTEP Clinical Development Plans highly variable
- Perception that the pre-IDSC process was not transparent
- Mixed perceptions of pre-IDSC trial quality
**Recommendations: **
- Focus future interviews on role of IDSC in enhancing transparency, collaboration  and  quality of Clinical Development Plans and trials

- DRAFT CONFIDENTIAL

## CTWG Prioritization InitiativesPhase III Cooperative Group Trials

**Data Source:**
- Cooperative Group trialists, CTEP/CIB, OCTR interviews
**Feasibility: **
- Perceptions can be determined by interviews
**Baseline Findings:**
- CTEP prioritization process perceived as opaque by some trialists and transparent by others
- CIB staff perceived the quality of Phase III trial concepts to be mixed
- CIB staff perceived little duplication in Phase III trials
**Recommendations:**
- Focus future interviews on role of Scientific Steering Committees in enhancing transparency and quality of trial concepts

- DRAFT CONFIDENTIAL

## CTWG Operational Efficiency Initiatives  Efficiency of Phase III Trial Accrual

**Data Source:**
- CTEP database; Phase III trialist interviews
**Feasibility:**
- Accrual data by trial and site readily available
**Baseline Findings:**
- 150  Phase III trials active in 2000-2005 with 1516 accruing institutions
- 67% of trials accrued less than 5 patients per site per trial
- 40% of institutions accrue 1-10 patients, representing 3% of patients
- 16% of institutions accrue 100+ patients, representing 64% of patients
- Most sites perceived below economically viable accrual levels
**Recommendations:**** **
- Repeat analysis at regular intervals

- DRAFT CONFIDENTIAL

## CTWG Operational Efficiency Initiatives  Efficiency of Phase III Trial Accrual

- DRAFT CONFIDENTIAL

## Next Steps

- Develop specific plan for future evaluation, refining baseline measures and developing protocols for future measures
- Incorporate additional information in clinical trials databases to strengthen future evaluation efforts
- Prepare initiative-specific timeline for future evaluation
- CTAC Subcommittee will be formed to oversee evaluation process

- DRAFT CONFIDENTIAL