+----------------------------------+-----------------------------------+
| **TEMPLATE FOR PRESENTERS---**   |                                   |
|                                  |                                   |
| **NOTE: THERE ARE NO RIGHT OR    |                                   |
| WRONG ANSWERS -- WE ARE BUILDING |                                   |
| A NEW SCIENTIFIC BASIS FOR QIIs  |                                   |
| AND ATTEMPTING TO UNDERSTAND     |                                   |
| CURRENT APPROACHES, THE          |                                   |
| TRADEOFFS AMONG THEM, AND        |                                   |
| CHALLENGES FACED BY QII          |                                   |
| EVALUATORS**                     |                                   |
+----------------------------------+-----------------------------------+
| SNAPSHOT OF THE QII              |                                   |
+----------------------------------+-----------------------------------+
| \-- Setting(s)                   | Variety of ambulatory care        |
|                                  | settings from community health    |
|                                  | centers to large medical group    |
|                                  | practices                         |
+----------------------------------+-----------------------------------+
| \--Change targets (who/what      | Pilot practices change delivery   |
| expected to change)              | in accord with the Chronic Care   |
|                                  | Model; care and outcomes of their |
|                                  | patients change for the better    |
+----------------------------------+-----------------------------------+
| \--QII goal (outcomes)           | Condition-specific process and    |
|                                  | outcome measures such as % with a |
|                                  | recorded self-management goal, %  |
|                                  | of diabetics with a recorded foot |
|                                  | exam, % of diabetics with HbA1c   |
|                                  | \<8%, % of asthmatics on          |
|                                  | controller medication, % of heart |
|                                  | failure patients on               |
|                                  | ACE-inhibitors, % of CHF patients |
|                                  | readmitted, etc.                  |
+----------------------------------+-----------------------------------+
| \--QII strategy (please also     | Use Model for Improvement (aim,   |
| fill in the chart in Appendix I) | measures, rapid-cycle changes)    |
|                                  | and Breakthrough Series           |
|                                  | collaboratives to redesign care   |
|                                  | delivery in accord with the       |
|                                  | Chronic Care Model.               |
+----------------------------------+-----------------------------------+
| \--Rationale for QII:            |                                   |
+----------------------------------+-----------------------------------+
| \-\-\--Evidence base (clinical   | The Chronic Care Model (CCM) is   |
| and/or QII)                      | based on evidence of              |
|                                  | interventions that improve care   |
|                                  | in various chronic illnesses.     |
|                                  | Like the Model for Improvement    |
|                                  | and Breakthrough Series, it had   |
|                                  | not been tested in an RCT .       |
+----------------------------------+-----------------------------------+
| \-\-\--Prior tests of QII        | Evaluations of prior Breakthrough |
|                                  | Series.                           |
+----------------------------------+-----------------------------------+
| \-\-\--Theoretical basis         | The CCM is not a large T theory   |
|                                  | describing some general           |
|                                  | principles for how the world      |
|                                  | works. It is empiric and          |
|                                  | heuristic, but has its foundation |
|                                  | in the premise that medical       |
|                                  | practice is oriented to react to  |
|                                  | undifferentiated acute disease,   |
|                                  | which is a poor fit with the      |
|                                  | needs of the chronically ill.     |
+----------------------------------+-----------------------------------+
| \-\-\--Other                     |                                   |
+----------------------------------+-----------------------------------+
| \--Sites and subjects and their  | Participating practices were all  |
| inclusion and exclusion          | volunteers, or at least their     |
| criteria. Describe what the      | organizations were volunteers.    |
| target for site selection was in | They had to pay for their         |
| terms of the diffusion curve for | involvement in the BTS. It is     |
| adoption of the intervention     | safe to say that most would       |
| (early adopters, early majority, | qualify as early adopters.        |
| late majority, or laggards)      |                                   |
+----------------------------------+-----------------------------------+
| \--Scope or scale of the         | Each organization involved in a   |
| intervention (such as single     | Breakthrough Series was           |
| site, multi-site, or national    | represented by a pilot population |
| rollout; single agency or        | of practices (generally 1 or 2    |
| cross-agency effort)?            | physicians or mid-levels) and     |
|                                  | their patients.                   |
+----------------------------------+-----------------------------------+
| \--Does the intervention cross   | Most participating practices were |
| organizational boundaries        | primary care, but teams often     |
|                                  | included relevant specialists.    |
| such as primary care and         | CHF teams frequently involved     |
| specialty care?                  | cardiology.                       |
+----------------------------------+-----------------------------------+
| \--Methods used to recruit sites | Recruitment of sites in the       |
| and participants                 | national collaboratives was done  |
|                                  | largely through the communication |
|                                  | vehicles and contacts of the      |
|                                  | Institute for Healthcare          |
|                                  | Improvement. Recruitment to the   |
|                                  | Washington state diabetes         |
|                                  | collaborative was done by the     |
|                                  | State Health Department and local |
|                                  | QIO.                              |
+----------------------------------+-----------------------------------+
| \--Length of study               | The Breakthrough Series each      |
|                                  | lasted about one year.            |
+----------------------------------+-----------------------------------+
| \--Did you implement change one  | Sites were encouraged to use PDSA |
| time, or did you use another     | cycles, and to make small,        |
| approach (eg rapid-cycle PDSA)?  | incremental changes based on      |
| In other words, how many         | early results. In fact, teams     |
| modifications were made during   | made an average of more than 30   |
| the course of the study and how  | significant practice changes.     |
| often?                           |                                   |
+----------------------------------+-----------------------------------+
| \--If you used an approach with  | No changes to the overall         |
| midcourse corrections, what      | strategy. We changed the agendas  |
| kinds of changes were made to    | for the Learning Sessions based   |
| QII during the project?          | on evaluations by faculty and     |
|                                  | teams, and established a parallel |
|                                  | set of activities for senior      |
|                                  | leaders in the participating      |
|                                  | organizations.                    |
+----------------------------------+-----------------------------------+
| \--What were the findings of the | Generally teams made major        |
| QII?                             | changes in accord with the CCM.   |
|                                  | Impacts on patients were          |
|                                  | generally positive.               |
+----------------------------------+-----------------------------------+
| \--Were you the instigator of    | Our RWJF national program,        |
| the QII and its evaluation or    | Improving Chronic Illness Care,   |
| was it someone else (e.g., a     | designed the program, and funded  |
| health plan, hospital) (please   | and oversaw a peer-reviewed       |
| specify)                         | process to select the evaluator.  |
|                                  | RAND Health designed the          |
|                                  | evaluation.                       |
+----------------------------------+-----------------------------------+
| \--Did you do both the QII and   | ICIC and IHI conducted the        |
| the evaluation?                  | program; RAND independently       |
|                                  | evaluated it.                     |
+----------------------------------+-----------------------------------+
| \--Please draw an organizational | See Appendix                      |
| chart showing the social and     |                                   |
| financial relationships in your  |                                   |
| study. An example is provided in |                                   |
| Appendix II.                     |                                   |
+----------------------------------+-----------------------------------+

+----------------------------------+-----------------------------------+
| EVALUATION                       |                                   |
+----------------------------------+-----------------------------------+
| \--Design (check one or more and |                                   |
| add comments as needed, e.g.,    |                                   |
| how you modified standard        |                                   |
| design):                         |                                   |
+----------------------------------+-----------------------------------+
| -   Post only                    |                                   |
+----------------------------------+-----------------------------------+
| -   Pre-post no comparison       |                                   |
+----------------------------------+-----------------------------------+
| -   Pre-post with comparison     | X Comparison group consists of    |
|     group (specify comparison    | practices (and their patients) in |
|     group)                       | the same or neighboring           |
|                                  | organizations that are not        |
|                                  | participating the collaborative   |
+----------------------------------+-----------------------------------+
| -   Group or cluster randomized  |                                   |
|     controlled trial             |                                   |
+----------------------------------+-----------------------------------+
| -   RCT -- individual as unit of |                                   |
|     analysis                     |                                   |
+----------------------------------+-----------------------------------+
| -   Interrupted time series      |                                   |
+----------------------------------+-----------------------------------+
| -   Multiple baseline            |                                   |
+----------------------------------+-----------------------------------+
| -   Regression discontinuity     |                                   |
|     design                       |                                   |
+----------------------------------+-----------------------------------+
| -   Other, please specify        |                                   |
+----------------------------------+-----------------------------------+
| \--Methods used (e.g., context   | Detailed evaluation of the        |
| measured, interaction between    | organizations involved in the     |
| context and intervention         | collaborative (QI activities and  |
| measured,[^1] other              | priority, culture, leadership,    |
| methodological approaches to     | etc.), the effectiveness of the   |
| strengthen research/evaluation   | QI teams, the nature and          |
| design.)                         | significance of the changes made, |
|                                  | and impacts on patients.          |
+----------------------------------+-----------------------------------+
| \--Analytic approach             | We used hierarchical regression   |
|                                  | for process and outcome measures  |
|                                  | with patients nested within       |
|                                  | sites, comparing intervention     |
|                                  | patients to comparison group      |
|                                  | patients. For some measures       |
|                                  | collected only in intervention    |
|                                  | sites, we regressed measures of   |
|                                  | downstream success on staff       |
|                                  | attitudes and activities in the   |
|                                  | intervention sites.               |
+----------------------------------+-----------------------------------+
| ACTIONS TAKEN TO AVOIDE BIAS IN  |                                   |
| DESIGN, DATA COLLECTION,         |                                   |
| ANALYSIS AND INTERPRETATION      |                                   |
+----------------------------------+-----------------------------------+
| \--Who was blinded to condition  | Data collectors were generally    |
| (circle NA if not applicable)    | unaware of whether the patient    |
|                                  | was in an intervention or control |
|                                  | practice.                         |
+----------------------------------+-----------------------------------+
| -   Participants (e.g.,          | Y [N]{.underline}A                |
|     providers)                   |                                   |
+----------------------------------+-----------------------------------+
| -   Organization(s)              | Y [N]{.underline} NA              |
+----------------------------------+-----------------------------------+
| -   Investigators involved in    | Y [N]{.underline} NA              |
|     data collection              |                                   |
+----------------------------------+-----------------------------------+
| -   Investigators involved in    | Y [N]{.underline} NA              |
|     data analysis                |                                   |
+----------------------------------+-----------------------------------+
| Was there separate funding for   | Y N NA                            |
| intervention and evaluation?     |                                   |
+----------------------------------+-----------------------------------+
| What other protections to        | \_x_formal protocols              |
| maintain independence of the     |                                   |
| evaluation were employed (check  | \_\_leadership support            |
| all that apply)                  | (specify:\                        |
|                                  | _\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ |
|                                  |                                   |
|                                  | \_x_functional separation of data |
|                                  | collection from implementation    |
|                                  |                                   |
|                                  | \_x\_ functional separation of    |
|                                  | data analysis from implementation |
|                                  |                                   |
|                                  | \_\_interpretation of data        |
|                                  | (specif                           |
|                                  | y:\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_) |
|                                  |                                   |
|                                  | \_\_separation of primary         |
|                                  | hypotheses from ancillary post    |
|                                  | hoc questions                     |
|                                  |                                   |
|                                  | \_\_\_other -- please describe:   |
+----------------------------------+-----------------------------------+
| \--Why was this evaluation       | Couldn't randomize organizations  |
| approach chosen?                 | that volunteered and paid for the |
|                                  | opportunity to participate        |
+----------------------------------+-----------------------------------+
| \--Was this evaluation design    | yes                               |
| your original choice?            |                                   |
+----------------------------------+-----------------------------------+
| \--What challenges did you face  | Problems in recruiting sites      |
| during the evaluation?           | because of IRB issues or concerns |
|                                  | about the burdens of data         |
|                                  | collection; lack of a comfortable |
|                                  | baseline period; poor response    |
|                                  | rates in some sites because of    |
|                                  | IRB requirements                  |
+----------------------------------+-----------------------------------+
| \--What are the strengths of the | Conservative control group;       |
| approach used?                   | reliance on research rather than  |
|                                  | QI data                           |
+----------------------------------+-----------------------------------+
| \--What are the weaknesses of    | Control groups may or may not be  |
| the approach used?               | comparable or uncontaminated;     |
|                                  | Absence of baseline values for    |
|                                  | some measures                     |
+----------------------------------+-----------------------------------+
| \--How in your opinion did the   | We believe that the selection of  |
| design and methods you used      | controls was conservative, i.e.,  |
| affect findings/conclusions      | may have mitigated                |
| about the effectiveness of the   | intervention-control differences  |
| intervention?                    |                                   |
+----------------------------------+-----------------------------------+
| \--Rate the internal validity of | 7                                 |
| your QII on a scale from 1-10    |                                   |
| (with 10 as the highest)         |                                   |
+----------------------------------+-----------------------------------+
| GENERALIZABILITY/EXTERNAL        |                                   |
| VALIDITY                         |                                   |
+----------------------------------+-----------------------------------+
| \--Which settings/populations    | Organizations committed to QI     |
| would your findings be           | whether fee-for-service or        |
| generalizable?                   | capitated, commercial or safety   |
|                                  | net                               |
+----------------------------------+-----------------------------------+
| \-- Please review the list of    | The ICIC Collaboratives used the  |
| QII strategies in Appendix I     | BTS and rapid cycle improvement   |
| below, then compare your         | methods in the standard way       |
| intervention to others in the    | promoted by IHI. We differed      |
| same category or class (e.g.     | perhaps in the use of a           |
| "didactic" or "rapid cycle       | well-defined change package based |
| improvement"). How is your       | on the CCM.                       |
| intervention similar to, or      |                                   |
| different from, other            |                                   |
| interventions that use this      |                                   |
| approach? How do you know?       |                                   |
+----------------------------------+-----------------------------------+
| SUMMARY QUESTIONS                |                                   |
+----------------------------------+-----------------------------------+
| \--If you had the evaluation to  | Basic design was sound, but       |
| do all over again, what would be | certain implementation problems   |
| your ideal in terms of design    | could and should be overcome. 1.  |
| and methods (if you hadn't had   | Increase participation in the     |
| the constraints you (may have)   | evaluation through                |
| had on implementing your ideal   | problem-solving and incentives    |
| evaluation)?                     | like helping identify target      |
|                                  | population; 2. Try to convince    |
|                                  | participants to use a common IRB; |
|                                  | 3. Initiate evaluation and        |
|                                  | baseline data collection before   |
|                                  | intervention underway             |
+----------------------------------+-----------------------------------+
| \--What would it take to do the  | 1\. It might help if there were   |
| evaluation differently? (what    | consensus guidance to delivery    |
| constraints would need to be     | organizations on HIPAA, IRB, and  |
| removed/modified?)               | related issues that would reduce  |
|                                  | their fears of participating in   |
|                                  | evaluations; 2. Longer lead time. |
+----------------------------------+-----------------------------------+
| \--Describe "cost/charges" data  | Participation costs averaged      |
| if cost data were collected      | \$140,000 per site including the  |
|                                  | costs associated with making the  |
|                                  | changes to their systems.         |
+----------------------------------+-----------------------------------+
| \--Describe cost effectiveness   | NA                                |
| analysis if it was done.         |                                   |
+----------------------------------+-----------------------------------+
| FUNDING CONTEXT                  |                                   |
+----------------------------------+-----------------------------------+
| \--How was your evaluation       | Contract from RWJF National       |
| funded?                          | Program                           |
+----------------------------------+-----------------------------------+
| \--Did you have any difficulties | Had to compete in an RFP process. |
| getting funding? Please          |                                   |
| describe.                        |                                   |
+----------------------------------+-----------------------------------+
| SUSTAINING THE STRATEGY          |                                   |
+----------------------------------+-----------------------------------+
| \--Do you/the organization       | ICIC supports participating       |
| (practice, health plan, State)   | organizations through its         |
| have funding and/or management   | biennial meeting and website.     |
| support to sustain the strategy, |                                   |
| if it was considered successful? |                                   |
+----------------------------------+-----------------------------------+
| \--Is the strategy being         | Limited evidence suggests that    |
| sustained and if so, how?        | over one-half of successful BTS   |
|                                  | participants are sustaining or    |
|                                  | increasing the improvements.      |
+----------------------------------+-----------------------------------+
| PROMOTION/ACADEMIC CONTEXT       |                                   |
+----------------------------------+-----------------------------------+
| -Is this kind of evaluation      | If published in credible,         |
| research acceptable for          | peer-reviewed journals            |
| promotion and tenure in your     |                                   |
| academic context?                |                                   |
+----------------------------------+-----------------------------------+
| \-\--Please share your thoughts  |                                   |
| on challenges or opportunities   |                                   |
| in your academic context         |                                   |
+----------------------------------+-----------------------------------+
| PUBLICATION CONTEXT:             |                                   |
+----------------------------------+-----------------------------------+
| \--Did you have any difficulty   | Some reviewers and editors still  |
| getting your work published?     | have difficulty dealing with      |
| Please describe.                 | non-randomized designs,           |
|                                  | multi-component interventions,    |
|                                  | and non-fixed interventions.      |
+----------------------------------+-----------------------------------+
|                                  |                                   |
+----------------------------------+-----------------------------------+
| DISSEMINATION AND DIFFUSION      |                                   |
+----------------------------------+-----------------------------------+
| \--If your QII was successful,   | Chronic care BTS guided by the    |
| do you know if anyone beyond the | CCM are quite common now.         |
| original organization is using   |                                   |
| it now?                          |                                   |
+----------------------------------+-----------------------------------+
| \--Is it being re-evaluated?     | Yes, the Health Disparities       |
|                                  | Collaborative of HRSA's BPHC is   |
|                                  | being evaluated as are several    |
|                                  | local ones.                       |
+----------------------------------+-----------------------------------+
| \--Is it being modified in any   | The major modifications have been |
| way and will that have any       | efforts to limit travel by        |
| impact on its success?           | shortening or reducing the number |
|                                  | of sessions, and/or conducting    |
|                                  | more of it electronically.        |
+----------------------------------+-----------------------------------+
|                                  |                                   |
+----------------------------------+-----------------------------------+
| SUMMARY COMMENTS:                |                                   |
+----------------------------------+-----------------------------------+
| \--If you could change the       | 1.  Research agencies should      |
| "system" (e.g., partnering,      |     cooperate with public         |
| funding, publishing,             |     (Medicaid, CMS) and private   |
| post-project dissemination),     |     payors to develop and support |
| what would be your top 3         |     physician practice networks   |
| priorities?                      |     committed to quality          |
|                                  |     improvement (e.g., the        |
|                                  |     practice networks supported   |
|                                  |     by North Carolina Medicaid).  |
|                                  |     These networks would be       |
|                                  |     expected to and rewarded for  |
|                                  |     improving care., and          |
|                                  |     participating in evaluations  |
|                                  |     of their efforts to improve   |
|                                  |     care.                         |
|                                  |                                   |
|                                  | 2.  Develop national consensus    |
|                                  |     guidelines and related tools  |
|                                  |     for organizations             |
|                                  |     participating in QI           |
|                                  |     evaluations that would        |
|                                  |     encourage development of and  |
|                                  |     participation in multi-site   |
|                                  |     IRB review; and help          |
|                                  |     alleviate fears of HIPAA or   |
|                                  |     IRB violations; breaches of   |
|                                  |     patient, provider or          |
|                                  |     organizational                |
|                                  |     confidentiality; exposure to  |
|                                  |     litigation; etc.              |
|                                  |                                   |
|                                  | 3.  A national program to         |
|                                  |     encourage the implementation  |
|                                  |     and use of automated disease  |
|                                  |     registries (whether part of   |
|                                  |     an EHR or separate software). |
+----------------------------------+-----------------------------------+

**Appendix I: QI Strategies.**

Please check the QI strategies used in your study.

  -----------------------------------------------------------------------
  **QI Strategy**                                     **Your study**
  --------------------------------------------------- -------------------
  *Overarching models*[^2]                            X CCM

  Chronic Care model implementation                   X

  "Defined change package" (SPECIFY COMPONENTS)       X varied by
                                                      condition

  *National guideline revision*                       

  *Data/measures development*                         X

  *Benchmarking/Audit and Feedback*                   

  Audit and feedback                                  

  Audit of surveillance data                          

  Facilitate data sharing within a state/region       

  *Technical Assistance from QI team/researcher*      X Breakthrough
                                                      Series faculty

  Access to implementation facilitator                Not in the practice

  Assistance with planning and coaching during        X List-serve,
  intervention                                        conference calls
                                                      plus learning
                                                      sessions

  *Leadership strategies*                             Senior leaders
                                                      involved

  Opinion-leader strategy                             

  *Reorganization of care/additional staff*           

  "Organizational change" NOS                         X Organization
                                                      change is the focus

  Practice change : Collaboration between primary     X
  care and specialty care                             

  Practice change/"office systems"/process            X
  improvement methods                                 

  Enhanced record keeping                             X Use of registries

  Px change: financial change                         

  Px change: informatics re-design                    X registries

  Px change: staffing changes                         

  Case management                                     X recommended but
                                                      no additional
                                                      support to sites

  Community volunteers?                               

  Data-based deployment of policy and fire personnel? 

  Linking patients with community resources?          X Key element of
                                                      CCM

  Provider reminder systems                           X
  -----------------------------------------------------------------------

**Appendix II: Organizational diagram.**

Please provide an organizational diagram indicating (as appropriate)
sponsorship, the PI and the research team, the staff (or levels of
staff) involved in implementing the intervention, and the target of the
intervention. Sample below. For your diagram, please provide more
details on the role of each party/group indicated in the chart. Please
indicate if some parts of the research illustrated by the diagram had
funding from different sources. Please also indicate who was blinded to
what.

100+ Partcipating

> Organizaions

[^1]: See Rychetnick.

[^2]: In addition to checking these, check off all the specific QI
    strategies used within these overarching strategies