## DRAFT

**Community Access Program**

Report on the Logic Model Process\*

November 7, 2001

I.  **Introduction**

In the federal fiscal year 2000, the U.S. Department of Health and Human
Services - Health Resources and Services Administration (HRSA) awarded
23 Community Access Program (CAP) grants to organizations in 22 states.
In March of the following year, 53 additional grants were made to
communities located in 23 states. The purpose of these grants is to
support public and private organizations in bridging service gaps and
improving health outcomes for the uninsured and underinsured.

The Center for Health and Public Services Research (CHPSR) of the Wagner
Graduate School of Public Service at New York University, in partnership
with the Center for State Health Policy at Rutgers University, is
providing technical assistance to the CAP sites and assisting HRSA in
the monitoring of the program. As part of this effort, CHPSR has worked
with HRSA to support each funded site\'s completion of a logic model, a
particular method for displaying the components of each site\'s proposed
project. In a logic model, the grantees display the relationship between
the activities they plan to undertake and the results, or \"outputs,\"
they expect to achieve. The logic model then shows the benefits or
\"outcomes\" that the outputs are expected to produce, and finally the
overall impact of the combination of these elements on the community in
which the project will take place. To complete the logic model, the
grantees articulate the assumptions that underlie their project
planning: that the output of a chronic disease information hotline, for
example, will only result in the outcome of better chronic disease
management if people actually call the hotline and act on the
information they receive. Finally, the logic model guides the community
in identifying appropriate interim as well as long-term measures of the
project\'s progress.

The research team provided an introduction and brief training in the
logic model tool to CAP grantees at their first meeting. Members of the
research team were available to answer grantees\' questions as each
project site developed a logic model for its CAP project. The team then
reviewed and evaluated each submitted logic model, sending written
comments with suggestions for improvement back to CAP grantees. Based
upon the models developed in the first year, as well as conversations
with federal program staff, CHPSR created a logic model for the entire
Community Access Program. Finally, to learn how the sites viewed the
process and whether they found the logic models to be a useful tool for
management or evaluation, CHPSR staff interviewed a number of the
year-two CAP sites about their experience creating the logic model.

**Overall Findings**

The models submitted by the sites in both waves showed a wide variation
in completeness, coherence, sophistication of thought and reflection of
the planned project as described in the original project proposal. The
following is an overview of strengths and weaknesses of the logic models
that were submitted by the March 2001 grantees as of mid-August 2001 (45
out of the 53 sites):

-   About 40% of the sites (18/45) submitted logic models that reflected
    an understanding of what a logic model should be and a thoughtful
    effort to display their plans for their project in this new way.

-   Most of the submitted models were incomplete. Only six of the sites
    submitted complete lists of assumptions and only seven submitted a
    fully realized set of measures. An additional 18 sites submitted a
    partial set of assumptions (e.g., defining assumptions for the
    project as a whole) and five sites submitted logic models that
    included only some measures (primarily outcome measures).

Other observations about the submitted logic models include:

-   A number of sites (8/45) submitted distinct logic models for
    different elements of their project but did not then create a
    summary model describing the project as a coherent whole.

-   Few sites (6/45) included with their logic models a description of
    the process they used to create the model that showed collaboration
    among the members of their health services coalition. The sites that
    did describe a collaborative logic model development process tended
    to submit stronger logic models.

-   Sites struggled with the distinction between outputs and outcomes:
    the need for further clarification of these elements came up in
    phone calls to CHPSR staff as well as in the review of the logic
    models themselves. A majority (25/45) of sites had at least some
    difficulty describing the link between what they expected their
    project to create and the benefit they expected to result. This was
    true even in some of the better logic models. In the follow-up
    interviews several sites commented that they, or their coalition
    members, had difficulty with this distinction.

II. **Development of the Program Logic Model**

Based upon the logic models that were submitted, the research team
developed a typology of seven main intervention approaches -- each with
multiple subcategories: (1) integration of service delivery systems, (2)
improvement of business practices and integration of financial systems,
(3) increased enrollment in health insurance plans, (4) expansion of the
delivery system, (5) implementation of community/patient education
programs, (6) improvements in service delivery, and (7) informing public
policy. These categories, together with other program documents, were
used to form the outlines of a logic model for the entire CAP
initiative. The research team then met with federal program staff,
refined the model, and articulated its assumptions. (Creation of program
measures was postponed pending clarification of any evaluation plans.)
Both the CAP Logic Model and Program Assumptions are available on the
CAP Community web site
([www.capcommunity.hrsa.gov](../www.capcommunity.hrsa.gov)).

This collaborative mapping process created a shared understanding of how
and why the Community Access Program was expected to work -- and what
anticipated outcomes it was expected to achieve -- given its resources
and planned activities. The resulting program logic model also formed
the basis of the monitoring tool that was subsequently developed by the
research team. (A report on the findings from the first six-month
monitoring period has been provided separately.)

### The Implications of These Findings

The logic model has the potential to be a valuable tool in developing
and managing large-scale new initiatives such as the CAP projects. In
its essence, use of the logic model guides project planners in applying
the scientific method - the articulation of a clear hypothesis to be
tested - to their project development work. Researchers (e.g., Weiss)
have described the value of theory-based evaluation. In demonstrating an
impact, describing a need or justifying an expenditure, being able to
state what was expected and what, in fact, occurred is essential.

The different components of the logic model allow practitioners and
evaluators to weigh the expected and actual result of a project and also
to take the evaluation to the next level: if our results were not what
we expected, why was this the case? Did we not carry out the activities
we planned, create the products we expected or did these products not
produce the benefits we hoped for? Were our assumptions faulty? Our
resources inadequate? Finally, what can we learn from these findings?
How do we need to adjust our hypotheses? What can we try next, with a
better chance of success? Through this learning, the health care
community can develop the best set of strategies to improve health
outcomes for the under- and uninsured.

a.  ***The Role of Assumptions***

Assumptions play an essential role in the logic model: to design a
project that has a good chance to succeed, project planners need to
articulate what they expect to be true so they and their colleagues can
highlight any gaps in the logic of the program and assess whether this
assumption will, in fact, turn out to be valid. Those sites that engaged
in this process and developed assumptions for at least part of their
programs found this to be the most valuable part of the exercise. In
several cases, it was through the articulation of the assumptions that
the sites were able to identify gaps in their program, sharpen their
thinking, or build a credible case in support of the program concept.

-   One site noted that after examining their assumptions about the
    relationship between quality management and clinical care
    coordination, they decided to change their staffing plans, creating
    two senior level positions instead of the one originally planned.

-   Another site commented that by articulating the assumptions of their
    program, they were able to identify what they "needed from each of
    the partners." Having a logic model in hand then gave them
    "credibility in asking for it."

-   One grantee noted that by showing their assumptions, they were able
    to make clear to their constituency and to government officials that
    they did not have to reduce quality in order to reduce costs.

-   One coalition has used their assumptions to help identify why a
    project component was not moving well. They realize that they had
    "assumed that knowledge was in place \[for one of their partners\]
    that was not there" and that these "partners were not yet ready to
    share information."

-   One site that modified their approach after examining their
    assumptions expressed some concern that "it is dangerous to do one
    of these \[logic models\] after the grant has been funded. By doing
    this, you may spot gaps in your original application and the you
    worry about tell the program office that you can't do what you said
    you would."

Yet few CAP sites submitted logic models with fully realized sets of
assumptions. For example, often the most unreliable assumptions are
those that state that people will, without coaching, change their work
or care-seeking habits because of the existence of a new type of
technology. Over 70% of the March 2001 CAP sites (33/45) plan some sort
of patient information system or computer-based referral system as part
of their CAP project. Of these, less than half (15) included in their
logic models any explicit assumptions that providers or patients will
use the new system. This evidence that sites have not planned fully for
the implementation of new technology raises concerns about the
likelihood that these projects will be successful.

Similarly, the success of a care coordination program (planned by 25 out
of the 43 respondents) assumes, among other things, that patients will
be responsive to recommendations from the care coordinators. Only nine
of the sites planning such a program are explicit about this underlying
assumption.

The fact that so few sites were able to articulate a full set of
assumptions underscores their importance and the need for more
assistance in this area. Several actions would improve sites'
incorporation of assumptions into their logic models:

-   More time should be spent on assumptions when sites are first
    introduced to logic models. Logic model training sessions should
    include enough time for each site to develop assumptions (perhaps
    based on a case study or an example from a previous project year)
    and to have these assumptions reviewed by session faculty.

-   In this year's training session, although faculty did emphasize the
    importance of assumptions, sites still did not include them in their
    models. In many cases, the people attending the conference at which
    the training was conducted did not work on the logic model back
    on-site. First, the staff who will complete the logic models need to
    attend the logic model training session and CAP management staff
    must make this requirement clear. Second, as written materials are
    heavily used as references by staff not attending the training
    session, these materials, especially the logic model template,
    should include clearer and more specific references to assumptions
    and their place in the model.

-   Since an in-depth examination of assumptions may well lead to
    program modifications, HRSA needs to have a system in place for
    reviewing changes in program design. If sites continue to use their
    logic models to assess their progress and revisit their assumptions,
    this need will continue for the duration of the grant.

b.  ***The Role of Measures***

The logic model format\'s inclusion of measures for activities and
outputs helps project managers figure out methods for tracking the
success of projects over the short-term, before outcomes or an impact
can be realized. Those sites that did focus on measures found this to be
a very useful exercise. One site that generally found the logic model
process to be burdensome and time-consuming, nevertheless noted that it
was useful to link program activities to measurements: "\[we\] are so
busy implementing, implementing, implementing, that it is easy to forget
to link activities to outcomes." Their coalition partners, while not
interested in the process of logic model development, were interested in
using the measures that emerged from the logic model. Another site
valued the measures since they allow them to "demonstrate interim
gains."

Some sites found this process difficult. One confessed that given the
uncertainty surrounding the future of the program, it was hard "to take
the outcomes measures seriously." That so few sites were able to
articulate short-term measures (although most had developed measures of
activities for the project management matrix of their original proposal
to HRSA) raises concerns about plans for short-term evaluation of the
overall project. We recommend the following:

-   As with assumptions, more time should be spent on measures at the
    logic model training sessions, including time for faculty to work
    directly with each site or smaller groups of sites on measure
    definition. The staff who will work on the logic model should be the
    staff attending the training, and written materials, including the
    logic model template, should be revised to emphasize the place of
    measures in the model.

-   The confusion many sites experienced in differentiating outputs and
    outcomes may have contributed to the difficulty in defining interim
    measures. As the output/outcome distinction is a useful one
    (especially for measuring and evaluation), the logic model training
    and materials should better clarify these two elements. One site
    suggested using a set of probing questions that would help
    differentiate between the two categories. Another site suggested
    that they be provided with more examples.

-   In addition, we recommend that the sites take the time now to
    develop a set of measures to match, at a minimum, the activities
    described in their logic model (listing activities was a part of the
    logic model most sites did well). This effort would be a
    particularly appropriate job for local evaluators. The resulting
    measures could then be circulated among sites, providing the
    opportunity for those using similar measures to meet to share their
    experiences, strategies, and effective practices.

c.  ***Collaborating to Develop the Logic Model***

One goal in asking CAP sites to develop logic models was to strengthen
the sense of collaboration among the different public and private sector
organizations participating in each sites\' project. The few sites that
did describe a collaborative development process uniformly characterized
it as positive. Most of the sites we interviewed, however, did not view
involving coalition members in the initial development of the draft as a
high priority. Generally, staff developed a draft model and then shared
it with a wider circle of coalition members. Several sites explained
that their coalition members are geographically dispersed and difficult
to convene. Many partners were already stretched thin in implementing
the project, and the staff was reluctant to burden them with this
additional task.

To facilitate a more collaborative process, we recommend the following:

-   The organizations sponsoring each CAP project gather regularly;
    these meetings present an excellent opportunity for review of the
    project logic model, perhaps beginning with a discussion of CHPSR's
    comments on the model.

-   An ongoing review of the logic model should take place at these
    collaborative meetings at least quarterly.

d.  ***Other Uses of the Logic Model***

In interviews, some CAP sites described specific ways in which they have
been able to use their logic models:

-   Several sites described using the logic models for ongoing program
    management and evaluation. One site plans to attach a timeline to
    the logic model and use it as a "business plan" for program
    implementation.

```{=html}
<!-- -->
```
-   Another site saw the logic model as being of limited use. They noted
    that they did not intend to use it as a "guiding" document, but that
    they did plan to use it to "keep track of measures."

-   One community has used their logic model for their own internal
    six-month evaluation -- "looking at priorities and goals, checking
    off activities and generally making sure they are on the right
    track."

-   One of the grantees found their logic model to be a useful tool in
    their application for state and federal waivers. They noted that the
    logic model is particularly helpful with funders and government
    agencies that are outcome-oriented since it makes clear that "you
    expect to achieve interim outcomes and milestones."

-   This same site plans to use the logic model to identify and approach
    foundations that may be interested in components of the initiative.
    They also use it at staff orientations.

-   One coalition has used the logic model as a clear summary of the
    project that "allows everyone to see their role."

CAP sites should be made aware of the potential for using their logic
models for these additional purposes.

III. **Conclusion**

The logic model is a powerful tool for project planning. Many sites
receiving funding from HRSA for CAP projects in March 2001 found at
least part of the logic model development process helpful to their
ability to achieve their projects\' goals. But the value of this process
has a direct relationship to the amount of time and energy that is
invested in it.

Part of the strength of the logic model may also be its greatest
difficulty: that it forces planners to think of their projects in a
conceptually different way. Participants in this round of CAP grants
conceived their projects and submitted their original proposals about a
year before they were expected to \"begin\" their projects and prepare
the logic model. This time lag may have led to changes in staff for the
projects; to work taking place in a different sequence than originally
planned and, most importantly, to the management staff\'s way of
thinking about the project becoming more fixed - and the conceptual
shift required by the logic model thereby becoming more difficult.

Although there will not always be such a long space of time between
submitting a proposal and beginning the work of the project, it would
benefit all future grantees to begin to think of their projects in the
logic model format as they actually begin to develop their project
components (that is, as they write their proposals). We understand that
this was required of the most recent group of CAP applicants. A
well-prepared logic model, with carefully stated assumptions and clear
interim measures, would also facilitate HRSA\'s work in deciding which
projects to fund.

Whether logic models are used before or after grants are awarded, this
year\'s experience clarifies the need for sites to have more resources
-- more training, more technical assistance and more reference materials
-- at their disposal as they create their models. Clarification of some
of the terminology, and additional direction in how to create logic
models, in the place of assumptions and measures, and in the importance
of collaboration on the model among all participants in the project,
will allow the value of the logic model to become more apparent to more
grantees, and will result in stronger projects that are more likely to
achieve their important goal.

\* Submitted to HRSA by Katherine E. Garrett, Sue A. Kaplan, John
Billings (the Center for Health and Public Service Research at New York
University), and Joel Cantor, Denise Davis (the Center for State Health
Policy at Rutgers University).