*U.S. Department of Health and Human Services*

*U.S. Department of Housing and Urban Development*

*U.S. Department of Veterans Affairs*

*U.S. Department of Labor*

**Establishing Meaningful Outcome Measures and Optimizing Program
Achievement: A Learning Community Workgroup**

*November 30 -- December 1, 2007*

*Houston, TX*

**Meeting Summary**

**OVERVIEW**

**Purpose:** Within the Policy Academy framework, the Learning Community
Workgroup (LCW) provided an opportunity for Policy Academy States to
focus on a particular topic (e.g., youth, employment, data and outcome
measures, discharge planning) in order to:

-   Maximize peer-to-peer learning

-   Address what helps and hinders implementation of State action plans

-   Highlight innovative approaches that States have implemented, with a
    focus on how they may be adapted in other States

-   Stimulate communication within the Learning Community that will
    prompt States to utilize and leverage different resources around
    them and across systems

**Participants:** Thirty-two attendees (one of whom participated via
phone) were convened in Houston from November 30 to December 1, 2006,
including:

-   27 State participants representing 15 States and Territories

-   Five additional attendees, including one State presenter and four
    facilitators and staff members

**Workgroup Learning Objectives:** A planning group comprising Federal
Agency and State representatives, as well as a Technical Assistance
Continuity Team (TACT) member (who had been providing technical
assistance to State Policy Academy Teams) held ongoing meetings to
identify key topic areas and create learning objectives in support of
the general LCW objectives that would enable participants to learn at
their appropriate level, and would result in participants bringing home
practical ideas and strategies. Objectives designed for the Data LCW
were to:

-   Optimize the design of data collection systems and collection of
    data as a means for achieving program objectives

-   Optimize reporting of program achievements for persons experiencing
    chronic homelessness as a means for assisting continuum of service
    delivery, including strategic planning, program design,
    implementation, monitoring, and reporting

-   Understand how social service systems and homeless systems
    interrelate and can implement concrete strategies for integrating
    data and enhancing collaboration within and across systems

-   Meet increased level of Federal performance reporting for partner
    agencies, including the Center for Mental Health Services
    (CMHS)/Substance Abuse Mental Health Services Administration
    (SAMHSA) (National Outcome Measures and State Outcome Measures
    (NOMS)), the Department of Housing and Urban Development (HUD)
    (Homeless Management Information Systems (HMIS)), and others

**AGENDA**

The 2-day LCW format included plenary sessions featuring State
presenters as well as small-group breakout sessions. Roundtable
discussions originally designed by the Planning Committee did not take
place on site, based on attendees' preferences to hold additional
full-group discussion. The closing session addressed next steps and
provided a final opportunity to raise questions or outstanding issues.

**Plenary Sessions:** Plenary sessions were designed to provide an
overview of and frame key issues, featuring State individuals in order
to maximize peer-learning opportunities, as well as to provide an
opportunity for national and Federal representatives to share insights
and remarks.

> ***Plenary #1:** Establishing Meaningful and Achievable Measures and
> Optimizing Program Achievement Through Data and Performance
> Measurement*

-   Overview: Barbara Ritter, Michigan

-   State Perspective

    -   Barbara Ritter, Michigan

    -   Pete Bailey, South Carolina (via telephone)

> ***Plenary #2:*** *Promising Practices to Implementing Workable Data
> Outcome Solutions*

-   Brian Smith, Vermont

> ***Plenary #3:*** *What Are the Keys to Success, and What Does It Take
> To Move Forward?*

-   State Perspective

```{=html}
<!-- -->
```
-   Charlene Moran Flaherty, Arizona

-   Michael Shank, Virginia

**Breakout Sessions:** Breakout sessions followed each plenary. Two were
structured prior to the Workgroup, and other small-group discussion
developed through onsite dynamics and outcomes. The two preformatted
breakout discussions focused as follows:

> ***Breakout Session #1***

-   What are participants currently tracking in their States?

-   Why are they tracking those data?

-   How are they doing it?

> ***Breakout Session #2***

-   What roadblocks impede States' progress? What are some strategies
    for overcoming them?

-   What key stakeholders need to be involved?

-   What resources, skills, and tools would be helpful?

**DAY 1 SUMMARY**

At the end of the first day of plenary, breakout, and full-group
discussions, the following summary was crafted to reflect two priority
areas discussed by participants and presenters:

***Priority Area 1: Using and Sharing Data Effectively***

-   **Using Data to Evaluate Program Performance, Reporting, and
    Accountability**

    -   Establishing relevant indicators

    -   Establishing local benchmark definitions relevant to local need

```{=html}
<!-- -->
```
-   **Sharing and Coordinating Data**

    -   Issues related to accessing and owning data

    -   Coordinating across different Federal, State, and program
        requirements and reporting systems

    -   Sharing data across agencies where there are confidentiality
        concerns

```{=html}
<!-- -->
```
-   **Working with Vendors**

    -   Ensuring flexibility to modify in house

    -   Ensuring accountability

    -   Considering self-hosting

***Priority Area 2: Collaboration and Political Will***

-   **Securing Leadership**

    -   Identifying a lead agency

    -   Identifying champions

```{=html}
<!-- -->
```
-   **Involving Partners**

    -   Determining who needs to be at the table

    -   Creating a process for working together to address policy issues
        (e.g., discharge planning)

    -   Creating and implementing a plan or strategy

```{=html}
<!-- -->
```
-   **Securing Funding**

```{=html}
<!-- -->
```
-   Exploring Federal, State, and private sources

-   Sharing and learning from other's experiences

```{=html}
<!-- -->
```
-   **Gaining Buy-In and Political Will**

```{=html}
<!-- -->
```
-   Getting programs and providers to view data and performance as a way
    of improving programs and not penalizing them

-   Gaining support from the Legislature, the Governor, and others

-   Collaborating with planning efforts (e.g., Interagency Council on
    Homelessness, Consolidated Plan)

```{=html}
<!-- -->
```
-   **Publishing and Marketing Data**

```{=html}
<!-- -->
```
-   Using data as a catalyst to stimulate policy questions

-   Using data to give advocates a voice

-   Creating tools and templates

```{=html}
<!-- -->
```
-   **Continuing Support for Efforts**

```{=html}
<!-- -->
```
-   Creating and enhancing peer support group

-   Follow-up from the Data LCW

**BREAKOUT SESSION DISCUSSIONS**

During the breakout sessions, participants identified the following key
policy issues:

-   Systems Change

-   Political Will

-   Collaboration and Strategy

-   Performance Reporting and Accountability

-   Governance of, Ownership of, Access to, and Sharing of Data

They identified the following general data and outcome performance data
collected:

-   Point-in-Time Data (MA, NC)

-   State Administrative Data for Research (Culhane study)

-   Research and Evaluation (including accountability) (AZ, IA)

-   HMIS Data Analysis (AZ, MI)

-   United Way Performance Reporting (NC)

-   Data Warehouse Projects (MI, AZ, CA Bay Area)

-   Cost Analysis Strategies (VA, MI, AZ)

The table below captures barriers to data collection and tracking, and
strategies for addressing those barriers. Where possible, States are
identified in parentheses.

+-------------------+--------------------------------------------------+
| **Barriers and    | **Strategies**                                   |
| Challenges**      |                                                  |
+-------------------+--------------------------------------------------+
| Entering data in  | -   Enter data at intake                         |
| a timely manner   |                                                  |
| (IA, MI)          | -   Scan cards for frequent service programs or  |
|                   |     overnight shelters                           |
|                   |                                                  |
|                   | -   Talk to other States implementing similar    |
|                   |     system                                       |
|                   |                                                  |
|                   | -   Put data entry requirement in contracts with |
|                   |     monthly draw-down request                    |
|                   |                                                  |
|                   | -   Monitor data entry monthly                   |
+-------------------+--------------------------------------------------+
| Identifying the   | -   Identify national information needs -- what  |
| appropriate       |     does everyone need to measure?               |
| outcomes and      |                                                  |
| related measures  | -   Talk to agencies about measures -- e.g., how |
| to put into       |     you define them, what targets mean           |
| system            |                                                  |
|                   | -   Develop realistic benchmarks and targets     |
|                   |                                                  |
|                   | -   Keep attuned to what is happening at Federal |
|                   |     level                                        |
|                   |                                                  |
|                   | -   Identify outcomes relevant to providers;     |
|                   |     offer menu of outcomes                       |
|                   |                                                  |
|                   | -   Meet with funders and providers              |
|                   |                                                  |
|                   | -   Recognize providers have multiple funders    |
|                   |                                                  |
|                   | -   Manage funders; help them respond to         |
|                   |     development of programs                      |
|                   |                                                  |
|                   | -   Publish data                                 |
|                   |                                                  |
|                   | -   Identify and work with willing group first   |
+-------------------+--------------------------------------------------+
| Gaining full      | -   Create broad, high status groups by          |
| participation of  |     providing information and engaging in        |
| State and         |     conversation about common issues             |
| community         |                                                  |
| agencies          | -   Develop and manage user groups, advisory     |
|                   |     councils, quality councils                   |
|                   |                                                  |
|                   | -   Find and groom "right" champion              |
|                   |                                                  |
|                   | -   Answer the question, "What's in it for us?"  |
|                   |     by interviewing people that control data     |
|                   |     elements to figure out what would help them  |
|                   |                                                  |
|                   | -   Work to address agency-specific barrier      |
|                   |     (e.g., domestic violence shelter providers   |
|                   |     are precluded from disclosing personal       |
|                   |     identifiers to HMIS, so HUD is working with  |
|                   |     a privacy expert and domestic violence       |
|                   |     agency utilize a separate database to track  |
|                   |     client data in MI)                           |
|                   |                                                  |
|                   | -   Market HMIS as more than reporting system by |
|                   |     teaching agencies how to look at reports for |
|                   |     grants accounting, supervision, program      |
|                   |     design, case management, and interfacing     |
|                   |     with legacy systems -- a champion can help   |
|                   |                                                  |
|                   | -   Develop innovative and useful workflows      |
|                   |     (e.g., matching HMIS to business workflow)   |
|                   |                                                  |
|                   | -   Establish communication channels (for        |
|                   |     positive and negative feedback) and use      |
|                   |     feedback                                     |
|                   |                                                  |
|                   | -   Make training available, cost-effective,     |
|                   |     timely                                       |
|                   |                                                  |
|                   | -   Write participation into your contracts and  |
|                   |     agreements                                   |
|                   |                                                  |
|                   | -   Be prepared to push data out of system       |
|                   |                                                  |
|                   | -   Get committed State technical staff -- the   |
|                   |     right staffing is critical                   |
+-------------------+--------------------------------------------------+

+-------------------+--------------------------------------------------+
| **Barriers and    | **Strategies (continued)**                       |
| Challenges        |                                                  |
| (continued)**     |                                                  |
+-------------------+--------------------------------------------------+
| Working           | -   Use vendor or system that allows for         |
| effectively with  |     flexibility without reprogramming, has       |
| vendors           |     process for trouble reporting and            |
|                   |     performance improvement, and allows for      |
|                   |     administrative interface                     |
|                   |                                                  |
|                   | -   Write detailed contract with financial       |
|                   |     penalties                                    |
|                   |                                                  |
|                   | -   Create a good, positive working relationship |
|                   |     with vendor so agencies can raise questions  |
|                   |     and be heard                                 |
|                   |                                                  |
|                   | -   Manage user base around communication with   |
|                   |     vendor, including process to screen what     |
|                   |     goes to vendor and for making suggestions    |
|                   |     for improvement                              |
|                   |                                                  |
|                   | -   Build a buyers' cooperative                  |
|                   |                                                  |
|                   | -   Communicate where people can plug into user  |
|                   |     groups, collective bargaining power, and     |
|                   |     collective renegotiation of contracts        |
|                   |                                                  |
|                   | -   Reach out to other communities using similar |
|                   |     system if you are having problem; when       |
|                   |     necessary, get technical assistance          |
+-------------------+--------------------------------------------------+
| Collecting and    | -   Explain HMIS as record keeping system: talk  |
| using data        |     about what it can do for partner agencies    |
| effectively       |                                                  |
|                   | -   Establish a data warehouse                   |
|                   |                                                  |
|                   | -   Provide training on data                     |
|                   |     collection/utilization (MI)                  |
|                   |                                                  |
|                   | -   Involve Department of Corrections (e.g.,     |
|                   |     covering some costs, importing data) (MI)    |
|                   |                                                  |
|                   | -   Have system administrator work for local     |
|                   |     Continuum of Care (MI)                       |
|                   |                                                  |
|                   | -   Secure staff funding out of balance of State |
|                   |     Continuum of Care                            |
|                   |                                                  |
|                   | -   Include community-based data in -- can look  |
|                   |     at issues over time                          |
|                   |                                                  |
|                   | -   Give advocates a voice with data             |
|                   |                                                  |
|                   | -   Secure HMIS funding from legislature         |
|                   |     (hardware, software, staffing) (VI)          |
|                   |                                                  |
|                   | -   Capitalize on turnover in administration     |
|                   |                                                  |
|                   | -   Develop political will and media; when you   |
|                   |     publish data, also tell story                |
+-------------------+--------------------------------------------------+
| Increasing        | -   Provide one-on-one training at computer with |
| comfort with      |     staff on data lookup and data entry and      |
| technology        |     update (NC)                                  |
|                   |                                                  |
|                   | -   Provide training on privacy protocols and    |
|                   |     how to talk with clients about consent (NC)  |
+-------------------+--------------------------------------------------+
| Moving from       | -   Add outcome measures to Emergency Shelter    |
| outputs to        |     Grant (ESG) application (IA)                 |
| outcomes          |                                                  |
+-------------------+--------------------------------------------------+
| Communicating     | -   Use online resources to communicate          |
| effectively       |                                                  |
+-------------------+--------------------------------------------------+

**NEXT STEPS AND RESOURCES**

As part of the closing session, participants identified what they would
bring back to their State from this Workgroup meeting and a few key
actions that they might take.

Key actions included making a presentation to the Agency of Human
Services IT Chief (based on Pete Bailey's presentation on how to serve
clients better if data are better integrated) and sharing Arizona's
(Charlene Moran Flaherty) self-sufficiency matrix with the Agency of
Human Services Executives, as well as the Continuum of Care and the
State Interagency Council on Homelessness. One participant wanted to
have the Workgroup presenters and the Workgroup's TACT representative
present to their State's Interagency Council. Several participants
wanted to focus on how to gain resources in their State.

All electronic resources and materials from the Workgroup are available
at [www.hrsa.gov/homeless](http://www.hrsa.gov/homeless) as part of the
Data LCW.