[Slide 1]{.underline}

**Data Learning Community Workgroup**

Charlene Moran Flaherty

Arizona Department of

Economic Security

[Slide 2]{.underline}

**Project Goals**

Support activities that make a real difference in lives of homeless
people

Evaluate the effectiveness of homeless projects

Implement method of understanding what works and what does not

Identify best practices

[Slide 3]{.underline}

**Project Goals**

Conscious shift from focus on outputs to investing in outcomes

Reduce the gap between knowledge and practice

Which programs are effective

What interventions produce results

Where do we target investment

[Slide 4]{.underline}

**Project Elements**

The Arizona Homeless Evaluation project involves:

A Self Sufficiency Matrix that has been tested for validity and
reliability

HMIS data, including demographics and length of stay

Statistical analysis of data

[Slide 5]{.underline}

**Project Elements**

Profiles may be developed for homeless programs

Identify agency strengths

Identify areas for agency improvements

> Objective feedback to assist agencies in determining if they want to
> specialize in a subpopulation niche or what needs strengthening to be
> more effective with their homeless clients

[Slide 6]{.underline}

**Arizona Self Sufficiency Matrix**

Two key features:

17 domains in the Self Sufficiency Matrix

Client's status on each domain is measured by a five-point scale

[Slide 7]{.underline}

Self Sufficiency Matrix

  ------------ ------- ------------- --------------- ------------- ------------ ---------------
  Domain       Score   **1**         **2**           **3**         **4**        **5**

  Income               No income     Inadequate      Can meet      Can meet     Income is
                                     income and-or   basic needs   basic needs  sufficient,
                                     spontaneous or  with subsidy; and manage   well managed;
                                     inappropriate   appropriate   debt without has
                                     spending        spending      assistance   discretionary
                                                                                income and
                                                                                benefits

  Employment           No job        Temporary,      Employed full Employed     Maintains
                                     part-time or    time;         full time    permanent
                                     seasonal;       inadequate    with         employment with
                                     inadequate pay, pay; few or   adequate     adequate income
                                     no benefits     not benefits  subsidized   and benefits.
                                                                   housing      

  Housing              Homeless or   In              In stable     Household is Household is
                       threatened    transitional,   housing that  in safe,     safe, adequate,
                       with eviction temporary, or   is safe but   adequate,    unsubsidized
                                     substandard     only          subsidized   housing.
                                     housing and-or  marginally    housing      
                                     current rent    adequate                   
                                     slash mortgage                             
                                     payment is                                 
                                     unaffordable;                              
                                     over 30 percent                            
                                     of income                                  

  Food                 No food or    Household is on Can meet      Can meet     Can choose to
                       means to      food stamps     basic food    basic food   purchase any
                       prepare it;                   needs, but    needs        food household
                       relies to a                   requires      without      desires.
                       significant                   occasional    assistance   
                       degree on                     assistance                 
                       other sources                                            
                       of free or                                               
                       low-cost food                                            

  Child Care           Needs child   Child care is   Affordable    Reliable,    Able to select
                       care, but     unreliable or   subsidized    affordable   quality child
                       none is       unaffordable;   childcare is  childcare is care of choice
                       available     inadequate      available,    available,   
                       slash         supervision is  but limited   no need for  
                       accessible    a problem for                 subsidies    
                       and-or child  child care that                            
                       is not        is available                               
                       eligible                                                 

  Safety               Home or       Safety is       Safety is     Safety is    Safety is
                       residence is  threatened      threatened    threatened   threatened
                       not safe;     slash temporary slash         slash        slash temporary
                       immediate     protection is   temporary     temporary    protection is
                       level of      available;      protection is protection   available;
                       lethality is  level of        available;    is           level of
                       extremely     lethality is    level of      available;   lethality is
                       high;         high            lethality is  level of     high
                       possible CPS                  high          lethality is 
                       involvement                                 high         

  Parenting            Safety        There are       There are     There are    There are
  Skills               concerns      safety concerns safety        safety       safety concerns
                       regarding     regarding       concerns      concerns     regarding
                       parenting     parenting       regarding     regarding    parenting
                       skills        skills          parenting     parenting    skills
                                                     skills        skills       
  ------------ ------- ------------- --------------- ------------- ------------ ---------------

[Slide 8]{.underline}

Self Sufficiency Matrix

  ------------ --- --------------- --------------- -------------- -------------- -------------
  Children's       One or more     One or more     Enrolled in    Enrolled in    All eligible
  Education        eligible        eligible        school, but    school and     children
                   children not    children        one or more    attending      enrolled and
                   enrolled in     enrolled in     children only  classes most   attending on
                   school          school, but not occasionally   of the time    a regular
                                   attending       attending                     basis
                                   classes         classes                       

  Adult            Literacy        Enrolled in     Has high       Needs          Has completed
  Education        problems and-or literacy and-or school diploma additional     educational
                   no high school  GED program     slash GED      education      slash
                   diploma slash   and-or has                     slash training training
                   GED are serious sufficient                     to improve     needed to
                   barriers to     command of                     employment     become
                   employment      English to                     situation      employable.
                                   where language                 and-or resolve No literacy
                                   is not a                       literacy       problems
                                   barrier to                     problems to    
                                   employment                     where they are 
                                                                  able to        
                                                                  function       
                                                                  effectively in 
                                                                  society        

  Legal            Current         Current         Fully          Has completed  No active
                   outstanding     charges/trial   compliant wit  probation      criminal
                   tickets or      pending,        probation      slash parole   justice
                   warrants        noncompliance   slash parole   successfully   involvement
                                   with probation  terms          within past 12 in more than
                                   slash parole                   months, no new 12 months
                                                                  charges filed  and-or no
                                                                                 felony
                                                                                 criminal
                                                                                 history

  Health Care      No medical      No medical      Some members;  All members    All members
                   coverage with   coverage and    EG, Children   can get        are covered
                   immediate need  great           on AHCCCS      medical care   by
                                   difficulty                     when needed,   affordable,
                                   accessing                      but may strain adequate
                                   medical care                   budget         health
                                   when needed.                                  insurance
                                   Some household                                
                                   members may be                                
                                   in poor health                                

  Life Skills      Unable to meet  Can meet a few  Can meet most  Able to meet   Able to
                   basic needs     but not all     but not all    all basic      provide
                   such as         needs of daily  daily living   needs of daily beyond basic
                   hygiene, food,  living without  needs without  living without needs of
                   activities of   assistance      assistance     assistance     daily living
                   daily living                                                  for self and
                                                                                 family

  Mental           Danger to self  Recurrent       Mild symptoms  Minimal        Symptoms are
  Health           or others;      mental health   may be present symptoms that  absent or
                   recurring       symptoms that   but are        are expectable rare; good or
                   suicidal        may affect      transient;     response to    superior
                   ideation;       behavior, but   only moderate  life           functioning
                   experiencing    not a danger o  difficulty in  stressors;     in wide range
                   severe          self slash      functioning    only slight    of
                   difficulty in   others;         due to mental  impairment in  activities;
                   day-to-day life persistent      health         functioning    no more than
                   due to          problems with   problems                      every day
                   psychological   functioning due                               problems or
                   problems        to mental                                     concerns
                                   health symptoms                               
  ------------ --- --------------- --------------- -------------- -------------- -------------

[Slide 9]{.underline}

**Using the Matrix**

Periodicity of data collection

Upon any program entry and at program exit

Transitional housing programs collect dataset every 6 months after entry

Permanent housing programs must collect dataset every 12 months after
entry

Subjects

All adults over the age of 18

[Slide 10]{.underline}

**Using the Matrix**

Should be done jointly by the client and case manager; the matrix is an
excellent engagement tool for family-centered practice

Select one and only one status level in each of the domains

If you and the client feel that the best score is somewhere between two
numbers, score using the lower number

[Slide 11]{.underline}

**Arizona Homeless Evaluation Pilot**

At the top of the slide are the words Permanent Supportive Housing. An
arrow points up to them from the words Transitional Shelter. An arrow
points up to those words from the words Emergency Shelter.

[Slide 12]{.underline}

**Arizona Homeless Evaluation Pilot**

More challenging clients show greatest increase in self-sufficiency

Lower expectations; greatest gains

Less challenging clients show smallest increase in self-sufficiency

Higher expectations; smaller gains

[Slide 13]{.underline}

**Measurement Model**

Answers the question:

Is this assessment technique valid and reliable?

Using factor analysis, the instrument was found to measure overall
self-sufficiency and two subscales:

Level of Independence

Level of Dysfunction

[Slide 14]{.underline}

**Measurement Model**

Reliability is essentially a measure of the amount of error in the scale

The three scores; Independence, Dysfunction, and Total Score; were all
found to have acceptable reliability

[Slide 15]{.underline}

**Measurement Model**

While the Arizona Self Sufficiency Matrix was found to be reliable and
valid within all three Continua of Care, the typical client in each
Continuum was found to score differently

[Slide 16]{.underline}

**Continua Self Sufficiency Scores**

On a point graph, the x-axis represents independence and the y-axis
represents dysfunction. A point labeled Maricopa is around the
coordinates negative 0.07 and 0.08. A point labeled Rural is around the
coordinates negative 0.04 and negative 0.48. A point labeled Pima is
around the coordinates 0.45 and negative 0.02.

[Slide 17]{.underline}

**Predictive Model**

Equations are generated from the Self Sufficiency Matrix data submitted
by homeless programs to determine the predictors of change in
Independence, Level of Difficulty of Clients Served, and Total Self
Sufficiency

[Slide 18]{.underline}

**Predictive Model**

These equations are then used to predict the amount of change in each
individual client if randomly assigned to a homeless program

Each individual's predicted change is uniquely determined based upon the
client's individual characteristics

These predicted changes constitute the expected change

[Slide 19]{.underline}

**Predictive Model**

Equations are generated from the data to determine the predictors of
change in Independence, Dysfunction, and Total Self Sufficiency

[Slide 20]{.underline}

**Predictive Model**

These equations are then used to predict the amount of change in each
individual client if randomly assigned to a homeless program

Each individual's predicted change is uniquely determined based upon the
client's individual characteristics

These predicted changes constitute the expected change

[Slide 21]{.underline}

**Predictive Model**

The predicted change for each individual is then compared with the
actual change

> Agencies whose clients consistently do better than predicted are the
> most successful
>
> Agencies whose clients consistently under-perform the predicted change
> are the least successful
>
> Agencies whose clients perform consistently with predictions are
> typical of their peers in the Continuum

[Slide 22]{.underline}

**Preliminary Overall Findings**

The distinction between emergency and transitional programs appears to
be an arbitrary definition; there is no difference between the two types
of programs in who they serve, the types and extent of problems their
residents exhibit, and the expected change is the same from both
programs

[Slide 23]{.underline}

**Preliminary Findings**

There is a window between 40 days and 72 days when program participants
experience positive change

Less than 17 days is insufficient to effect positive change and clients
staying more than 72 days tend to regress; difference for families

This suggests that stabilization between 6 and 10 weeks followed by
placement in long-term housing with supports when necessary is likely to
maximize client impact

[Slide 24]{.underline}

**Effectiveness of Intervention on Life Skills over Time**

On a point graph, the x-axis is labeled Days and the y-axis, which gives
numbers from negative 30 to positive 30, is labeled Change. Countless
dots collectively labeled Series 1 are concentrated primarily within the
first 200 days and with change between negative 10 and positive 10. The
last dot is a little less than 500 days in.

[Slide 25]{.underline}

At the top of the chart is written Eff Life. Underneath this term is a
box containing the following information:

Mean: Negative 0.3258

Standard Deviation: 6.9122

N: 744; 100 percent

Predicted: Negative 0.3258

Running down from the box is a line labeled Stay: P Value Equals 0.0009,
F Equals 8.7261, DF Equals 3,740. The line then branches into four lines
corresponding to certain ranges. The first line, marked 0 to 17, leads
to a box containing the following information:

Mean: Negative 2.1893

Standard Deviation: 4.7387

N: 163; 21.91 percent

Predicted: Negative 2.1893

The second line, marked More than 17 to 40, leads to a box containing
the following information:

Mean: Negative 0.1542

Standard Deviation: 6.4835

N: 159; 21.37 percent

Predicted: Negative 0.1542

The third line, marked More than 40 to 72, leads to a box containing the
following information:

Mean: 1.6943

Standard Deviation: 7.5553

N: 156; 20.97 percent

Predicted: 1.6943

The fourth line, marked More than 72 to 480, leads to a box containing
the following information:

Mean: Negative 0.4711

Standard Deviation: 7.5634

N: 266; 35.75 percent

Predicted: Negative 0.4711

[Slide 26]{.underline}

**Additional Variables**

Maricopa Continuum of Care supplements the data set with measures of
history of homelessness and primary reason for homelessness

These additional variables considerably strengthen the ability to
identify clients particularly appropriate or inappropriate for referral
to a specific agency

[Slide 27]{.underline}

**Implications and Further Research**

Are gains in self-sufficiency achieved in shelter sustained in housing

Are people continuing in shelter due to institutional needs or resource
constraints; IE, lack of housing affordable to population

What are appropriate incentives slash disincentives for maximizing
results and resources

[Slide 28]{.underline}

**Referral Profiles**

Agencies can be ranked from most to least effective from the results
obtained thus far

Profiles can be generated describing the clients most likely and least
likely to benefit from a given agency

[Slide 29]{.underline}

**Profile A**

This agency is the most successful in Maricopa County in having positive
impact with those recently released from jail or prison

Agency is among the least effective in working with homeless Hispanics

[Slide 30]{.underline}

**Profile B**

This agency is very effective in meeting the needs of homeless families
who were relatively high functioning prior to homelessness

Least effective agency in having positive impact with homeless Native
Americans

While somewhat effective with mentally ill homeless, ineffective with
homeless substance abusers

[Slide 31]{.underline}

**Profile C**

This agency primarily serves older and-or disabled homeless persons

Highly effective with older clients

Much less effective with disabled clients

Needs to determine if they should further develop the older niche or
make programmatic changes to be more effective with the disabled persons

[Slide 32]{.underline}

**Next Steps**

Increased participation

Refining models

Ongoing training

Identify and replicate best practices

Feedback

Technical Assistance: peer-to-peer, et cetera

Use results to make a conscious shift from focus on outputs to investing
in outcomes