>> 12/13/21

3:00 Meeting 

>> Good afternoon or Good morning for some of you.  My name is Susan Jenkin, and I'm with the U.S. Department of Health & Human Services.  We are waiting for some more people to sign in, so we'll get started in just a couple of minutes.  It looks like people are still signing in, so we'll give it another 30 seconds or so, then we'll get started.  Thank you all.

It looks like the numbers are starting to stabilize.  So again, welcome.  Good afternoon or Good morning as the case may be.  My name is Susan Jenkins, and I am with the U.S. Department of Health and Human Services.  And we're here today to talk about measuring progress toward achieving your State Plan goals and objectives focusing on outcomes, how to really measure whether and how your State Plan is meeting the goals and objectives that it was designed to meet.  So not just how many people were served for example, but are those people doing better as a result of your State Plan activities?  Next slide.  Today we're going to focus on what does the State Plan guide itself says about results?  What are the outputs versus outcomes?  Because that's an important distinction to make.  What are categories of outcomes which is really the so-what, the results, the success, the value of your State Plan?  And how can they be measured and reported?  And what are some baseline data?  Baseline data are very important for us to really have a good sense as to whether we are meeting the goals that we have, and then, what's next?  Some ideas for what are some steps that you can take if you have some areas for improvement in this type of activity.  And then there will be time for questions at the end.  If you have questions at the end, you can put them in the chat, or if you have questions throughout the presentation, you can put them in the chat and I will try to answer them as we go along, if you have something that's very timely to a particular slide.  Next slide, please.  I do like to start off with a little story about why outcomes and performance measurement is important.  And, Alice in Wonderland, this is a lesson that is so.  What helps us to remember why it's important to measure outcomes.  If you may remember the story of Alice in Wonderland, once she gets into Wonderland, once he falls down into the rabbit hole she sees the Cheshire cat, she asks, which way am I to go?  The cat says it depends on where you want to go.  And, Alice chimes in, well it doesn't matter really where.  The cat interrupts and says "well then it doesn't matter which way you go."  Alice then clarifies and says, "well, as long as I get somewhere" and the cat says "well, you're sure to do that" that's the same idea if we don't measure our outcomes we don't measure our progress towards the success that we say outcomes, the results that we want from our State Plans or any of our programs really.  We're sure to have done something.  But we won't really know what it is and we won't know if we've made the difference that we're intending to make.  Next slide.  So to think about specifically what does the State Plan guidance, the new State Plan guidance say about results?  And it says a number of things.  And so you are required to report on outcomes as part of the State Plan process.  The State Plan is to document tangible outcomes so those aren't theories or we're going to improve the lives of older adults in our state or territory, but tangible outcomes.  How many people's lives have been improved?  How have they been improved in really measurable quantitative outcomes? And they're still within that range of different things you can do that fit your program and fit your capacity for measuring outcomes, but tangible outcomes are important.  We talk about translating activities’, datas, and outcomes into best practices.  A lot of times we use the language of "best practices."  But many times that term is used incorrectly.  We think about best practices in terms of what we think works.  What makes us feel good?  What appears to have a good outcome without really the data to back it up.  And that's where measuring outcomes comes into play.  C talks about providing data on the number and types of people in an organization that will be targeted for such efforts.  And measures to determine if those efforts have been successful.  So not just what we're doing or how much of that we're doing, but is what we're doing successful in improving the organizations that serve older adults as well as the lives of those older adults, the communities in which they live, their family members and caretakers.  And D talks about the plan should describe objectives and measures, and progress towards the areas that the plan is to address.  So knowing where we started and where it is that we want to go, and how far towards that goal did we get?  Next slide.  The Older Americans Act in this text is also in the State Plan guidance, also specifies that we must measure outcomes and so, the State Plan shall provide that the state agency will conduct periodic evaluations including evaluations of the effectiveness of services.  And for the sake of this conversation, we're not talking about full-blown project and program evaluation, but really the dictionary definition of evaluation which is an assessment of the value of something.  So we don't have to necessarily do a randomized control trial or other fancy methodological program evaluation, but really, evaluating the extent to which our programs, the programs and initiatives funded under this stat plan are meeting the goals as outlined in the State Plan.  Next slide.  The language in the older Americans act goes on to say further that area agencies on gauge aging will collect data to -- the area agencies on aging will collect data to determine what are the services needed for older adults, that maybe our Needs Assessment or other things.  We'll talk about that a little bit more when we talk about baseline data near the end of this presentation.  But then also, B the effectiveness of programs, policies and services provided by such agencies on aging in assisting such individuals as are targeted by the plan.  So, numerous places in the State Plan guidance as well as in the Older Americans Act, there is a requirement for measuring the extent to which the funds expended under the Older Americans Act, the programs funded by the policies by, the initiatives funded by the Older Americans Act are meeting the goals as defined under the Older Americans Act, and then further specified under your State Plans.  Next slide.  So before we get into defining outputs and outcomes, are there any particular questions about what I've covered so far?  What the State Plan guidance says about measuring outcomes, measuring the so-what, the results?  I'll look in the chat for a moment.  I don't see anything yet.  So we'll move on to talking about outputs versus outcomes.  And with your SPR your stat plan report data -- State Program Report data you're reporting on outputs, you're reporting on the level of activity that you have under your State Plans.  Not all of your activities of course, but the ones that are specifically required under the SPR for Older Americans Act funding.  But those are outputs those are counts of the expenditures, counts of the numbers of people served by program type, and counts of the numbers of units of service, there are other things in the State Program Reports of course, but those are some of the main things, and those really are what we think of as outputs.  Those are measures of the level of activity that you have.  Outcomes on the other hand are what we sometimes call the so-what.  So you served people meals, transportation, caregiver respite services.  You've gone to visit long-term care facilities as part of the ombudsman program, things like that.  Are people better?  Do they know something differently?  Are they aware of something?  Do they perceive things differently?  Those are short-term outcomes.  Are their behavior's different?  Are they eating more healthy foods and are they working with more preventative care solutions?  Those are changes in behavior which we think of as intermediate-term outcomes, and then, are they better able to stay in their homes?  Are they better able to live where they want to live with the people that they want to live with?  Is their health better or maintained?  Those are really the longer-term outcomes.  And I do see a question in the chat, do all goals have to have measurable outcomes?  What if there is no data to support a goal?  That is an excellent question.  And so, yes, our goals should have outcomes, we should have an idea of what we want to measure, but, there may not be existing data.  It may not be realistic or feasible for us to collect those data.  And that's something that you can discuss with your regional administrator as well as within your state to determine which outcomes you are going to measure.  But for every goal, there should be some level of outcome.  And so, what I have on the screen now is something that comes from and the source is listed below, in most cases, on these slides, the source of the information is listed on the slide.  So if you want to refer to the original document you can do so.  And this particular slide which has a Logic Model shown, but has nice definitions for outputs and outcomes comes from a tool kit that ACL put together, and it is on the ACL website.  There's also a link on one of the last slides in this presentation to all of the toolkits that ACL has recently put out related to evidence building, evaluation, and performance measurement.  And a link to this toolkit is there as well.  And so, as you think about what problem you're planning to solve, or address, what need you want to meet, which is on the left-hand side of this Logic Model, you can then think about the inputs that you have, the outputs and activities that you have, and then on the far right-hand side of the slide, you have those outcomes and those definitions that I just reviewed.  Next slide.  The State Plan guidance also has language that defines short-term, intermediate-term, and long-term outcomes as I've described.  And it says, again, that short-term outcomes may include improvements in knowledge.  So nutrition education, you can imagine nutrition education, the people that take advantage of that service, the people that receive that service, they improve their knowledge about nutrition, that's a short-term outcome.  An intermediate-term outcome is typically a change in behavior or process.  And so you can imagine somebody receives nutrition education, they've increased their knowledge about nutrition.  Then you would hope that they could change their behaviors and eat more healthful foods.  That would be the intermediate-term outcome.  And then the long-term outcome is ultimately, you would hope that they would be healthier.  They would have better control maybe over dietary-based illnesses such as diabetes or high blood pressure.  That I may be better at controlling those things because of the knowledge and the behavior changes that they had based on the nutrition education program.  That's just one example but all of those services could have a short-term, leading to better mediate term, leading to long-term outcome.  And the time period related to short-term intermediate and long-term outcomes is going to differ based on the program itself.  Some programs you'll see knowledge challenges very quickly and behavioral changes may be relatively straightforward so you will see those relatively quickly.  And they may have a very quick result in terms of health, ability to stay in the community, things like that.  Other programs, it may be a longer-term knowledge gain or perception or awareness gain.  It may take longer for those behavioral changes.  So it really does depend on the specific nature of the need that's being addressed, and the program and service that's being delivered as to what exactly those time frames would be to realize the short-term, intermediate-term and long-term outcomes.  But one important thing to keep in mind is the relationship between the three.  Short-term outcomes are what you expect to see first and immediately, most immediately.  If you see those, then you may see the intermediate-term, and then the long-term.  So it's the order of the things that's also very important.  Next slide.  And going back to explain a little bit why we do this, we can always say, well, we offer various different services to help older adults to live independently in the community, to improve quality of life, to improve health, ability, function, those kinds of things.  But if we don't measure it, then it really is just hope.  We hope, with the theory, that if we offer these services to individuals, communities, family members, they're going to be better.  But without measuring it, it really is just hope.  And hope is not a strategy in and of itself.  Next slide.  As we think about how we come up with what outcomes we're going to measure, and this goes a little bit back to the question that was in the chat about all goals having measurable outcomes, the "if'' really is the goal.  And so, you can think about it if this helps you to think about it as an if-then statement.  So if the Older Americans Act was developed to address a lack of community social services for older persons, which it was, then examples of outcomes which are changes in condition, would be increased services that meet the needs of older persons.  So we would need to measure what those needs are.  That can come through our Needs Assessment, or the information that we have that justifies why your State Plan is going to focus on the areas that it's going to focus on.  Then how do we know we met those needs and how do we know we met them?  You have your now and then your then.  By answering those two questions in your "then" statement, that helps you off what data you may need, and what your outcomes are that you're looking for.  The direction that Alice was asking the Cheshire cat for.  Next slide.  There are two broad, broad categories of outcomes that are most relevant to you as you think about measuring the outcomes of your State Plan.  And they're not always objects to you.  The individual-level which is in yellow on the right-hand side, that's something that we think of a lot.  That's something that we're asked for a lot in terms of how many people did we serve?  How many people may have reduced their emergency room visits?  How many people may have been able to stay in the community?  How many people did we transition from a nursing facility or other long-term care facility back into the community?  And we talk about those things a lot.  So we have the individual-level outcomes and the statements on the slide are from some of the topic areas in the State Plan, but for individual-level outcomes, addressing social isolation, reducing social isolation, improving caregiving, reducing malnutrition, and focusing on making changes in our priorities for individuals.  Addressing the needs of formal and informal caregivers and strengthening the direct care workforce which is made up of individuals and helping people who live in institutions to return to living in the community.  Those would all -- can all be defined as individual-level outcomes, because they're focused on individual people in your state and territory.  And you may have examples of outcome measures related to health of older adults, the health of caregivers for example, also the health of various other people that you may be serving but primarily focused on the older adults, their well being, and their ability, the ability to stay in the community, the ability to live independently those kind of things.  The other categories of outcomes that you should consider measuring are the organizational outcomes, because the State Plan guidance and the cop call areas that are listed under the various categories, also talk a lot about improving the service network, improving service delivery.  So for example, better integrating health and social service delivery systems.  Ultimately we want that to affect individuals better service, better service integration should help to provide better services, quicker services, more efficient services for example to individuals, but it's also important to measure that progress towards that organizational outcome of better integrating health and social services.  Increased capacity for long-term care efforts improving coordination of community-based and long-term care services.  These are organizational outcomes.  Outcomes for the system itself, and so, those are also things that it's important to measure because the Older Americans Act focuses on it, the State Plan guidance also focuses on it.  And we want to have credit for all of the various different things that you all are doing.  You are making some organizational change which is not easy, which is not short-term, which is not cheap in many cases, and so, measuring that organization change is important, as well as measuring of course the individual change.  And so examples of organizational change type measures would be measuring staffing, do you have the right staff?  Do you have the right number of staff?  Do staff have the needed capacity?  Efficiency of services, capacity in terms of number of service slots.  Do you have enough services for the people who need them?  And service quality, those could be some outcomes from your system change operations, your system change initiatives.  Next slide.  So to try to make this more concrete, there are a few real-world examples.  We're going to go over two examples now that come from State Plans that I was able to review.  And one of them, and I won't identify any states in these examples, you may recognize yourself, but I've taken out all of the state names or the specific names.  One is a quote that was in one of the State Plans, a previous State Plan, and it says "from housing to transportation, we have streamlined and improved the existing program to make sure they reach people of all ages and abilities in a more effective way." That is a good statement but it can be made measurable.  In deciding this kind of a statement, there should also be the data that support this statement, or that elaborate on this statement.  So one of -- so some ideas of things that you can do in your State Plans as you draft your State Plans to make them more outcome-focused, more quantifiable is in this particular example, defining streamlined and improving.  Because it's hard to measure the debt to which you streamline your -- it's hard to measure the extent to which you streamline or improve your services or programs if you don't have a concrete definition of streamline or improvement.  To streamline means that people who want services have one comprehensive assessment instead of going to three or four different places and being assessed three or four different times?  Or does it mean reducing the amount of time from an assessment to receiving services?  What does streamline mean in defining that in a way that is measurable?  And improve, so improving existing programs, does that mean improving the amount of service that's available, the quality of service, the match between services available and the needs of the communities?  Defining these things, streamlined and improved in ways that are measurable is really important.  It's also important to set targets.  And so, if you're talking about reducing the time to serve as maybe how you're going to streamline your services, if it takes a month to receive services now, you can set a target.  Are you going to reduce that by one day?  Or are you going to reduce that by two weeks for example?  It gives you something to aim toward to really say yes, we have been successful.  You can measure the status now and over time.  So as you measure the nature of the problem and you think about these are the needs that we have, these are the problems that we're going to address, these are the challenges that we're going to overcome with our State Plan over the State Plan period, a measure that now, and continually measure that throughout the State Plan period so that you can look at it and you pick the time frame, do you measure it monthly, quarterly, semiannually, annually, that will depend on the nature of what you're measuring, how quickly and what your data sources are.  You know those kinds of things.  But measure the status now, and measure it periodically over time.  And that's an easy way to show those outcomes.  Because if the problem is reduced over time, if the need is met increasingly over time, that's proof that you are having at least some of the outcomes, some of the results that you want to have.  So compare those values over time.  And report quantitative data.  So in addition to these qualitative type statements which we have in the example here, then put those data with those statements.  So you could say from housing to transportation, we've streamlined improved existing programs to make sure they reach people of all ages and abilities in a more effective way, specifically for housing, we now serve 30% more people 20% more quickly.  I made up those particular numbers.  But that would be a much stronger outcome focused statement to accompany this first statement that we have here.  Next slide.

The next real-world example and when you have the slides, I realize it's very small on the screen right now.  When you have the slides you'll have all of the words.  But it talks about a husband who has dementia and he's been falling a lot.  The specialist helped the family to define their concerns and goals.  Their concerns were related to how things were, accessibility difficulties, a need for respite.  The program, the AAA helped the family to identify both formal and informal caregiving resources, made referral to the Alzheimer's disease caregiver service initiative, and at the end of the family report that the Alzheimer's program was providing temporary in-home and respite care technology, safety, services and education.  If there's an ability to follow up with the family, it would also be good to know not just that they are receiving services, but are those services meeting their needs?  So as long as you are already following up with the family, ask them the degree to which their needs have been met.  And that is more outcome focused.  Because right now, following up to determine that they received the services is more of an output.  Yes, they received the services, these are the activities that we are funding.  But the outcome is then, is the respite care helping to relieve the caregiver?  Is the technology and safety service helping to improve the lives of the family, the caregiver, and the older adult, the husband in this case as defined in the beginning of the comment?  And then, it goes on to say per the family's request, the program is working with the family to explore possible veterans in-home respite and explore Medicaid long-term care.  Additionally the family chose to have a referral made to the Department of Aging who assisted with referrals related to proofing assistance and with a ramp.  That is a good description of what has happened.  But it really doesn't tell us, did these services, did this interaction help the family?  Did it keep the husband, in this case, at home and this the community?  Did it offer relief to a caregiver?  You know, what were the outcomes?  So moving the statement, which is good, and it is good to have the qualitative information towards an outcome focus is really important.  Next slide.  So some ways that we can improve this statement or add to this statement from a performance perspective is by recognizing that there are problems identified but no baseline or outcomes.  And so, there are problems with respite is needed.  How much respite is needed?  How much respite is given?  And then to what degree did that respite meet the need that the caregiver had?  Did it help to improve the caregiver's health?  Reduce the caregiver's stress?  Allow the caregiver to give care longer than they otherwise would have?  Those are some of the outcomes that can be measured.  A need was related to falls.  But there wasn't data provided about the severity of the falls, the number of falls, and any changes in falls after the service was provided.  The same for housing and accessibility.  We don't know if the services met the needs.  So, the problems are identified but there are no baselines as to what degree of problem is it?  And what are the outcomes of the services related to those problems?  Activities were identified.  In-home respite care technology and safety services, education, and referrals for the roof and assistance with the ramp.  With education in particular, it's unclear which activity education addresses.  Was education related to respite and caregiver services?  Was it related to accessibility?  It's really not clear where education fits in in this issue.  So that could be made clearer.  And again, measuring the outcomes, did the respite help to improve the caregiver's ability to give care?  Did the safety services reduce the number and severity of falls?  Those would be some nice additions to this kind of a statement to make it more of an outcome-related statement.  Next slide.  Another question that is asked a lot is, okay, we have individual stories, which are again, very important.  They really bring a human face to the services that we provide.  And that is a very good thing.  So we should continue with the individual community level, service provider level stories.  But make them more measurable, make them more quantifiable.  But that doesn't necessarily help you at a state or territory level because, each individual story you can't report that as part of your outcomes, you know, really what has the state achieved?  It hasn't achieved a million individual stories.  You need to aggregate that information into a state or territory level.  This is what we've been able to achieve.  And how can you do that?  How can you move from the individual story to a state-level statement of what we've been able to achieve really at scale, if you will?  And so, one thing you can do is link the pieces of the individual example.  So respite care, safety and security, the housing services as an example to your current or planned data collections.  And so, in theory, and I know for the State Program Report, you are collecting some data in your other services.  But you may be collecting, should be collecting data for your evidence-based falls prevention program if that's one of the services that was provided to this individual and his family.  But link parts of your individual example to your broader data collections.  Categorize them based on the categories that you're already collecting data on, or already reporting data on.  Add context from your existing data.  So, in this case, for the second example, how many people do you serve who have housing issues?  Accessibility difficulties or need for respite?  And so you know if you have large percentages of the people you serve that have these things you can provide that contextual data, then say here's an illustrative example of that.  Because the illustrative examples are interesting.  But when they're put in the context of, they can be that much more meaningful.  You can also ask how many formal and informal caregiving resources you provide.  Because that was brought up in the second example.  And how many referrals are made to caregivers, to Alzheimer's disease caregiver service initiative, and look at the outcomes related to that, of the referrals that are made to that?  How many people then receive the services?  How many people then report that those services help them to stay in the community longer, reduce crises that may happen, reduce the need for emergency room visits, and other various different things that you may already be collecting?  And build your individual examples around planned or existing data collections.  Sometimes the examples that we use are the ones that people gave us that are most poignant.  And that's one way to select your examples, but it's hard to tie data to those examples because they may be so unique.  We may not collect data on them.  And so, trying to craft examples illustrative examples based on what we do collect data on, then tying those together can be more powerful in some cases.  And so, think about the categories of data that you already collect.  Next slide.  Another thing to do is to map the categories from your examples to the topic areas in the State Plan guidance.  So the State Plan guidance, the revised guidance has a number of topic areas that talk about these are the areas that you should be focused on as part of your -- under your State Plan.  Explain how you're going to focus on these, what you're going to do in these areas, and what outcomes you're looking for.  So if you have a series of examples or you have existing data, mapping them back to the categories in the State Plan can help you then to aggregate those data so you don't have one million disparate examples anymore, you now have these examples and they're grouped under those topical areas in the State Plan and that you can add those up.  All the people that were served related to this topic area, that topic area, oh are the other topic areas, all of the people who reported that the service helped them related to these different topic airs.  So that can be a way to categorize your examples.  And your different individual program data, and other things like that.  So for example, expanding access to HCBS topic area that's one of the topic areas in the State Plan.  It talks about securing the opportunity for older individuals to receive managed in-home community-based long-term care services, and the services related to housing, the services related to respite likely, the services related to the roof and the ramp would be in-home services.  So you could combine those under this category with other services that also fit under this category to then say, for our state or territory, we serve X number of people, they indicated that these services were helpful to them instead of having again a million different examples.  This topic area also talks about incorporating aging network services with HCBS funded by other entities such as Medicaid, and that second example did specifically mention linking between the services from under the Older Americans Act with Medicaid.  And so that could count if you will, towards that category and be added to other examples and count towards that category to come up with a state-wide or territory-wide number.  Next slide.  Another way to think about how services can be measured and reported, and this slide on the left-hand side is part of one of the topic areas related that's in the guidance.  And in the green box, highlighted it says preventing detecting, assessing intervening and/or investigating elder abuse, neglect and financial exploitation.  So if that's an area that you're working in, or given that that is an area that you're working in I should probably say, you can think about it this way.  This is another way to think about how do you get to outcomes. And how do you then define them and measure them from there?  So your activity under here could be one of your activities providing training and resources to staff to better prevent abuse, neglect and exploitation.  A relevant short-term outcome, these would be in the organizational category, not the individual category which we had been focused on is increased theft, knowledge or skills.  Short-term outcome, increase knowledge or skills.  Intermediate-term outcome, improving staff intervention in suspected abuse, neglect or exploitation.  Then the long-term outcome would be individuals experiencing fewer issues and improved well being.  So you're improving the organization to ultimately improve the lives of individuals.  Next slide.  A few places that you can look for outcomes, and I realize that this slide also is very small, but the link is at the bottom.  So that you can get all of the Logic Models that were developed related to the long-term care ombudsman program.  You can lack at all of the outcomes there.  So I just wanted to show this to you so you're familiar with what they look like.  But it has a number of -- a lot of information about the activities, the outputs, and the different levels of outcomes at the facility level, the individual level, the state level, and at the federal level for the long-term care ombudsman program and the link is on the bottom of the slide so you can get to the report that has these different Logic Models in there, you can pull the relevant outcomes for yourself, for your program.  And so I show this to you to point you to that resource.  But looking at the Logic Model and other program documentation, you can usually find a relevant outcome or set of outcomes that you want to measure towards.  So you don't have to invent them yourselves or create them from scratch for this particular example of training, which we just -- which I just talked about, that would fit under the education and outreach activity, which is the leftmost orange arrow for the activity.  And the relevant outcomes would be the facilities line related to improved prevention, provision of resources and improved quality of life of individuals.  You can improve all of your education and training activities under these outcomes to compile a statewide program-wide statistic related to training of any type that's relevant to the prevention of abuse, neglect and exploitation.  Next slide.  This is the Logic Model for the Title VI program and the link to the report that it comes from is on the bottom of the slide.  And similarly, it has a number of outcomes that you may want to consider measuring that tie back both to the topic areas and State Planned guidance, but also are relevant to the Title VI program itself specifically.  And so, you see a range of different outcomes related to the different service types, nutrition services, caregiver services, and other types of services in the rows that go across.  When you get to the outcomes you see outcomes such as increases in social contacts as a result of the program, increases in empowerment as a result of the program.  Increases in cultural and community integration, and a range of different outcomes that you may already be measuring or that you want to measure and then report quantitative information through your State Plan process.  Another source, next slide, another source of outcomes can be from the national survey of Older Americans Act participants, NSOAAP, and the link here is to the survey tools themselves.  And for example, there are questions if you go to the next slide from the national survey of Older Americans Act participants, this is related to the case management survey specifically, if you go to the next slide, please.  And it talks about as a result of the services you received, are you able to live independently?  That's an outcome living independently?  Do you feel more secure?  Are you better able to care for yourself?  And since you started receiving the services, do you have a better idea of how to get additional help that you would need?  Those would be some outcomes you may want to measure, those are proven survey questions that you could use if you do follow up with your consumers that you serve.  Any questions about what we've covered so far?  We're going to move onto baseline data and why it's important in just a minute.  But I don't know if there are any questions?  I'll look at the chat for a moment.  Seeing none move to the next slide, please.  There we go.  Baseline data and why they're important.  So with baseline data, that really is the information that you collect, you should collect as you're developing your State Plan or as you're implementing a new initiative, a new program, changing a program, something like that.  And so, that's the state, where are you now?  Before you implement the change to the program, before you implement the program or initiative, at the start of your State Plan period, where are you now?  What is your baseline?  It frames the level of problem that you're going to address, it explains the level of problem, frame it numerically in terms of how many people for example does it affect?  What level of effect does it have?  How much does this program or how much is this problem costing the state or costing insurance or costing individuals?  And it could be financial costs, but it could also be costs in terms of health or other resources and capacities.  You compare then the baseline of what you have at the start to where you are throughout your program, and where you are at the end of your State Plan period.  And then that shows you how far you have come.  Hopefully you've come a long way, and the problem is smaller than it was at the beginning of this period.  But you need to know that.  It can help you to set realistic targets in terms of how much change you can really realistically expect of the State Plan period.  Generally we cannot eradicate an entire problem, maybe over the State Plan period, but we can reduce it and we can reduce it by realistic amounts if we have a good sense of our baseline data.  Next slide.  It's always important to have that comparative information.  The baseline data can be our comparative information.  Looking at this cartoon, the little cat comes in and says, I drew this picture, the father dad cat says, that's great, I give you three stars.  Well, is three stars good or bad?  If it's three stars out of three stars it's excellent.  If it's three stars out of five stars it's mediocre, or if it's three stars out of the billion stars in the universe it's pretty bad.  In order to know what that three means, we need to have that comparative information, that baseline information, or that community sort of census population information.  So you need that comparative information.  Much of which can come from your baseline data.  And the baseline data can also help us to set targets.  Next slide.  So as we think about what changes we want to effect over the course of the State Plan period or the course of an intervention or an initiative, you can do it proportionately.  If you have an equity goal and there are topic areas in the State Plan guidance related to equity, you could look at what are the proportions of different demographic groups in your communities, in your states, your territories, your counties, and you can have a goal of proportionally serving members of those groups.  And so you can set your targets proportionately.  So maybe now one of your State Plan goals is to increase the number of people living in rural communities or the people living under the poverty line, the people from racial and ethnic minority groups that are receiving services to reach that proportionality, their rates in the population, in your states and your communities.  So that's one way to set targets.  Then measure, are we meeting those targets?  Are we getting closer to that proportionality? Another way to set targets is a constant increase from a previous year.  And so, you could say that you've reduced time to service by, you're going to reduce it by 1%.  So, if it takes a month, you're going to reduce it by 30 days, or you're going to reduce it by 1% each year for each six months, you know.  So you can do that consistent increase from the previous reporting period.  Or you can change.  Change is based on prior year averages.  So you can look at how many people you served, how many people rated the services as successful and helping them to stay in the community?  And you can look at those averages and say okay, we're going to keep up these averages or we're going to increase these averages by a certain amount.  So those are some ways to set targets that you can then put in your State Plan, you can measure towards those targets, that helps to keep you focused on outcomes.  Next slide?  It's also important to think about what might be your data sources for this information?  And so, you do have a lot of administrative data.  And, I can't comment on whether this administrative data are suitable if they're good quality.  I think I'm sure that everybody has administrative data challenges.  I think in the federal government we definitely do.  So, that's something to look at the quality and usability of your administrative data.  But you do have administrative data some of which I'm sure is very, very good and very suitable for this type of measurement.  But looking at the number of people that receive the services which are the outputs, then the number of people that may be reduced emergency room visits, if you have administrative data on emergency room visits.  If you have other administrative data on how people are faring as a result of receiving the services, and when you focus on the organizational outcomes related to improving service quality, improving staffing, improving capacity, it's likely that you have very good administrative data and a lot of administrative data about those program operations that you can use.  You can also do surveys or follow-up telephone calls with some or all of the consumers that you serve using the NSOAAP or National Survey of Older Americans Act Participants Survey questions or other survey questions that you may have.  You can use census data to help you with that proportionality to know what the rates of your communities are related to racial and news minority group members, people living in rural areas, people living at or below the poverty line, other things like that.  And I've mentioned several times the Needs Assessments that you're conducting to determine what needs you're going to focus on in your State Plan period. You can use those same data collections and keep collecting them over the course of the State Plan period and look for changes over time. Are the needs being met?  Are the needs fewer than they were before?  Are they changing in some way?  You can report those as outcomes depending on what exact data you have.  So, the next, the last slide, or second to last slide if you can skip to the next slide, is really what is next.  So I've presented a lot of different information, and all of you are in different places in terms of working on these types of things with your State Plan as you write your State Plan.  But a couple of ideas and use what works well for you if you have methods for doing this that are successful and that are working to really have quantitative outcome data, then those are good approaches.  But if you're not sure where to start, or how to go about this, one way to do it is would be to have your State Plan for the changes you're expecting to see based on your efforts.  So if you have more of a qualitative, many more of a textural State Plan that talks about what you intend to achieve, pull out the things that are the outcomes, the improved streamlining, improving services from that first example earlier in the session.  Or looking at the example of the individual who needed the housing services and things like that.  Pull out what might be considered outcomes from the text.  Then look at your Needs Assessment and identify the needs that you will need.  And so based on the Needs Assessment and based on what you say you're going to do in your State Plan, you're looking at the previous State Plan, look and see what are the areas that you're going to focus the most on?  Pull those out and those are the ones that you're going to want to develop the outcome measures for.  And so that again goes back to the question that Lisa asked at the beginning about do all goals have to have measurable outcomes.  The things that -- if you need to start somewhere, you can't have outcomes from this point for all of your goals, start with the ones that are addressing the most priority questions, the most critical needs, maybe where you're spending the most money.  And so start there, ultimately you can have goals and outcomes for all of your goals.  You can start with the ones that are highest priority.  Map each program or funded activity to the changes or needs that you've identified in the Needs Assessment.  Because you shouldn't be doing activities that do not map directly to meeting those needs.  You need to link those things up and either address, identify another need that that activity does need, or maybe drop that activity if it's not meeting an identified need that you've identified through the Needs Assessment and that you are highlighting in your State Plan.

Map the results of steps 1-3 to the topic areas that are in the State Plan guidance.  Identify the existing data and data gaps that you have related to measuring those needs and the changes in those needs over time and the outcomes that you've identified.  And use the data that you have while you start to work on filling in some of those data gaps.  Maybe you have some follow-up to ask people about knowledge.  So you have short-term outcome data.  It would be helpful over time to be able to develop some ways to collect the long-term outcome data to go back to those people over time.  Have they changed their behaviors?  Their processes?  Their procedures?  And are their conditions better?  Are they better able to stay in the community as a result of your programs?  So it is an iterative process.  Start where you are.  Start where you can be, focus on your priorities.  And build this evaluation, this outcome, this evidence-building capacity.  And then the last step is to plan how you will fill your data gaps, especially related to the intermediate and long-term outcomes.  Because that's typically the hard it's to get for us.  And so just build on it.  Start where you are and take steps in these directions.  It doesn't have to be 100% perfect right away.  But show progress, and show a plan for getting to where you need to go to measure these outcomes.  So, next slide.  Have some tools and resources that can be helpful to you in thinking about these things.  That easy link to toolkits that ACL has put out and that are on the website.  So close out, data can show us where we've been, where we're going and help us to stay on track of it is very important in addition to being required by the State Plan, it's very important for running programs successfully and making sure that we're meeting the needs that our communities have and that individuals have by measuring the degree to which we're actually doing that.  So with that we have a couple of minutes left, I don't know if there are any questions, any comments that people would like to make before we reach the top of the hour?  I put those in the chat that the PowerPoint will be posted to the State Plan guidance page within two weeks along with the transcript and that the State Plan guidance, there's a link to the State Plan guidance in the chat as well.  And within the Power Point there are links to the Logic Models that I showed and the reports that described those Logic Models and the outcomes that are related to them.  There are links to the national survey.  So if you wanted to use some of those questions in some surveys that you may be administering you can access all of those things.  Seeing no questions, I want to thank everyone for your attendance and your participation in today's webinar.  And I think we are -- I hope everyone has a good rest of the day.