Evaluability Assessment of the 

Title VI Grant Program

Final Report 

September 30, 2015
CONTENTS

ACKNOWLEDGMENTS i

BACKGROUND 1

Older Americans Act and Administration on Aging 1

Title VI Program 2

Title VI Program Grantees 2

Evaluability Assessment of the Title VI Program 2

ENHANCING READINESS FOR EVALUATION—TITLE VI PROGRAM RECOMMENDATIONS 7

Federal Recommendations 7

Tribal Recommendations 14

Other Stakeholder Recommendations 15

TITLE VI EVALUATION DESIGN RECOMMENDATIONS 15

Design Summary 15

Title VI Implementation Study 17

The Title VI Outcomes Study 18

Design Framework and Methodology—A Community-Driven Approach 20

REFERENCES 25

APPENDIX A—EVALUABILITY ASSESSMENT METHODOLOGY A-1

APPENDIX B—LOGIC MODEL SUPPLEMENT B-1

Inputs B-1

Activities B-2

Outputs B-4

Proximal Outcomes B-4

Distal Outcomes B-6
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page i

ACKNOWLEDGMENTS

This report was prepared by ICF Macro, Inc. (an ICF International company) for the 

U.S. Department of Health and Human Services (HHS), Administration for Community Living 

(ACL)/Administration on Aging (AoA) under contract number GS23F9777H with AoA, HHS. 

Leadership and coordination of this publication was provided by Susan Jenkins, Ph.D., Social 

Science Analyst and Contracting Officer’s Representative from ACL. 

ICF would like to acknowledge the invaluable contributions of many individuals from ACL and 

the Title VI Evaluability Assessment (EA) Advisory Group that provided guidance and feedback 

on the Title VI EA and final evaluation recommendations. 

U.S. Administration for Community Living Title VI EA Advisory Group

Cynthia LaCounte

Margaret Graves

Jean Lloyd 

Courtney Hoskins

Bruce Finke

Kristen Robinson

James DelaCruz

Steve Wilson

Colette Browne 

Kathryn Braun

Kay Branch

Mary Weston

Wendell Holt

Laurai Atcitty

Charmaine Mix

Karen Cook 

Mae Gilene Begay

Larry Curley

Terry Duffin

Paula Morin-Carter

Kevin Billiot

Cheryl Frye-Cromwell

Nancy FurlowShelly Zylstra

In addition, we would like to acknowledge and thank the 18 representatives from Title VI grant 

programs who participated in key informant interviews. 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page ii

DISCLAIMER

The views, opinions, and content of this publication are those of the author and do not 

necessarily reflect the views, opinions, or policies of ACL, AoA, or HHS. 

ACCESSIBILITY

If you encounter access issues with this material, please contact Susan Jenkins by telephone at 

202.357.3591, for additional assistance.

PUBLIC DOMAIN NOTICE

All material appearing in this publication is in the public domain and may be reproduced or 

copied without permission from ACL. Citation of the source is appreciated. However, this 

publication may not be reproduced or distributed for a fee without the specific, written 

authorization of ACL, HHS. 

RECOMMENDED CITATION

Administration for Community Living. (2015). Evaluability assessment of the Title VI Grant 

Program. Washington, DC: Author. 

CONTRIBUTING AUTHORS

Robin Davis, PhD1

Gretchen Clarke, MPH1

Ruth Westby, MPH1

Beth Bahe, MPH1

 

1

ICF Macro, Inc.
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 1

BACKGROUND

Older Americans Act and Administration on Aging 

The Older Americans Act (OAA), established by the United States Congress in 1965 in response 

to concerns over the lack of social service programs and protections for older persons, 

promotes the well-being of older Americans by providing a range of community and homebased services to help them live independently (Administration for Community Living [ACL], 

n.d.a, p. 3). Through the OAA legislation:

The Congress hereby finds and declares that, in keeping with the traditional 

American concept of the inherent dignity of the individual in our democratic 

society, the older people of our Nation are entitled to, and it is the joint and 

several duty and responsibility of the governments of the United States, of the 

several States and their political subdivisions, and of Indian tribes to assist our 

older people to secure equal opportunity to the full and free enjoyment of the 

following objectives...

Among these objectives are the right to an adequate income in retirement; the best possible 

physical and mental health available; suitable housing; restorative services for those requiring 

institutional care; the opportunity for employment; and retirement in “health, honor, and 

dignity” (OAA Sec 101: 42 U.S.C. 3001). To achieve this, the OAA authorizes grants to states for 

community planning and social services, research and development projects, and personnel 

training in the field of aging. The Administration on Aging (AoA), housed within the 

U.S. Department of Health and Human Services, is the administrative body that oversees the 

OAA programming. Through the OAA, a large network of service providers, the National Aging 

Network, was established. This broad partnership, comprising 56 State/Territory Units on Aging,

655 Area Agencies on Aging, 241 tribal organizations, two organizations serving Native 

Hawaiians, and nearly 30,000 service providers and volunteers, links federal, state, tribal, and 

local partners in the service of nearly 7 million Elders and their caregivers (Administration for 

Community Living, n.d.b.). 

American Indian and Alaska Native (AI/AN) and Native Hawaiian populations continue to 

experience health and socioeconomic disparities. Compared to other U.S. populations, the 

AI/AN population has the highest rate of disabilities (Goins, Moss, Buchwald, & Guralnik, 2007; 

Centers for Disease Control and Prevention, 2008). Although AI/AN populations also have the 

lowest life expectancy compared to the average for the overall population, the increase in life 

expectancy across all populations over the past few decades means that the proportion of 

AI/AN adults living with a disability is expected to increase. The number of AI/ANs aged 75 and 

older who will need long-term care is expected to double in the next 25 years (Goins et al.,

2007). In addition, the proportion of AI/ANs living in poverty is twice the rate for the overall 

U.S. population (U.S. Census Bureau, 2005). Taken together, these data point to a need for a 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 2

strong health and social service support system; however, a wide disparity in the availability of 

services for Elders in tribal communities remains.

The OAA is divided into sections, or Titles, with each Title corresponding to a component of the 

legislation. In the years since the initial OAA legislation, reauthorizations have amended and 

revised the original language and have expanded programming to include AI/AN as well as 

Native Hawaiians. Title II directs the AoA and establishes the Office for American Indian, Alaska 

Natives and Native Hawaiian Programs; Title III regulates grants for state and community 

programs and includes specific provisions for administration (Part A), supportive services (Part 

B), nutrition services (Part C), disease prevention and health promotion (Part D), and caregiver 

support program (Part E). In 1978, the Older Americans Act (OAA) was amended to include Title 

VI, which established programs for the provision of nutrition and supportive services for Native 

Americans (American Indians, Alaska Natives and Native Hawaiians)

(http://www.aoa.acl.gov/Index.aspx). Title VI has since been expanded to include caregiver 

support services as well. 

Title VI Program

Title VI specifically outlines the care for AI/AN and Native Hawaiian Elders. Under Title VI, 

programs provided to AI/AN and Native Hawaiians are to be “comparable to services provided 

under Title III” (OAA Section 601: 42 U.S.C. 3057a) and include supportive, nutrition, disease 

prevention and health promotion, and caregiver support services. Title VI Part A is the Indian 

Program and serves federally recognized tribes with at least 50 members ages 60 years and 

above; Part B is the Native Hawaiian Program, serving public or nonprofit private organizations 

that serve Native Hawaiians and represent at least 50 individuals ages 60 years and above; Part 

C is the Native American Caregiver Support Program, which serves all programs within Parts A 

and B (AoA, 2008, p. 5). To qualify for Title VI, programs must provide either Part A or B 

(Nutrition Services and Supportive Services) as well as Part C (Caregiver Support Services) (AoA, 

2008, p. 6). 

Title VI Program Grantees

In fiscal year 2014, 263 Title VI grants were awarded to tribes/tribal organizations and one 

organization serving Native Hawaiian Elders for the provision of Nutrition and Supportive 

Services; 231 grants were awarded for the Native American Caregiver Support Program (Title VI 

Part C). Grantees represent over 400 tribes and span the continental U.S., Alaska, and Hawaii. 

Evaluability Assessment of the Title VI Program

In 2014, ICF International (ICF) was contracted by ACL/AoA to conduct the Evaluability 

Assessment (EA) of the Title VI Grant Program. EAs are used as a preevaluation assessment to 

guide and inform investments in evaluation and research. The purpose of the EA of the Title VI 

Grant Program was to systematically examine the Title VI Program’s characteristics, context, 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 3

activities, processes, implementation, evaluation, outcomes, and logic to determine the extent 

to which the theory of how the program is intended to work aligns with the program as it is 

implemented and perceived in the field. In this case, the EA was done to examine the program 

characteristics of Title VI grantees’ nutritional, supportive, and caregiver support services to 

assess the feasibility of, and best approaches for, further evaluation of the Title VI Program. As 

part of the EA, a 20-member stakeholder advisory group, consisting of representatives from the 

National Indian Council on Aging, the National Resources Center on Native American Aging, as 

well as Title VI grantees, was convened to review and provide feedback on each step of the EA 

process, including: (1) program description; (2) selection of a smaller group of intended users 

for direct in-depth interviews; (3) EA design, instruments, implementation, and findings; (4) 

program logic model; and (5) recommendations for a rigorous evaluation, questions, design,

and feasibility of implementation. Potential respondents were selected to participate in indepth interviews to gain a more nuanced understanding of the Title VI service areas (Nutrition 

Services, Supportive Services, and Caregiver Support Services). Distinct interview guides were 

created for each service area, and six grantees were selected within each interview group, for a 

total of 18 in-depth interviews. Grantees were selected based on geography and size in order to 

gain a richer and more representative perspective of tribes. (An overview of the EA process can 

be found in Appendix A.)

As part of this process, a comprehensive Title VI Logic Model was developed. It includes the 

following information: 

 Title VI Inputs and Resources support the implementation of Title VI grant programs that 

include monetary and other resources (e.g., equipment, personnel, time); 

 Title VI Program Activities comprise work done by program staff, including volunteers, to 

achieve program goals and objectives (e.g., delivering meals, providing transportation, 

educating caregivers);

 Title VI Program Outputs measures the level of activity (e.g., the number of people 

reached or the number of hours of service provided);

 Title VI Proximal Outcomes are direct results for those served (e.g., reductions in stress,

improved nutrition/food security); and

 Title VI Distal Outcomes include fundamental goals of Title VI implementation guided by 

the OAA (e.g., improved health, independence, quality of life). 

A logic model (Exhibit 1) outlines the expected activities and outcomes of Title VI Programming 

and is divided by service area (Nutrition, Supportive Services, Caregiver Support, and Program 

Management). As a whole, Title VI Program inputs include a mix of federal, state, tribal, local, 

and regional resources, such as funding, training, and technical assistance. Nutritional services 

include the provision of congregate meals and home-delivered meals, nutrition screening, and 

education to Elders. Supportive services include the provision of information and assistance, 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 4

transportation, chore services, and other services that support Elder welfare. Caregiver support 

services assist caregivers of Elders or grandparents caring for grandchildren. Outputs include 

process measures, such as numbers reached and number of meals provided. Proximal 

outcomes vary by service area but include increased linkages to needed services, increases in 

social connectedness, and stasis in well-being. Distal outcomes include increased 

empowerment, cultural and community integration, and maintenance of Elder independence 

and ability to remain in the home. Appendix B serves as a compendium to the logic model and 

provides greater detail as well as specific examples of the types of inputs and activities

referenced by the broader language found within the logic model. 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 5

Exhibit 1. Title VI Program Logic Model
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 6

Because many tribal communities do not conceptualize their programming in the linear frame 

laid out by the logic model, a medicine wheel was developed to orient the proximal and distal 

outcomes across the traditional quadrants of indigenous practice: Spiritual, Mental, Emotional, 

and Physical. Each quadrant is housed within the context of community, family, and 

intergenerational connection—highlighting the importance of each to the spiritual, mental, 

emotional, and physical well-being of indigenous communities and Elders. Exhibit 2 presents 

the Title VI Program Medicine Wheel.

Exhibit 2. Title VI Program Medicine Wheel

The program and design recommendations that follow are also informed by a review of existing 

and potential data sources and benefits and limitations of primary and secondary data 

collection. In addition, both program evaluability recommendations and design

recommendations are guided by literature and best practices in public health evaluations in and 

with tribal populations. The recommendations in this report are guided by a participatory 

evaluation framework, grounded in a community-based participatory evaluation model, which 

is essential to conducting culturally relevant evaluation in tribal communities.
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 7

ENHANCING READINESS FOR EVALUATION—TITLE VI 

PROGRAM RECOMMENDATIONS

Conducting an evaluation of the Title VI grant program requires a significant commitment and 

investment for federal, tribal, and local Title VI stakeholders. It requires an agreement about

program goals and priorites as well as the value and necessity of demonstrating program 

outcomes to program funders. It requires a shared understanding about how data can be used 

to inform, improve, and guide programs so that limited resources can be prioritized in their 

application. To ensure successful implementation of future evaluation efforts and to nuture the 

readiness for evaluation, ICF has identified multiple priorities and recommendations to support 

and enhance the evaluation of the Title VI grant program. The following presents specific 

recommendations for federal, tribal, and local stakeholders for future Title VI evaluation. 

Federal Recommendations

Federal program stakeholders and funders have a critical role to play in ensuring evaluation 

readiness of Title VI grant programs. AoA and ACL have a shared role in demonstrating the 

value of evaluation to Title VI Program grantees as well as serving as an advocate for the 

importance of evaluation to other stakeholders who will be essential to evaluation 

implementation (e.g., training and technical assistance providers, service providers, federal 

staff, Title III stakeholders). The following presents key recommendations to enhance the 

readiness of Title VI grant programs for evaluation. 

 Engagement of Stakeholders. Provide 

opportunities to involve Title VI stakeholders 

in formulating the utility and need of 

program evaluation—demonstrating 

performance, progress, and outcomes of the 

program. Interest, knowledge, and 

awareness of evaluation will diminish if 

grantees and other Title VI stakeholders are 

not regularly engaged about its importance.

The first step for engaging tribal stakeholders is education on the proposed process (e.g., 

roles and responsibilities, time frame, burden), with a focus on the concrete and specific 

value of the evaluation data to ACL and tribal grantees. It is important to explain how 

data from the evaluation will be shared with participating sites and that participating 

sites will receive support in utilizing their own data (either by itself or in combination 

with other data available to the tribe). Title VI regional staff, who (in some cases) have 

Participatory Evaluation

Partnership approach to 

evaluation where stakeholders 

are active participants in all 

phases of implementation.

Zukoski & Luluquisen (2002)
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 8

more direct contact with tribes than do federal staff, are an important stakeholder 

group.

 Participatory Evaluation. Community-based participatory evaluation should be 

considered for future Title VI evaluation. Subsequent engagement of stakeholders

(e.g., tribal leadership, line staff and volunteers, administrative staff, and Elders) for 

evaluation can include explanation of this model. Early engagement can also be used to 

gain buy-in from future evaluation participants, ensure their future participation, and get 

their input into framing evaluation questions. Participatory evaluation can identify data 

collection processes that would be appropriate and yield meaningful information for Title 

VI Program administrators, project directors, and stakeholders, and guiding 

interpretation of evaluation data gathered about the Title VI Program. It will be critical to 

have communication, especially early communication, come through a “trusted friend.”

 Framing the Evaluation. AoA/ACL should give serious consideration to how the 

evaluation is framed, named, and promoted to get buy-in from Title VI grantees. Certain 

terminology may be met with concern over whether it will impact tribal Title VI funding, 

distrust about what will be done in an evaluation and how information will be used, and 

active or passive resistance to participation in an evaluation. The terms evaluation, 

research, and surveillance can have negative connotations in tribal communities. Thus, 

avoiding these terms and framing the evaluation with culturally relevant and positive 

terms can gain 

buy-in from community members and frame the project in a meaningful way to tribal 

communities. In addition, it is important to frame the evaluation within the value that 

stakeholders see in the program. For example, stakeholders have indicated that they 

value the program for its ability to promote access to and development of home and 

community-based services and for having the “pulse” of tribal Elders. ICF recommends an 

evaluation that is framed around storytelling. Initial components of the evaluation should 

provide opportunities for Title VI grantees to tell their stories to share the context, 

meaning, challenges, and implementation of their programs (Implementation Study). To 

date, grantees have not had the opportunity to fully tell their story about the need for 

and outcomes of the Title VI Program in tribal communities. In addition, the evaluation 

should include information about promising and/or evidence-based programs that 

AoA/ACL can highlight and tribes could adopt to improve their own outcomes. 

 Performance Reporting. The Title VI Program Performance Report (Title VI PPR) could be 

revised to include additional performance measures/program outcomes to ensure 

consistent grantee reporting on program specific outputs and outcomes. The Title VI PPR 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 9

is a potential tool for gathering critical information about program activities and outputs 

that can be used as a data source for the implementation study (presented below) and 

can be measured and reported consistently across all grantees. For Supportive Services, 

additional indicators can be added to gather the number of referrals for supportive 

services during the reporting period. In addition, numbers reached through group 

activities (such as tribal community events and/or celebrations) could be added to the 

individual outreach collected. Finally, additional details on the type of activities should be 

considered so that grantees are cataloging or reporting on more details about the 

specific nature of program activities within each program. Exhibit 3 provides an overview 

of additional data elements that could be reported on by Title VI Program grantees. 

Exhibit 3. Recommendations for Additional Title VI PPR Data Elements

Program PPR Enhancements

Nutrition Services

(see Title VI Logic Model 

Activities)

▪ Screening assessments (number and type of assessments used)

▪ Menu planning activities

▪ Food safety activities

▪ Provision of related supportive services

Caregiver Support Services

There is an “other” category currently listed on the PPR. Additional services 

could be specifically listed out to include: 

▪ Palliative care services (consider alternative name)

▪ Grandparent support services

▪ Dementia awareness

Supportive Services

Distinction between direct supportive services provided and coordination of 

supportive services should be gathered on the Title VI PPR including: 

▪ Cultural/intergenerational activities

▪ Socialization and recreation activities

Management Activities

Consider adding a management section on the PPR for management-related 

activities. Activities to be considered include:

▪ Training and management

▪ Program evaluation activities

▪ Performance monitoring

▪ Program recruitment and outreach

▪ Quality assurance

▪ Coordination between Title VI Program and other tribal programs

▪ Policy and protocol development

▪ Advocacy activities

If expanded, training and technical assistance (TTA) on collection and reporting can be 

provided to grantees with the revised tool. Tracking tools and training on their use can 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 10

be shared with grantees to reduce the burden on increased performance indicator 

requirements. 

Finally, AoA/ACL should consider expanding the frequency of reports. Currently, grantees 

submit once annually for program activities conducted from April 1 through March 31. 

Although increasing the frequency of reporting may modestly increase reporting burden, 

regular submissions may ultimately enhance the accuracy and completeness of the data 

reported. By requiring more frequent reporting, mistakes in data can be corrected more 

easily because less time will have passed between date of service and the reporting 

period.

 Needs Assessment. Assessment data are an invaluable source of information that can be 

used to guide and prioritize program activities as well as serve as baseline data for 

demonstrating program outcomes. AoA/ACCL should consider expanding assessment 

requirements to year three of grants. This addition would allow for a comparison 

between the initial assessment conducted at the time of grant application and the 

assessment data collected at the end of the grant period. AoA/ACL should gather 

feedback from grantees on how these assessments should be conducted (e.g., with 

support from the National Resource Center on Native American Aging as with the pregrant assessments, by the tribe or tribal organization with TTA support from an external 

evaluator). 

Currently, grantees can either use the needs assessment (and TTA support) provided by 

the National Resource Center on Native American Aging (Identifying Our Needs: A Survey 

of Elders) or use their own needs assessment. AoA/ACL should consider requiring 

grantees to use the same measures (either use the National Resource Center on Native 

American Aging needs assessment survey or ask needs assessments questions the same 

way, using their own survey protocols). Because grantees are not required to use the 

same instrument at baseline (i.e., pre-grant), there is limited comparability across 

measures of these data. Thus, consistent data are only available with a subset 

(approximately two thirds) of grantees who use the National Resource Center on Native 

American Aging survey. In addition, grantees (who do not elect to use the Identifying Our 

Needs: A Survey of Elders) may not collect data on relevant program outcomes at 

baseline (pre-grant) (i.e., they might not collect the information at all as opposed to 

asking it differently than the Identifying Our Needs Survey). Exhibit 4 presents data 

elements that should be gathered consistently from all grantees at baseline (and year 

three of grants, if the recommendation above is adopted). 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 11

Exhibit 4. Required Indicators for Title VI Program Needs Assessment 

Program Outcomes2Required Measures for Baseline and Follow-up 

Nutrition Services

▪ Screening (Qs 7–11)

‒ About how long ago has it been since you last visited a doctor/health care 

provider for a routine checkup?

‒ How long has it been since you had your blood stool tested for blood using 

a home kit?

‒ (FOR WOMEN ONLY) How long has it been since you had your last 

mammogram?

‒ (FOR WOMEN ONLY) How long has it been since you had your last pap 

smear?

‒ (FOR MEN ONLY) How long has it been since you had your last prostatespecific antigen test (PSA), a blood test used to check MEN for prostate 

cancer?

▪ Social Support/Housing (Q 45)

‒ How often do you get out and socialize? (Attend church/religious meetings, 

clubs/organizations you belong to or cultural activities/traditional 

ceremonies)?

▪ Weight and Nutrition (Qs 41–43)

‒ How much do you weigh today?

‒ Are you presently trying to lose or gain weight?

‒ Over the past 30 days, what vigorous exercises did you do?

Caregiver Support Services

▪ Social Support/Housing (Qs 51, 52)

‒ Do you take care of grandchildren?

‒ Are you the primary caregiver of grandchildren?

Supportive Services

▪ General Health Status (Q 4)

‒ How many falls, if any, have you had in the past year?

▪ Activities of daily living (Qs 5, 6)

‒ Because of a health or physical problem that lasted more than 3 months, 

did you have any difficulty...

‒ Because of a health or physical problem that lasted longer than 3 months, 

did you have any difficulty...

▪ Weight and Nutrition (Qs 43)

‒ Over the past 30 days, what vigorous exercises did you do?

▪ Social Functioning (Q 59)

‒ Are you now using, or if at some point you become unable to meet your 

own needs, would you be willing to use the following services?

▪ Health Care Access (Qs 30–33)

‒ What type of health care coverage do you have?

‒ Do you have one person you think of as your personal doctor or health care 

provider?

‒ When you are sick or need professional advice about your health, to which 

of the following places do you usually go?

‒ Have any of the following kept you from medical care in the past 12 

months?

Title VI Program Distal ▪ General health status (Qs 1, 2, 3)

 

2

See Title VI Program Logic Model.
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 12

Program Outcomes2Required Measures for Baseline and Follow-up 

Outcomes ‒ Would you say your health in general is excellent, very good, good, fair, or 

poor?

‒ During the last 12 months, how many different times did you stay in the 

hospital overnight or longer?

‒ Has a doctor ever told you that you had any of the following diseases?

▪ Social functioning (Qs 53–58)

‒ During the past month, how much of the time were you a happy person?

‒ How much of the time, during the past, month have you felt calm and 

peaceful?

‒ How much of the time, during the past month, have you been a very 

nervous person?

‒ How much of the time, during the past month, have you felt downhearted 

and blue?

‒ How much of the time, during the past month, have you felt so down in the 

dumps that nothing could cheer you up?

‒ We would like to ask the extent to which you feel you can personally 

influence things by what you do or say. How much influence do you feel 

over your life in general?

▪ Screening (Qs 7–11)

‒ About how long ago has it been since you last visited a doctor/health care 

provider for a routine check-up?

‒ How long has it been since you had your blood stool tested for blood using 

a home kit?

‒ (FOR WOMEN ONLY) How long has it been since you had your last 

mammogram?

‒ (FOR WOMEN ONLY) How long has it been since you had your last pap 

smear?

‒ (FOR MEN ONLY) How long has it been since you had your last PSA, 

prostate-specific antigen test, a blood test used to check MEN for prostate 

cancer?

▪ Social and support/housing (Qs 44, 46–50)

‒ Do you participate in cultural practices that include traditional food, music, 

and customs?

‒ How long have you lived at your present address?

‒ What type of housing do you presently have?

‒ Are you living with family members, nonfamily members, or alone?

‒ How many (including yourself) live in your household?

‒ Do you have a family member who provides care for you?

Demographic and Other Data 

Elements

▪ Demographics (Qs 60–70)

‒ Gender

‒ Age

‒ Current marital status

‒ What is your personal annual income?

‒ Have you been employed full or part time during the past 12 months?

‒ What is the highest grade or year of school you completed?

‒ What zip code and county/borough do you currently reside?

‒ Are you American Indian, Alaska Native, Native Hawaiian, mixed race, or 

other?

‒ Do you reside on/in a reservation, trust land, Alaska village, or Hawaiian 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 13

Program Outcomes2Required Measures for Baseline and Follow-up 

homestead?

‒ Are you an enrolled member of a federally recognized tribe?

‒ Have you ever served on active duty in the U.S. Armed Forces, Military 

Reserves, or National Guard?

Finally, if the needs assessment survey is to be used as a proxy for baseline and follow-up 

on Title VI outcomes, additional measures not currently gathered through the 

assessment but tied to explicit program goals should be considered. Exhibit 5 provides an 

overview of additional items to be considered for expansion of the assessment. 

Exhibit 5. Additional Indicators for the Title VI Program Needs Assessment3

Program Outcomes4Required Measures for Baseline and Follow-up 

Nutrition Services5

▪ Number of social contacts within the last 60 days

▪ Explicit social connectedness and social isolation questions

▪ Food security question

▪ Nutritional intake 

▪ Overall mental health question

Supportive Services ▪ Linkage (or availability) of needed services (beyond willingness to use services 

in Q 59)

Title VI Program Distal 

Outcomes

▪ General health status (Qs 1, 2, 3)

‒ Would you say your health in general is excellent, very good, good, good, 

fair, or poor?

‒ During the last 12 months, how many different times did you stay in the 

hospital overnight or longer?

‒ Has a doctor ever told you that you had any of the following diseases?

▪ Screening (Qs 7–11)

‒ About how long ago has it been since you last visited a doctor/health care 

provider for a routine check-up?

‒ How long has it been since you had your blood stool tested for blood using 

a home kit?

‒ (FOR WOMEN ONLY) How long has it been since you had your last 

mammogram?

‒ (FOR WOMEN ONLY) How long has it been since you had your last pap 

smear?

‒ (FOR MEN ONLY) How long has it been since you had your last PSA, 

prostate-specific antigen test, a blood test used to check MEN for prostate 

cancer?

 

3

Caregiver Support Service outcome measures are not recommended for enhancement of the community needs 

assessment survey, as respondents may not be caregivers. 

4

See Title VI Program Logic Model.

5

Consider using questions on these topics from the National Evaluation of the Title III-C Nutrition Services 

Program.
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 14

 Program Goals and Intended Outcomes. The Title VI Program manual includes allowable 

activities and services, and references the purposes of Nutrition Services. However, 

program specific outcomes (beyond Nutrition Services) should be articulated and shared 

with grantees. For example, the Title VI Logic Model has clear program outputs and 

outcomes associated with the three primary Title VI Program areas as well as 

management activities. The OAA references Title VI Program objectives with no explicit 

link to program outputs and outcomes. The link between the Act and the specific 

program outcomes anticipated through Title VI should be included as part of the AoA 

Title VI Manual for grantees. 

 Evaluation TTA. Evaluation TTA, distinct from programmatic TTA, should be supported to 

ensure consistent cross-community data collection capacity, including the development 

of tracking tools by Title VI grantees to observe and record program outputs. The 

following provides an overview of specific evaluation TTA that should be considered to 

prepare grantees for participation in a national evaluation of the Title VI Program. 

– Regulatory TTA (institutional review board [IRB], data collector confidentiality 

agreements, data use agreements, Memoranda of Understanding [MOUs], tribal 

resolutions)

– Data collection TTA (primary data collection and storage, respondent recruitment,

protection of personally identifiable information, tracking and monitoring tools,

performance monitoring)

– Data use to refine program activities (using data to inform and refine program 

activities)

 Title VI Resource Manual. The resource manual can be expanded to include resources 

for local evaluation and tools for gathering required evaluation data. Current 

requirements for any evaluation are vague. 

Tribal Recommendations

 Participatory Evaluation. Title VI grantees can be engaged and recruited to help develop

evaluation questions beyond the draft questions presented in the program

recommendations, above. AoA/ACL should engage tribal grantees to participate in an 

advisory group to identify priorities for building capacity and readiness for evaluation 

participation. Community members should be engaged regularly to provide input on the 

evaluation design recommendations proposed by the external evaluator, the TTA needs 

associated with the evaluation design, and the data collection and procedures involved in 

the evaluation design. Tribal communities should maintain involvement to provide 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 15

context and interpretation to evaluation findings and to participate in dissemination 

activities. 

 Engagement of Title VI Program Partners. Title VI grantees can engage Indian Health 

Services (IHS) and other service providers to secure MOU to access and use existing data 

and resources to inform and guide program prioritiIHes and refine program efforts. 

 Tribal Colleges and Other Local Evaluation Resources. Title VI grantees, with support 

from the TTA provider, can engage tribal colleges to identify potential and existing 

opportunities to demonstrate program outcomes at the local level. Partnerships between 

tribal organizations and tribal colleges or local universities can build capacity of local Title 

VI Program staff to engage in data collection and program evaluation. Tribes, with 

support from the TTA provider, also should identify data resources (e.g., state vital 

statistics, tribal administrative data; IHS/data from Epidemiology Centers) that could be 

used to reduce the need for primary data collection. Tribes should perform an 

environmental scan of surveys being conducted with their Elders to assess burden and 

identify other possible sources of data.

Other Stakeholder Recommendations

 Opportunities for Local Engagement. Program partners (e.g., state health departments, 

public health nursing, local businesses) play a critical role in the success of Title VI 

Programs through program funding, service access, and service provision. ACL should 

engage local program partners to determine their interest and participation in a 

community-based approach to evaluation of Title VI Programs. 

TITLE VI EVALUATION DESIGN RECOMMENDATIONS

In reviewing the Title VI Logic Model and inputs to support Title VI Program implementation, 

two distinct areas for evaluation were identified: (1) a national setting to understand the overall 

implementation and outcomes of Title VI grant programs and (2) Title VI tribal-specific settings 

and program contexts for Title VI grant programs to implement grant activities (e.g., by tribal 

size and number of Elders reached). The following presents an overview of the evaluation 

design recommendations to evaluate Title VI grant programs. 

Design Summary

A multimethod evaluation design is recommended to understand the implementation and 

outcomes of the Title VI Program. The recommended design includes two interconnected 

studies to assess Title VI implementation and its outcomes at the national and tribal levels. The 

study design includes an implementation and outcomes study as well as special analyses to 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 16

Implementation of Title VI Study

What are the context and operation of 

Title VI Programs at the national level? 

How do Title VI Programs operate at the 

tribal level? 

assess the reach and outcomes of program activities of grantees that have sole-sourced versus 

blended funding efforts. For both studies, we propose a combination of primary and secondary 

(extant) data collection and analysis to determine Title VI implementation and outcomes. 

The recommended evaluation design is guided by three overarching evaluation questions: 

 What is the context of the Title VI Programs at the national and tribal levels? How are the 

Title VI Programs implemented at the national and tribal levels? 

 What are the outcomes and impact of Title VI Programs at the national level and by tribe 

or tribal groups? 

 What are the Title VI Program outcomes for programs that rely solely or primarily on 

Title VI funds compared to cost-shared programs that receive a significant proportion of 

their resources from other programs/agencies?

Although a comprehensive evaluation would address each type of question, the evaluation 

design and specific data collection and analysis requirements are markedly different. The first 

question would assess the nature and scope of program inputs, activities, and outputs across 

both the national and regional Title VI Program levels. The second question would examine the 

results of these inputs, activities, and outputs in terms of direct change in program participants’ 

and staff behaviors and knowledge as well as program’s capacity for expansion. In addition, the 

question would examine the impact of these changes on critical Title VI outcomes—for 

example, reduced risk of nursing home placement, loss of independent living, and reduction in 

emergency department/acute care visits. Thus, addressing the first question is a prerequisite 

for any attempt to address the second.

The Implementation Study includes grantee 

document reviews, a primary survey data 

collection with Title VI grant management, 

and the use of secondary data to 

understand program implementation and 

program context over time including Title VI 

PPR. The Implementation Study also 

includes direct data collection with Title VI 

Program staff to understand the implementation of each Title VI Program component, the 

challenges and barriers to program implementation, and how program sources are shared or 

sole sourced for service implementation. The following provides an overview of the 

Implementation Study components. 

 Monitoring the Implementation of Title VI through Secondary Data—provides, at minimal 

cost, aggregate counts of nutrition, caregiver support, and supportive services activities 

as well as those reached through the Title VI Program through Title VI PPR data.
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 17

 Primary survey data collection through key informant interviews (KIIs)—assesses the 

implementation of Title VI Programs at the national level and regional level. This 

component gathers critical information on the nature and context of grant programs and 

management structures as well as the use of sole or blended funding to accomplish 

program-specific activities.

 Program implementation inventory—gathers primary implementation data on the 

services and activities of the Title VI Programs at the grantee level by documenting all 

Title VI resources, inputs, and activities by the Title VI Program component. 

The Outcome Study primarily relies on extant data and analysis to assess the outcomes of Title 

VI implementation. This strategy would rely on three sources of data—needs assessment data6

at baseline and year three of grant implementation and acute care/ED visit data from IHS

regions. While the use of extant data provides less precise measures of Title VI distal and 

proximal outcomes than those that could be obtained through primary data collection of 

program participants, it has the advantage of providing a lower cost and lower burden methods 

for getting data from the sizable samples necessary to assess change in program outcomes by 

variation in implementation (learned and coded through primary and secondary data in the 

Implementation Study) and Title VI grantee.

Title VI Implementation Study

Understanding implementation of Title VI (including program specific activities) is fundamental 

to assessing whether its intended outcomes (and objectives) are being achieved. Title VI

implementation involves coordination of a range of resources and inputs, reliance on 

volunteers and donations, management strategies, and program specific practices—none of 

which are systematically and/or broadly captured in any extant data. So, while some aspects of 

the Title VI contexts and process can be gathered through extant program data, additional 

implementation data are necessary to fully understand program implementation. The following 

describes a particular instrument for such an assessment. A strategy to monitor program 

outputs, based on secondary data analysis, is also presented. Finally, we suggest a way to 

gather information on barriers and facilitators to program implementation as well as program 

sourcing mechanisms (sole or blended sourcing of program activities). 

Title VI Program Inventory7

We recommend all Title VI grantees (program administrators) complete a biannual inventory to 

document program supports and resources, program activities by program type, and 

populations reached through these efforts. The inventory would result in a comprehensive data 

 

6

If adopted, the needs assessment would need to be expanded to gather additional proximal and outcome data. 

7

If enhancement of the Title VI PPR is adopted, the Title VI Program Inventory would not be necessary. Expanded 

Title VI PPR data requirements and bi-annual submission would replace this recommendation. 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 18

set of all Title VI Program activities within Nutrition Services, Caregiver Support Services, and 

Supportive Services as well as program management (what were the activities conducted, when 

were they conducted, who was reached through the activities). We recommend a program 

tracking tool (in Excel) that would enable program administrators to compile information 

regularly (e.g., monthly) for submission on a biannual basis. The inventory would also include 

assessments to document management systems and processes necessary to ensure functioning 

of the Title VI Program, including gathering information about collaboration and program 

outreach as well as oversight and fiscal monitoring. 

Title VI PPR Data Extraction

We recommend analyzing all Title VI PPR data (from currently funded grantees) to demonstrate 

program specific implementation and Elders reached through these efforts. Information can be 

used to demonstrate program reach over the course of grant programs and be used to 

understand the outcomes of Title VI efforts. Data include numbers of congregate meals, homedelivered meals, those who received meals, other nutrition services, supportive service units 

provided, number and type of staff, and caregiver support services. In Exhibit 3, we have 

provided specific recommendations on ways to expand Title VI PPR reporting requirements to 

gather detailed information on the inputs and resources of programs as well as the specific 

activities and outputs of Title VI Programs. If adopted, the inventory would not be necessary, as 

these PPR data would yield the same information. 

Title VI Tribal KIIs

Storytelling is an integral component of tribal culture and community. To understand the 

context, challenges, successes, and components of Title VI Programs, tribal communities need 

to tell their unique stories. We recommend conducting primary data collection (qualitative)

with a purposeful sample of project administrators across Title VI grantee programs. The 

samples will be based on the number of interviews to be reached before saturation of themes 

as well as the following strata: size of tribal community, tribal location (rural, frontier, village),

number of Elders projected to be reached through the tribal program, and type of tribal 

grantee (lower 48, Hawaiian, Alaskan). Interviews will be used to document challenges and 

barriers to program funding and program implementation, to gather detailed information on 

the sourcing of activities (sole or blended), and to understand critical services that are 

contingent on blended funding sources. 

The Title VI Outcomes Study

In order to assess the impact of Title VI implementation, information on both proximal and 

distal program outcomes is required as well as on natural program variation, such as program 

specific activities. Furthermore, since implementation of Title VI is not the result of random 

assignment, additional information on potentially confounding variables is indispensable, 

including baseline information from before Title VI implementation occurred. Multiple design 

options were considered to understand the outcomes and impact of Title VI, including primary 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 19

data collection on outcomes as well as the use of secondary or extant data. After thorough 

review, an outcome evaluation using secondary data as outcomes is recommended using a 

nonexperimental, pre-post design. 

Using Secondary Data to Assess Outcomes

Primary data collection on pertinent Title VI Program outcomes would involve census, 

screenings, or surveys of a large number of Title VI eligible participants and follow up with 

those identified for prolonged periods of time to assess reach before and after program 

activities. Secondary data sources can offer a relatively inexpensive option to obtain this large 

longitudinal sample. Two specific types of secondary data sources to assess outcomes are 

presented.8The proposed design would require extensive engagement and agreements with 

tribal communities to secure 

de-identified individual level data from community needs assessments. While the data would 

be de-identified prior to analysis, it would be necessary to include participant IDs that could be 

tracked at the community level so that pre and post-program data could be linked.

Needs Assessment

Identifying Our Needs: A Survey of Elders VI provides a unique opportunity to assess changes in 

tribal population outcomes. The needs assessment requirement of the grant serves as a 

necessary tool to identify health disparities within tribes to prioritize and refine program plans. 

In addition, these data can be used as baseline measures of a range of proximal and distal 

program outcomes. As administered, the survey yields robust estimates of health risk and 

protective conditions within communities and can be used to assess change in these conditions 

over time. In addition, implementation data (gathered through the program inventory) can be 

incorporated to understand how funding source(s) may moderate these program outcomes. 

Data elements include:

 General health status

 Tribe (or tribal region)9

 Vision, hearing, and dental

 Memory and disability

 Tobacco and alcohol use

 Weight and nutrition

 Social supports and connections

 Social functioning

 Elder demographics

 Health care access

 Activities of daily living

 History and frequency of screenings

 

8

Secondary data for post-Title VI Program efforts assumes adoption of the program recommendation to expand 

requirements for needs assessment to third year of Title VI grant.

9 MOU would be required among the Title VI tribal organization, AoA, and the evaluation contractor to ensure 

tribal ownership of data and no tribal or individual level data were published. 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 20

A community survey, such as the Identifying Our Needs Survey provides an opportunity to 

gather baseline and follow-up (year three) data on Title VI Program efforts and can be used to 

understand the relationship of key program outcomes with variation in program 

implementation. In addition, data analysis can incorporate variation in strata used to gather 

implementation data to understand the program contexts of significant program outcomes 

(e.g., type of tribal grantee, size of tribal grantee, location of tribal grantee). 

Indian Health Service Acute Care and Emergency Care

In addition to needs assessment baseline and year three data, we recommend exploring 

availability and access of IHS acute care (emergency) data within Title VI Regions. If available, 

these data may serve as both baseline and follow-up data on distal program outcomes (e.g., 

reduction in emergency or acute care). As with needs assessment data, implementation data 

can be used to frame and categorize Title VI Regions by nature and reach of program activities 

on these outcomes. 

Caregiver Support KIIs

KIIs should be used to understand the context, barriers, and implementation of caregiver 

support services from caregivers’ perspectives. Similar to the KII approach above, a sample of 

Title VI grantees should be selected purposefully to ensure a range of tribal programs (using the 

same strata above: type of grantee, location, size, Elder population). Within each grantee, a 

small number of caregivers (2–4) should be recruited for KIIs to better understand 

implementation of program activities and outcomes of these activities. Interviews should 

gather information (qualitative and quantitative) around the following program outcomes: 

While the KIIs are intended to be implemented face-to-face, the recruitment and interviewing

of participants should be determined with the evaluator in coordination with the tribal 

community. 

Design Framework and Methodology—A Community-Driven Approach

As described in the recommendations, a community-based participatory evaluation should be 

considered for the Title VI Program. AoA/ACL should work with the existing advisory group to 

identify strategies that engage tribal community members to tell their stories about the Title VI 

 Physical activity of the caregiver

 Linkage to needed services

 Number and nature of social contacts

 Social connectedness

 Social isolation

 Overall well-being and mental health 

of caregivers and care recipients

 Referrals and access to care and 

services

 Physical safety
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 21

Program, the context of program activities, challenges and needs to Title VI Programs, and Title 

VI successes. The design framework assumes ongoing participation and engagement of tribal 

community members and active involvement in all phases of the evaluation (evaluation design, 

data collection measure development and piloting of primary data collection, participant 

recruitment and engagement strategy development, resource development for evaluation 

challenges experienced by tribal communities, interpretation of Title VI data and analysis, and 

disseminating evaluation findings). 

Methodology of Primary Data Collection

Evaluation readiness to conduct Title VI Program evaluation activities will vary across Title VI 

tribal communities as will preferences as to how they want their community members to be 

surveyed, questioned, and engaged. In addition, tribal grantees will have a range of regulatory 

requirements (e.g., tribal resolutions, MOUs, IRBs) that will dictate methodology of 

participation. As a result, the design should be focused on the core data elements (and 

domains) that should be collected as part of the evaluation. However, the mode of data 

collection should be flexible to ensure participation of the largest number of tribal grantees. It 

should also be expected that tribal grantees may not have human resources to support primary 

data collection, and the evaluation team should have supplemental TTA for direct data 

collection, if possible. Exhibit 6 presents an overview of the Title VI Program indicators to be 

gathered, the affiliated study components, and the recommended sources of data. 

Exhibit 6. Title VI Evaluation Data Sources and Logic Model Indicators

Implementation Study

Study Component Logic Model Variables

Title VI Program Inventory

▪ Nutrition Service Activities

▪ Caregiver Support Service Activities

▪ Supportive Services

▪ Program Management Activities

Title VI PPR Data Extraction10

▪ Nutrition Service Outputs

▪ Caregiver Support Outputs

▪ Supportive Services Outputs

▪ Program Management Outputs

▪ Nutrition Service Activities

▪ Caregiver Support Service Activities

▪ Supportive Services

▪ Program Management Activities

Title VI Tribal KII Storytelling

▪ Title VI Program Inputs

▪ Nutrition Service Activities

▪ Caregiver Support Service Activities

▪ Supportive Services

▪ Program Management Activities

 

10 Nutrition Services, Caregiver Support Services, and Supportive Service activities would be available if Title VI PPR 

revisions are adopted as described in program recommendations. 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 22

Outcome Study

Study Component Logic Model Variables

Title VI Needs Assessment (Pre- and 

Postdesign)

▪ Nutrition Service Outcomes

▪ Caregiver Support Service Outcomes

▪ Supportive Outcomes 

▪ Program Management Activities

▪ Title VI Program Distal Outcomes

IHS Urgent Care Data ▪ Emergency Department and Acute Care Visits

Title VI Caregiver Support KIIs

▪ Experience with caregiver support activities and services

▪ Challenges and barriers to accessing caregiver supports

▪ Nutritional intake 

▪ Linkage to needed services

▪ Ability to provide care

▪ Caregiver well-being

▪ Caregiver physical, emotional, and financial stress

▪ Quality of life

▪ Level of independence

▪ Level of community integration

▪ Social connectedness

Timeline

The overall timeline for an evaluation of tribal programs is 36 months. This timeline allows for 

initial engagement and recruitment of tribal communities into the evaluation process; regular 

engagement of tribal communities across the lifespan to ensure community-based 

participation, including recruitment and maintenance of a tribal advisory group; storytelling 

with tribal communities as part of an Implementation Evaluation; and sufficient ongoing 

Outcome Evaluation data collection to document progress and program outcomes across their 

three-year grant cycle. The process includes components that ensure ongoing feedback of 

evaluation findings to tribal grantees and TTA on the use of data to refine and prioritize 

program activities. The timeline also includes early feedback from tribal communities and the 

piloting of specific instruments, OMB approval/clearance, and program-wide evaluation of Title 

VI Programs. Exhibit 7 provides an overview of activities to be conducted within and across 

each year. 
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 23

Exhibit 7. Title VI Three-Year Evaluation Design

In addition to the general timeline of evaluation activities, the evaluation should be flexible 

relative to the specific implementation of evaluation activities. Some communities may be 

prepared to begin surveys (KIIs) of caregivers early on in the process, whereas other 

communities may take longer to secure local, tribal approvals. The evaluation must have 

flexibility and not assume all communities participate in evaluation activities at the exact same 

time. Although overall evaluation milestones will need to be achieved each year, the evaluation 

plan must be flexible enough to accommodate grantees who experience serious challenges to 

participation in a national evaluation of the Title VI Program. 

Evaluation Framework

The community-based participatory evaluation assumes full participation by the community in 

telling their Title VI stories—implementation and outcomes. However, the nature of that 

participation (direct data collection by tribal members) will vary across Title VI grantees. The 

evaluation framework is participatory in nature but must be flexible to support the variation in 

direct participation. For this evaluation, we propose that Title VI communities (as part of their 

participation) work with an evaluation team to determine the model of evaluation to be 

implemented, which will, in turn, guide the evaluation TTA required of each participating tribal 

grantee. Exhibit 8 presents a continuum of tribal engagement in the evaluation and the 

associated evaluation TTA necessary to support the levels of participation. 

Evaluation Year 1

 Community engagement and recruitment

 Advisory group membership invitation

 Community researcher or liaison identified

 Tribal resolutions

 Tribal regulatory and IRB approvals

 Data collection TTA

 Pilot of instruments and OMB clearance

Evaluation Year 2

 Tribal storytelling of Title VI 

program implementation

 Tribal documentation of 

program activities through 

Title VI Program Inventory

 Ongoing data collection 

and analysis

 Data automation and 

infographic generation

 Special analysis and reports

Evaluation Year 3

 Ongoing data collection and 

analysis

 Data automation and 

infographic generation

 Community and advisory 

panel review of findings

 Sharing and dissemination of 

Title VI findings
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 24

Exhibit 8. Title VI Evaluation Framework Models

Evaluation Framework Description and TTA Needs TTA Needs

Tribal Framework

▪ Tribal grantee leads the evaluation 

activities for all primary data collection

▪ Tribal grantees receive training and 

support on the overall evaluation 

design

▪ Evaluation TTA is available for specific 

questions and challenges that arise 

through the evaluation

▪ Financial support/stipends are 

required to support tribal members for 

specific activities and deliverables 

▪ Regulatory TTA (IRB and regulatory 

support)

▪ Data collection TTA (implementation 

and procedures trainings)

▪ Using data to inform and guide 

programs TTA

▪ Ongoing point of contact for data 

collection questions and challenges

▪ Evaluation provides ongoing 

monitoring, data quality assurance, 

and quality control procedures

Tribal and External Mixed 

Evaluation Framework

▪ Tribal grantee works with external 

evaluator to implement data collection 

▪ Tribal grantee designates community 

researcher or community liaison to 

facilitate data collection 

▪ Community researcher is supported 

through a stipend from the evaluation 

contractor, with payment upon 

specific deliverables (e.g., human 

subjects training, implementation of 

data collection training, data sharing,

and storage training)

▪ Both external evaluator and 

community researcher may be 

involved in primary data collection 

activities

▪ Evaluation TTA is available for specific 

questions and challenges that arise 

through the evaluation

▪ Regulatory TTA (IRB and regulatory 

support)

▪ Data collection TTA (implementation 

and procedures trainings)

▪ Using data to inform and guide 

programs TTA

▪ Ongoing point of contact for data 

collection questions and challenges

▪ Evaluation provides ongoing 

monitoring, data quality assurance, 

and quality control procedures

External Evaluation 

Framework

▪ External evaluator leads the evaluation 

activities for all primary data collection

▪ Tribal grantees designate tribal liaison 

to work with external evaluator to 

obtain local-level approvals (IRB, tribal 

resolutions, data use agreements)

▪ Evaluation TTA is available to tribal 

community to obtain tribal approvals

▪ Using data to inform and guide 

programs TTA

▪ Ongoing point of contact for data 

collection questions and challenges

▪ Evaluation provides ongoing 

monitoring, data quality assurance, 

and quality control procedures
EVALUABILITY ASSESSMENT OF THE TITLE VI GRANT PROGRAM FINAL REPORT Page 25

REFERENCES

Administration for Community Living. (n.d.a.) Administration on Aging (AoA): Older Americans 

Act. Retrieved September 16, 2015, from http://www.aoa.gov/AoA_programs/OAA 

Administration for Community Living. (n.d.b.) Office for American Indian, Alaska Natives and 

Native Hawaiian Programs. In Administration on Aging (AoA). Retrieved September 16, 2015, 

from http://www.aoa.acl.gov/Index.aspx

Administration on Aging. (n.d.) Title VI Resource Manual. Washington, DC: Author. 

Centers for Disease Control and Prevention. (2008). Racial/ethnic disparities in self-rated health 

status among adults with and without disabilities — United States, 2004–2006. Morbidity and 

Mortality Weekly Report, 57, 1069–1073.

Goins, R. T., Moss, M., Buchwald, D., & Guralnik, J. M. (2007). Disability among older American 

Indians and Alaska Natives: An analysis of the 2000 Census public use microdata sample. The 

Gerontologist, 47, 690–696.

Zukoski, A., & Luluquisen, M. (2002). Participatory evaluation. What is it? Why do it? What are 

the challenges? Community-Based Public Health Policy & Practice, 5(April), 1–6. 
APPENDIX A

Evaluability Assessment Methodology
APPENDIX A: Evaluability Assessment Methodology Page A-1

APPENDIX A—EVALUABILITY ASSESSMENT METHODOLOGY

ICF used a six-step methodology to conduct the Evaluability Assessment (EA) of the Title VI 

Grant Program. The process included: 

 Identifying intended users of evaluation findings;

 Including a subset of users for in-depth interviews on program reality and context, 

measurability of program goals, and an extensive review of the Title VI Program; 

 Development of a Title VI Program logic model; 

 Continued involvement of the stakeholder advisory group to assess program reality and 

to provide review and feedback on the logic model and potential evaluation designs; 

 Analysis and synthesis of program information, including primary and extant data 

collection; and

 Development of evaluation design options for the Title VI Program. 

Exhibit B-1 presents an overview of the six-step process. 

Exhibit A-1. EA of Title VI Grant Program Six-Step Process

EA Step Description

Identify and Engage Title VI 

Program Grantees 

▪ Define intended users of evaluation (e.g., Title VI Program grantees)

▪ Develop a list of potential users that represents a breadth of perspectives with 

input from Administration for Community Living/Administration on Aging 

(ACL/AoA) and advisory group

▪ Assure multiple levels and range of perspectives for feedback

▪ Conduct in-depth interviews across the three Title VI service areas (Nutrition 

Services, Caregiver Support Services, Supportive Services)

Review Title VI Program

Materials

Conduct extensive review of Title VI Program documents provided by ACL/AoA 

and gather information related to: 

▪ Program history, goals, and objectives

▪ Program organizational structures and operations processes 

(e.g., management, staff qualifications and years of involvement, number of 

dedicated staff, agency affiliation, use of volunteers)

▪ Program implementation resources and needs

▪ Program services and implementation activities

▪ Barriers to and facilitators of program implementation

▪ Program evaluations to date

▪ Existing electronic and paper-based information systems and relevant data 

sources

Develop Title VI Program Logic 

Model 

▪ Develop a draft logic model reflecting three Title VI service areas

▪ Clarify the rationale and the expectations of the program 

▪ Delineate the logical relationships among Title VI Program goals, activities, and 

outputs

▪ Expand and revise logic model based on interviews and input from ACL/AoA 
APPENDIX A: Evaluability Assessment Methodology Page A-2

EA Step Description

and advisory group

Ensure Involvement of Key 

Stakeholders through EA 

Advisory Group

▪ Solicit input from advisory group members on each EA activity 

▪ Develop materials and facilitate meetings with advisory group members to 

gather input on the EA process and recommendations

Analyze of Title VI Program

Information

▪ Develop a spreadsheet to catalog information gathered through the document 

review, advisory group feedback on logic models, and information gathered 

through in-depth interviews

▪ Synthesize information across Title VI service areas and evaluation domains,

including key populations, goals and objectives, program components and 

services, evaluation capacity and evaluation needs, and management and 

implementation of Title VI Programs

▪ Prepare memo summarizing findings 

Develop Evaluation 

Recommendations

▪ Develop overarching Title VI Program recommendations

▪ Develop Title VI Program evaluation recommendations

▪ Identify training and technical assistance needs for Title VI grantees to fully 

participate in an evaluation

▪ Draft report summarizing program and evaluation recommendations

▪ Conduct briefings to obtain input on report and evaluation designs

▪ Prepare final report
APPENDIX B

Logic Model Supplement 
APPENDIX B: Logic Model Supplement Page B-1

APPENDIX B—LOGIC MODEL SUPPLEMENT

Due to space restrictions, the logic model is not able to list every programmatic input and 

activity. Instead, broad categories are given. The list below expands on those categories and 

includes examples of the types of inputs and activities referenced by the broader category. This 

is not meant to be a definitive list but rather indicative of the types of inputs and activities 

found in that category. These overarching examples were selected based on feedback from the 

expert advisory group and interviewees.

INPUTS

Inputs are the human, financial, organization, and community resources that support Title VI.

 Funding includes federal, state, and local funding sources.

 Program staff includes individuals paid with Title VI funds, such as directors, grant 

managers, drivers, chefs, resource specialists, and caregivers.

 Volunteers include unpaid individuals performing activities related to Title VI Programs,

such as drivers for meal delivery and caregivers.

 Donations include donations of food, equipment, vehicles, and supplies, etc.

 Tribal resources & community goodwill include advisory boards, committees, coalitions, 

and Elders.

 Tribal buy-in includes Elders’ need for services and desire for socialization and tribe’s 

interest in serving Elders.

 Tribal & local business supports include tribal financing; tribal provision of ancillary 

services; administrative support from tribal agencies; and support from casinos, school 

districts, etc.

 Federal & state partners include Administration on Aging, IHS, Bureau of Indian Affairs, 

Veterans Affairs, Centers for Medicare & Medicaid Services, Centers for Disease Control 

and Prevention, Department of Justice, United States Department of Agriculture, 

American Cancer Society, AmeriCorps, Office of Minority Health, state health 

departments, public health nursing, etc.

 National foundations include private philanthropic and grant-making foundations.

 University of North Dakota Needs Assessment & Technical Assistance/National 

Resource Center on Native American Aging includes surveys, assessments, and guidelines 
APPENDIX B: Logic Model Supplement Page B-2

for administration from the University of North Dakota School of Medicine & Health 

Sciences, Center for Rural Health, National Resource Center on Native American Aging, 

https://www.nrcnaa.org.

 Title VI Program & training and technical assistance includes guidelines, webinars, 

toolkits, manuals, and the AOA Web site, http://olderindians.aoa.gov. 

ACTIVITIES 

Activities are the work of the program and include the processes, tools, and events that 

constitute program implementation.

Nutrition Services

 Screening assessments (optional) include comprehensive nutritional screening 

assessments or other assessments done through health care providers.

 Nutrition education & counseling include presentations, meal preparation

demonstrations, and wellness public service announcements (PSAs).

 Menu planning includes planning healthy lunches according to Title III/Title VI nutritional 

guidelines and the use of nutritionists, dieticians, chefs and/or caterers.

 Food safety includes ServSafe certification, adherence to meal preparation and storage 

procedures, and annual inspections by the state and/or IHS.

 Meal production includes cooking or catering healthy nontraditional and traditional 

foods.

 Meal delivery includes transporting boxed meals, uncooked meals, and prepared meals

to Elders in need of home delivery.

 Socialization includes interactions at congregant meal sites and special meals, 

gatherings, and activities hosted at the meal site.

 Supportive assistance includes person-centered care and referrals to other services.

Caregiver Support Services

 Respite care provides caregivers a rest from caregiving through a temporary care 

provider. It includes specialty respite care, based on a flat level of need or a continuum of 

need.
APPENDIX B: Logic Model Supplement Page B-3

 Palliative care is specialized medical care for those with serious illness, with a focus on 

relief from symptoms. Palliative care support services include partnerships with and 

referrals to other service providers.

 Grandparent care includes the provision of information and assistance for grandparents 

raising grandchildren.

 Dementia awareness includes educational workshops, conferences, and a broad array of 

support, including stress relief and reduction for caregivers.

 Counselling/support groups include information and assistance services, training, 

supplemental services, quality assurance, education, caregiver honoring/conference, 

loan closet, library, and specialty training.

Supportive Services

 Health promotion & wellness includes chronic disease self-management education, fall 

prevention programs, health fairs, health screenings, and exercise programs, etc.

 Education & outreach includes PSAs, health fairs, Facebook posts, and newsletters.

 Home care includes in-home services for frail Elders, homemaker and chore-work 

services.

 Transportation includes transportation to access medical appointments, congregant 

meals, or other approved locations.

 Information & assistance include information about clinical services, veterans’ services, 

utility and energy bill assistance, legal services, food stamps, and more.

 Cultural/intergenerational activities include promotion of cultural activities, such as 

beading, regalia making, language classes, and intergenerational activities.

 Socialization & recreation include promoting tribal games and other activities to increase 

social interaction.

Program Management

 Program evaluation includes UND Needs Assessments, satisfaction surveys, screening 

assessments, and informal discussion.

 Training & management includes subject specific (e.g., Elder abuse awareness) training 

as well as staff trainings (e.g., ServSafe certification, driver monitoring & assessment) and 

volunteer recruitment, training, and management.

 Performance monitoring includes evaluation and reporting requirements.
APPENDIX B: Logic Model Supplement Page B-4

 Program recruitment & outreach includes methods of outreach, including flyers, health 

fairs, and referrals.

 Quality assurance includes data collection, data analysis, and utilization data.

 Coordination between program and others includes coordination between Title VI 

services and other clinical or ancillary services, coordination of multiple grants for service 

provision, and emergency management (e.g., alert and evacuate Elders during a natural 

disaster).

 Policies and procedures include written protocol for administration of Title VI Programs.

 Advocacy includes promoting and organizing centers or services for tribal Elders.

OUTPUTS

Outputs are the direct results of program activities and include types, levels, and numbers 

served/reached. Outputs are recorded through client intake forms, needs assessments, case 

management records, daily logs, and attendance records among others.

 #s reached/served includes unduplicated counts of Elders receiving congregant meals, 

home-delivered meals, and supportive services as well as unduplicated counts of 

caregivers receiving services.

 # of services provided includes total number of meals, transportation services, and 

supportive and caregiver services. 

 # of referrals is the actual count of referrals by type provided through Title VI 

Programming.

 # of activities is the actual count of outreach activities (e.g., health fairs, PSAs) provided 

through Title VI Programming.

PROXIMAL OUTCOMES

Outcomes are the specific changes in knowledge, skills, and behavior as a result of the Title VI 

Program; proximal outcomes are short-term and attainable, within 1–3 years of program 

implementation.

 ↑ in social contacts as a result of Title VI services, means Elders will have increased 

social contacts.
APPENDIX B: Logic Model Supplement Page B-5

 ↑ in social connectedness as a result of Title VI services, means Elders will have 

increased social connectedness.

 ↑ in linkage to needed services as a result of Title VI services, means Elders and 

caregivers will be referred/connected to needed services, including legal, energy 

assistance, health, and respite.

 ↑ in food security as a result of Title VI services, means Elders will have increased food 

security.

  meal and service satisfaction as a result of Title VI services, means Elders will 

maintain their satisfactions with meal and supportive and caregiver services.

  nutritional intake as a result of Title VI nutrition services, means Elders will maintain 

their nutritional intake.

 ↓in social isolation as a result of Title VI services, means Elders and caregivers will 

experience decreased social isolation.

 ↓in mental overload as a result of Title VI caregiver support and supportive services, 

means caregivers will experience lessened mental overload.

 ↑ in ability to provide care as a result of Title VI caregiver support services, means 

caregivers will be better able to provide care.

  caregiver well-being as a result of Title VI caregiver support services, means 

caregivers will maintain their sense of well-being.

  care recipient well-being as a result of Title VI services, means Elders will maintain 

their sense of well-being.

 ↓ in caregiver physical, emotional, and financial stress as a result of Title VI caregiver 

services, means caregivers will experience lessened physical, emotional, and financial 

stress.

 ↑ in physical safety as a result of Title VI supportive services, including Elder abuse 

awareness, means Elders will be safer.

  physical activity as a result of Title VI supportive services, including health promotion 

and wellness activities, means Elders will maintain their physical activity levels.

 ↑in participation in cultural/community events as a result of Title VI services, means 

Elders and caregivers will be able to participate in cultural and community events.
APPENDIX B: Logic Model Supplement Page B-6

 ↑ in referrals and access to care/services as a result of Title VI services, means Elders 

and caregivers will be referred to and have greater access to needed care/services.

 ↑ in staff knowledge and skills as a result of Title VI Program management activities,

including trainings and technical support/assistance, means staff knowledge and skills 

will increase.

 ↑ in implementation of best practices as a result of Title VI Program management 

activities, including program evaluation, policies and procedures, and advocacy, means 

Title VI Programs will adopt current best practices within the field.

 ↑ in meeting meal standards (i.e., DRI) as a result of Title VI Program management 

activities, including quality assurance and policies and procedures, means there will be 

an increase in the number of meal standards met.

 ↑ in partnerships with range of service providers as a result of Title VI Program

management activities, including coordination between programs, means there will be 

an increase in partnerships.

 ↑ in financial management as a result of Title VI Program management activities (e.g., 

quality assurance, and policies and procedures, performance monitoring, training and 

management), means there will be an increase in financial management.

 ↑ in financial monitoring of program activities as a result of Title VI Program

management activities, including quality assurance and policies and procedures, 

performance monitoring and training and management, means there will be an increase 

in financial monitoring and oversight.

DISTAL OUTCOMES

Outcomes are the specific changes in knowledge, skills, and behavior as a result of your 

program; distal outcomes are long-term and attainable within 5 or more years of program 

implementation.

 ↑ in empowerment as a result of Title VI Programming, means Elders and caregivers will 

feel more empowered in their daily lives.

 ↑ in cultural/community integration as a result of Title VI Programming, means those 

served will feel more integrated both culturally and within their community 

  overall health as a result of Title VI Programming, means Elders and caregivers will be 

able to maintain their overall health.
APPENDIX B: Logic Model Supplement Page B-7

  independence as a result of Title VI Programming, means Elders will be able to 

maintain their independence.

 ↓ in risk of nursing home placement as a result of Title VI Programming, means Elders 

will be at decreased risk of out-of-home placement.

 ↑ in quality of life as a result of Title VI Programming, means Elders and caregivers will 

experience an increase in overall quality of life.

 ↑ in quality of care/services as a result of Title VI Programming, means service providers 

will be able to provide higher quality care/services.

 ↑ in coordination of local/state/federal services as a result of Title VI Programming, 

means there will be an increase in coordination of services between local, state, and 

federal agencies.

 ↓ in emergency department and acute care visits as a result of Title VI Programming, 

means Elders will experience less need for acute care/emergency room visits.

 ↓ in health care system costs for clients (chronic condition management) as a result of 

Title VI Programming, means Elders’ health will be better managed, resulting in 

decreased health care (urgent/acute) costs.