APPENDIX B

TITLE VI EVALUATION METHODS
1

Methods

The Evaluation of the ACL Title VI Programs uses a mixed-methods approach to assess the impact of the 

Title VI Programs on stakeholders, including elder program participants, caregivers of elders, and 

program staff. As the evaluation is implemented over time (2018–2020), various data sources will be 

used to describe and demonstrate the implementation of the Title VI Programs and outcomes across 

nutrition services, supportive services, and caregiver support services. As described in the year 3 interim 

report, the evaluation design includes two interconnected studies to assess the Title VI Programs: 

Primary and secondary data collection and analysis approaches are designed to ensure that the 

objectives of the evaluation are achieved. The evaluation is designed to address the following questions: 

• What is the context of the Title VI Programs at the national and tribal levels? How do tribes operate 

their Title VI Programs?

• What are the outcomes and impact of Title VI Programs, nationally and by tribe or tribal groups? 

What is the effect of the Title VI Programs on elders in the community, and are there differences 

nationally or by tribe/tribal group?

• Do Title VI Programs that rely only on Title VI funds have a different community impact than programs 

that have money from other programs or agencies? 

The data sources used to inform the year 3 interim report in 2019 are described in the following 

sections. 

Data Sources 

QUALITATIVE

Primary Data—Caregiver Focus Groups/Interviews

During site visits held in March, April, and May of 2019, the ICF team conducted focus groups and 

interviews with caregivers participating in the Title VI Program. The purpose of the data collection was 

to understand caregivers’ experiences with the Title VI Program, including met and unmet needs related 

to spirituality; social connectedness and isolation; physical, mental, and emotional health and wellness; 

and independence and quality of life. ICF collaborated with ACL and the Title VI Evaluation Steering 

Committee to design moderator and interview guides to elicit information related to the evaluation 

questions (the guides appear at the end of this summary). 

Caregivers were asked to describe the length of time they had participated in the Title VI Program, the 

services they had received through the program, what they liked best about the services, how the 

program helps them, and the ways in which they would change or enhance the program. Before the site 

The implementation study is designed to understand the extent of implementation of the Title VI 

Programs at the national and tribal levels, contextual factors that affect implementation, and the 

barriers and facilitators to program implementation.

The outcomes study is designed to assess the impact of program implementation, including the 

proximal and distal program outcomes outlined in the program logic model.
2

visits, ICF held a webinar presentation with the grantee program staff to share the purpose of the site 

visits and coordinate planning of the data collection activity. Caregivers were identified and recruited for 

the focus groups and interviews with assistance from local Title VI Program staff. During the site visits, 

the ICF team conducted focus groups or interviews with caregivers receiving caregiver services at each 

grantee program. Incentives were provided to caregiver participants by local Title VI program staff and 

varied across grantee programs (e.g., incentives often included a meal or a gift card). All interviews and 

focus groups were audio recorded and transcribed for analysis. As a backup data source, ICF staff took 

notes during the interviews and focus groups and met to debrief after data collection activities to 

document the content and initial impressions of the findings. In all, the team conducted 8 focus groups 

and 11 interviews. The focus groups lasted an average of 56 minutes, and the interviews lasted an 

average of 20 minutes. Participants were primarily unpaid family caregivers; however, at one grantee 

site, the participants were formal, paid caregivers who also had experience as informal caregivers for 

their own families. Participants ranged in age from early adult to elder and included adults caring for 

their aging parents, husbands or wives caring for their spouse, siblings caring for another sibling, adults 

caring for a disabled family member, and grandparents caring for grandchildren. 

Caregiver Program Assessment

The ICF team developed the caregiver program assessment—an 

Excel workbook that catalogs information related to grantees’

caregiver programs, including program participants, services 

provided, and program monitoring and evaluation. The 

caregiver program assessment was designed to support a 

thorough understanding of the Part C Caregiver Support 

Program as it is implemented at the local level. The caregiver 

program assessment was conducted through conversations 

with program directors and other program staff as well as 

through direct program observation during site visits held in 

March, April, and May 2019. 

QUANTITATIVE

Secondary Data—Elder Needs Assessment 

The National Resource Center on Native American 

Aging/University of North Dakota (NRCNAA/UND) Identifying 

Our Needs: A Survey of Elders (Elder Needs Assessment) survey 

and data is administrated by NRCNAA, located at the Center for Rural Health at UND. Cycle VI data were 

collected from April 2014 to March 2016 and include information related to the health status of all 

elders receiving services through the Title VI Programs. It covers domains such as demographics, overall 

health and wellness, healthcare access, weight and physical activity, social functioning, social 

connectedness, and social supports and housing of all program recipients. The Elder Needs Assessment 

also includes domains related to caregiving, including whether elders have a family caregiver, whether 

they are raising grandchildren, as well as their current and anticipated future use of caregiver services.

To the extent that survey participants are elders, responses pertaining to the caregiver experience are 

assumed to reflect the experience of elders who are themselves caregivers (of other elders or of 

grandchildren) and/or their perception of their family caregiver’s experience. As the evaluation grantees 

Caregiver Program

Assessment Topics

• Overall caregiver program 

description, including 

participants, staff, and services 

provided

• History of the program

• Community partnerships and 

collaborations

• Program challenges and 

facilitators

• Program monitoring and 

evaluation

• Grantee Part C 

questions/concerns 
3

are representative of all Title VI grantees (as detailed on page 4 of the Year 2 interim report), in this 

report our sample for data analysis was the evaluation grantees. 

Prior to receiving frequency tables from UND, ICF undertook the following steps to ensure data 

agreements and protections were in place:

1. Tribal resolutions. ICF worked with each of the 12 evaluation grantees to secure a tribal 

resolution to access their tribal level Elder Needs Assessment data. ICF staff sent an advance 

notification email to the Title VI evaluation grantee program directors to inform them of the 

purpose of the request for a tribal resolution. ICF staff followed this with personal calls to each 

program director to address any questions or concerns related to data sharing and to identify 

the tribal resolution process for their tribe. To assist with the tribal resolution, ICF developed an 

optional resolution template for grantees. The template clearly stated what data were

requested, how the data would be used, and who would have access to the data. ICF received 

tribal resolutions from each grantee. 

2. Data use agreement (DUA). ICF established a DUA with NRCNAA/UND. The DUA described the 

purpose of the evaluation; the proposed usage of the Elder Needs Assessment data; data access, 

transfer, and storage; how local level approvals (tribal resolutions) were obtained; and noted 

that only aggregated data would be reported.

Secondary Data—n4a Title VI Program Survey

The n4a Title VI Program Survey is collected and administered by Scripps Gerontology Center (Scripps). 

The survey gathers information from program staff about which services are available, how the services 

are delivered, partnerships established to support service delivery, strategies for record keeping and 

budgeting, and challenges experienced with Title VI implementation. In order to access the Title VI 

Program Survey data for the 12 evaluation grantees, ICF undertook the following steps: 

1. Informed consent. In November of 2017, ICF staff worked with Scripps to develop an informed 

consent letter to the Title VI evaluation grantee program directors to inform them of the 

purpose of the request for their Title VI Program Survey data. This letter explained the purpose 

of the evaluation, the risks and benefits to sharing the data, who would have access to the data, 

and how the data would be stored and transferred. The letter was emailed to the program 

directors in January 2018.

2. Program data. In January 2018, Scripps sent a PDF version of each tribe’s completed 2016 Title 

VI Survey to each of the 12 evaluation grantees. ICF staff then reached out to each of the 

grantees to answer any questions. 

3. Data access. ICF supported grantees in sharing their 2016 data with the evaluation team. 

4. Data extraction. ICF extracted data for each grantee manually and consolidated the data in an 

Excel file based on data cleaning and management plans. Following ethical procedures, only 

aggregated data for the overall evaluation sample is ever reported.

Secondary Data—Title VI Program Performance Report (PPR)

The Title VI Program Performance Report (PPR) is maintained by the Administration on 

Aging/Administration for Community Living (AoA/ACL). The PPR collects information on the number of 

clients and Title VI service units for nutrition, supportive, and caregiver support services delivered by 

each Title VI grantee. 
4

Data Analysis 

QUALITATIVE

Caregiver Interviews and Focus Groups

ICF’s approach to the qualitative analysis of interview data and focus group data included the following 

steps: 

1. Data capture and management. Three team members working on the analysis managed a 

comprehensive data inventory in Excel to monitor and track for each grantee: dates of data 

collection activity, number of participants per data collection effort, number and type of data

files (i.e., digital recordings, notes, transcripts), and steps and progress related to monitoring 

data quality. The analysis team reviewed transcripts and notes for completeness and to 

minimize errors before the analysis process began. All data files and documents produced or 

collected before, during, and after data collection activities (e.g., interview and focus group 

notes, proprietary documents associated with a grantee, digital recordings and transcripts) were 

stored in password-protected electronic files accessible only by the ICF project team members.

1

Audio recordings from each interview and focus group were transcribed and imported into the 

qualitative software package ATLAS.ti v7.5.18, useful for facilitating the organization and 

management of textual data. Three ICF team members used this software to analyze the data 

for themes, patterns, and interrelationships relevant to the evaluation questions. 

2. Codebook development. Codebook development was a multistep and iterative process 

involving the development and definition of codes and pretests to refine the codes and 

definitions. A draft codebook, with an initial set of codes, was first developed on the basis of 

evaluation questions. Next, each of the three team members reviewed the transcripts and 

developed summary memos to document initial impressions of the findings. This process helped 

to inform the development of new codes to capture unexpected issues and themes not included 

in the preliminary coding scheme. Upon completion of a revised codebook, team members 

separately coded a single transcript and then came together to discuss how they applied the 

codes to the document. Through these discussions, codes were added, removed, or refined to 

best capture emerging themes. This process was repeated through a similar review of a single 

transcript. Team members debated confusion or disagreement over code definitions or 

applications until all members had the same analytical understanding of the codes as measured 

by sufficient intercoder reliability. The final codebook included codes, subcodes, definitions, and 

links to evaluation questions; a version of the codebook, excluding sub codes (for brevity), is 

presented in Table 1. 

3. Code application. Following codebook development, each team member created their own 

Hermenutic Unit (HU) in ATLAS.ti, where they uploaded the transcripts and codebook. Team 

members applied the codes to the transcripts to facilitate data analysis using ATLAS.ti software. 

ICF team members coded at the level of a whole sentence or paragraph. 

4. Analysis and documentation of themes. Upon completion of the coding process, team 

members used ATLAS.ti to search for, retrieve, and classify the coded data. The team produced 

output documents associated with each code and sub code. Two team members participated in 

 

1 See additional detail about security procedures in the section titled “Ethnical Considerations and Regulatory 

Review.”
5

a thematic analysis approach in which each team member conducted an in-depth review of a set 

of output documents reflecting groupings of interrelated codes and sub codes. Next, each team 

member produced a comprehensive summary memo providing an assessment of key themes 

and relationships among themes for the assigned set of output documents. Following this 

process, team members met to discuss the analysis, findings, and themes, some of which were 

identified by a single analyst and some by multiple analysts. Team members then compiled and 

reduced the results of the output review in preparation for developing a triangulated approach 

to reporting, featuring multiple data sources. Themes in responses that were posed repeatedly 

by interview and focus group participants are presented in the year 3 interim report. Themes 

that were more subtle or less often voiced are also described. Ultimately, the analysis facilitated 

the development of explanatory narratives related to Title VI Program staff and elders’ 

experiences with and insights about the Title VI Program. 

Table 1. ACL Title VI Caregiver Interviews and Focus Groups: Codebook

Code Definition Related Evaluation Question(s)

Caregiver service 

availability

Discussion of availability of caregiver 

resources and supports in the community 

and surrounding area.

Non-Title VI: Availability of caregiver 

services offered through other programs.

Lack of: Discussion of lack of caregiver 

resources.

• What is the context of the Title 

VI Programs at the national and 

tribal levels?

Caregiver 

awareness 

services

of 

Discussion of caregivers’ awareness (or 

lack of awareness) of services, resources, 

and supports for caregivers; how 

caregivers are currently recruited or 

provided information about 

services/supports; and how respondents 

think caregivers should be recruited or 

informed.

Current Outreach: References to how the 

respondent learned about Title VI 

caregiver services and/or how outreach 

and information occurs currently.

Suggestions for Outreach: Respondents’ 

suggestions for outreach strategies to 

build awareness of caregiver services and 

resources (i.e., how respondents suggest 

getting the word out).

Lack of: Discussion of lack of awareness 

of resources or supports for caregivers in 

the community among caregivers. 

• What is the context of the Title 

VI Programs at the national and 

tribal levels?
6

Code Definition Related Evaluation Question(s)

Caregiver role Description of caregiving duties, 

individual(s) receiving care, and caregiver 

characteristics. 

• What is the context of the Title 

VI Programs at the national and 

tribal levels?

Caregiver-Elder: An elder providing 

informal care for another elder or for an 

individual of any age with Alzheimer’s 

disease or a related disorder.

Caregiver-Adult: An adult (non-elder) 

providing informal care for an elder or for 

an individual of any age with Alzheimer’s 

disease or a related disorder.

Grandparent: Grandparents raising 

grandchildren. 

Disabilities-Elder: Elder providing 

informal care for an adult with 

disabilities.

Note: When the details are not specific 

enough to select a sub code, simply use 

“caregiver role.” 

Caregiver 

experience

Description of 

caregiver and 

impacted.

what it’s like to be a 

how the caregiver is 

• What is the context of the Title 

VI Programs at the national and 

tribal levels?

Key quotes Use to document key quotes. 

Length of time 

program

at Discussion of how long caregiver has 

been receiving Title VI services or has 

been a caregiver.

• What is the context of the Title 

VI Programs at the national and 

tribal levels?

No Title VI Description of what would happen or 

what caregivers would miss if Title VI 

services went away. 

• What are the outcomes and 

impacts of the Title VI 

Programs, nationally and by 

tribe or tribal groups?

• What are the met and unmet 

needs of consumers and 

program stakeholders?

Services 

Part C

received— Description of services that caregivers 

receive that are designed for caregivers 

and appear to be supported by Part C.

Information and Outreach: A public and 

media activity that conveys information 

to caregivers about available services, 

which can include an in-person 

interactive presentation to the public. 

Assistance: Helping caregivers access 

services, including through:

•

•

•

How are the Title VI Programs 

implemented at the national 

and tribal level?

What are the program inputs, 

resources, and activities 

implemented through Title VI?

How are nutrition, supportive, 

and caregiver support services 

provided through the Title VI 

Programs?
7

Code Definition Related Evaluation Question(s)

• Individual one-on-one contact linking 

family caregivers with services, 

supports, and other opportunities.

• Follow-up to ensure services are 

received.

• Case management—Caregiver 

Assessment, wrap-around services, 

person-centered planning.

Counseling, Training, or Support Groups 

for Caregivers: A service designed to 

support caregivers and assist them in 

their decision-making and problem 

solving.

• Individual counseling, organization of 

support groups, and caregiver training 

to assist the caregivers in the areas of 

health, nutrition, and financial literacy, 

and in making decisions and solving 

problems relating to their caregiving 

roles.

Respite Care: A service for caregivers 

that offers temporary, substitute 

supports or living arrangements for care 

recipients in order to provide a brief 

period of relief or rest for the caregivers. 

Also includes opportunities for 

caregivers to “get a break” via fieldtrips 

or other activities

Supplemental Services: (meant to be 

provided on a limited basis)

• Home Modifications 

• Assistive Technologies 

• Emergency Response Systems 

• Equipment 

• Incontinence Supplies 

• Transportation 

• What is the array of Title 

practices available to 

caregivers?

VI 

Services received—

Part A/B

Discussion of services provided through 

Part A/B (e.g., meals, transportation) or 

provided in other ways through the elder 

center.

Supports Respondent: Respondent sees 

service as helping them generally—e.g., 

•

•

How are the Title VI Programs 

implemented at the national 

and tribal level?

What are the program inputs, 

resources, and activities 

implemented through Title VI?
8

Code Definition Related Evaluation Question(s)

they directly benefit as a participant in 

the meal, transportation, or activity.

Supports Respondent as Caregiver: 

Respondent recognizes that service helps 

them to be a better caregiver by relieving 

them of the stress, worry, and 

responsibility of meal preparation, finding 

transportation, etc.

•

•

How are nutrition, supportive, 

and caregiver support services 

provided through the Title VI 

Programs?

What is the array of Title VI 

practices available to 

caregivers?

Services 

Other

received— Discussion of services that caregivers 

receive that seem to be non-Title VI 

services.

• What is the context of the Title 

VI Programs at the national and 

tribal levels?

Challenges Description of challenges associated with 

caregiving in general or caregiver 

program implementation. 

• What is the context of the Title 

VI Programs at the national and 

tribal levels?

Aging Population: As individuals are living 

longer, elders are now caring for older 

elders. There is a growing need for longterm care and in-home nursing. 

Dementia: Discussion of prevalence of 

and impact related to caring for 

individuals with dementia, Alzheimer’s, 

and related illnesses.

Title VI-like best Description of what caregivers like best 

about the Title VI Program that supports 

them in their caregiving role.

• What are the outcomes and 

impacts of the Title VI 

Programs, nationally and by 

tribe or tribal groups?

• What are the met and unmet 

needs of consumers and 

program stakeholders?

• What are the physical, 

emotional, mental, and 

spiritual outcomes of the Title 

VI Programs?

Ways 

helps

program Description of the various ways in which 

the program helps caregivers. Consider 

the medicine wheel: 

• Spiritual

• Mental

• Emotional

• Physical

•

•

•

What are the outcomes and 

impacts of the Title VI 

Programs, nationally and by 

tribe or tribal groups?

What are the met and unmet 

needs of consumers and 

program stakeholders?

What are the physical, 

emotional, mental, and 
9

Code Definition Related Evaluation Question(s)

spiritual outcomes 

VI Programs?

of the Title 

Wish list Description of other services or supports 

that respondent wishes for, that could be 

provided for caregivers (either by the 

elder center or by other entities in the 

community in general).

•

•

•

What are the outcomes and 

impacts of the Title VI 

Programs, nationally and by 

tribe or tribal groups?

What are the met and unmet 

needs of consumers and 

program stakeholders?

What are the physical, 

emotional, mental, and 

spiritual outcomes of the Title 

VI Programs?

Cultural- Discussion of cultural or community • What is the context of the Title 

community norms, expectations, trends, or other 

issues rooted in culture that affect 

caregiving, attitudes about caregiving, or 

help-seeking among caregivers.

Reluctance to Seek Help: Reluctance of 

caregivers to seek or access services or 

ask for help (e.g., embarrassment, 

stigma, desire to maintain 

independence). 

Women Caregivers: Notion that women 

should provide caregiving. 

Family Caregivers: It is the family’s 

responsibility to provide caregiving to 

avoid facility care or, for various reasons, 

families end up providing caregiving (e.g., 

often family members take shifts to 

cobble together care for a parent).

Identity-Caregiver: Sense that people 

don’t identify as a caregiver because it’s 

just part of their job as a family member.

VI Programs at the national and 

tribal levels?

Caregiver Program Assessment

ICF staff reviewed the caregiver program assessment results to identify commonalities and areas of 

divergence among the grantees. Findings were organized by topical area. 

QUANTITATIVE

Elder Needs Assessment

ICF program staff revised frequency tables generated by NRCNAA/UND containing evaluation grantees’ 

data and consolidated the variables of interest for Title VI Part C (Table 2). Then, for each variable of 
10

interest, ICF generated an average percentage for all evaluation grantees. As the Elder Needs 

Assessment survey categorizes questions related to “respite” and “caregiver services”—including all 

other caregiver services—separately, findings are presented in the report using this approach to 

categorizing services. 

Table 2. Elder Needs Assessment Data Title VI Year 3 Evaluation Indicators

Elder Needs Assessment Survey Question 

Q69. Are you an enrolled member of a federally recognized tribe? 

Q50. Do you have a family member who provides care for you?

Q51. Do you take care of grandchildren?

Q52. Are you the primary caregiver of grandchildren?

Q59. Are you now using or, if at some point you became unable to meet your own needs, would you 

be willing to use the following services? 

• Caregiver Program

• Respite Care

n4a Title VI Program Survey

In April 2018, ICF received the dataset for the 12 evaluation grantees’ Title VI Program Survey responses. 

ICF staff developed a codebook with key questions aligned with the evaluation plan and consolidated 

responses in one aggregated dataset containing recoded responses for all grantees. The recoded 

responses included both objective and descriptive answers. Table 3 includes a list of the selected survey 

questions used in this report along with a description of the data management approach for each. 

ICF focused on an initial descriptive analysis (frequencies, means). Data cleaning (e.g., excluding invalid 

entries) and examination of missing data and out-of-range values were performed prior to data analysis. 

Additional recoding and transformation of variables were conducted and values were collapsed into 

meaningful categories. Due to the nature of many “multiple response” items on the survey, data were 

analyzed based on multiple response sets for the relevant questions. Therefore, the sum of percentages 

may add up to more than 100%.
11

Table 3. n4a Data Title VI Year 3 Evaluation Indicators

n4a Survey Question Data Management

Q9. Which services 

Respite care

are Title VI funded, wholly or partly— Responses recoded to 

Title VI funds = 1 

Provided partly with

Title VI funds = 2

Provided through

non-Title VI funds = 3

Not available = 0 

categorical 

Q9. Which services are Title VI funded, 

Family caregiver support services

wholly or partly—

Q10. Rate the capacity 

of your elders—Family 

of services to actually meet 

caregiver support services

the needs Responses recoded to categorical 

Significant unmet needs = 0

Some unmet needs = 1

Needs met = 2 Q10. Rate the capacity of services to 

of your elders—Respite care

actually meet the needs 

Q10a. Please identify 

needs—Respite care

up to five of the most significant unmet Responses recoded to categorical 

Family caregiver support services 

Respite care = 20

=19

Q10a. Please identify up 

needs—Family caregiver

to five of the most 

 support services

significant unmet 

Q26a. Evidence-based program has been offered through 

Title VI or another program within the last two years—The 

Savvy Caregiver in Indian Country

Responses recoded to dichotomous 

Not familiar with this program = 0 

Familiar with this program = 1 

Q26a. Evidence-based program has been offered through 

Title VI or another program within the last two years—

Powerful Tools for Caregivers

Q31a. 

about 

What information does the Title VI program maintain 

each elder? Caregiver/informal support information

Responses recoded 

Yes = 1 No = 0

to dichotomous
12

PPR

In June 2019, the Title VI Part C Performance Data Report for fiscal years 2010–2017 was shared with 

ICF. Review of the data identified extreme variances within and across grantees’ data, suggesting an 

overall lack of reliability. In consultation with ACL, the evaluation team determined to exclude Part C 

PPR data from data synthesis. 

Ethical Considerations and Regulatory Review 

Office of Management and Budget (OMB) Clearance

In 2017, ICF prepared, in consultation with ACL, an Evaluation of the ACL Title VI Programs OMB 

Information Collection Request (ICR) package. The OMB package included standard forms, a 

comprehensive supporting statement, 60- and 30-day Federal Register notices (FRNs), and a list of 

attachments such as data collection instruments. The summary statement described the objectives of 

the Evaluation of the ACL Title VI Programs, the evaluation questions being pursued, and the domains 

and/or data elements to be collected. The statement included clear descriptions of each data collection 

activity and instrument as well as burden estimates. The 60-day FRN was posted for public comment in 

the Federal Register. Comments were solicited over a 60-day period on the need for and proposed use 

of the study, respondent types, and annualized burden. No comments were received at the conclusion 

of the 60-day comment period. ICF then resubmitted the package, along with the 30-day FRN, to ACL for

review and approval. ICF worked with the contracting officer’s representative to obtain necessary 

approvals and otherwise move through the approval chain for submission to OMB. OMB approval was 

received on February 5, 2018.

2

 

Institutional Review Board (IRB) Review and Approval

To ensure the protection of human subjects, including the confidentiality of data compiled and collected 

during the Evaluation of the ACL Title VI Programs, evaluation data collection protocols and instruments 

were reviewed and approved by the ICF IRB prior to the collection of protected data. This review 

ensured compliance with the spirit and letter of U.S. Department of Health and Human Services (HHS) 

regulations governing such projects. 

The ICF IRB is committed to protecting the rights, welfare, and privacy of individuals who participate in 

ICF-supported research and evaluation studies, as well as the confidentiality of the data. The IRB’s 

approach to protecting human subjects is guided by the ethical principles and guidelines outlined in the 

Belmont Report: respect for persons, beneficence, and justice. The ICF IRB (IRB00000954; expires July 

12, 2023) complies with all requirements specified in the Code of Federal Regulations (45 CFR 46) on the 

protection of human subjects and has a Federalwide Assurance (FWA00000845). In addition to the 

Federal regulations, the IRB takes into consideration any State or local laws regarding human subjects 

that may be more protective than the Federal statutes. It is the responsibility of the IRB, as well as the 

evaluation team, to ensure that these regulations and other applicable laws are followed in the conduct 

of ICF-supported research and evaluation. 

 

2 OMB No.: 0985-0059, Expiration Date: 02/28/2021
13

To ensure the protection of human subjects in this evaluation, ICF undertook the following steps:

1. Respondents were given the opportunity to refuse to answer questions or to stop the focus 

group or interview or leave the focus group at their discretion. 

2. Focus group moderators were trained about their responsibility to maintain the privacy of 

respondents’ answers. Moderators were instructed not to disclose any information obtained 

during the focus group to any other individual outside of the evaluation team. 

3. Interviewers were trained about their responsibility to maintain the privacy of respondents’ 

answers. Interviewers were instructed not to disclose any information obtained during the 

interview to any other individual outside of the evaluation team. 

4. To minimize the risk of emotional or mental distress or general discomfort related to the topics 

discussed, all focus group moderators and interviewers received in-depth training on how to ask 

questions and respond appropriately, particularly regarding sensitive topics. 

5. Focus group moderators were trained to conduct focus groups in private locations with only the 

respondents, the moderator and the note-taker(s) present. 

6. Interviewers were trained to conduct interviews in private locations with only the respondent 

and interviewer present. 

7. The Evaluation Working Group (EWG) received human subjects protection training at the EWG 

in-person meeting prior to beginning data collection. 

8. Comments made during the focus groups and interviews were not attributed to any one 

respondent. 

9. Focus group and interview digital recordings and notes are kept in password-protected 

electronic files at ICF. 

10. All evaluation reports and publications that result from these data include only group-level 

analyses that fully protect the confidentiality of individual participants.

 

Approval for the Evaluation of the ACL Title VI Programs was obtained from ICF’s IRB, including both an 

original protocol (approved on March 1, 2017) and a modified protocol (approved on February 9, 2018). 

ICF received continuing review approval on January 15, 2019. The submission to the IRB included a 

summary statement addressing the key required criteria for the IRB outlined above and the evaluation 

protocol, including data collection instruments. Data collection did not begin until the ICF team member 

had reviewed the informed consent form with all eligible respondents, including program staff and tribal 

elders. All participants received a copy of the informed consent form, which emphasized the voluntary 

nature of participation and the right to stop participating at any time, and noted that all individual data 

would be kept private. Focus groups and interviews were audio recorded with the permission of the 

participants.

Local-Level IRB Review and Approval

In addition to ICF IRB, ICF worked with each evaluation grantee to identify and obtain the local level 

approval(s) necessary to participate in the Evaluation of the ACL Title VI Programs through an IRB, 

institution at large, or other governing or advisory body, such as the tribal council. Each of the 

evaluation grantees obtained a tribal resolution confirming their tribe’s commitment to participate in 

the evaluation as well as share their Title VI data with ICF. All grantees received a copy of ICF’s 

1) approved IRB package/application, 2) IRB approval letter; 3) summary page of IRB steps; 

4) instruments and informed consents; and 5) summary of instruments by research question.
Title VI Tribal Caregiver Focus Group/Interview

Moderator Guide
Form Approved 

OMB No.: 0985-0059 

Expiration Date: 02/28/2021

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently 

valid OMB control number. The OMB control number for this project is 0985-0059. Public reporting burden for this collection of information is estimated to average 5 

minutes per respondent, per year, including the time to review instructions. Send comments regarding this burden estimate or any other aspect of this collection of 

information, including suggestions for reducing this burden, to Office of Performance and Evaluation, Center for Policy and Evaluation, 330 C Street, SW, Rm 1229A, 

Washington DC, 20201.

Evaluation of the ACL Title VI Programs 

Title VI Tribal Caregiver Focus Group/Interview Moderator Guide 

Questions Probes Time Guidelines 

Introduction of 

Moderator/Guests and 

Purpose of Focus Group/ 

Logistics 

5 minutes 

Read Consent Form/Confirm 

Verbal Consent/ Confirm 

Permission to Audio Record 

5 minutes 

Opening Question 

1. Please tell us your first 

name and how long you 

have been using Title VI 

(or local program name) 

services. 

5 minutes 

Introductory Question 

2. Tell me a little bit about 

the services you get as a 

caregiver. 

Provide examples of services specifically provided by 

the program: 

 Information 

 Counselling 

 Education 

 Support group 

 Respite care 

 Grandparent support

10 minutes 

Transition Questions

3. What do you like about 

the caregiver program? 

 Why is that? What about that service makes you 

say that? 

 Can you give me an example or tell a story about it? 

10 minutes 

Key Questions

4. How does the program 

help you as a caregiver?

 How does the program help you with stress, time, 

resources, etc.? 

 How/does the program ease mental overload? 

 How/does the program improve your quality of life? 

 How/does the program improve the quality of life of 

the person you care for? 

 Are there other ways that the program helps you 

feel connected to your community? 

10 minutes 

5. What else do you wish 

the program had for 

caregivers? 

 What other types of information might be valuable 

to you as a caregiver? 

 What other types of services might you want to 

receive?

10 minutes 

6. If you could change 

something about the 

program for caregivers, 

what would that be? 

10 minutes 
Form Approved 

OMB No.: 0985-0059 

Expiration Date: 02/28/2021

7. What would happen to 

you as a caregiver if you 

didn’t have this program?

Components to talk about: 

 Employment 

 Stress 

 Difficulty with providing care 

 Chances to socialize/visit with others 

 Getting out of the house, etc. 

 Not be able to continue providing care 

 Not be able to provide as good care

10 minutes 

8. If you were telling a friend 

about the experiences 

you’ve had as a caregiver 

with the program, what 

would you would tell 

them? 

10 minutes 

Ending Question 

9. Thank you so much for 

sharing your stories with 

us today. Is there 

anything that we have 

missed? Is there 

anything that you came 

wanting to say that you 

didn’t get a chance to?

5 minutes 

Total Time 90 minutes 
Title VI Caregiver Program Assessment
Title VI Caregiver Program Assessment, Codebook

Topic Description

General Program Description

Who participates in the  Understand who receives services and verify that the participating caregivers

caregiver program

A

are unpaid.     

caregiver can be:  

1) An unpaid family member, friend or neighbor that cares for an elder.  

2) Grandparents caring for grandchildren as the primary care provider.

(Definition is stated briefly here; see additional details in the caregiver

chapter of the Title VI manual).

Number

program

of caregivers in the  Understand how many caregivers are served, including how many of which

caregiver type (i.e., family members, grandparents caring for grandchildren)

Caregiver outreach—how  Explore how grantee programs identify caregivers and the process for

caregivers are identified recruitment and enrollment into the program.

Related description from caregiver chapter of Title VI manual:

 “In Indian Country, the most likely caregivers are family members (spouses,

daughters, granddaughters, and occasionally, male relatives), friends of the

family, neighbors, or members of a church or social club who are close to the

person needing care. Word of mouth is the best way to find caregivers, but

many caregivers don’t identify themselves as such. Since caring for elders is a

traditional activity in Indian Country, caregivers are simply doing what needs

to be done.”

Caregiver services provided  Describe whether the program includes some, all, or none of the five

caregiver services (listed below), and which services are most often used.

 Describe what each service type looks like.   

 Describe types of services provided to grandparents caring for grandchildren.  

 Describe types of services provided for family members or friends caring for

elders.

The five types of caregiver services include:

1) Information and outreach (information to caregivers about available

services)  

2) Assistance to caregivers in accessing the services

3) Counseling or support groups for caregivers and caregiver trainings

4) Respite care (i.e., allows the caregiver to be relieved of their duties)  

5) Supplemental services like lending closets (e.g., supplies like toiletries or

school supplies) or home modifications (e.g., install a wheelchair ramp at

front entrance)

(Description is stated briefly here; see additional detail in the caregiver

chapter of the Title VI manual).

Who provides

services

caregiver  Describe whether caregiver services are provided by

other staff, volunteers, partner agencies, or others.  

the program director,

How often

provided

services are  Describe how the program is organized in terms of frequency of service

provision. Specify for all five types of caregiver services. For example, if

respite services are provided, note the frequency of service provision for

specific service (e.g., two hours per week).  

that
Title VI Caregiver Program Assessment, Codebook

 Describe whether all caregivers in the program are provided the same

services at the same frequency or if there is any variation depending on

caregiver type or other factors (and if so, why).  

Where services are provided  Describe where services are provided. Specify for each of the five caregiver

service types.   

History of the Program1

When caregiver services

began

 Describe when the grantee began providing caregiver services and how long

caregiver services have been provided (specify for all five types of caregiver

services).

How caregiver services have

changed over time (if at all)

 Provide a description of the changes.

 Describe reasons or context for the changes.

If some or all caregiver

services are not provided

 Describe perceived barriers to providing caregiver services.

 Describe what it would take, from the grantee’s perspective, to provide such

services.

Partnerships/Collaborations

Caregiver program partners  Describe agencies, organizations, tribal departments or other entities that

serves as a partner to the grantee in the provision of caregiver services (e.g.,

the transportation department may provide transportation services for

respite care workers).  

Other caregiver service

provider in the area/region

 Describe whether other caregiver services are provided in the local area or

region.

Related description from the caregiver chapter of the Title VI manual:  

 “The OAA requires that caregiver support funds supplement, and not

supplant, any federal, state, or local funds for caregiver services. In other

words, if some other program was paying for caregiver support services,

funds from the Native American Caregiver Support Program cannot be used

for those same services. However, Title VI, Part C funds can be used to

provide additional services.”

Use of volunteers  Explore whether the grantee uses local volunteers to help deliver the

program.  

Related description from caregiver chapter of Title VI manual:  

 “In addition to the required (caregiver) services, the 2005 Amendments to the

OAA require the programs to use trained volunteers to expand the required

services.  If possible, you are to coordinate with organizations that have

experience in providing training, placement, and stipends for volunteers, such

as programs administered by the Corporation for National and Community

Service. These would include VISTA Volunteers and Senior Corps Volunteers.”

Program Challenges & Facilitators

1 Note: many of our program directors are relatively new in their position and may not be able to provide much insight into

the history of the program. If this is the case, just note as such. Do not press on the issue or require the director to go in

search of the information.
Title VI Caregiver Program Assessment, Codebook

Challenges providing

caregiver services

 Describe challenges the grantee identifies related to providing caregiver

services (e.g., engaging caregivers, getting them to participate, securing

background checks).  

Solutions/strategies

attempted to address

challenges identified

 Describe any solutions grantees have found related to the challenges or

strategies they have tried to address the challenges (if any).

What helps grantees

provide services

 Describe what helps grantees provide the caregiver services (e.g., partners,

particular outreach strategies).

Program Monitoring and Evaluation

How the grantee tracks their

caregiver program

 Describe how the grantee tracks their caregiver program, building off of what

we learned last year as part of the data needs assessment (e.g., caregiver

needs assessment, approaches to monitor caregiver services provided,

number of participants served, caregiver satisfaction with services).

How collected data informs

programming

 Describe how the grantee uses the program monitoring/evaluation data they

collect to inform their program, if they do (again, building off of what we

learned last year).   

How program decisions

regarding services are made

 Describe how program decisions regarding services are made (e.g., how

priorities are identified, how decisions are made about service approach given

limited resources, who participates in decision‐making).

Grantee Questions/Concerns/Ongoing Needs

Questions the grantee may

have about the program

 List any questions the grantee has about the caregiver program (e.g., this may

include questions about what should be included as a part of caregiver

services, how to implement a particular service, how to request support).

Current needs of caregivers

in the grantee community

 Describe the current needs of caregivers in this community based on the

perceptions of the program director or other staff.

 Note how caregiver needs are identified by the grantee (relates to program

monitoring and evaluation).

Related description in caregiver chapter of Title VI manual:

 “The Native American Caregiver Support Program can help caregivers in a

variety of ways. It is important to ask caregivers what would help them best.

The suggestions included here are just suggestions. Be certain to include clinic

staff and churches in your discussion of what caregivers need. Both may be

able to support your programs with training, volunteers, and identifying

caregivers.”

Resources or assistance the

grantee needs to implement

their caregiver services

more fully

 Describe the resources or assistance the grantee needs to implement their

caregiver services more fully (specify whether the resource identified was

identified by the grantee or by the site visitor/liaison).

Technical assistance areas of need may include:  

 Starting a caregiver program

 Strengthening or expanding their caregiver program

 Additional information or expertise (e.g., best practices in providing caregiver

services)

 Training needs for staff

 Other issues identified by the grantee
Title VI Caregiver Program Assessment, Codebook

Additional Liaison Observations/Questions

General notes  Add any notes here about the grantee’s caregiver program that do not seem

to belong elsewhere in the excel file.