Initial Findings from the 

Nutrition Services Program 

Outcomes Evaluation

September 14, 2017

National Evaluation Results Webinar

The National Resource Center on Nutrition and Aging
2

Administration for Community Living Team

Holly Greuling, RDN

Heather Menne, Ph.D.
Initial Findings from the 

Nutrition Services Program 

Outcomes Evaluation

National Evaluation Results Webinar

The National Resource Center on Nutrition and Aging

James Mabli, Ph.D.

Liz Gearan, M.S., RDN

September 14, 2017
4

AoA Nutrition Programs Evaluation Objectives

1. Process study

– Provide information to support program planning

– Analyze program structure, administration, staffing, coordination, 

processes, and service delivery

2. Cost study

– Estimate the average cost of a congregate and a home-delivered meal

– Assess variation in costs by select characteristics of local providers

3. Outcomes evaluation (ongoing)

– Assess program effectiveness in improving food security, socialization, 

and diet quality

– Assess program effectiveness in improving longer-term health and 

delaying or avoiding institutionalization
5

Objectives of the Outcomes Evaluation

1. Describe participants’ demographic and household 

characteristics, health status, mobility, eating behaviors, 

diet quality, food security, and socialization

2. Describe participants’ experiences with and impressions of 

the program and their valuation of meals and supportive 

services received through the program

3. Determine the impact of meals and related services on 

participants’ nutrition, food security, and diet quality

4. Determine the impact of meals and nutrition services on 

overall wellness and well-being
6

Study Design of Outcomes Evaluation
7

Study Design of Outcomes Evaluation
8

Data Collection Instruments

• Outcomes survey

• 24-hour dietary recall

– Automated Self-Administered 24-hour dietary recall (ASA-24) module

– Administered in-person by interviewer
9

Survey Timeline and Sample Sizes
10

Outcome Measure: Food Security

• Having access at all times 

to enough food for an 

active, healthy life for all 

household members

• Based on USDA’s six-item 

food security module 

based on 30-day recall

• Food insecurity and very 

low food security
11

Outcome Measure: Socialization

• Revised UCLA Loneliness 

Scale (R-UCLA) based on 

responses to three questions 

related to how often one feels 

lack of companionship, left 

out, and isolated from others

• Patient Health Questionnaire 2 

(PHQ-2) based on two 

questions assessing 

frequency of depressed mood 

over past two weeks. Used to 

screen for depression

• Self-reported satisfaction with 

opportunities to spend time 

with other people
12

Outcome Measure: Diet Quality

• Percentage contribution 

program meals made to 

participants’ nutrient intakes

• Usual intakes of vitamins, 

minerals, and macronutrients 

relative to recommendations

• Healthy Eating Index 2010 

scores (HEI-2010) to assess 

overall diet quality
13

Descriptive Analysis Methods

• Describe characteristics of older adults, impressions of 

program, valuation of meals and services

• Use percentages, means, and medians

• Describe characteristics separately for congregate meal (CM) 

and home-delivered meal (HDM) participants

• Based on weighted data, participant findings are nationally 

representative of the population of CM and HDM participants
14

Characteristics of Program 

Participants and Impressions of Meals 

and Services
15

Demographic Characteristics

• CM and HDM participants similar in terms of gender, veteran 

status, whether they lived alone, and race and ethnicity

– More than 2/3rds were women

– 15 to 17 percent were veterans

– About 60 percent lived alone

– 14 to 18 percent non-Hispanic black; 9 to 13 percent Hispanic

• Compared with CM participants, HDM participants were older, 

had less education, and were more likely to be widowed

– Average age was 77 (CM) versus 82 (HDM)

– 24 to 40 percent had not completed high school

– 47 to 52 percent were widowed

Source: AoA NSP outcomes survey, weighted data, Tables III.1.
16

Monthly Income Relative to Poverty Guidelines
17

General Health Status
18

Doctor-Diagnosed Chronic Health Conditions
19

Diet and Eating Behaviors
20

Percentage of Participants on 

Special or Therapeutic Diets
21

Frequency of Participation
22

Food Security
23

Adequacy of Income and Food Coping Strategies
24

Socialization Outcomes
25

Impressions of the NSP
26

Impressions of the NSP Staff
27

Impressions of Meal Delivery
28

Impressions of Meals

• Many congregate meal participants were satisfied with:

– Attractiveness of dining area (96 percent)

– Overall meals (95 percent)

– Amount of food (91 percent)

– Proper temperature of food (91 percent)

– Appearance of food (86 percent)

– Way food smells (85 percent)

– Variety of food (84 percent)

– Taste of food (81 percent)

– Foods provided (79 percent)

– Meets special dietary needs or restrictions (73 percent)

• Similar findings for home-delivered meal participants

Source: AoA NSP outcomes survey, weighted data, Table III.35.
29

Effects of Program Participation 

on Food Security and 

Socialization Outcomes
30

Analysis Methods

• Selected matched comparison group using Medicare records 

and geography

– Collected participants’ SSNs as part of outcomes survey

– Obtained Medicare records for participants

– Identified potential nonparticipants in same geographic service area with 

similar characteristics to participants

– Screened nonparticipants for eligibility

– Conducted interview with nonparticipants

• Multivariate regression analysis to account for observed 

differences between participants and nonparticipants

• Propensity-score matching based on machine-learning 

algorithm
31

Effects of CM Program Participation on 

Being Food Insecure or Having Very Low Food Security
32

Effects of HDM Program Participation on 

Being Food Insecure or Having Very Low Food Security
33

Effects of HDM Program Participation on Being Food Insecure or Having 

Very Low Food Security, by Number of Meals Received per Week
34

Effects of CM Program Participation on 

Socialization Outcomes

Outcome Participants Nonparticipants Difference

R-UCLA loneliness score . . .

Average score 4.1 4.1 0.0

PHQ-2 depression screener questions . . .

Percentage affirmed 4 out of 6 2.3 6.5 -4.2**

Satisfaction with socialization 

opportunities

. . .

Percentage that were satisfied 94.0 85.8 8.2***

Percentage that were very satisfied 67.5 55.5 12.0***

Source: AoA NSP outcomes survey, 2015-2016, weighted data, Table IV.4.

***/**/*Difference between participants and nonparticipants is significantly different from zero at the 0.01/0.05/0.10 level.
35

Effects of HDM Program Participation on 

Socialization Outcomes

Outcome Participants Nonparticipants Difference

R-UCLA loneliness score . . .

Average score 4.5 4.3 0.2*

PHQ-2 depression screener questions . . .

Percentage affirmed 4 out of 6 11.5 11.6 -0.1

Satisfaction with socialization 

opportunities

. . .

Percentage that were satisfied 82.3 85.7 -3.3

Percentage that were very satisfied 44.5 53.4 -8.9**

Source: AoA NSP outcomes survey, 2015-2016, weighted data, Table IV.5.

***/**/*Difference between participants and nonparticipants is significantly different from zero at the 0.01/0.05/0.10 level.
36

Effects of HDM Program Participation on Socialization 

Outcomes, by Number of Meals Received per Week

Outcome Participants Nonparticipants Difference

R-UCLA loneliness score (average) . . .

Receive fewer than five meals 4.6 4.2 0.4*

Receive five or more meals 4.5 4.3 0.2

Percentage satisfied with socialization 

opportunities

. . .

Receive fewer than five meals 79.7 87.2 -7.6**

Receive five or more meals 84.1 85.2 -1.1

Percentage very satisfied with 

socialization opportunities

. . .

Receive fewer than five meals 34.5 55.0 -20.5***

Receive five or more meals 49.7 53.0 -3.4

Source: AoA NSP outcomes survey, 2015-2016, weighted data, Table IV.6.

***/**/*Difference between participants and nonparticipants is significantly different from zero at the 0.01/0.05/0.10 level.
37

Diet Quality Analysis
38

Background on the Diet Quality Analysis

• Objectives of the analysis

– Describe the quality of participants’ diets

– Determine the impact of participation on diet quality

• 24-hour dietary recall data 

– Collected detailed information on all foods and beverages 

consumed during preceding 24 hours

– Subset of participants and nonparticipants completed 2nd recall 

– Provide data on the amounts of nutrients and food groups 

consumed over 24 hours
39

Outcome Measure: Diet Quality

• Percentage contribution program 

meals made to participants’ 

nutrient intakes

• Usual intakes of vitamins, 

minerals, and macronutrients 

relative to recommendations

• Healthy Eating Index-2010 scores 

(HEI-2010) to assess overall diet 

quality
40

Contribution of Program Meals to Participants’ 

Daily Nutrient Intakes

• Identified foods consumed from program meals versus 

other sources

• Both congregate and home-delivered meals 

contributed substantially to participants’ diets

• Program meals made largest contributions to 

participants’ intakes of protein, vitamin C, vitamin A,

alpha-linolenic acid, and sodium

Source: AoA NSP 24-hour dietary recall (Day 1), 2015-2016, weighted data, Table III.9.

. CM participants HDM participants

Percentage of daily calories 41 38

Percentage of daily nutrients 39 to 47 35 to 47
41

Assessing Whether Participants’ Usual Nutrient 

Intakes Met Recommendations

• Federal nutrition standards provide recommendations 

for amounts of nutrients to consume

– Dietary Reference Intakes

– Dietary Guidelines for Americans

• Nutrient recommendations should be met over time 

and applied to measures of usual intake

• Estimated usual nutrient intakes using method 

developed by the National Cancer Institute 

– Provides estimates of the percentage of participants with usual 

nutrient intakes that met recommendations
42

Usual Intakes of Vitamins and Minerals
43

Usual Intakes of Vitamins and Minerals (cont.)
44

Usual Intakes of Macronutrients 
45

Usual Intakes of Saturated Fat and Sodium
46

Healthy Eating Index-2010

• Diet quality index that assesses conformance to the 

2010 Dietary Guidelines for Americans

• Consists of 12 components and a total score

– 9 adequacy components 

• Total fruit

• Whole fruit

• Total vegetables

• Greens and beans

• Whole grains

• Dairy 

• Total protein foods

• Seafood and plant proteins

• Fatty acids

– 3 moderation components 

• Refined grains

• Sodium

• Empty calories
47

Healthy Eating Index-2010 (cont.)

• HEI-2010 scoring

– Each component has a maximum score 

– Scores assigned based on amounts of foods and calories 

consumed 

– Total score is sum of component scores 

• Higher scores indicate better conformance with Dietary 

Guidelines recommendations and higher diet quality 

• Estimated mean HEI-2010 scores using method 

developed by the National Cancer Institute 

– Scores are expressed as percentage of maximum possible score
48

Total HEI-2010 Scores 
49

HEI-2010 Scores for Adequacy Components
52

HEI-2010 Scores for Moderation Components
53

Effects of Program Participation on 

Diet Quality Outcomes 
54

Effects of CM Program Participation on 

Usual Nutrient Intakes
55

Effects of HDM Program Participation on 

Usual Nutrient Intakes
56

Effects of CM and HDM Participation on 

Overall Diet Quality 

• CM participants had significantly higher HEI-2010 

scores than nonparticipants for: 

– Total HEI-2010 score (66 versus 59 percent)

– Total fruit (97 versus 72 percent)

– Dairy (69 versus 57 percent)

– Total vegetables (90 versus 78 percent)

– Refined grains (78 versus 60 percent)

• HDM participants had significantly higher HEI-2010 

scores than nonparticipants for: 

– Dairy (72 versus 58 percent)

– Refined grains (74 versus 64 percent)

Source: AoA NSP 24-hour dietary recall (Day 1), 2015-2016, weighted data, Table IV.8.

Note: All differences between participants and nonparticipants were significantly different from zero at the 0.10 level or lower.
57

Conclusion
58

Summary of Evaluations’ Findings for Key Outcomes

Outcome 2016

Evaluation

1995 

Evaluation

Congregate meal program . .

Participants had greater food security than nonparticipants Not 

measured

Participants had higher levels of socialization than 

nonparticipants

Participants had higher diet quality than nonparticipants. Program 

meals made substantial contribution to participants’ diets

Home-delivered meal program . .

Participants had similar food security as nonparticipants No effect Not 

measured

Participants had similar levels of socialization as nonparticipants Mixed

Participants had higher diet quality than nonparticipants. Program 

meals made substantial contribution to participants’ diets



 

 



 
59

Thank You!

• Mathematica extends our sincere thanks to all of the SUA, AAA, 

and LSP staff who completed study surveys, provided data for the 

meal cost analysis, and helped facilitate a successful outcomes 

survey

• Holly Greuling (ACL/AoA National Nutritionist)

– Holly.Greuling@acl.hhs.gov

• Heather Menne (ACL/AoA Project Officer)

– Heather.Menne@acl.hhs.gov

• Susan Jenkins (ACL/AoA Director of Performance and Evaluation)

– Susan.Jenkins@acl.hhs.gov

• James Mabli (Evaluation Project Director) 

– JMabli@mathematica-mpr.com

• Liz Gearan (Evaluation Co-Principal Investigator) 

– LGearan@mathematica-mpr.com
60

Evaluation Reports

• Process study report

– http://www.acl.gov/sites/default/files/programs/2017-02/NSP-Process-Evaluation-Report.pdf

• Cost study report

– http://www.acl.gov/sites/default/files/programs/2017-02/NSP-Meal-Cost-Analysis.pdf

• First outcomes evaluation report

– http://www.acl.gov/sites/default/files/programs/2017-07/AoA_outcomesevaluation_final.pdf

• Nutritional quality of program meals issue brief (anticipated fall 2017)

• Second outcomes evaluation report (anticipated summer 2018)

– Present participants’ healthcare utilization and behavior characteristics

– Estimate effect of participation on hospital admissions and readmissions, emergency 

department visits, primary care physician visits, home health episodes, admittance to a 

skilled nursing facility, admittance to a nursing home, and total Medicare costs