JBS INTERNATIONAL INC. MEMORANDUM 1 | Page

Contract Number: CNSHQ14A0007-95332A18F-0023

Contract Title: Senior Corps Systematic Evidence Review of Agency Supported 

Programs 

Submitted To: Anthony Nerino 

Date: July 31, 2019 

Subject: Senior Corps Meta-Analysis Feasibility

Introduction

Meta-analysis can provide important insights about effectiveness across programs and 

interventions. Usually derived from a systematic review, meta-analysis is a useful tool for 

assessing whether an evidence base offers quantitative evidence of effect with more power and 

accuracy than can be obtained from individual studies.1 JBS systematically reviewed the 

evidence base of Senior Corps programs to determine the extent to which a meta-analysis is

possible. Through this systematic review and feasibility analysis, we found that it would be 

technically possible to conduct a meta-analysis with a limited number of outcomes (17 outcomes 

from 6 studies that have the necessary data elements: sample sizes, directionality, and p-values). 

However, the notable shortage of rigorous evaluation studies indicates that a meta-analysis 

should not be conducted at this time. 

Methods

JBS identified the hallmarks of a quality meta-analysis and defined precisely what would be 

needed to make a meta-analysis of Senior Corps programs feasible. To complete our assessment, 

we relied on the results from our systematic review of 27 Senior Corps impact and outcome 

evaluation studies from 1980-2019. The results of the systematic review are examined in the 

corresponding report, A Systematic Review of Senior Corps’ Impact of Volunteers and Program 

Beneficiaries. The 27 studies selected for inclusion in the systematic review and report met 

certain criteria for recency and quality (see Figure 1). 

The systematic review process extracted key

information about study design, program type, 

participant type, and outcome data, which included

statistical significance and directionality. The 

systematic review considered the quality of the 

studies and categorized them by study design. 

Specifically, if rigorous study designs did not 

address threats to internal validity, they were 

categorized as non-experimental. 

1 Ahn, E. and Kang, H. Introduction to Systematic Review and Analysis. Korean Journal of Anesthesiology, 2018 Apr;

71(2): 103-112.

Figure 1. Inclusion Criteria for the 

Systematic Review

• Conducted between 1980 and the Present

• Evaluation report with findings (no theory, 

review, or design papers)

• Focused on Senior Corps volunteers or 

beneficiaries (excluding Experience Corps)

• Relevant to the report’s research questions

• An impact or outcome evaluation (no 

implementation-only reports)
JBS INTERNATIONAL INC. MEMORANDUM 2 | Page

We compiled 110 outcomes from the 27 studies included in the Senior Corps systematic review 

and report into one dataset. The information in the dataset as well as additional information from 

individual evaluation reports, such as exact sample sizes, were used to determine the presence of

sufficient data for a meta-analysis. We also assessed the extent to which effect sizes were already 

provided in evaluation reports and whether certain outcomes have an evidence base more suited 

to an eventual meta-analysis than others. 

Hallmarks of a Quality Meta-Analysis

A quality meta-analysis is derived from a systematic review of studies collected on a particular 

topic. Typically, the more information across the topic and within individual studies, the more 

thorough the meta-analysis. Meta-analysis is not usually just one analysis, but a series of metaanalyses examining different groups of outcomes that have similar characteristics (e.g., outcome 

types, program type, or beneficiary type). Using advanced statistical techniques, meta-analyses 

provide information, such as average effect size, and identify moderating variables that cause 

variation in effect sizes across studies. Measures of effect can be calculated and reported by 

several standard approaches, including Cohen’s d, Hedges’ g, eta-squared, r, and r-squared. 

Typically, the effect size is considered very small if it is below 0.20, small between 0.20 and 

0.50, medium between 0.50 and 0.80, and large if it is greater than 0.80. A meta-analysis that 

calculates overall effect sizes using multiple data points is generally a reliable source of 

information about effectiveness across interventions.2 

Since there are multiple ways to conduct meta-analyses, different types of data are required for 

different meta-analysis approaches. The data used is generally extracted from final evaluation 

reports, and the type of meta-analysis run is dependent on the amount of data reported. Metaanalyses can be conducted using information such as statistical significance, directionality, 

sample sizes, variance, and/or standard errors. The meta-analysis technique that requires the data 

elements most often available in final evaluation reports is the correlation coefficient (r)

calculation converted to and back from the Fischer’s z scale (the Fischer’s z transformation). To 

undertake this particular meta-analysis, variance is not necessary since Fischer’s z score and its 

variance are used in the analysis.3 Therefore, the necessary data elements include: p-value 

(statistical significance), directionality, and sample size. Though it can be helpful if reports 

include their own effect sizes, it is not necessary.4 Although standardized effect sizes may not be 

needed to conduct meta-analyses, when an evaluation report includes effect sizes, it is highly 

likely to have enough data elements available to use in a meta-analysis.

A useful meta-analysis must be able to rely on the quality of the studies included. The studies 

must be rigorous, such as a quasi-experimental design (QED) that, ideally, uses a comparison 

2 Leppink, Jimmie; O’Sullivan, Patricia; and Winston, Kal. Effect size – large, medium, and small. Perspectives on 

Medical Education, 2016 Dec; 5(6): 347–349. 3 Comprehensive Meta-Analysis User’s Manual. Source: https://www.meta-analysis.com/downloads/Metaanalysis%20Effect%20sizes%20based%20on%20correlations.pdf 4 As the evidence base expands, it may be helpful to encourage the inclusion of standardized effect sizes. 

Evaluations that provide standardized effect sizes help strengthen evidence of effectiveness even without metaanalysis and provide a helpful quick reference of effect. 
JBS INTERNATIONAL INC. MEMORANDUM 3 | Page

group or a randomized controlled trial (RCT) and must have addressed any potential bias, threats 

to internal validity, or methodological issues. The systematic review is a useful tool for 

identifying the strongest studies with high internal validity, which generally produce more 

reliable results. The outcomes from these studies can be positive or negative, but statistical 

significance is an important factor. 

Meta-analyses commonly assess results by outcome domain. Meta-analyses can be run using 

multiple types of groupings, such as by Senior Corps program or Senior Corps overall. But, the 

information gleaned from outcome analyses within a single Senior Corps program and type of 

outcome offer the most specific and potentially useful information, such as the effect of FGP 

programs on pre-K school readiness. Additionally, using the same participant type is important 

to get an accurate assessment of the evidence base. For example, caregiver outcomes should be 

separated from volunteer outcomes. Within such specific groups, researchers can also identify 

moderating variables to determine how another factor of the study, such as study design, effects 

the evidence. The number of outcomes can be small to run a meta-analysis; in fact, it is possible 

to run a meta-analysis with only two outcomes, but a larger sample is better. Finally, the 

outcomes should not all come from the same study as the results would not give an accurate 

representation of the evidence base. 

Findings

Our feasibility assessment takes into consideration the findings from the Senior Corps systematic 

reviews report, which found that while there is ample positive evidence of the effectiveness of 

Senior Corps programs, fewer outcomes are associated with rigorous studies or are statistically 

significant. Nearly two-thirds of the outcomes compiled in the systematic review (65 percent) did 

not report statistical significance. Overall, the total of 110 outcomes yielded just 17 outcomes 

from rigorous studies reporting statistical significance. 

Outcome Quality

In total, 110 outcomes from 27 studies were reviewed and included in the synthesis report. Of the 

110 outcomes, 38 included statistical significance, and 17 of those were based on rigorous 

designs with internal validity to be considered for inclusion in a meta-analysis (QEDs or RCTs). 

Regardless of whether they have effect sizes in their reports, these 17 outcomes, from 6 studies, 

could be included in a meta-analysis. However, more factors must be considered in determining 

the feasibility of a meta-analysis, including details such as sample sizes, distribution of outcomes 

across programs and study participants, and concentration of data within outcome types. See 

Table 1 for an overview of feasibility factors and the elements included. 
JBS INTERNATIONAL INC. MEMORANDUM 4 | Page

Table 1: Feasible Outcomes

Study Author(s)

Study 

Design

Progr

am

Participant 

Type

Outcome 

Type(s)

Sample 

sizes 

(treatme

nt and 

control)

pvalue

Direc

tional

ity

Effect 

sizes 

incl* 

Longitudinal Study of Foster Grandparent 

and Senior Companion Programs: Service 

Delivery Implications and Health Benefits to 

the Volunteers (2018)

JBS 

International, 

Inc.

QED

FGP Volunteer Physical 

health x x x Yes

SCP Volunteer Physical 

health x x x Yes

FGP Volunteer

Psychosocial 

health: 

depression 

and mental 

health

x x x No

SCP Volunteer

Psychosocial 

health: 

depression 

and mental 

health

x x x No

The 2013-2014 Senior Corps Study: Foster 

Grandparents and Senior Companions (2016)

JBS 

International, 

Inc.

QED

FGP Volunteer

Physical 

health x x x No

SCP Volunteer

Physical 

health x x x No

FGP Volunteer

Physical 

health x x x No

SCP Volunteer

Physical 

health x x x No

FGP Volunteer

Psychosocial 

health: life 

satisfaction

x x x No
JBS INTERNATIONAL INC. MEMORANDUM 5 | Page

Study Author(s)

Study 

Design

Progr

am

Participant 

Type

Outcome 

Type(s)

Sample 

sizes 

(treatme

nt and 

control)

pvalue

Direc

tional

ity

Effect 

sizes 

incl*

SCP Volunteer

Psychosocial 

health: life 

satisfaction

x x x No

Fostering social ties through a volunteer role 

Implications for older-adults' psychological 

health (2003)

Rook, K.S., 

& Sorkin, 

D.H.

RCT FGP Volunteer

Psychosocial 

health: 

social 

connectedne

ss

x x x Yes

FGP Volunteer

Psychosocial 

health: 

depression 

and mental 

health

x x x No

Senior Companion Program impact 

evaluation: Final report (1985)

SRA 

Technologies 

(for 

ACTION)

RCT SCP Client

Overall 

functioning 

x x x No

Volunteer 

Overall 

functioning 

x x x No

Evaluating the Observable Effects of Foster 

Grandparents on Hospitalized Children

(1982)

Suzanne 

Ziegler and 

Johanna King

RCT FGP Child

Social, 

emotional, 

behavioral 

development

x x x No

Can Senior Volunteers Deliver Reminiscence 

and Creative Activity Interventions? Results 

of the Legacy Intervention Family Enactment 

Randomized Controlled Trial (2014)

R.S. Allen et 

al. RCT RSVP Client

Psychosocial 

health: 

depression 

and mental 

health

x x x Yes
JBS INTERNATIONAL INC. MEMORANDUM 6 | Page

Study Author(s)

Study 

Design

Progr

am

Participant 

Type

Outcome 

Type(s)

Sample 

sizes 

(treatme

nt and 

control)

pvalue

Direc

tional

ity

Effect 

sizes 

incl*

Caregiver

Psychosocial 

health: 

depression 

and mental 

health

x x x No

*Optional
JBS INTERNATIONAL INC. MEMORANDUM 7 | Page

Participants 

The data in the studies identified in Table 1 include all programs and beneficiaries. A successful 

meta-analysis would, ideally, be run by group. Each group would contain outcomes with similar 

data, with separate groups for volunteers, clients, caregivers, and children. While a meta-analysis 

with Senior Corps data could be run for volunteers and, technically, for clients (since there are 

two outcomes), it could not be run for the other types of beneficiaries. While it is possible to run 

the meta-analysis for all beneficiaries together, this analysis could be misleading due to the vast 

differences between some of the beneficiaries, such as caregivers (who are often also seniors) 

and children. 

Programs

Without taking outcome type and other factors into consideration, there are a sufficient number 

of outcomes for a meta-analysis for both SCP and FGP, and most are volunteer outcomes, 

making the data relatively similar. However, while the outcomes are evenly distributed by SCP 

and FGP, there are only two RSVP outcomes. Again, it is possible to run a meta-analysis on two 

outcomes, but the outcomes are from the same study, which would be problematic since it does 

not provide enough information about the larger base, and the outcomes also differ in type and 

participant. 

Outcome Types

The ideal meta-analysis would be run using various groupings, including program and 

participant, but it would be driven by the smallest groups with the most in common. These 

groups would include outcomes in the same program, with the same participants, and the same 

outcome type. With the 17 total outcomes, few such groupings can be formed. The two outcome 

types with more than one outcome are physical health and several psycho-social outcomes, 

which include depression and mental health and life satisfaction. With three outcomes in each 

program (SCP and FGP), physical health may be the best option for meta-analysis; however, 

physical health outcomes vary widely. Although both depression/mental health and life 

satisfaction might be better candidates due to specificity, there are not enough outcomes within 

each program. An analysis within outcome types but across programs is possible, but it would 

not be ideal.

Conclusion

Overall, we determined that the available Senior Corps data are not sufficient to conduct a metaanalysis. Only a small number of usable outcomes from studies conducted with rigor have the 

necessary data elements for meta-analysis (e.g., p-values, sample sizes, etc.). To perform a metaanalysis, it would be important to have additional rigorous evaluations distributed more evenly 

across participants, programs, and outcome types. For example, RSVP lacked a sufficient 

number of outcomes for an adequate meta-analysis. In fact, the two potentially usable RSVP 

outcomes were limited to the same study and differed across both participant and outcome type. 

For these reasons, conducting a meta-analysis at this time would not provide enough useful 

information about any or all of the Senior Corps evidence base.