I

Mental Health Task-Shifting

in Community-Based Organizations 

Implementation, Impact, and Cost—

Evaluation of the Connections to Care Program 

Lynsay Ayer and Dana Schultz 

EDITORS

Michele Abbott, Dionne Barnes-Proby, Wing Yi Chan, Michael Stephan Dunbar, 

Emily Hoch, Harry H. Liu, Monique Martineau, Elie Ohana, Daniel Siconolfi,

Joshua Snoke, Clare Stevens, Vivian L. Towe 

CONTRIBUTORS
 

 

 

 

November 12, 2020

Dear Reader,

New Yorkers turn to social service providers in their neighborhoods for many forms of support. They may expect help

with a resume, connections to potential employers, safe and enriching childcare to permit them to work, or guidance 

about educational opportunities. In 2016, the Mayor’s Fund to Advance New York City, the Mayor’s Office for Economic 

Opportunity, the NYC Department of Health and Mental Hygiene, and ThriveNYC set out to assess whether combining

these services with new mental health supports would be possible, improve social service outcomes, and increase the

use of mental health services. The Connections to Care (C2C) program explored whether equipping trusted staff at social

service agencies with mental health knowledge and skills could create new opportunities for New Yorkers to access

mental health services, while also improving their economic and social outcomes. This public-private partnership was

supported by the federal Social Innovation Fund, New York City funding, and private donors.

We asked the RAND Corporation to evaluate the program’s effectiveness – through an implementation study that

examined how and whether providers integrated mental health support into their work and an impact study that 

compared the mental health and social service outcomes of participants in C2C programs with participant outcomes at

comparison organizations without C2C programs. RAND also estimated the costs of the model to both government and

non-profit providers. 

The implementation evaluation demonstrated that the community-based organizations (CBOs) participating in C2C were

able to incorporate mental health practices into their work. CBOs were able to expand the skillsets of their staff to

identify and address mental health needs and promote positive mental health. These organizations trained large

numbers of staff, implemented practices to support those staff in adopting skills, and in doing so changed the culture at 

their organizations. This implementation evaluation shows that our communities have valuable assets that can be 

deployed to meet residents’ mental health needs.

The impact evaluation focused on C2C participants who screened positive for mental health needs. Overall, the 

researchers did not find that these C2C participants experienced different mental health or social service outcomes than

New Yorkers who accessed services at a comparison CBO. However, the impact evaluation did find a positive effect on

mental health and social services outcomes for some groups included in this study. Specifically:

• Expecting parents and parents of young children who were engaged in C2C were three times less likely to use 

emergency services and inpatient care than the comparison group. 

• C2C participants in youth development programs were three times more likely to seek outpatient care than the 

comparison group. 

• C2C participants in workforce development CBOs saw greater increase in hours worked per week than the

comparison group. 

Based on these findings, RAND researchers concluded that CBOs are capable and well-suited to integrate mental health 

support into their usual services, when connected to the right expertise. RAND also offers recommendations for how 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

practitioners can refine and evaluate the C2C model to realize improved mental health outcomes for additional groups 

and settings.

There are many challenges in implementing and measuring the impact of real-world interventions. The evaluation began

while the providers were still ramping up training and support, when the program was not fully mature. In addition, the 

study sample, by design, does not reflect the full population of 50,000 people who received C2C services, as it focused

only on those who were screened as having symptoms of mental health challenges. An additional limitation in evaluating

this complex intervention is the great variation in service delivery. Providers were asked to customize based on individual 

and community needs but the study was not able to examine differences in outcomes for participants who received 

more or fewer services. Nonetheless, the report's findings, including qualitative results from interviews and feedback

from leadership, staff and participants, can shape the implementation of C2C and similar programs that aim to increase 

mental health integration into social services.

The needs continue, and the opportunity is clear. COVID-19 has laid bare the effects of racism and inequality across New 

York City neighborhoods. The pandemic has also ushered in new mental health needs, as isolation, anxiety, fear, and

economic distress have increased across New York City. The City’s social service providers are being called on to respond

to ongoing and new mental health needs. And the lessons of C2C can help them meet those needs. 

In addition to these evaluation reports, RAND and DOHMH published a practitioner’s guide to share the core 

components of the task-sharing approach used in the C2C model. Released in May 2020, “Helpers in Plain Sight: A Guide 

to Mental Health Task-Sharing in Community-Based Organizations,” is a resource for CBOs and mental health providers 

who are interested in partnering to address mental health issues in their communities. The guide offers 

recommendations, tools, and best practices to support promoting mental health in community-based settings. 

We are excited to partner with the New York City Human Resources Administration to invest in the next version of C2C. 

Beginning in early 2021, providers of Jobs-Plus, an employment program that serves residents of the New York City

Housing Authority, will integrate mental health supports into their model. We will build and improve on and learn from

the C2C model through this initiative. 

C2C is a noteworthy example of the potential of public-private partnerships and collaboration across City agencies. We 

thank you for your interest in this evaluation. 

Sincerely,

Susan Herman

Director, Mayor’s Office of Thrive NYC

Hillary Kunins

Executive Deputy Commissioner of Mental Hygiene at 

the New York City Department of Health and Mental 

Hygiene

Matt Klein Toya Williford

Executive Director, Mayor’s Office for Economic Executive Director, Mayor’s Fund to Advance New York

Opportunity City
For more information on this publication, visit www.rand.org/t/RR3083 

Published by the RAND Corporation, Santa Monica, Calif. 

© Copyright 2020 RAND Corporation 

R® is a registered trademark. 

Cover image: Lyubov Ivanova/GettyImages 

Limited Print and Electronic Distribution Rights 

This document and trademark(s) contained herein are protected by law. This representation of RAND 

intellectual property is provided for noncommercial use only. Unauthorized posting of this publication 

online is prohibited. Permission is given to duplicate this document for personal use only, as long as it 

is unaltered and complete. Permission is required from RAND to reproduce, or reuse in another form, any of 

its research documents for commercial use. For information on reprint and linking permissions, please visit 

www.rand.org/pubs/permissions. 

The RAND Corporation is a research organization that develops solutions to public policy challenges to help make 

communities throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit, 

nonpartisan, and committed to the public interest. 

RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors. 

Support RAND 

Make a tax-deductible charitable contribution at 

www.rand.org/giving/contribute 

www.rand.org 
 

Preface

Mental health and substance use problems affect at least one in five adults in the United 

States, and yet access to treatment is not guaranteed. Some of our most vulnerable communities, 

including low-income communities of color, face the greatest challenges finding evidence-based 

services. In 2016, New York City (NYC) policymakers came together with federal and private 

funders to support an innovative program intended to open doors to quality mental health support 

within these high-risk communities. The program, called Connections to Care (C2C), used a 

mental health task-shifting approach to integrate evidence-informed mental health supports into 

the work of community-based organizations (CBOs) serving low-income communities in NYC. 

Partnering with local mental health providers (MHPs), 15 CBOs served more than 40,000 New 

Yorkers over the course of 4 years. RAND Corporation conducted an evaluation of C2C, 

examining its implementation across 15 organizations, its effect on mental health and other 

outcomes, and its costs. This report presents the results of that evaluation. The results should be 

of interest to CBOs, MHPs, health policymakers, and communities struggling to overcome 

barriers to mental health care. 

Data collection for this study began in 2017 and ended in February 2020, just before the 

COVID-19 pandemic led to stay-at-home orders and shutdowns of nonessential businesses in 

NYC. Specific activities included a literature review and development of a logic model; site 

visits and qualitative interviews with CBO and MHP staff and clients; a survey of CBO staff 

involved in the C2C program; collection of CBO financial reports and other spending information; 

and a three-wave, quasi-experimental study of C2C and comparison clients. 

At the time of this writing, the United States was the epicenter of the COVID-19 pandemic, 

where these same communities were—largely due to systemic, social inequalities— 

disproportionately at risk of contracting and dying from the virus. This was the case in NYC, 

one of the initial virus hot spots and where the C2C program was implemented and tested. 

Leveling the playing field when it comes to health care access (including mental health care) was 

more critical than ever. Findings from the evaluation of C2C—a program specifically designed 

to address this problem—could help inform communities inside and outside NYC as they 

considered new ways to bring mental health support to their most vulnerable members. 

This research was sponsored by the Mayor’s Fund to Advance NYC under the contract 

“Evaluation and Research Services Relating to the Connections to Care Initiative” and carried 

out within the Access and Delivery Program in RAND Health Care. The Mayor’s Fund received 

funding through the Social Innovation Fund (SIF), which was a program that received funding 

from 2010 to 2016 from the Corporation for National and Community Service, a federal agency 

that engages millions of Americans in service through its AmeriCorps, Senior Corps, and 

Volunteer Generation Fund programs, and leads the nation’s volunteer and service efforts. Using 

iii 
public and private resources to find and grow community-based nonprofits with evidence of 

results, SIF intermediaries received funding to award subgrants that focus on overcoming 

challenges in economic opportunity, healthy futures, and youth development. Although CNCS 

made its last SIF intermediary awards in fiscal year 2016, SIF intermediaries will continue to 

administer their subgrant programs until their federal funding is exhausted. 

RAND Health Care, a division of RAND, promotes healthier societies by improving health 

care systems in the United States and other countries. We do this by providing health care 

decisionmakers, practitioners, and consumers with actionable, rigorous, objective evidence to 

support their most complex decisions. For more information, see www.rand.org/health-care, or 

contact 

RAND Health Care Communications 

1776 Main Street 

P.O. Box 2138 

Santa Monica, CA 90407-2138 

(310) 393-0411, ext. 7775 

RAND_Health-Care@rand.org 

iv 
 

Study Directors 

Lynsay Ayer 

Clare Stevens 

Vivian L. Towe 

Management Team Implementation Evaluation Team 

Lynsay Ayera Michael Stephan Dunbara 

Emily Hoch Wing Yi Chana 

Susan Lovejoy Michele Abbott 

Clare Stevens Dionne Barnes-Proby 

Vivian L. Towe Emily Hoch 

Sarah Lieff 

Lisa Wagner 

Rebecca Weir 

Impact Evaluation Team Cost Evaluation Team 

Dana Schultza Harry Liua 

Daniel Siconolfia Michele Abbott 

Dionne Barnes-Proby Thomas Goughner 

Boyer Bazelais 

Dianne Egelhoff Technical Assistance Team 

Naomi Hale Clare Stevensa 

Emily Hoch Andrew Cleeka 

Polina Kats-Kariyanakatte Dionne Barnes-Proby 

Serafina Lanna Lydia Franco 

Rebecca Lawrence Priya Gopalan 

Susan Lovejoy 

Ammarah Mahmud Communications Support 

Elie Ohana Monique Martineau 

Robert Reynoso 

Massiel Ubilus Rivera 

Ricardo Sanchez 

Joshua Snoke 

a Team leader. 

v 
 

Contents

Preface............................................................................................................................................iii 

Summary....................................................................................................................................... xx 

Acknowledgments...................................................................................................................... xxxi 

Abbreviations............................................................................................................................ xxxii 

Figures............................................................................................................................................ ix 

Tables............................................................................................................................................. xi 

PART I. INTRODUCTION..................................................................................................................................1 

1. Introduction................................................................................................................................. 2 

Consequences of Untreated Mental Health Problems ................................................................ 2 

Access to Mental Health Care .................................................................................................... 3 

Increasing Access to Mental Health Care Through C2C............................................................ 4 

Evaluating the C2C Program ...................................................................................................... 5 

2. Background and Rationale for C2C............................................................................................ 6 

Mental Health Service Utilization in Low-Income Populations................................................. 6 

Barriers to Mental Health Care................................................................................................... 8 

Mental Health Task-Shifting ...................................................................................................... 8 

3. C2C Program and Logic Model................................................................................................ 13 

Program Overview.................................................................................................................... 13 

Building the Program................................................................................................................ 14 

Logic Model.............................................................................................................................. 17 

C2C Evaluation......................................................................................................................... 21 

References................................................................................................................................. 25 

PART II. IMPLEMENTATION ..........................................................................................................................36 

4. Adaptation of C2C at Community-Based Organizations in Early-Stage Implementation ....... 37 

Key Findings............................................................................................................................. 37 

Introduction............................................................................................................................... 38 

Methods .................................................................................................................................... 41 

Results....................................................................................................................................... 44 

Discussion................................................................................................................................. 63 

Limitations................................................................................................................................ 65 

Summary................................................................................................................................... 65 

References................................................................................................................................. 66 

5. Evolution of C2C Implementation............................................................................................ 67 

Key Findings............................................................................................................................. 67 

vi 
Introduction............................................................................................................................... 68 

Methods .................................................................................................................................... 69 

Results....................................................................................................................................... 71 

Discussion............................................................................................................................... 108 

Limitations.............................................................................................................................. 112 

Summary................................................................................................................................. 113 

References............................................................................................................................... 114 

6. How C2C Transformed Organizational Culture and Approaches to Client 

Care Coordination................................................................................................................. 116 

Key Findings........................................................................................................................... 116 

Introduction............................................................................................................................. 117 

Methods .................................................................................................................................. 118 

Results..................................................................................................................................... 119 

Discussion............................................................................................................................... 140 

Limitations.............................................................................................................................. 144 

Summary................................................................................................................................. 145 

References............................................................................................................................... 146 

PART III. IMPACT .......................................................................................................................................148 

7. Impact of C2C on Mental Health Care Access and Utilization.............................................. 149 

Key Findings........................................................................................................................... 149 

Introduction............................................................................................................................. 150 

Methods .................................................................................................................................. 152 

Results..................................................................................................................................... 159 

Discussion............................................................................................................................... 169 

Limitations.............................................................................................................................. 175 

Summary................................................................................................................................. 176 

References............................................................................................................................... 177 

8. Impact of C2C on Mental Health Symptoms.......................................................................... 181 

Key Findings........................................................................................................................... 181 

Introduction............................................................................................................................. 182 

Methods .................................................................................................................................. 184 

Results..................................................................................................................................... 185 

Discussion............................................................................................................................... 191 

Limitations.............................................................................................................................. 195 

Summary................................................................................................................................. 196 

References............................................................................................................................... 197 

9. Impact of C2C on Employment, Education, Housing, and Incarceration .............................. 200 

Key Findings........................................................................................................................... 200 

vii 
 

Introduction............................................................................................................................. 201 

Method.................................................................................................................................... 204 

Results..................................................................................................................................... 205 

Discussion............................................................................................................................... 213 

Limitations.............................................................................................................................. 218 

Summary................................................................................................................................. 219 

References............................................................................................................................... 220 

PART IV: COST ...........................................................................................................................................224 

10. Resources Required to Implement and Maintain the C2C Program..................................... 225 

Key Findings........................................................................................................................... 225 

Introduction............................................................................................................................. 226 

Methods .................................................................................................................................. 227 

Results..................................................................................................................................... 234 

Discussion............................................................................................................................... 249 

Limitations.............................................................................................................................. 254 

Summary................................................................................................................................. 255 

References............................................................................................................................... 257 

PART V. CONCLUSIONS AND RECOMMENDATIONS ...................................................................................258 

11. Conclusions and Recommendations ..................................................................................... 259 

Conclusions............................................................................................................................. 260 

Recommendations................................................................................................................... 263 

Final Thoughts ........................................................................................................................ 266 

References............................................................................................................................... 267 

Appendix A. C2C Program Summaries...................................................................................... 269 

Appendix B. Implementation Evaluation Methods .................................................................... 326 

Appendix C. Impact Evaluation Methods................................................................................... 338 

Appendix D. Cost Evaluation and Sensitivity Analyses............................................................. 461 

viii 
 

 

 

 

Figures

Figure 1.1. C2C Evaluation and Data Collection Timeline ............................................................ 5 

Figure 3.1. C2C Program Logic Model ........................................................................................ 20 

Figure 4.1. Key C2C Program Implementation Events from 2016 Through 2019....................... 39 

Figure 4.2. Conceptual Model of C2C Program Implementation Phases in CBO Settings.......... 40 

Figure 5.1. CBO Staff and Supervisors Trained in C2C Skills .................................................... 74 

Figure 5.2. CBO Staff Trained by C2C Skills .............................................................................. 75 

Figure 5.3. C2C Skills Trainings Received by CBO Staff ........................................................... 75 

Figure 5.4. CBO Staff Trained in Each C2C Skill (Year 4) ......................................................... 76 

Figure 5.5. Perceptions of C2C Program Functioning by CBO Type .......................................... 81 

Figure 5.6. Coaching and Supervision Hours by Core C2C Skill ................................................ 82 

Figure 5.7. New C2C Program Clients Served per Quarter, by Year........................................... 88 

Figure 5.8. Average C2C Program Clients Served per CBO, by CBO Type ............................... 89 

Figure 5.9. Average Percent of Annual New Client Targets Met per CBO, by CBO Type......... 91 

Figure 5.10. Eligible CBO Clients Who Received at Least One C2C Skill ................................. 91 

Figure 5.11. Use of C2C Skills by CBO Program Staff ............................................................... 92 

Figure 5.12. Clients Screened for Mental Health Symptoms per Quarter.................................... 93 

Figure 5.13. Clients Screened per CBO, by CBO Type ............................................................... 94 

Figure 5.14. Screenings and Other C2C Services Delivered to CBO Clients .............................. 96 

Figure 5.15. Number of Screenings for Mental Health Symptoms, Overall and by Type ........... 97 

Figure 5.16. Client Accepted Referrals and Kept Referrals by Type ......................................... 101 

Figure 5.17. Percentage of Kept Referrals to MHPs .................................................................. 102 

Figure 5.18. Client Referrals by CBO Type ............................................................................... 103 

Figure 6.1. CBO Staff Satisfaction with C2C Training (Year 4)................................................ 120 

Figure 6.2. Organizational Support for Addressing Client Mental Health................................. 127 

Figure 6.3. Staff Perceptions of Community Mental Health Stigma by Wave........................... 130 

Figure 6.4. Staff Perception of Why Clients Decline Mental Health Referrals.......................... 131 

Figure 6.5. Staff Perception of Client Comfort with Accepting Referrals ................................. 134 

Figure 10.1. Program-Level Cost Data Collection ..................................................................... 228 

Figure 10.2. Average Number of Staff Members Who Received C2C Training per CBO, 

by Job Category, C2C Project Year, and CBO Type.......................................................... 235 

Figure 10.3. Average Number of Unique Clients Served per CBO, by CBO Type and 

Project Year ........................................................................................................................ 236 

Figure 10.4. Average Weekly C2C Labor Hours per Staff Member, by Job Category, C2C 

Activity, and Project Year................................................................................................... 238 

Figure 10.5. Average Annual C2C Labor Hours per CBO, by CBO Type and Project Year .... 239 

ix 
 

 

Figure 10.6. Average Annual Staff Compensation in Year 3, by CBO Type............................. 240 

Figure 10.7. Average Annual Labor Cost per CBO, by CBO Type and Project Year ............... 240 

Figure 10.8. Average Annual C2C Program Cost per CBO, by Cost Component, Project 

Year, and CBO Type........................................................................................................... 241 

Figure 10.9. Average Annual Program Cost per CBO, by Funding Source and Project Year... 242 

Figure 10.10. Average C2C Program Cost per Client Served, by Cost Component, Project 

Year, and CBO Type........................................................................................................... 243 

Figure 10.11. Average Program Cost per Client Served, by Cost Component, MHP 

Partnership Model, and Project Year.................................................................................. 244 

Figure 10.12. Average Program Cost per Client Served, by Cost Component, CBO 

Employment Size, and CBO Project Year.......................................................................... 245 

Figure C.1. Data Collection Process at C2C CBOs.................................................................... 350 

Figure C.2. Data Collection Process at Comparison CBOs........................................................ 352 

Figure C.3. Study Enrollment Timeline by CBO ....................................................................... 356 

Figure C.4. Impact Study Analytic Database Development Process.......................................... 359 

Figure D.1. Average Annual Program Cost per CBO Using Invoice-Based Overhead Cost 

Rates, by Cost Component, Project Year, and CBO Type ................................................. 461 

Figure D.2. Average Annual Program Cost per CBO Based on Invoices, by Cost 

Component, Project Year, and CBO Type.......................................................................... 462 

Figure D.3. Average Program Cost per Client Served Based on Invoices, by Cost 

Component, Project Year, and CBO Type.......................................................................... 463 

Figure D.4. Average Annual Program Cost per Client Served, Adjusted for Incremental 

Time Spent on Delivering CBO Services Due to Utilizing C2C Skills, by Cost 

Component and Project Year.............................................................................................. 464 

Figure D.5. Average Weekly Labor Hours per Staff Member in Year 4, by Job Category 

and C2C Activity ................................................................................................................ 465 

Figure D.6. Average Annual Labor Hours per CBO in Year 4, by CBO Type.......................... 465 

x 
 

 

 

 

Tables

Table S.1. Participating CBOs, by Type..................................................................................... xxii 

Table 3.1. Overview of CBOs Participating in the C2C Program................................................ 15 

Table 3.2 Implementation Evaluation Research Questions 

Table 5.5. Percent of Clients Who Received Any C2C Skill That Received at Least a 

.......................................................... 21 

Table 3.3. Impact Evaluation Research Questions ....................................................................... 23 

Table 3.4. Cost Evaluation Research Questions ........................................................................... 24 

Table 4.1. Required Components of C2C Program ...................................................................... 38 

Table 4.2. Number of Key Informants Interviewed by Year and Type of Informant................... 42 

Table 4.3. CBOs with Integrated and Stand-Alone Delivery of C2C Skills................................. 53 

Table 5.1. CBOs by Type ............................................................................................................. 71 

Table 5.2. Distribution of Training Provided During Year 4 (2019)............................................ 73 

Table 5.3. Annual New C2C Client Targets, by Year and CBO Type ......................................... 90 

Table 5.4. Delivery of Specific C2C Skills by Lay Staff.............................................................. 92 

Screening, by CBO Type and Project Year .......................................................................... 95 

Table 7.1. Baseline Characteristics............................................................................................. 154 

Table 7.2. Sample Sizes for Subgroup Analysis......................................................................... 156 

Table 7.3. Strength of Baseline Barriers to Seeking Professional Care for a Mental Health 

Problem—Domain Scores (Weighted) ............................................................................... 159 

Table 7.4. Number of Endorsed Baseline Barriers to Seeking Professional Care for a Mental 

Health Problem—Individual Barrier Endorsement (Weighted) ......................................... 160 

Table 7.5. Baseline Mental Health Care Utilization Among Persons with Baseline Unmet 

Need (Weighted)................................................................................................................. 161 

Table 7.6. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted)............ 162 

Table 7.7. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted)......... 163 

Table 7.8. Within-Group Mental Health Care Utilization over 1 Year Among Persons with 

Need at Baseline—Binary (Weighted) ............................................................................... 164 

Table 7.9. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous (Weighted) ..................................... 165 

Table 7.10. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Binary (Weighted)............................................. 166 

Table 7.11. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous (Weighted) ..................................... 166 

Table 7.12. Summary of Mental Health Care Barriers and Utilization Outcomes in the 

Overall Sample.................................................................................................................... 170 

Table 8.1. Clinically Significant and Reliable Improvement for Mental Health Outcomes....... 185 

xi 
 

 

 

Table 8.2. Baseline Mental Health Symptoms (Weighted) ........................................................ 186 

Table 8.3. Within-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 188 

Table 8.4. Between-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 189 

Table 8.5. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 12 Months ............................................................................. 189 

Table 8.6. Summary of Mental Health Outcome Results in the Overall Sample ....................... 192 

Table 8.7. Summary of Mental Health Status of Study Participants at Baseline........................ 193 

Table 9.1. Recall Period for Non–Mental Health Outcomes...................................................... 205 

Table 9.2. Baseline Indicators for Employment, Housing, Educational Attainment, and 

Incarceration (Weighted) .................................................................................................... 206 

Table 9.3. Within-Group Differences in Full-Time or Part-Time Employment at 

12 Months (Weighted) ........................................................................................................ 207 

Table 9.4. Within-Group Mean Changes in Hours Worked and Employment Income at 

12 Months (Weighted) ........................................................................................................ 207 

Table 9.5. Within-Group Differences in Housing Status at 12 Months (Weighted) .................. 208 

Table 9.6. Within-Group Changes in Homelessness at 12 Months (Weighted)......................... 208 

Table 9.7. Within-Group Differences in Educational Attainment at 12 Months (Weighted)..... 208 

Table 9.8. Within-Group Changes in Incarceration at 12 Months (Weighted) .......................... 209 

Table 9.9. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 12 Months (Weighted) .............................................................................. 210 

Table 9.10. Summary of Employment, Educational Attainment, Housing, and Incarceration 

Outcomes in the Overall Sample ........................................................................................ 214 

Table A.1. AAANY Training, Coaching, and Supervision ........................................................ 272 

Table A.2. BSRC Training, Coaching, and Supervision ............................................................ 275 

Table A.3. CAMBA Training, Coaching, and Supervision........................................................ 278 

Table A.4. CEO Training, Coaching, and Supervision............................................................... 281 

Table A.5. HMI Training, Coaching, and Supervision............................................................... 286 

Table A.6. Hudson Guild Training, Coaching, and Supervision................................................ 288 

Table A.7. NMIC Training, Coaching, and Supervision ............................................................ 293 

Table A.8. RHI Training, Coaching, and Supervision................................................................ 295 

Table A.9. Safe Horizon Training, Coaching, and Supervision ................................................. 301 

Table A.10. Sheltering Arms Training, Coaching, Supervision, and Continuous 

Quality Improvement.......................................................................................................... 305 

Table A.11. STRIVE Training, Coaching, and Supervision....................................................... 309 

Table A.12. The Door Training, Coaching, and Supervision ..................................................... 315 

Table A.13. HOPE Training, Coaching, and Supervision .......................................................... 319 

Table A.14. Voces Latinas Training, Coaching, and Supervision.............................................. 322 

xii 
 

 

 

 

 

Table B.1. Number of Key Informants Interviewed by Year and Type ..................................... 327 

Table B.2. Quarterly Progress Report Data Elements ................................................................ 335 

Table C.1. Comparison Group CBOs ......................................................................................... 340 

Table C.2. CBO Information ...................................................................................................... 341 

Table C.3. Minimum Detectable Effect Size with 80-Percent Power, Accounting for 

Propensity Score Adjustments ............................................................................................ 341 

Table C.4. Impact Evaluation Eligibility Criteria on Screening Measures................................. 349 

Table C.5. Balance Table for Propensity Score Weighting of Study Participants at Baseline, 

Demographic Characteristics.............................................................................................. 364 

Table C.6. Balance Table for Propensity Score Weighting of Study Participants at Baseline, 

Mental Health Symptoms ................................................................................................... 366 

Table C.7. Balance Table for Propensity Score Weighting of Retained Study Participants at 

6 Months, Baseline Demographic Characteristics.............................................................. 367 

Table C.8. Balance Table for Propensity Score Weighting of Retained Study Participants at 

6 Months, Baseline Mental Health Symptoms ................................................................... 369 

Table C.9. Balance Table for Propensity Score Weighting of Retained Study Participants at 

12 Months, Baseline Demographic Characteristics............................................................ 370 

Table C.10. Balance Table for Propensity Score Weighting of Retained Study Participants at 

12 Months, Baseline Outcomes .......................................................................................... 372 

Table C.11. Balance Table for Propensity Score Weighting of Retained Study Participants at 

Either 6 or 12 Months, Baseline Demographic Characteristics, Unmet Need Sample for 

Pooled 1-Year Utilization-Related Outcomes .................................................................... 373 

Table C.12. Balance Table for Propensity Score Weighting of Retained Study Participants at 

Either 6 or 12 Months, Baseline Outcomes, Unmet Need Sample for Pooled 1-Year 

Utilization-Related Outcomes............................................................................................. 375 

Table C.13. Final Study Enrollment ........................................................................................... 378 

Table C.14. Required Versus Actual Enrollment for a Small Effect Size.................................. 379 

Table C.15. Retention of Participants Eligible to Participate at Each Time Point ..................... 380 

Table C.16. Study Participants by Time Point............................................................................ 381 

Table C.17. Baseline Sample Characteristics by Study Retention at 6 Months (Unweighted) ... 382 

Table C.18. Baseline Sample Characteristics by Study Retention at 12 Months (Unweighted)... 384 

Table C.19. Sample Characteristics: Full Unweighted Sample at Baseline ............................... 387 

Table C.20. Sample Characteristics: Full Unweighted Subsample with Unmet Mental 

Health Need at Baseline...................................................................................................... 388 

Table C.21. Baseline Barriers to Seeking Professional Care for a Mental Health Problem— 

Individual Barrier Endorsement (Weighted) ...................................................................... 389 

Table C.22. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ........... 391 

Table C.23. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ........ 391 

xiii 
 

 

 

 

 

 

 

Table C.24. Within-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) .......................................................................................................... 392 

Table C.25. Between-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) .......................................................................................................... 392 

Table C.26. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 6 Months ............................................................................... 393 

Table C.27. Within-Group Differences in FT/PT Employment at 6 Months (Weighted).......... 394 

Table C.28. Within-Group Mean Changes in Hours Worked and Employment Income at 

6 Months (Weighted) .......................................................................................................... 394 

Table C.29. Within-Group Differences in Housing at 6 Months (Weighted) ............................ 394 

Table C.30. Within-Group Differences in Educational Attainment at 6 Months (Weighted).... 394 

Table C.31. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 6 Months (Weighted) ................................................................................ 395 

Table C.32. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ........... 396 

Table C.33. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted) ......... 396 

Table C.34. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ........ 397 

Table C.35. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted) ...... 397 

Table C.36. Within-Group Mental Health Care Utilization over 1 Year Among Persons 

with Need at Baseline—Binary (Weighted) ....................................................................... 398 

Table C.37. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous (Weighted) ..................................... 398 

Table C.38. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Binary (Weighted)............................................. 399 

Table C.39. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous (Weighted) ..................................... 399 

Table C.40. Within-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) .......................................................................................................... 400 

Table C.41. Within-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 400 

Table C.42. Between-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) .......................................................................................................... 401 

Table C.43. Between-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 401 

Table C.44. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 6 Months ............................................................................... 402 

Table C.45. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 12 Months ............................................................................. 402 

Table C.46. Within-Group Differences in FT/PT Employment at 6 Months (Weighted).......... 403 

xiv 
 

 

 

 

 

 

 

 

Table C.47. Within-Group Mean Changes in Hours Worked and Employment Income at 

6 Months (Weighted) .......................................................................................................... 403 

Table C.48. Within-Group Differences in FT/PT Employment at 12 Months (Weighted)........ 403 

Table C.49. Within-Group Mean Changes in Hours Worked and Employment Income at 

12 Months (Weighted) ........................................................................................................ 404 

Table C.50. Within-Group Differences in Housing at 6 Months (Weighted) ............................ 404 

Table C.51. Within-Group Differences in Housing at 12 Months (Weighted) .......................... 404 

Table C.52. Within-Group Changes in Homelessness over 1 Year (Weighted) ........................ 404 

Table C.53. Within-Group Differences in Educational Attainment at 6 Months (Weighted).... 405 

Table C.54. Within-Group Differences in Educational Attainment at 12 Months (Weighted)... 405 

Table C.55. Within-Group Changes in Incarceration over 1 Year (Weighted).......................... 405 

Table C.56. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 6 Months (Weighted) ................................................................................ 406 

Table C.57. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 12 Months and 1 Year (Weighted)............................................................ 407 

Table C.58. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ........... 408 

Table C.59. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted) ......... 408 

Table C.60. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ........ 409 

Table C.61. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted) ...... 409 

Table C.62. Within-Group Mental Health Care Utilization over 1 Year Among Persons 

with Need at Baseline—Binary (Weighted) ....................................................................... 410 

Table C.63. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous (Weighted) ..................................... 410 

Table C.64. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Binary (Weighted)............................................. 411 

Table C.65. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous (Weighted) ..................................... 411 

Table C.66. Within-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) .......................................................................................................... 412 

Table C.67. Within-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 412 

Table C.68. Between-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) .......................................................................................................... 413 

Table C.69. Between-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 413 

Table C.70. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 6 Months ............................................................................... 414 

Table C.71. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 12 Months ............................................................................. 414 

xv 
 

 

 

 

 

 

 

 

Table C.72. Within-Group Differences in FT/PT Employment at 6 Months (Weighted).......... 415 

Table C.73. Within-Group Mean Changes in Hours Worked and Employment Income at 

6 Months (Weighted) .......................................................................................................... 415 

Table C.74. Within-Group Differences in FT/PT Employment at 12 Months (Weighted)........ 415 

Table C.75. Within-Group Mean Changes in Hours Worked and Employment Income at 

12 Months (Weighted) ........................................................................................................ 416 

Table C.76. Within-Group Differences in Housing at 6 Months (Weighted) ............................ 416 

Table C.77. Within-Group Differences in Housing at 12 Months (Weighted) .......................... 416 

Table C.78. Within-Group Changes in Homelessness over 1 Year (Weighted) ........................ 416 

Table C.79. Within-Group Differences in Educational Attainment at 6 Months (Weighted).... 417 

Table C.80. Within-Group Differences in Educational Attainment at 12 Months (Weighted)... 417 

Table C.81. Within-Group Changes in Incarceration over 1 Year (Weighted).......................... 417 

Table C.82. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 6 Months (Weighted) ................................................................................ 418 

Table C.83. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 12 Months and 1 Year (Weighted)............................................................ 419 

Table C.84. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ........... 420 

Table C.85. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted) ......... 420 

Table C.86. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ........ 421 

Table C.87. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted) ...... 421 

Table C.88. Within-Group Mental Health Care Utilization over 1 Year Among Persons 

with Need at Baseline—Binary (Weighted) ....................................................................... 422 

Table C.89. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous (Weighted) ..................................... 422 

Table C.90. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Binary (Weighted)............................................. 423 

Table C.91. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous (Weighted) ..................................... 423 

Table C.92. Within-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) ......................................................................................................... 424 

Table C.93. Within-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 424 

Table C.94. Between-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) .......................................................................................................... 425 

Table C.95. Between-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 425 

Table C.96. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 6 Months ............................................................................... 426 

xvi 
 

 

 

 

 

 

 

 

Table C.97. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 12 Months ............................................................................. 427 

Table C.98. Within-Group Differences in FT/PT Employment at 6 Months (Weighted).......... 428 

Table C.99. Within-Group Mean Changes in Hours Worked and Employment Income at 

6 Months (Weighted) .......................................................................................................... 428 

Table C.100. Within-Group Differences in FT/PT Employment at 12 Months (Weighted)...... 428 

Table C.101. Within-Group Mean Changes in Hours Worked and Employment Income at 

12 Months (Weighted) ........................................................................................................ 428 

Table C.102. Within-Group Differences in Housing at 6 Months (Weighted) .......................... 429 

Table C.103. Within-Group Differences in Housing at 12 Months (Weighted) ........................ 429 

Table C.104. Within-Group Changes in Homelessness over 1 Year (Weighted) ...................... 429 

Table C.105. Within-Group Differences in Educational Attainment at 6 Months (Weighted) ... 429 

Table C.106. Within-Group Differences in Educational Attainment at 12 Months (Weighted)... 430 

Table C.107. Within-Group Changes in Incarceration over 1 Year (Weighted)........................ 430 

Table C.108. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 6 Months (Weighted) ................................................................................ 431 

Table C.109. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 12 Months and 1 Year (Weighted)............................................................ 432 

Table C.110. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ......... 433 

Table C.111. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted) ....... 433 

Table C.112. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ...... 434 

Table C.113. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted) .... 434 

Table C.114. Within-Group Mental Health Care Utilization over 1 Year Among Persons 

with Need at Baseline—Binary (Weighted) ....................................................................... 435 

Table C.115. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous (Weighted) ..................................... 435 

Table C.116. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Binary (Weighted)............................................. 436 

Table C.117. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous Weighted........................................ 436 

Table C.118. Within-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) .......................................................................................................... 437 

Table C.119. Within-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 437 

Table C.120. Between-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) .......................................................................................................... 438 

Table C.121. Between-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 438 

xvii 
 

 

 

 

 

 

 

 

Table C.122. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 6 Months ............................................................................... 439 

Table C.123. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 12 Months ............................................................................. 439 

Table C.124. Within-Group Differences in FT/PT Employment at 6 Months (Weighted)........ 440 

Table C.125. Within-Group Mean Changes in Hours Worked and Employment Income at 

6 Months (Weighted) .......................................................................................................... 440 

Table C.126. Within-Group Differences in FT/PT Employment at 12 Months (Weighted)...... 440 

Table C.127. Within-Group Mean Changes in Hours Worked and Employment Income at 

12 Months (Weighted) ........................................................................................................ 441 

Table C.128. Within-Group Differences in Housing at 6 Months (Weighted) .......................... 441 

Table C.129. Within-Group Differences in Housing at 12 Months Weighted ........................... 441 

Table C.130. Within-Group Changes in Homelessness over 1 Year (Weighted) ...................... 441 

Table C.131. Within-Group Differences in Educational Attainment at 6 Months (Weighted) ... 442 

Table C.132. Within-Group Differences in Educational Attainment at 12 Months (Weighted)... 442 

Table C.133. Within-Group Changes in Incarceration over 1 Year (Weighted)........................ 442 

Table C.134. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 6 Months (Weighted) ................................................................................ 443 

Table C.135. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 12 Months and 1 Year (Weighted)............................................................ 444 

Table C.136. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ......... 445 

Table C.137. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted) ....... 445 

Table C.138. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted) ...... 446 

Table C.139. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted) .... 446 

Table C.140. Within-Group Mental Health Care Utilization over 1 Year Among Persons 

with Need at Baseline—Binary (Weighted) ....................................................................... 447 

Table C.141. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous (Weighted) ..................................... 447 

Table C.142. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Binary (Weighted)............................................. 448 

Table C.143. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year 

Among Persons with Need at Baseline—Continuous (Weighted) ..................................... 448 

Table C.144. Within-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) .......................................................................................................... 449 

Table C.145. Within-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 449 

Table C.146. Between-Group Mean Changes in Mental Health Symptoms at 

6 Months (Weighted) .......................................................................................................... 450 

xviii 
 

 

 

 

Table C.147. Between-Group Mean Changes in Mental Health Symptoms at 

12 Months (Weighted) ........................................................................................................ 450 

Table C.148. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 6 Months ............................................................................... 451 

Table C.149. Clinically Significant/Reliable Change for Mental Health Symptoms and 

Group-Level Comparison at 12 Months ............................................................................. 452 

Table C.150. Within-Group Differences in FT/PT Employment at 6 Months (Weighted)........ 453 

Table C.151. Within-Group Mean Changes in Hours Worked and Employment Income at 

6 Months (Weighted) .......................................................................................................... 453 

Table C.152. Within-Group Differences in FT/PT Employment at 12 Months (Weighted)...... 453 

Table C.153. Within-Group Mean Changes in Hours Worked and Employment Income at 

12 Months (Weighted) ........................................................................................................ 453 

Table C.154. Within-Group Differences in Housing at 6 Months (Weighted) .......................... 454 

Table C.155. Within-Group Differences in Housing at 12 Months (Weighted) ........................ 454 

Table C.156. Within-Group Changes in Homelessness over 1 Year (Weighted) ...................... 454 

Table C.157. Within-Group Differences in Educational Attainment at 6 Months (Weighted) ... 454 

Table C.158. Within-Group Differences in Educational Attainment at 12 Months (Weighted)... 455 

Table C.159. Within-Group Changes in Incarceration over 1 Year (Weighted)........................ 455 

Table C.160. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 6 Months (Weighted) ................................................................................ 456 

Table C.161. Between-Group Differences in Employment, Housing, Education, and 

Incarceration at 12 Months and 1 Year (Weighted)............................................................ 457 

xix 
 

 

 

Summary

Monique Martineau, Lynsay Ayer, Dana Schultz, Michael Stephan Dunbar, and Harry H. Liu 

Mental health problems affect a substantial portion of the U.S. population: National surveys 

estimate that every year about 20 percent of Americans deal with mental health problems such 

as depression or anxiety. Among young adults (age 18–25) that figure is even higher, at just 

more than 25 percent. Yet the distribution of these problems is uneven across segments of the 

U.S. population. Mental health problems disproportionately affect low-income individuals, racial 

and ethnic minorities, and those with low English proficiency in comparison to other populations 

in the United States. Left unaddressed, mental health problems can profoundly affect people’s 

lives, from their ability to engage in healthy relationships and secure employment, to their ability 

to care for themselves physically and emotionally. Mental health disorders represent a serious 

public health problem in the United States with a societal cost approximated at $200 billion per 

year in lost earnings. 

Easy-to-administer screening can detect mental health problems, and early treatment can stop 

or slow the progression of mild symptoms to more severe illness. But obtaining early treatment 

can be difficult because of a nationwide shortage of mental health care professionals. In both 

rural and urban areas, mental health professionals—particularly those who can deliver evidencebased treatments, such as cognitive behavioral therapy—are difficult to come by. In New York 

City (NYC) alone, the Health Resources Services Administration designated 17 areas as mental 

health provider (MHP) shortage areas in 2019. 

To fill gaps in the mental health care workforce and lower barriers to accessing mental health 

care, regional governments and coalitions have been exploring new strategies to address mental 

health care delivery. In one such effort that began in 2016, the Connections to Care (C2C) 

Collaborative in NYC built an innovative model of delivering mental health screening and 

evidence-based treatments through staff at community-based organizations (CBOs) that already 

serve low-income and at-risk populations. This report describes the C2C model and RAND 

Corporation’s evaluation of how it was implemented; whether it had the intended effect on 

mental health treatment, symptoms, and related outcomes; and how much it costs CBOs. 

The C2C Model

C2C sought to address inequality in access to mental health care and improve mental health 

and well-being by integrating mental health skills into the usual work of CBOs trusted and 

established in their NYC communities. To address access to mental health care in an environment 

with limited mental health workforce resources, the C2C model borrowed a concept of delivering 

xx 
care under constrained resources from global health initiatives called “task-shifting” and applied 

it to low-income communities in NYC. Using the task-shifting concept in this manner had 

theoretical grounding but, before C2C, had not been implemented and evaluated in CBO settings. 

Both the C2C program and its evaluations of diverse participants, implementation, and cost are a 

first in mental health interventions and research. 

At its core, the C2C program expands the mental health care workforce by pairing CBOs 

with MHP organizations to improve mental health and well-being by integrating mental health 

skills into CBO workflows. In C2C, trained and supervised lay (nonspecialist) CBO staff screen 

for mental health problems and deliver evidence-based treatments—“C2C skills”—including 

mental health first aid (MHFA), motivational interviewing (MI), and psychoeducation (PE). 

By integrating these skills into regular CBO programming and allowing CBOs to customize 

processes and procedures within the C2C framework, CBOs were able to deliver evidence-based, 

client-centered mental health care to participants. For clients with mild symptoms, CBO staff 

could deliver most C2C skills within typical client interactions, after participating in training, 

coaching, and supervision activities. But C2C was not meant to substitute for professional mental 

health care. For clients needing more intensive treatment, CBOs could facilitate a “warm handoff” 

referral to mental health professionals at their MHP partners, which were sometimes co-located 

with the CBO. 

By situating the intervention in CBOs already working with the target populations, and setting 

up mutually beneficial relationships with MHPs, C2C could leverage existing relationships with 

clients to build the trust necessary to enable conversations about mental health and stressful 

experiences. Addressing mental health challenges among CBO clients could improve outcomes 

for existing CBO programs, as well. For instance, a CBO that focuses on job training could see 

greater success with its programming by using C2C to address underlying mental health issues 

that may be barriers to employment. Further, MHPs could benefit from CBOs’ cultural expertise 

to better serve their surrounding communities. 

C2C is one initiative of ThriveNYC, an $850 million commitment by the City of New York 

to address the mental health needs of New Yorkers. The C2C Collaborative, a $30 million 

public-private partnership that includes the Mayor’s Fund to Advance NYC (Mayor’s Fund), the 

Mayor’s Office for Economic Opportunity (NYC Opportunity), the NYC Department of Health 

and Mental Hygiene (DOHMH), and federal and private donors, oversaw the design of C2C and 

its implementation. Over 5 years, the C2C Collaborative expected the program to reach up to 

40,000 New Yorkers who are clients of the CBOs and in one of the three overlapping targeted 

population groups: 

• youth/young adults ages 16 to 24 who are not in school and are not employed 

• adults age 18 or older who are unemployed or underemployed 

• parents/primary caregivers who are expecting or who have children up to the age of 4. 

xxi 
 

 

 

 

 

CBO and MHP Participants

The C2C Collaborative began its initiative with a competitive process of selecting 15 CBOs 

that serve low-income and at-risk populations (e.g., through job training and employment, 

domestic violence shelters, homeless shelters, youth development, services for immigrants) to 

participate in the C2C program (Table S.1). CBOs that applied to participate were required to 

pair with an MHP that could help provide technical assistance in the form of coaching and 

supervision, as well as direct professional services to the CBO. 

Table S.1. Participating CBOs, by Type

CBO Type CBOs

Job training and employment The HOPE Program

Bedford-Stuyvesant Restoration Corporation

Northern Manhattan Improvement Corporation

STRIVE

Center for Employment Opportunities

Youth development The Door

Red Hook Initiative

Hetrick Martin Institute

Parent/caregiver-serving Hudson Guild

Sheltering Arms

Committee for Hispanic Children and Families

Other Safe Horizon

CAMBA

Voces Latinas

Arab American Association of New York

Participating CBOs received half of their funding for C2C in years 1 through 3 from a federal 

grant from the Corporation for National and Community Service (CNCS) made to the Mayor’s 

Fund, and they were required to match these funds with nonfederal grants. CNCS closed midway 

through the 5-year program; the DOHMH provided grants to CBOs in year 4. MHP partners 

were typically paid through a contract with the CBO. 

Program Monitoring

Within the C2C model, CBOs had the following six required implementation elements: 

• establishment of a formal CBO-MHP relationship 

• plans for staffing, training, and supervision 

• engagement of specific client populations in C2C services 

• integration of C2C into existing CBO programming 

• establishment of a process for mental health referrals 

• continuous quality improvement (CQI) of C2C. 

xxii 
To monitor progress on these elements and to measure changes over time, CBOs were required 

to submit data to the Mayor’s Fund on a regular basis, such as operational plans, administrative 

data on progress toward utilization and service targets (for instance, number of staff members 

trained, rates of “kept” referrals), screening data, and invoices of expenses for both staff labor 

and material goods. CBOs received technical assistance to implement CQI and to learn how to 

use data for organizational goal setting. Because implementation of C2C progressed beyond the 

startup phase, some CBOs became more comfortable with using their data on C2C skill delivery, 

for instance, to make decisions about staff or client offerings to facilitate improvement. For 

example, CBOs and MHPs carefully monitored data on rates of kept client referrals to MHPs in 

conjunction with qualitative feedback from staff and clients to inform adaptations to referral 

protocols. Such efforts were critical in improving rates of kept referrals over time. 

CBOs and clients were also invited to participate in several surveys, focus groups, and (for 

some) key informant interviews to gauge the program’s implementation and impact at key time 

points of the evaluations. RAND used these quantitative and qualitative data to help inform 

progress updates to the CBOs and the C2C Collaborative during implementation and also to 

conduct the mixed-methods evaluations presented in this report. 

Evaluations

To measure the effects of the C2C program, the C2C Collaborative tasked RAND with 

conducting an evaluation of C2C in three parts: the implementation, impact, and cost of C2C. 

RAND used data submitted by the CBOs throughout the program and fielded surveys to CBO 

staff, leadership, and clients to gather information on experiences with the C2C program. RAND 

also conducted several focus groups and interviews with key stakeholders. 

By studying the program’s implementation, impact, and costs, RAND sought to get a holistic 

picture of program outcomes from multiple perspectives, for instance, at the individual level 

from CBO staff and C2C client participants, at the organizational level among CBOs and MHPs, 

and at the systems level in terms of health care services utilization and access to care. Below we 

summarize the main objectives and findings of each evaluation. 

Implementation

The implementation evaluation sought to understand how CBOs implemented the C2C 

program within and across CBOs, whether participation in the program changed CBO staff 

knowledge, behaviors, and attitudes about mental health, and whether CBO clients’ access to 

mental health services changed as a result of C2C. It also examined the key facilitators and 

barriers to effective implementation. The primary research questions were: 

• How were the C2C program strategies implemented? 

• To what extent have the CBOs identified clients with mental health or substance use 

issues as a result of C2C implementation? 

xxiii 
 

 

• Do CBO staff have improved knowledge of mental health and C2C modalities, as well as 

attitudes, and behaviors about mental health issues and services? 

• What are the key facilitators of and barriers to effective implementation of C2C program 

strategies within and across CBO and MHP partnerships? 

All 15 CBOs participating in C2C were included in the implementation evaluation. Data 

collection included: abstracting data from CBO operational plans; conducting three rounds of 

key informant interviews in 2017 (n = 82), 2018 (n = 158) and 2019 (n = 40) with CBO leaders, 

MHP leaders, CBO frontline staff, and CBO clients; reviewing C2C model summaries; compiling 

data on key program services and outcomes, program performance against targets, and other 

aspects of program and contract management from CBO quarterly reports; and conducting three 

waves of annual staff surveys in 2017 (n = 140), 2018 (n = 252), and 2019 (n = 320) to gain a 

broader view on program implementation from the perspective of CBO program staff who were 

trained in C2C skills and provided services to clients. For the quantitative data, we conducted 

univariate analyses (e.g., means, percentages, counts) to describe implementation measures and 

assess variability (e.g., range, standard deviations) in the metrics across CBOs and performed 

sub-analyses by CBO type. For qualitative data, we used a mixed-method software environment 

to conduct thematic analysis and identify recurring patterns, or themes, in the interview data. 

From our analyses, we found that in years 1 and 2 of C2C implementation, the C2C 

Collaborative provided detailed implementation guidance and technical assistance activities to 

help support CBOs and MHPs in developing plans for C2C implementation. CBOs had many 

factors to consider in balancing how to implement the required components of C2C while also 

tailoring the C2C program to their particular settings and clientele. These included existing 

staffing structures, resources, organizational processes, and workflows. Adapting the C2C model 

to specific CBO settings required considerable upfront planning and consideration of sitespecific staff and client cultural needs, available resources, and organizational processes. 

In total, 14 of the 15 CBOs implemented the required components of C2C within their 

organizations and remained active participants in the C2C initiative through implementation 

year 4. One CBO discontinued the program at the end of year 3 due to persistent challenges with 

meeting programmatic requirements, which resulted in the funder terminating their participation 

in C2C. Between March 2016 and December 2019, CBOs and MHPs trained more than 1,700 

CBO staff members and provided C2C services to more than 41,000 unique clients. A little more 

than 60 percent of all staff members at participating CBOs had been trained in at least one C2C 

skill by implementation year 4, with MHFA as the most commonly reported training. CBOs were 

quick to ramp up training and service delivery early on; these activities plateaued in year 2 and 

beyond, suggesting that initial C2C rollout took slightly over a year to achieve. Implementation 

of C2C also evolved over time, with CBOs/MHPs refining implementation strategies to address 

challenges and optimally align with staff, client, and organizational needs. Diligent efforts 

among CBOs and MHPs to improve referral processes ultimately led to very high rates of kept 

first appointments for clients referred to MHPs: rates surpassed 70 percent by year 3 and exceeded 

xxiv 
 

80 percent in year 4 of implementation. Also, by year 4, all organizational leaders reported 

commitment to sustaining at least some elements of C2C into the future. 

In many respects, C2C participation transformed CBO (and sometimes MHP) organizational 

cultures and approaches to client mental health. Through C2C, CBO staff and leadership 

experienced a cultural shift toward client mental health and adopted a “common language” 

with each other and with clients that was more conscious about mental health and well-being in 

general. The majority of CBO staff (77 percent) reported feeling that C2C training helped them 

address their clients’ mental health-related problems, and most (64 percent) felt satisfied with the 

continuing supervision they received to help with skill maintenance. Of the four C2C skills, CBO 

staff highlighted MI as the most effective tool to address client mental health needs and provide 

better services to clients. 

Throughout implementation, CBO staff perceived community mental health stigma to be 

relatively high and thought it was a primary barrier to clients accepting referrals to seek mental 

health services outside CBOs. Other client barriers to accepting referrals included practical 

barriers (e.g., lacking the time, transportation, and childcare to see an off-site provider, inability 

to pay, or lack of insurance), previous negative experience with mental health services, and 

general discomfort with the idea of receiving mental health services. CBOs and MHPs worked 

collaboratively to develop solutions to minimize these client barriers, such as distributing metro 

cards to clients or adding MHP walk-in hours devoted to C2C clients. 

Over time, CBOs and MHPs made adjustments to their organizational structures and policies 

to improve coordination surrounding clients’ mental health needs. The C2C model diffused both 

locally and outside the NYC area, such that CBO and MHP leaders reported sharing their 

experiences with C2C with staff members from other organizations outside the C2C network. 

CBOs shared best practices with each other, and many reported plans to extend the formal 

CBO-MHP relationship beyond the life of the C2C contract. 

Impact

The impact evaluation measured the effect of the C2C task-shifting approach on CBO 

clients’ access to and utilization of mental health care, on mental health symptoms, and on 

outcomes related to other indicators of well-being: employment, education, housing, and 

incarceration. The primary research questions were: 

• Do C2C participants have greater reductions in barriers to mental health care and greater 

increases in utilization of mental health services relative to comparison group participants? 

• Do C2C participants show greater positive improvement in depression, generalized 

anxiety, PTSD, alcohol use, substance use, and general psychological distress relative to 

comparison group participants? 

• Do C2C participants show improved outcomes in the domains of employment, housing, 

education, and incarceration relative to comparison group participants? 

xxv 
 

 

 

 

 

The quasi-experimental evaluation design allowed us to examine the pooled effect of C2C 

across all treatment participants (i.e., C2C clients) compared with all comparison participants 

(clients of CBOs who did not participate in C2C). We also examined the effect of C2C on 

specific subgroups of participants, including the three C2C target populations (i.e., adults age 

18 or older who are unemployed or underemployed, young adults ages 16 to 24 who are not in 

school and are not employed, and parents/primary caregivers who are expecting or who have 

children up to the age of 4) and the two CBO service types with sufficient sample sizes (e.g., job 

training and employment programs and youth development programs). Not all C2C CBOs were 

included in this evaluation; two CBOs were excluded (one because it was not able to implement 

the program, and the other because its program model differed substantially than the others and 

was offered in a language that would have required extensive translation resources). Participants 

from the remaining 13 CBOs had to meet a minimum threshold for one of the five mental health 

issues to be eligible for the study. This evaluation design decision was made to focus the impact 

evaluation on the individuals for whom the C2C intervention was most likely to have a 

measurable effect. In addition, the impact evaluation included a matched comparison group— 

clients of CBOs similar to the C2C-participating CBOs—to measure the effect of C2C 

programming compared with usual CBO programming. 

For the analyses for the primary research questions, we used baseline survey data collected 

as part of study enrollment (n = 1,838) and follow-up client survey data collected at 6 months 

(n = 688; 37 percent retention rate) and 12 months (n = 732; 40 percent retention rate). In our 

analysis, we used propensity scores that estimate the probability that each individual received 

C2C (versus services as usual) based on their baseline characteristics to adjust for any differences 

between the C2C and comparison groups at baseline. We examined both differences within 

groups over time, comparing each participant retained at each follow-up (i.e., 6 or 12 months) 

with that participant’s score at the baseline assessment; and (2) intervention effects over time 

using an intent-to-treat approach in which we compared all individuals in the C2C group with all 

those in the comparison group, regardless of the actual amount of C2C skills received, in both 

propensity score weighted models and doubly robust models. 

We expected intervention effects to be small in size due to the characteristics of the 

intervention itself (i.e., less intensive than individual therapy, focus on a broad range of mental 

health symptoms and severity rather than only clinically significant symptoms), that it was being 

compared with “as usual” services, and that the sample was highly heterogeneous. These latter 

two factors commonly pose challenges to detecting significant group differences in intervention 

studies. That is, “treatment as usual” groups often perform just as well as the intervention group, 

even for interventions with demonstrated efficacy compared with no-treatment or waitlist control 

conditions. Further, interventions typically work better for certain individuals than for others— 

a phenomenon known as “treatment heterogeneity.” Recognizing this phenomenon, we also 

explored treatment heterogeneity by examining the effect of C2C on different subgroups of 

participants for each set of outcomes. Effects measured in this evaluation compared self-reported 

xxvi 
 

 

 

responses from participants at baseline and at follow-up 6 and 12 months later. (In this report, we 

report the 12-month follow-up results, unless the two follow-up results were in conflict.) 

Access and Utilization

We first looked at barriers to mental health care in terms of logistical, attitudinal, and stigma 

barriers, and also changes in mental health care utilization, attributable to the C2C program. From 

baseline to follow-up, the C2C and comparison groups reported fewer logistical, attitudinal, and 

stigma barriers to mental health care, increased utilization of clinical outpatient care, and decreased 

utilization of inpatient and emergency care. When we examined the effect of C2C, we found that 

C2C did not result in greater improvement for C2C participants on measures of access to and 

utilization of mental health services compared with usual CBO services. This suggests that C2C 

had no effect on mental health care barriers or utilization overall. There are several alternative 

explanations to consider. For example, at the time of the C2C intervention, there were also 

concurrent mental health programs running in the city through the ThriveNYC Initiative, which 

may have made it difficult to detect intervention effects in our sample. 

We also examined whether we could observe different effects of C2C among subgroups 

within the overall sample, namely among each of the three targeted C2C populations and among 

two of the CBO types for which we had enough data from participants (CBOs offering job 

training and employment services, and CBOs offering youth development programs). There was 

some evidence that C2C may have been more effective for some of these subgroups. Among 

youth and young adults, C2C appeared to have reduced attitudinal barriers to mental health care 

relative to the comparison group. Participants from youth development CBOs who received C2C 

used more outpatient mental health services than the youth development CBO participants in the 

comparison group. Among the parent and caregiver target population and the youth and young 

adult target population, emergency department use was lower in the C2C group relative to the 

comparison group. In addition, there was evidence that C2C reduced residential treatment use at 

job training and employment CBOs and among unemployed/underemployed adults and parents 

and caregivers. Though these effects were not observed across all subgroups, these findings 

show that an innovative task-shifting intervention can meet the needs of some populations that 

are often underserved and marginalized. 

Mental Health Symptoms

Next, we looked at the effect of the C2C program on mental health symptoms. Overall, 

C2C did not result in greater improvement for C2C participants on mental health symptoms of 

depression, generalized anxiety, posttraumatic stress disorder, alcohol use, other substance use 

(i.e., drug use), and general psychological distress, compared with usual CBO services. We did 

see high symptom levels at baseline and improvement over time for both groups. Across the 

board, the baseline prevalence of mental health issues in the study sample exceeded the prevalence 

rates from epidemiological studies of similar populations. From baseline to follow-up, both C2C 

xxvii 
 

 

 

participants and those in the comparison group experienced statistically significant decreases in 

symptoms across all of the mental health conditions. At 12 months, C2C did not result in greater 

improvements in any of the mental health symptoms compared with usual CBO services. At 

6 months, there was evidence that C2C may have been more effective in decreasing substance 

use symptoms, although the effect size was very small. Together, these findings suggest that 

C2C had no effect on mental health symptoms. Alternative explanations to consider include a 

natural recovery process or added mental health services at comparison CBOs. 

Although C2C may not have had an effect on improving mental health symptoms in the overall 

sample, there was some evidence that C2C may have been more effective for certain subgroups, 

as with the access and utilization outcomes. Participants at job training and employment CBOs 

showed greater reductions in alcohol use (small effect size) in the C2C group relative to the 

comparison group. At youth development CBOs, participants’ depressive symptoms decreased 

(small-to-medium effect size) more in the C2C group relative to the comparison group. Among 

the targeted population of youth and young adults, psychological distress decreased (small-tomedium effect size) more in the C2C group than comparison group. 

Employment, Education, Housing, and Incarceration

Finally, we looked at C2C’s effect on outcomes in the domains of employment, housing, 

education, and incarceration for those individuals receiving services from CBOs. From baseline 

to follow-up, both C2C participants and those in the comparison group reported more full or 

part-time employment, more weekly work hours, higher monthly pay, more stable housing, and 

more high school or general equivalency diploma completion. Across all of the employment, 

housing, education, and incarceration outcomes, there was one statistically significant intervention 

effect: at the 12-month follow-up, C2C participants experienced larger increases in weekly work 

hours relative to the comparison group (small effect size). Otherwise, the findings indicate that 

C2C did not have an effect on the other outcomes. Other explanations to consider include the 

positive effect of CBO programming outside the C2C skills, as well as concurrent ThriveNYC 

initiatives at the time of C2C implementation. 

As with the previous analyses for the impact evaluation, there was evidence that C2C may 

have been more effective for certain subgroups. For those at job training and employment CBOs, 

C2C participants improved at a greater rate than comparison participants in weekly work hours 

increase (small-to-medium effect size) and monthly pay (small effect size). Weekly work hours 

also increased (very small effect size) more for C2C group than comparison group participants 

among the target population of adults 18 or older and for those at CBOs serving parents or 

caregivers (small effect size). 

Cost

The cost evaluation sought to quantify the resources CBOs needed to both implement and 

maintain the C2C program, as well as the average cost per client. It also measured change in the 

xxviii 
 

 

average cost per client over time. This evaluation factored in both labor and nonlabor costs as 

well as overhead and payments to vendors and consultants, including MHP partners. The primary 

research question was: 

• What are the costs to CBOs associated with implementation of the C2C program, overall 

and average per program client? 

To capture data on program-level resource information, we designed four data collection 

forms: a cost study section embedded in the implementation evaluation’s annual staff survey, a 

cost of labor survey, a staff compensation report, and a non-labor expenses report. We also 

reviewed quarterly CBO invoices and quarterly CBO quarterly reports. All 15 CBOs were 

included in the cost evaluation. Because of data availability, this analysis was limited to changes 

between year 2 and year 3 data. We conducted a main analysis to determine the costs associated 

with the C2C program, as well as three sensitivity analyses. 

Overall, because CBOs started to implement the program, the number of CBO staff who 

received C2C training or were involved in other C2C activities increased by 49 percent from 

year 2 to year 3. In the same time period, the number of clients receiving C2C skills increased by 

31 percent, up from 817 per CBO in year 2 to 1,066 in year 3. Due mostly to the increase in 

staff labor, the average annual cost per CBO rose about 18 percent, from $437,546 in year 2 to 

$514,142 in year 3. The average cost per C2C client served decreased from $536 in year 2 to 

$482 in year 3. 

Nearly half of the resources used in the C2C program were not part of the planned budget. 

Because of the structure of the financial arrangement to fund C2C—a 50-percent grant through 

the Mayor’s Fund and DOHMH with matching arranged by CBOs to cover the other half of the 

planned budget—many CBOs had to cover the remaining costs for C2C out of their own funds. 

Although we were not able to determine the marginal cost per client, defined as the increase 

in variable costs––costs that change when service volume increases––due to serving additional 

clients, we were able to see that several CBOs were beginning to move out of the startup phase 

after the second year of implementation. But not all CBOs progressed to program “maturity” in 

terms of cost on the same timeline. By year 3, only six of the 15 CBOs achieved increasing 

returns to scale, where the average cost per client declined as the program scaled up. With a 

longer time horizon, future evaluations would be able to provide more clarity on when the 

remaining CBOs were able to achieve increasing returns to scale. 

Future Considerations

To our knowledge, this is the first U.S.-based evaluation of a mental health task-shifting 

intervention on this scale. Overall, the results of the impact evaluation did not provide evidence 

of the effectiveness of the C2C model of task-shifting. We found no differences between the 

C2C group and the comparison group overall, although we did find some benefits of C2C for 

certain subpopulations and in certain settings. While the findings from the impact evaluation do not 

xxix 
 

provide evidence for broad scale-up of the C2C model of task-shifting, additional improvements 

in the design, targeting, and evaluation of C2C (or another task-shifting model) could demonstrate 

effectiveness in improving outcomes in future studies. From the implementation evaluation, we 

learned that implementing C2C required CBOs to balance many factors to tailor the program to 

their clientele, but most CBO staff and leaders noted a marked change in how they approached 

mental health with clients and among themselves, due to C2C. And the cost evaluation found 

that labor costs accounted for a substantial portion of overall costs, and that less than half of 

CBOs were able to achieve economies of scale sufficient to reduce the cost per client by year 3. 

This evaluation has limitations, as noted, and is unable to answer every question about C2C’s 

impact, implementation, and cost. Yet the positive feedback from CBO and MHP staff and 

leadership and the effects of C2C on certain subgroups within the impact evaluation—however 

small—demonstrate that CBOs are capable and well suited to integrate mental health supports 

into their usual services, with help from partnering MHPs. Over the course of the 5-year project, 

C2C brought mental health services to tens of thousands of low-income New Yorkers and 

continues to do so, adapting to the new reality of delivering these services remotely during the 

COVID-19 pandemic. 

Together, the three evaluations underscore that mental health task-shifting in communitybased organizations is feasible. Moreover, these evaluations can provide useful information to 

those interested in further refining how task-shifting is designed, targeted to different populations 

and settings, and studied. For those interested in building from our work, we outline seven 

recommendations stemming from our evaluation findings that might inform the design, 

implementation, and evaluation of future task-shifting efforts which are as follows: 

1. Design the model with evidence-based content and alternative delivery modes (e.g., 

telehealth) to reduce barriers to mental health care. 

2. Consider replacing or augmenting the original four skills that made up the C2C model 

(screening, MHFA, MI, PE) with other evidence-based strategies. 

3. Examine the role of systemic barriers to implementation and sustainment of the model 

and other nontraditional mental health delivery models. 

4. Invest resources in reducing barriers to care among youth and young adults. 

5. Plan for at least an initial 1-year ramp-up period, and prepare for increasing costs over 

time until the program gets to scale. 

6. Consider the population, setting, and outcomes when determining whether and how to 

implement the model. 

7. Design future evaluations of the effectiveness of task-shifting models to account for 

individual- and site-level heterogeneity, and to examine mechanisms of change. 

Together, these recommendations can help CBOs, MHPs, policymakers, funders, 

researchers, and communities because they consider further development of mental health taskshifting as a way to expand the mental health workforce and reach vulnerable, low-income 

communities with mental health supports. 

xxx 
Acknowledgments

The authors would like to acknowledge several individuals without whom this study and 

research report would not have been possible. We thank Harold Pincus, Jeanne Ringel, Peter 

Hussey, Paul Koegel, Anita Chandra, and Jeffrey Wasserman for their advice and guidance on 

this report and throughout the project. We are grateful for the careful review of the document 

provided by Joshua Breslau, Mary Acri, Emmett Keeler, and Lynn Karoly, and for Lisa Turner’s 

excellent work shepherding the document through peer review. We thank Yamini Bhalla, Mary 

Gendron, and Steve Oshiro for their editorial assistance and for managing production of this 

report. 

This work would not have been possible without the many New York City (NYC)–based data 

collection team members (especially Emily Hoch, Polina Kats-Kariyanakette, Robert Reynoso, 

and Massiel Ubilius Rivera) who worked unusual hours, at times traveling across all five NYC 

boroughs, to conduct study assessments. We are also indebted to RAND Corporation’s Survey 

Research Group for their persistence in following up with participants and completing telephone 

assessments. 

Our colleagues at NYC Opportunity, NYC Department of Health and Mental Hygiene, and 

the Mayor’s Fund worked with us tirelessly to ensure the evaluation and findings were of the 

highest possible quality. In particular, this work could not have been completed without Kate 

Dempsey, Sinead Keegan, Brigit Beyea, Elise Tosatti, and Gary Belkin. Our colleagues at 

McSilver Institute for Poverty Policy and Research at the New York University Silver School of 

Social Work—especially Andy Cleek, Lydia Franco, and Priya Gopalan—provided outstanding 

technical assistance to the participating community-based organization (CBOs) and mental 

health providers (MHPs) as they implemented the C2C program. Funding for program oversight, 

evaluation, technical assistance and learning community activities was provided by private 

funders through the Mayor’s Fund including the Ford Foundation, Astoria Energy, LLC, The 

Marc Haas Foundation, NY Community Trust, the Marilyn J Simons Foundation, the Stavros 

Niarchos Foundation, The Chapman Perelman Foundation, SOMOS Community Partners, 

Allergan Foundation, Otsuka American Pharmaceutical, Wainwright Berger Philanthropic Fund, 

and others. 

And finally, we would like to thank the clients, staff, and leadership at the CBOs and MHPs 

who participated in this project and who have carried on in their critical work to improve the 

well-being of vulnerable communities in NYC. 

xxxi 
CC 

Abbreviations

95% CI 95% confidence interval 

AAANY Arab American Association of New York 

ALGEE Assess for risk of suicide/harm 

Listen nonjudgmentally 

Give reassurance and information 

Encourage appropriate professional help 

Encourage self-help and other support strategies 

AUDIT-10 Alcohol Use Disorder Identification Test–10 

AUDIT-C Alcohol Use Disorder Identification Test–Concise 

BACEv2 Barriers to Access to Care Evaluation 

BSRC Bedford Stuyvesant Restoration Corporation 

C2C Connections to Care 

CASES Center for Alternative Sentencing and Employment Services 

CBO community-based organization 

CBT cognitive behavioral therapy 

clinical coordinator 

CEO Center for Employment Opportunities 

CHCF Committee for Hispanic Children and Families 

CNCS Corporation for National and Community Service 

CPIC Community Partners in Care 

CQI continuous quality improvement 

C-SSRS Columbia-Suicide Severity Rating Scale 

DAST-10 Drug Abuse Screening Test–10 

DOHMH Department of Health and Mental Hygiene 

DSM-5 Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition 

EBP evidence-based practice 

ECE Early Childhood Education 

ECEI Early Care and Education Institute 

xxxii 
 

GAD generalized anxiety disorder 

GAD-7 Generalized Anxiety Disorder 7-Item Scale 

GED general equivalency diploma 

HIPAA Health Insurance Portability and Accountability Act of 1996 

HIV human immunodeficiency virus 

HMI Hetrick-Martin Institute 

K-6 Kessler 6 

LCSW licensed clinical social worker 

LGBTQ lesbian, gay, bisexual, transgender, and questioning 

LMSW licensed master social worker 

LOA Legal, Organizing, and Advocacy 

MHFA mental health first aid 

MHP mental health provider 

MI motivational interviewing 

MSSI-SA Modified Simple Screening Instrument for Substance Abuse 

NAMI National Alliance on Mental Illness 

NMIC Northern Manhattan Improvement Corporation 

NYC New York City 

NYS New York State 

NYU McSilver McSilver Institute for Poverty Policy and Research at the New York 

University Silver School of Social Work 

OR odds ratio 

PC-PTSD Primary Care PTSD Screen for DSM-5 

PCL-5 PTSD Checklist for DSM-5 

PHQ-8 Eight-item Patient Health Questionnaire 

PHQ-9 Nine-item Patient Health Questionnaire 

PTSD posttraumatic stress disorder 

PWS participant wellness specialist 

QI quality improvement 

RHI Red Hook Initiative 

xxxiii 
RS retention specialist 

SHCC Safe Horizon Counseling Center 

SRG Survey Research Group 

SIF Social Innovation Fund 

SSM supportive services manager 

SSSW Silberman School of Social Work 

TC Teacher’s College 

UHP Urban Health Plan, Inc. 

US Union Settlement 

xxxiv 
 

 

Part I. Introduction

The three chapters in this section provide an overview of the Connections to Care (C2C) 

program and the problems it tries to address. They provide background on what is known about 

mental health care in low-income populations and potential interventions to lower barriers to 

care, introduce the concept of task-shifting in the context of C2C, and provide detail on the logic 

behind C2C as a foundation to the detailed descriptions of the three C2C evaluations in 

subsequent sections. 

Chapter 1 describes the prevalence of mental health problems in the United States, the 

consequences of letting those problems go unaddressed, and the difficulties in accessing mental 

health care in many parts of the country. It also describes the motivation for the C2C program. 

Chapter 2 presents a review of the evidence based on interventions to address mental health 

problems among low-income, underserved populations such as those targeted by the C2C program. 

It also examines the evidence behind task-shifting, a central feature of the C2C program. 

Chapter 3 provides more detail on how the C2C program was built and its main features. It 

also reviews the program’s logic model and its expected outcomes, as well as the key research 

questions in the implementation, impact, and cost evaluations. 

The sections and chapters that follow describe the methods and results for each evaluation. In 

Part II, three chapters report on the qualitative and quantitative findings from the implementation 

evaluation. In Part III, three chapters present findings from the impact evaluation. Part IV describes 

cost evaluation findings. A summary of overall conclusions and recommendations follows these 

groups of chapters. In addition, four appendices provide greater detail on how the C2C program 

was implemented at each community-based organization (CBO) site (Appendix A) and the 

methods and analytical approaches used for the implementation (Appendix B), impact 

(Appendix C), and cost (Appendix D) evaluations. 

1 
1. Introduction

Lynsay Ayer 

Mental health problems are common in the United States (Kessler et al., 2012): Approximately 

one-fifth of the U.S. population experiences one, according to national survey estimates from 

2017 (Substance Abuse and Mental Health Services Administration, 2018). Among young adults 

(ages 18–25), mental illness prevalence was even higher, at 25.8 percent, than for other adult age 

groups (Substance Abuse and Mental Health Services Administration, 2018), and mental health 

symptoms often continue into later adulthood as well (Essau et al., 2010). 

Mental health problems disproportionately affect low-income communities (Chow, Jaffee, 

and Snowden, 2003; Duncan, Magnuson, and Votruba-Drzal, 2017), and racial/ethnic minorities 

are more likely to live in poverty in the United States (Elliott, 2016). Likely stemming from 

multiple disadvantages in U.S. society, low-income individuals, racial/ethnic minorities, and 

those with lower English language proficiency are least likely to have access to mental health 

services (Akincigil et al., 2012; Chow, Jaffee, and Snowden, 2003; Kataoka, Zhang, and Wells, 

2002; Saloner and Cook, 2013; Sentell, Shumway, and Snowden, 2007), compared with other 

segments of the U.S. population. 

Consequences of Untreated Mental Health Problems

Mental health problems left unaddressed can lead to myriad negative consequences to the 

individual, family, and society. For example, depression is related to many different physical 

health problems (Stubbs et al., 2017), and mental illness in one or both members of a couple can 

contribute to marital dissolution and divorce (Butterworth and Rodgers, 2008). Globally, the 

burden of disability related to mental health disorders is substantial, accounting for as much as 

32 percent of the years individuals live with a disability and 13 percent of disability-adjusted life 

years (a measure of the effect of disability on quantity and quality of life) (Vigo, Thornicroft, and 

Atun, 2016). Major depression is the second-largest cause of disability worldwide (Vos et al., 

2015). 

Untreated mental health problems (including substance misuse) can have far-reaching 

impacts. For example, individuals experiencing such problems may have trouble obtaining a job 

or achieving academic success, and they are more likely to be involved with the justice system 

(Watkins et al., 2004). When mental health symptoms become severe, those without access to 

mental health services often seek care from emergency rooms, a costly approach that often does 

not effectively address mental health problems (Ayangbayi et al., 2017). Left untreated, these 

problems can even be life threatening: Depression and other mental health problems are among 

2 
 

 

the most consistent predictors of suicide (Gournellis et al., 2018; Krysinska and Lester, 2010), 

and drug and alcohol use are risk factors for accidental death and injury (Hingson and Howland, 

1987; Olsson et al., 2016). 

Together the consequences of untreated mental health disorders lead to societal costs estimated 

at nearly $200 billion in lost earnings in the United States (Kessler et al., 2008). Thus, the unmet 

need for mental health treatment, overall and in underserved communities, represents a serious 

public health and societal problem. 

Access to Mental Health Care

Early detection and treatment of mental health problems, ideally before they become severe 

and chronic, is critical to slow or stop their progression and associated costs. Recent studies have 

demonstrated that proactive effort to screen for mental health problems is related to greater 

likelihood of mental health service utilization among those who have symptoms (Petrenko et al., 

2011; Shippee et al., 2014). However, racial/ethnic minorities and individuals who are affected 

by poverty are less likely than their peers to receive mental health screening in clinical settings 

(Hahm et al., 2015). Mental health screenings can also be performed in nonclinical settings, such 

as CBOs, and offering screenings in these settings has the potential to reduce such disparities 

(Thomas and Staiger, 2012). Other populations at high risk for mental health problems, such as 

those who misuse substances, tend to receive mental health screenings at relatively low rates. 

With adequate training, substance use treatment centers can successfully implement mental 

health screening for youth and adults (Aitken et al., 2008; Lubman et al., 2008). 

Once such problems have been identified, evidence-based interventions have generally been 

effective in treating a wide range of mental health problems, including depression, anxiety, and 

posttraumatic stress disorder (PTSD) (Bisson et al., 2007; Hofmann and Smits, 2008; Miranda et 

al., 2003; Roy-Byrne et al., 2010; Shear et al., 2005). However, the United States has been 

experiencing a critical shortage of mental health providers (MHPs). One in five U.S. counties 

had an unmet need for mental health professionals according to the most recent data publicly 

available (Thomas et al., 2009). For example, 96 percent of the nation’s 3,140 counties do not 

have adequate access to psychiatric services (Thomas et al., 2009). As of November 2019, the 

Health Resources Services Administration (HRSA) had designated 17 areas in New York City 

(NYC) as MHP shortage areas (ThriveNYC, 2019). 

Amid this country-wide shortage of mental health clinicians, not all specialists have access to 

the training and supervision needed to deliver evidence-based interventions (Pagoto et al., 2007). 

Consequently, regions throughout the nation and across all distributions of income have been 

exploring new delivery models, many of which leverage nonspecialist health workers such as 

peers (American Hospital Association, 2016; Kazdin, 2017; Kazdin and Rabbitt, 2013). 

These new delivery models are particularly relevant to low-income and racial/ethnic minority 

communities because of their elevated mental health risk and particularly limited access to care. 

3 
 

 

Increasing Access to Mental Health Care Through C2C

In response to the need for greater access to mental health care in NYC’s low-income, 

underserved communities with provider shortages, the C2C Collaborative began a publicly and 

privately funded initiative to expand access to mental health services. C2C is one initiative of 

ThriveNYC, an $850 million commitment by the City of New York to address the mental health 

needs of New Yorkers. The C2C Collaborative—the Mayor’s Fund to Advance NYC (Mayor’s 

Fund), the Mayor’s Office for Economic Opportunity (NYC Opportunity), and the NYC 

Department of Health and Mental Hygiene (DOHMH)—drew funding from the federal Social 

Innovation Fund (SIF) of the Corporation for National and Community Service (CNCS), the 

City of New York, and private donors to create the C2C program and support this evaluation.1 

In 2016, the C2C Collaborative began implementing the C2C program, an innovative model 

of integrating mental health support into the work of CBOs that serve low-income and at-risk 

populations (e.g., through job training and employment, domestic violence shelters, homeless 

shelters, youth development, services for immigrants). The C2C program uses a “task-shifting” 

approach (described in more detail later) to maximize the capabilities of CBOs and MHPs to 

promote community mental health. It serves clients from the following three target populations 

of low-income New Yorkers: 

• youth/young adults ages 16 to 24 who are not in school and are not employed 

• adults age 18 or older who are unemployed or underemployed 

• parents/primary caregivers who are expecting or who have children up to the age of 4. 

The C2C program’s $30 million public-private partnership was expected to reach up to 

40,000 New Yorkers over a 5-year span through CBOs and MHPs. At the time of this writing, 

this goal has been surpassed, and CBOs are funded to continue the C2C program through 

February 2021. New York University’s (NYU) McSilver Institute for Poverty Policy and 

Research provided technical assistance to the CBOs and MHPs from 2016 through 2019. In 

addition to addressing immediate mental health needs, the C2C Collaborative and other 

stakeholders—including the NYC government—aim to determine whether the C2C program 

works as intended to remove barriers to mental health care and identify, assess, treat, and 

prevent mental health disorders among these populations. 

1 The SIF was a federal grant program of the Corporation for National and Community Service that received 

funding from 2010 to 2016. Using public and private resources to find and grow community-based nonprofits with 

evidence of results, SIF intermediaries received funding to award subgrants that focus on overcoming challenges in 

economic opportunity, healthy futures, and youth development. The Corporation for National and Community 

Service awarded an SIF intermediary award to the Mayor’s Fund in 2015 for use to administer the first 3 years of a 

5-year subgrant program. 

4 
 

 

 

 

Evaluating the C2C Program

In a previously published interim report, RAND Corporation summarized initial findings on 

the C2C program using data from its initial one and one-half years of implementation (Ayer et 

al., 2018). Further, in collaboration with DOHMH, RAND published a guide for CBOs interested 

in designing and implementing task-shifting (also called “task-sharing”) approaches similar to 

C2C in their own organizations (Stevens et al., 2020). 

This final report summarizes the qualitative and quantitative findings in the three key domains 

of the C2C program evaluated by RAND: (1) its effect, (2) its implementation, and (3) its costs 

for CBOs. Figure 1.1 provides a high-level timeline of the start and end of key evaluation and 

data collection activities. A more detailed outline of C2C program implementation activities is 

included in Chapter 4 (Figure 4.1). As illustrated in Figure 1.1, all data collection was complete 

by the end of February 2020, before the COVID-19 pandemic and related policies (e.g., stay at 

home orders and closure of nonessential businesses) affected NYC. 

Figure 1.1. C2C Evaluation and Data Collection Timeline

5 
 

2. Background and Rationale for C2C

Lynsay Ayer, Wing Yi Chan, and Michael Stephan Dunbar 

As described in the previous chapter, mental health problems affect about 20 percent of the 

U.S. population overall, with a disproportionate effect on low-income communities of color 

(Allen et al., 2014; Chow, Jaffee, and Snowden, 2003; Saloner and Cook, 2013). These 

communities also tend to be less likely to access mental health care, a problem worsened by 

national MHP shortages and limited access to high-quality, evidence-based care (Akincigil et al., 

2012; Chow, Jaffee, and Snowden, 2003; Sentell, Shumway, and Snowden, 2007). Untreated 

mental health conditions confer risk for a range of consequences such as physical disability 

(Brenes et al., 2008; Penninx et al., 1999), substance misuse, involvement with the justice 

system, suicide, accidental injury or death, and lowered life expectancy (Chang et al., 2011). 

Therefore, interventions that increase engagement in mental health care, including early 

screening to detect problems before they become severe, could save both lives and money. 

In this section, we review the literature on mental health service utilization in low-income 

populations, including the three populations targeted by C2C; barriers to mental health service 

utilization; and the use of the task-shifting model for mental health, which formed the basis for 

the C2C program. The studies cited in the sections below form the base of evidence supporting 

the approach used in the C2C model. 

Mental Health Service Utilization in Low-Income Populations

Poverty is consistently linked to poor mental health and adverse outcomes (Santiago, 

Kaltman, and Miranda, 2013). Individuals affected by poverty are more likely than those with 

higher incomes to experience chronic and acute stressors, including community violence, lack of 

community supports, economic and financial difficulties, and unstable housing, all of which 

increase the risk for mental health problems (Santiago, Kaltman, and Miranda, 2013). Despite 

this greater risk, most individuals living in low-income communities do not receive mental health 

care. A recent study analyzing nationally representative data from the National Health Interview 

Survey found that participants with lower incomes were 1.5 times more likely to have unmet 

mental health care needs than higher-income participants. In addition, those without health 

insurance coverage were almost four times more likely to report unmet mental health needs in 

comparison to those with private health insurance (Roll et al., 2013). 

Unmet need for mental health services is prevalent in NYC. The city government estimated 

that less than 40 percent of New Yorkers report receiving needed mental health treatment (City 

of New York; Office of the Mayor, 2015). The national rate of unmet need for mental health 

treatment is similar. One study found that only about 41 percent of individuals who were 

6 
diagnosed with a mental health problem had received treatment in the past 12 months (Wang et 

al., 2005). Rates of mental health care utilization are even lower among historically underserved 

individuals, including racial/ethnic minority and other socioeconomically disadvantaged groups, 

which may contribute to or exacerbate mental health problems and other disparities, such as 

economic and educational inequalities (Wang et al., 2005). 

Populations affected by poverty may be particularly vulnerable to mental health problems. 

Therefore, increased access to mental health care is especially important for these groups. 

The three target groups of the C2C program include some of these populations (under- and 

unemployed adults, out-of-school and out-of-work youth and young adults, and expectant 

parents and parents/caregivers of young children). The literature on the challenges unique to 

these three groups supports the need for their inclusion as the primary focus populations of the 

C2C program. 

Under- and Unemployed Adults

Unemployment is a risk factor for mental health problems (Cygan-Rehm, Kuehnle, and 

Oberfichtner, 2017; Milner, Page, and Lamontagne, 2014; Pharr et al., 2012; Strandh et al., 

2014). For example, a meta-analysis found that mental health generally worsens after job loss 

and then improves after reemployment (Paul and Moser, 2009). Further, mental health problems 

make it more difficult to obtain employment, potentially leading to even worse mental health 

problems (Olesen et al., 2013; Terza, 2002). Yet mental health services are often out of reach for 

unemployed and underemployed adults (i.e., those who do not have enough paid work or work 

that makes full use of their skills). For instance, the typical structure of mental health services 

(e.g., weekly visits, weekday appointments) makes it difficult for those who have unpredictable 

hours, work on a shift schedule, or do not own a car to travel to mental health care (Anderson et 

al., 2017; Santiago, Kaltman, and Miranda, 2013). 

Out-of-School and Out-of-Work Youth and Young Adults

A study of a large British cohort found that about 60 percent of 18-year-olds who were not in 

employment, education, or training had already experienced a mental health problem, compared 

with 35 percent of their peers (Goldman-Mellor et al., 2016). Youth who have dropped out of 

high school are more likely to be depressed and to have attempted suicide than high school 

graduates (Liem, Lustig, and Dillon, 2010; Maynard, Salas-Wright, and Vaughn, 2015). In 

addition, out-of-school adolescents and young adults are less likely to have health insurance 

than their student counterparts (Cadigan, Lee, and Larimer, 2019). 

Expectant Parents and Caregivers of Young Children

Expectant parents and caregivers of young children are particularly vulnerable to depression 

during pregnancy and the postpartum period because of (for postpartum women) hormonal 

changes and (for all caregivers) the stressors associated with childbirth and child rearing 

7 
 

 

 

(Brummelte and Galea, 2010; Cunningham and Zayas, 2002). A prior history of mental health 

problems, lack of social support, and adverse life events also contribute to risk for mental health 

problems for caregivers (Biaggi et al., 2016; Milgrom et al., 2008). Expectant parents and 

caregivers with young children who have low incomes report higher rates of mental health 

problems—and are less likely to seek mental health treatment—than those with higher incomes 

(Lazear et al., 2008). Many note that a fear of losing custody of their children is a barrier to 

seeking care (Anderson et al., 2006). In addition, competing priorities such as childcare and their 

children’s mental health needs can add to the challenge of participating in their own mental 

health services (Anderson et al., 2006). The bidirectional relationship between maternal and 

child mental health makes providing mental health services to parents especially important, 

because better parental well-being is related to more positive mental and behavioral health in 

children (Kvalevaag et al., 2013; Zalewski et al., 2017). 

Barriers to Mental Health Care

There are many barriers to accessing mental health services. These barriers are often 

categorized into two groups: logistical or “concrete” and perceptual or stigma-related (Clement 

et al., 2012). Examples of logistical barriers include cost, lack of health insurance or inadequate 

insurance, limited availability of MHPs (e.g., due to a shortage of providers in a given region), 

lack of transportation, inflexible mental health services, and lack of culturally and/or linguistically 

appropriate care (Santiago, Kaltman, and Miranda, 2013). Some of these barriers (e.g., cost, lack 

of health insurance and transportation) disproportionately affect low-income communities. 

Mental health stigma refers to “the stigma and discrimination that individuals believe to be 

associated with receiving care for a mental health problem” (Clement et al., 2012). Stigmarelated barriers include individuals’ concerns about the potential negative consequences 

associated with seeking treatment for a mental health problem. For example, someone in need of 

mental health care may avoid seeking it due to shame, embarrassment, or fear that others would 

find out about seeking treatment (Hadfield and Wittkowski, 2017; Lazear et al., 2008). C2C is 

designed to address both types of barriers. 

Mental Health Task-Shifting

To broaden access to mental health support and address many of the logistical and stigmarelated barriers to mental health care, C2C implements a model of service delivery known as 

“task-shifting” or “task-sharing.”1 In the context of mental health, task-shifting is a way of 

1 Some researchers, practitioners, and policymakers use the term “task-sharing” to reflect that tasks may remain 

shared between the specialist and lay providers. In this report, we use “task-shifting,” as it is the more commonly 

used term, but many of the C2C models implemented by CBOs and MHPs may be best described as task-sharing. 

8 
 

 

 

expanding the mental health workforce by training lay staff (i.e., staff who are not professional 

mental health workers) to deliver basic mental health screening and evidence-informed 

psychosocial intervention strategies (Chibanda et al., 2011; Govindarajan and Ramamurti, 

2018; Huang et al., 2014; Legha et al., 2015). Under this model, mental health specialists 

(e.g., psychiatrists, psychologists, social workers) provide training, supervision, and fidelity 

monitoring for nonspecialists (e.g., community outreach and early childhood workers, teachers, 

employment specialists, shelter staff). Training nonspecialists to deliver mental health 

interventions through trusted CBOs already working within the community has the potential to 

increase mental health access among underserved populations. Task-shifting expands the mental 

health workforce and delivers mental health supports and interventions through organizations 

that are culturally competent, accessible, and lack the stigma associated with formal mental 

health settings. Evidence has shown that task-shifting enables referrals, detection of mental 

health issues, psychoeducation (PE), and follow-up care with positive outcomes for clients 

(Kakuma et al., 2011). 

Integrating mental health support into community-based settings also has the potential to 

improve the usual services delivered by CBOs, as well as the overall health care system (Patel et 

al., 2013). For example, a CBO that focuses on job training and employment may find that clients’ 

mental health and substance use issues are barriers to program participation and successful 

employment outcomes. When staff are able to identify and address clients’ mental health 

concerns swiftly and effectively, CBOs may find that their clients are better able to participate in 

programs and ultimately better prepared for employment. With mental health supports delivered 

by nonspecialists, task-shifting could also improve the efficiency of the mental health care 

system, allowing mental health professionals to focus their time on delivering specialty care to 

those with the most intensive needs. 

The concept of task-shifting emerged from global health initiatives that focused on disease 

prevention and treatment. Task-shifting has been adopted in mental health because of its ability 

to be scaled up to provide services to individuals with limited access to care as well as its 

adaptability to diverse cultures and local conditions (dos Santos et al., 2016; Kazdin and Rabbitt, 

2013; Matsuzaka et al., 2017). Several factors have been hypothesized to increase the likelihood 

of success of mental health task-shifting interventions: (1) assessment and integration of local 

context into the intervention; (2) referral and care pathways that follow a protocol to address a 

mental health issue and achieve a specific health goal; (3) access to training, ongoing supervision, 

and decision supports that help lay workers know when to take action and which actions to take; 

(4) integration of quality improvement (QI) practices that allow for assessment and rapid 

improvement of care; and (5) planning for capacity building and sustainability (Belkin et al., 

2011). The C2C initiative incorporates these elements and seeks to build on the current 

evidence of effective delivery of mental health interventions, as well as inform further use and 

implementation of the C2C model in NYC. 

9 
 

 

Four Skills for Nonspecialists

Four skills—screening for detection of mental health conditions, mental health first aid 

(MHFA), motivational interviewing (MI), and PE—have been shown to be effective on a range 

of outcomes (e.g., substance use and misuse, depression symptoms) when delivered by trained 

lay people (Acri et al., 2014; Acri et al., 2015; Barrowclough et al., 2001; Cunningham et al., 

2012; de Roten et al., 2013; Hohmna, Doran, and Koutsenok, 2009; Jensen et al., 2011; Kagee et 

al., 2013; Schwalbe, Oh, and Zweben, 2014; Smith et al., 2012; Wong, Collins, and Cerully, 

2015). Other skills can also be effective, but C2C focused on these four skills for delivery by 

CBOs and MHPs (described further in Chapter 3). To date, this specific combination of skills 

has not yet been studied in the literature. 

Mental Health Screenings

Mental health screenings for common mental health and substance use and misuse issues are 

often used to determine the level and type of additional supports or services that a client needs. 

In low-income settings, there is emerging evidence that screenings for mental health issues— 

conducted by nonclinical staff who have undergone adequate training—result in population-level 

gains, including greater mental health services coverage, more effective use of health care staff 

and resources, and reductions in stigma (Kagee et al., 2013). This appears to be especially true 

when there is also a clear referral strategy in place for individuals who screen positive for 

common mental health issues (Kagee et al., 2013), as offered through C2C with the CBO-MHP 

partnerships (described in more detail in Chapter 3). 

Mental Health First Aid

MHFA was designed specifically for use by lay people and non–mental health specialists. 

The behavioral health problems covered in MHFA trainings include depression, anxiety, other 

psychiatric disorders (e.g., schizophrenia, eating disorders), and substance use disorders. The 

objective of MHFA training is to facilitate trainees’ ability to identify and respond to clients’ 

behavioral health problems. More specifically, MHFA training is intended to 

• increase awareness of signs and symptoms of behavioral health problems, including 

mental health problems and substance use 

• enhance active listening skills 

• provide skills in immediate intervention, crisis response, and referrals to mental health 

services. 

To support these skills, MHFA training also seeks to build trainees’ mental health literacy 

(knowledge and vocabulary) and normalize and destigmatize mental health problems. MHFA has 

been found to be effective for improving trainees’ knowledge and attitudes and for promoting 

helping behavior toward individuals with mental health disorders and/or symptoms (Wong, 

Collins, and Cerully, 2015). Several studies support the effectiveness of MHFA in improving 

recognition of mental health symptoms, knowledge of mental health support and treatment 

10 
 

 

 

resources, attitudes about social distance, and confidence in providing help by staff trained in 

MHFA (Wong, Collins, and Cerully, 2015). 

Motivational Interviewing

MI is a collaborative, person-centered method that aims to explore and resolve ambivalence 

about maintaining or changing unhealthy behaviors and increases motivation to facilitate 

behavior change (Miller and Rollnick, 2012). For example, rather than arguing with a client 

about whether their alcohol use is a problem, or telling a client that they need to take specific 

actions to change, in MI the therapist/interventionist would approach the client’s ambivalence 

about changing their drinking in a nonjudgmental way, eliciting reasons to change and affirming 

the client’s autonomy over decisions to change. Originally developed in the context of substance 

use treatment, multiple studies have demonstrated the effectiveness of MI in changing behaviors 

related to a variety of health and mental health issues (Lundahl and Burke, 2009). MI has been 

used effectively to facilitate behavior change in multiple medical and psychiatric conditions, 

including anxiety, depression, and PTSD (Burke, Arkowitz, and Menchola, 2003); comorbid 

psychiatric and substance use and misuse issues; and a wide range of other issues that affect 

well-being. For example, MI has been applied in education settings to improve a range of 

behaviors that include (but are not limited to) dropout rates, marijuana use, and obesity. MI has 

also been increasingly applied in corrections settings (Miller and Rollnick, 2013). There is 

moderate evidence to suggest that non–mental health professionals can be trained in MI and 

implement it with fidelity. Mental health and non–mental health professionals alike are most 

likely to develop MI skillfulness through opportunities for continued learning over time, 

especially coaching that includes individualized feedback based on observed practice (Miller 

and Rollnick, 2013; Schwalbe, Oh, and Zweben, 2014). 

Psychoeducation

PE integrates education and strengths-based approaches to empower people to improve their 

mental health and well-being (Donker et al., 2009). PE includes providing emotional support, 

accurate information about mental health issues and helping participants develop new tools and 

competencies to manage mental health conditions and other challenges in optimal ways. It can be 

delivered in group or individual settings, and it may target individuals affected by or at risk for 

mental health issues or their families. PE is a flexible mental health support that has shown 

potential utility and efficacy when incorporated into treatment programs for a range of mental 

health and related problems (e.g., severe mental illness, adjustment to medical diagnoses) 

(Lukens and McFarlane, 2004; Pekkala and Merinder, 2002). In addition, there is emerging 

evidence that lay providers can successfully deliver PE. Individuals in India diagnosed with 

schizophrenia who received PE from lay health workers combined with usual care showed 

increased adherence to medications after 12 months in comparison to participants in the usual 

care group (Chatterjee et al., 2014). 

11 
Other Nonspecialist Interventions

International research in resource-limited settings also indicates that other mental health 

interventions delivered by nonspecialist lay workers can produce positive outcomes, such as 

decreased depression, reduced PTSD symptoms, decreased alcohol consumption, and reduced 

stigma (Bolton et al., 2003; van Ginneken et al., 2013). Further, studies suggest that because 

known lay staff are viewed as more credible and trustworthy than unfamiliar mental health 

professionals, engagement in mental health treatment is improved when these staff perform 

some tasks of care (Gopalan et al., 2010). 

Other programs have applied mental health task-shifting to primary care medical settings 

(Patel et al., 2013). For example, lay staff provided depression screening and PE in primary care 

sites (a model referred to as “collaborative care”) in two underserved communities in Los 

Angeles (Chung et al., 2010). A randomized controlled trial found that men who received the 

collaborative depression care, including from nonclinical staff, reported better mental health and 

fewer hospital visits than those who received usual care (Mehta et al., 2017). 

12 
 

 

 

 

3. C2C Program and Logic Model

Lynsay Ayer and Vivian L. Towe 

The C2C Collaborative designed the C2C program to lower barriers to mental health care 

among three low-income, underserved populations in NYC: under- and unemployed adults, outof-school and out-of-work youth and young adults, and expectant parents and caregivers of 

young children. It also applied a task-shifting model to mental health care, so that lay staff at 

CBOs could provide some essential mental health skills in an environment of mental health 

care provider shortages. C2C designers thought that, as entities trusted by their surrounding 

communities and client populations, CBOs could be an ideal setting for delivering this mental 

health task-shifting intervention and may, in the process, improve their existing offerings by 

promoting mental health among their clients. 

Program Overview

The C2C Collaborative chose four nonspecialist skills described in Chapter 2—mental health 

screening, MHFA, MI, and PE—to become the core mental health skills delivered by CBO staff 

in C2C. Although the C2C model requires that some combination of staff at each CBO receive 

training in all four skills, each CBO developed a customized plan for incorporating them into 

their existing services. CBO staff received training, ongoing coaching, and support from their 

MHP to implement these four skills, at a minimum, though CBOs could choose to implement 

additional skills if desired. Not every staff member was required to be trained in and deliver all 

four skills; the skills could be delivered by a combination of staff members as appropriate. Not 

all clients were expected to receive all four supports, and clients could receive a different mix of 

C2C skills in an order that best served their needs and the CBO’s implementation plan. 

The C2C Collaborative understood that some clients would also need access to stepped, 

specialty mental health care, and that CBOs would need training, coaching, and supervision from 

mental health professional organizations (MHPs) to deliver C2C skills effectively. To facilitate 

this, C2C enabled and funded CBOs to partner with MHPs to develop and strengthen referral 

pathways to specialty care. (See “Building the Program” below for more on the selection process 

for CBOs and MHPs.) 

In addition to adopting the four C2C skills, each CBO and its MHP were expected to develop 

and strengthen referral pathways from the CBO to the MHP for clients who request or agree to 

additional care. The CBO was expected to work with the MHP to ensure that clients receive 

services and to follow up with clients about appointments and their progress. Streamlining the 

referral process and ensuring warm handoffs to MHP providers who have training and licensure 

13 
 

 

to provide specialized clinical care is particularly important for low-income individuals in need 

of treatment, because they face numerous barriers to accessing quality care. 

C2C was not intended to be a substitute for professional mental health services (e.g., 

psychological evaluation for diagnostic purposes; psychotherapy delivered by a clinical 

psychologist, licensed clinical social worker [LCSW], counselor, psychiatrist) or to be focused 

exclusively on those with clinically significant symptoms. Instead, the program is designed 

to meet the needs of people with mild or no symptoms with mental health promotion and 

prevention, and to increase the likelihood that people who are in need of clinical care will get it 

by preparing them for engagement and retention in such services (e.g., by reducing stigma and 

other barriers to care). 

Ultimately, the C2C program is designed to address a wide range of unmet mental health 

needs, thus improving outcomes for individuals served. CBO staff can also apply the personcentered conversation skills and strategies they develop through their C2C training to improve 

the everyday, non–mental-health-related conversations with the clients they serve. 

Building the Program

The C2C Collaborative brought together many stakeholders and partners to achieve its goals 

of promoting the well-being and mental health of program participants by increasing client 

access to and improving CBO capacity for delivery of evidence-informed mental health supports. 

In fall 2015 and winter 2016, the C2C Collaborative used a competitive process to choose 15 CBOs 

operating throughout NYC to participate in the C2C program. The 15 CBOs each applied with an 

MHP organization licensed to deliver professional mental health services in NYC. 

CBOs chose their MHP partners; they could partner with culturally responsive MHPs, or if 

that was not possible, CBOs could leverage their expertise in cultural responsiveness to build 

those skills among MHP staff. In most cases, CBOs contracted with a third-party MHP to 

provide training, technical assistance (often in the form of “coaching and supervision” of mental 

health support skills), and direct services to the CBO. In other cases, the CBO had an existing 

mental health clinic within its agency that was engaged as the MHP to provide services where 

they otherwise were not occurring within the CBO social service settings. A description of the 

specific way each CBO and MHP implemented the C2C program is provided in Appendix A. 

The chosen CBOs, all of which are nonprofits, provide a wide range of services to their 

clients, including job training and employment, youth-oriented programming, immigration 

services, human immunodeficiency virus (HIV) testing, early childhood education, homeless 

shelters, and domestic violence interventions. Table 3.1 summarizes key characteristics of CBOs 

that participated in the C2C program. 

14 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 3.1. Overview of CBOs Participating in the C2C Program

Borough(s)/

Geographical

CBO Area(s) Served Brief Description of CBO Type Example Services Provided by CBO C2C Target Group(s)

Arab American 

Association of

New York

BedfordStuyvesant

Restoration 

Corporation

CAMBA

Center for 

Employment

Opportunities

Committee for 

Hispanic 

Children and 

Families

Hetrick-Martin

Institute

Hudson Guild

Northern 

Manhattan

Improvement

Corporation 

Brooklyn

BedfordStuyvesant

East Elmhurst

Citywide

Bronx

Citywide

Chelsea, Hell’s 

Kitchen, Lincoln 

Square

Washington

Heights/Inwood/

South Bronx

Grassroots center situated in a

storefront.

Multiservice community

development organization.

Community-based social service 

agency. C2C was implemented in 

one of CAMBA’s homeless

shelters (The Landing).

Organization serving men and 

women returning home from

incarceration.

Facilitates youth development and 

afterschool programs to support

early childhood educators in 

underserved communities.

Provides lesbian, gay, bisexual, 

transgender, and questioning 

youth with the skills necessary for a

productive transition into healthy 

adulthood.

Social service agency. C2C was

implemented in Hudson Guild’s

early childhood education 

programs.

Community-based social services

provider.

• Adult education

• Naturalization services

• Immigration and legal services

• Youth programming

• General casework and domestic violence services

• Financial education and economic self-sufficiency 

training

• Family services

• Arts and cultural programming

• Economic development services

• Education and youth development services

• Family support, health, housing, and legal services

• Employment reentry and workforce training 

programs

• Early childcare and education programs

• Youth development programs

• Homeless outreach services

• Health and wellness education

• Job readiness

• Academic enrichment; arts and cultural

programming

• Early childhood education

• Youth development and education

• Public housing assistance

• Community building and outreach services

• Adult education and workforce development

programs

• Legal services

• General social services

• Community organizing

Expectant and new mothers and 

parents of young children, out-ofschool and out-of-work youth, and 

unemployed or underemployed 

low-income working individuals

Low-income underemployed and 

unemployed working individuals

Expectant and new mothers and 

parents of young children

Expectant mothers and parents

of children, out-of-school and 

out-of-work young adults

Expectant mothers and parents

of children and unemployed or

underemployed low-income 

working-age adults

Out-of-school, out-of-work young 

adults, and unemployed or

underemployed low-income 

working-age adults

Expectant and new mothers and 

parents of children

Out-of-school, out-of-work young 

adults ages

15 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

CBO

Borough(s)/

Geographical

Area(s) Served Brief Description of CBO Type Example Services Provided by CBO C2C Target Group(s)

Red Hook 

Initiative

Red Hook Agency providing three youth 

programs and a community

building program.

•

•

•

•

Intensive academic support

Job readiness and leadership training

College application assistance

and college retention support

Youth and young adult development and wellness

programming

Out-of-school, out-of-work young 

adults

Safe Horizon Citywide Domestic violence shelter

program.

•

•

•

•

24-hour domestic violence hotline

Food and clothing assistance

Childcare and safety assistance

Financial support and housing assistance

Expectant and new mothers and 

parents of children 

Sheltering Arms Queens, Bronx,

Manhattan

Social service agency. C2C was

implemented in Sheltering Arms’ 

early childhood education 

programs.

•

•

•

•

Early childhood education and afterschool programs

Community school program

Programming in foster care/group homes

Family preservation, juvenile justice placement,

and aftercare services

Expectant and new mothers and 

parents of children

• Services for developmentally disabled adults

STRIVE East Harlem Job training and employment

agency.

•

•

•

•

Job training and placement services

Parenting programs

Economic stability workshop

Job placement services

Unemployed or underemployed 

low-income working-age adults

The Door Citywide Multiservice youth development

agency.

•

•

•

•

•

•

Career and education services

Adolescent health services

Legal services

Runaway and homeless youth services

Supportive housing

Arts and cultural programming

Low-income young people and 

out-of-work young adults, as well

as expectant mothers and 

parents of children and 

unemployed or underemployed 

low-income working-age adults

The HOPE

Program

Brooklyn Job training and employment

agency.

•

•

Career and workforce readiness workshops

Adult education

Unemployed or underemployed 

working-age adults

• General social services

Voces Latinas Queens Program focused on HIV

prevention and services for

Latina women.

•

•

•

•

•

Adult educational workshops

Health education

Peer training

Economic empowerment services

Case coordination services

Unemployed or underemployed 

low-income working-age adults

•

•

•

Domestic violence support groups

HIV testing

Community mobilization

16 
 

 

Funding support for C2C implementation at CBOs was provided through multiple 

mechanisms. In the first 3 years of the 5-year project, half of the C2C funding came from a 

federal grant from CNCS to the Mayor’s Fund. CBOs were required to raise matching nonfederal 

dollars for the other half of the funding they needed. Annual CBO budgets for C2C ranged from 

$200,000 to $450,000 depending on the amount requested and justified by each CBO. CBOs 

were required to submit quarterly progress reports and financial statements to the Mayor’s Fund 

as part of their programmatic requirements. In the fourth year of implementation, the federal 

funding portion of the budget was replaced by support from NYC DOHMH. Programmatic 

reporting requirements remained the same throughout the 5 years. 

Logic Model

RAND and the C2C Collaborative developed the C2C logic model through an iterative 

process. Figure 3.1 presents this logic model, which connects inputs (C2C actors, such as 

funders, the C2C Collaborative, CBOs and their MHPs, and resources); C2C program strategies 

(mental health supports and referrals); outputs (“products” delivered by implementing C2C 

program strategies); and intended outcomes at the individual, program, and systems levels. The 

content of this logic model is consistent with both C2C-specific goals and current literature on 

best practices in implementing sustainable, evidence-based practices (EBPs) in community 

settings (Aarons, Hurlburt, and Horwitz, 2011; Scheirer and Dearing, 2011). 

Inputs

Inputs are all participants, community assets, and resources, including programs and policies, 

that can support program implementation or client access to mental health services. Consistent 

with current best practices on creating sustainable impacts on complex health challenges, C2C 

brings together a breadth of partners from multiple sectors, including those outside the health 

system (Erickson and Andrews, 2011). Collaboration from such a group of partners can lead to 

community-driven approaches to solving health challenges, the buy-in needed to sustain such 

changes, and the political will to enact policy changes that ultimately lead to wider and longerlasting change (Towe et al., 2016). This multisectoral group of C2C partners supports a set of 

strategies to reach the desired C2C outcomes. 

C2C Program Support Strategies

C2C program strategies include a range of activities to promote the integration of new mental 

health skills and services into routine CBO work, including the four core C2C skills, as well as 

referral pathways between the CBO and MHP. C2C’s core skills were selected from an array of 

possible approaches based on evidence of each to effect positive change on mental health 

outcomes and to be implemented by individuals without specialized (e.g., clinical) training in 

mental health care delivery. As the program matured, the C2C Collaborative encouraged 

17 
 

participating organizations to build on this initial set of skills through the addition of other 

context-driven skills or services that can be implemented by non–mental health specialists with 

the right supports. 

Another key C2C strategy was the implementation of a systematic and ongoing QI process 

to assess and continually enhance outcomes for program participants (Hunter et al., 2009). 

Although participating organizations entered the program with varying levels of experience 

implementing QI strategies, the C2C Collaborative provided technical assistance on key 

components, including setting program goals and targets; systematically collecting and 

reviewing data to gauge progress; interpreting program data; and using data to make targeted, 

informed choices about program changes or enhancements. 

C2C Outcome

As illustrated in Figure 3.1, C2C implementation was expected to produce outcomes at the 

staff, participant, organization, and system levels. Many, but not all, outcomes, were investigated 

in the C2C evaluation. 

At the CBO staff level, C2C was expected to 

• improve knowledge and attitudes related to mental health (e.g., stigma) 

• improve knowledge and confidence to address mental health issues 

• increase use of new C2C skills with CBO participants 

• improve staff well-being and retention 

• facilitate the use of QI methods to focus and improve the effect of the incorporated skills 

and methods, and to further expand on them. 

At the participant level, C2C aimed to demonstrate that implementing C2C program 

strategies into CBO workflows can 

• improve participants’ knowledge and attitudes about mental health issues 

• improve engagement in and use of CBO services 

• increase engagement in and use of mental health services, when needed 

• improve mental health outcomes and general functioning of participants 

• increase participants’ ability to achieve other targeted program-specific outcomes in areas 

such as education, housing stability, and employment. 

At the organizational level, C2C was expected to 

• increase capacity, awareness, and confidence to promote mental health care and support 

an organizational culture shift to integrate mental health awareness 

• improve awareness and skills of MHP clinicians to support and partner with CBOs 

• strengthen referral systems between CBOs and MHPs 

• improve buy-in (at both CBOs and MHPs) for task-shifting 

• improve ability to use QI techniques 

• provide more effective general CBO services 

• reduce stigma. 

18 
Systems-level outcomes of C2C were expected to include 

• greater mental health services coverage for low-income populations 

• more efficient and effective use of mental health specialists and nonspecialists in MHPs 

and CBOs 

• shift in focus toward promotion and prevention services 

• increased presence of MHPs in communities and an increase in MHPs partnering with 

CBOs to meet needs 

• increased capacity of the mental health workforce to provide culturally competent care 

• reduced health services spending and government outlays 

• reduced mental health disparities 

• reduced disability. 

19 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

INPUTS 

Funders 

• CNCS Social Innovation Fund

• private funders

C2C Collaborative 

• Mayor's Fund to Advance NYC

• Mayor's Office for Economic

Opportunity

• NYC DOHMH

Evaluators/TA Providers 

• RAND Corporation

• NYU McSilver Institute

15 Cross-Sector CBO-MHP 

Relationships 

15 competitively selected CBOs

contractwith at least one MHP.

CBO-MHPs leverage the existing

strengths of CBOs and MHPs,

while building the capacity of

each to take on new roles to 

advance community mental

health.

Community Assets & Resources 

• ThriveNYC

– MHFA training

• mental and behavioral health 

services providers 

• policies (e.g., Medicaid

redesign)

• multi-sectoral stakeholders

STRATEGIES/ACTIVITIES 

MHFA to increase recognition and

improve response to MH symptoms 

and signs of distress

MI to improve trust and engagement

of CBO participants and support

positive behavior changes

PE to provide MH information and

support optimal follow-up

Screening to identify mental and

behavioral health needs and provide

follow-up support through referrals 

and other interventions

Core MH Interventions 

CBOs and MHPs develop and adapt

training and implementation

strategiesfor the CBO context

CBOs staff integrate the core

interventions into their routine CBO

activities

CBO staff trained in core interventions

by MHPs 

CBOs and MHPs establish formal

working relationship and scope of

work

CBOs identify and connect

participantsin need of clinical

supports withstepped clinical care

through the MHP and other

MHPs provide CBO staff with 

structured ongoing coaching and 

support

Additional skills: CBOs and MHPs

encouraged to identify and embed

additional MH skills/support that

meet the needs of their clients into

their services

specialists 

MHPs and CBOs use quality

improvement strategies to enhance 

implementation

OUTCOMES 

CBO Staff Outcomes 

Improved knowledge and attitudes

relatedto MH (e.g., stigma)

Improved knowledge and confidence

to address mental health

Use new skills effectively with CBO

participants

Improved well-being

Improved retention

CBO Participant Outcomes 

Improved knowledge and attitudes Improved engagement in and Improved mental health, functioning,

toward mental health, personal utilization of CBO services and well-being

mental/behavioral health needs, and • decrease in symptoms

help seeking Improved engagement in and • increase in healthy behaviors

utilization of MH services (when

needed) Improved social outcomes linked to

CBO services

Organization Outcomes 

Increased CBO organizational

capacity, awareness, and confidence

to promote mental/behavioral health

and support culture shift to integrate

MH into CBO

Improved awareness and skills of MHP 

clinicians to support and partner with 

CBOs

Strengthening referralsystems 

between the CBOs and MHPs

More effective CBO services

Reduce stigma

Reduced government outlays 

Reduced disparities in MH outcomes

Workforce buy-in for task-shifting

Ability to use QI to iterate,add skills

and activities, and improve

performance

Greatermental health services

coverage for low-income populations,

including

• improved access to care in

community settings

• more holistic care, including

prevention, health promotion,

and treatment options

• culturally relevant care

• evidence-informed care

More efficient and effective use of

MH specialists and nonspecialists in

the CBO and MHPworkforce

Reduced health services spending

• reduced avoidable

hospitalizations and emergency

room visits 

Systems Outcomes 

Shift toward promotion and

prevention services

Reduced disability

 

 

Figure 3.1. C2C Program Logic Model

SOURCE: Adapted from Ayer et al., 2018, Figure 1.2.

20 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

C2C Evaluation

The C2C evaluation sought to understand how the C2C program can promote mental health, 

prevent mental health issues from developing or worsening, and improve mental health outcomes 

overall. For long-term changes to occur, several early and intermediate steps needed happen 

first. The early phase of C2C focused on providing staff at CBOs with the four core mental 

health skills. Under the task-shifting framework, staff used these skills to identify their clients’ 

unmet mental health needs. As CBO staff became more adept at using these C2C skills, (with 

supervision and coaching provided by the MHP or other CBO staff with these capabilities), they 

would in theory be able to address many of their clients’ mental health needs and ease introductions 

when referring to MHPs for clients with more serious issues. The MHP remains connected with 

the CBO during implementation through continued promotion of task-shifting, ongoing coaching, 

monitoring, and support. 

In this section, we summarize the confirmatory and exploratory research questions that test 

the expectations outlined in the logic model. The overall C2C evaluation is divided into three 

evaluations that assess the implementation, impact, and cost of the C2C program. Each evaluation 

is covered in detail in Parts II to IV, respectively. Confirmatory research questions are the 

primary, hypothesis-driven questions, whereas exploratory questions are secondary and tertiary 

questions or those where there is little scientific basis for the formation of clear hypotheses. 

Implementation Evaluation

The goals of the C2C implementation evaluation were to examine (1) how C2C was 

implemented within and across CBOs; (2) whether CBO staff exhibit improved knowledge, 

behaviors, and attitudes about mental health issues and services; (3) how C2C implementation 

changes CBO client access to mental health services; and (4) facilitators of and barriers to C2C 

implementation. 

The primary and secondary implementation evaluation research questions are displayed in 

Table 3.2. 

Table 3.2 Implementation Evaluation Research Questions

Primary Secondary

How were the C2C program strategies

implemented? (Chapters 4 and 5)

To what extent have the CBOs identified clients

with mental health or substance use issues as a 

result of C2C implementation? (Chapter 5)

What components of the C2C program strategies were

implemented, how were they implemented, and to what extent?

What was the role of the MHP in terms of training and supporting

the CBO to implement C2C program strategies over time? 

To what extent have the CBOs referred clients to mental health 

services as a result of C2C implementation?

To what extent have C2C clients who screened positive for

mental health or substance use/misuse issues received mental

health services from the MHP and/or the C2C modalities

beyond screening (i.e., PE, MI, or MHFA)?

21 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Primary Secondary

Do CBO staff have improved knowledge of 

mental health and C2C modalities, as well as

attitudes, and behaviors about mental health 

issues and services? (Chapter 6)

What are the key facilitators of and barriers to 

effective implementation of C2C program

strategies within and across CBO and MHP 

partnerships? (Chapters 5 and 6)

How have the CBOs reduced barriers (e.g., perceptual and 

logistical barriers) to clients’ access to and utilization of mental 

health services?

What are the key facilitators of and barriers to CBO engagement

of clients in C2C program strategies within and across CBOs?

How have CBOs used continuous QI (CQI) as part of C2C

program implementation?

What are the best practices learned from sites demonstrating

effective implementation with regard to key objectives (e.g.,

CQI, identification of client mental health or substance

use/misuse issues, client engagement in C2C program

strategies, and working with MHPs to optimize task-shifting)?

Impact Evaluation

The overarching goal of the C2C impact evaluation was to examine the effect of C2C’s 

task-shifting approach of integrating mental health skills and support into the work of CBOs on 

participants’ access to and utilization of mental health care, mental health and functioning, and 

non–mental health outcomes targeted by the programs and services offered by the CBOs (e.g., 

employment, education, housing, incarceration). The primary inclusion criterion for CBO clients 

participating in the impact study was meeting a low symptom threshold for one or more of the 

following common mental health conditions: depression, anxiety, PTSD, alcohol misuse, or drug 

misuse. The impact evaluation focused on these conditions because these are the most common 

issues among the target populations, the CBOs identified these as the most relevant for their 

specific target populations, and they are the conditions for which CBOs were screening clients as 

part of C2C. 

There are two key reasons that minimum symptom threshold was an inclusion criterion. First, 

this criterion is consistent with the intention of C2C, because the program focuses on increasing 

access to mental health care for those with or at high risk for a mental health condition. Having 

symptoms of a mental health condition is evidence that the individual is in this higher risk group. 

Second, it is unlikely that individuals with no mental health symptoms would exhibit any 

detectable change (i.e., measurable impact) in mental health symptoms or service utilization. 

Thus, the impact study focuses on individuals who met a minimum threshold for a common 

mental health condition. The primary and secondary impact evaluation research questions are 

shown in Table 3.3. 

For all primary research questions, we expected C2C participants to show greater 

improvement in outcomes relative to the comparison group at the 6- and 12-month follow-up 

assessments. However, we anticipated that intervention effects would be small in size for a few 

22 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 3.3. Impact Evaluation Research Questions

Primary Secondary

Do C2C participants have greater reductions in 

barriers to mental health care and greater increases

in utilization of mental health services relative to 

comparison group participants? (Chapter 7)

Do C2C participants show greater positive 

improvement in depression, generalized anxiety,

PTSD, alcohol use, substance use, and general

psychological distress relative to comparison group 

participants? (Chapter 8)

Do C2C participants show improved outcomes in 

the domains of employment, housing, education,

and incarceration relative to comparison group 

participants? (Chapter 9)

Does program impact on barriers and utilization vary 

between subpopulations (e.g., underemployed or

unemployed adults, unemployed/disconnected youth,

caregivers of children ages 0–4) or CBO service types

(e.g., job training and employment program, youth 

development program)?

Does program impact on mental health symptoms vary 

between subpopulations (e.g., underemployed or

unemployed adults, unemployed/disconnected youth, 

caregivers of children ages 0–4) or CBO service types

(e.g., job training and employment program, youth 

development program)?

Does program impact on employment, housing,

education, and incarceration vary between subpopulations

(e.g., underemployed or unemployed adults, unemployed/

disconnected youth, caregivers of children ages 0–4) or 

CBO service types (e.g., job training and employment

program, youth development program)?

reasons. First, by design, the intervention is not a long-term or intensive intervention like 

individual therapy—so we would not expect C2C alone to show effects on mental health 

symptoms similar to formal mental health treatment, for example. In addition, C2C was available 

to CBO clients regardless of their initial symptom levels, and the evaluation did not require a 

clinically significant level of symptoms; it is more difficult to detect an improvement in 

symptoms when the starting point is low. Further, the evaluation compared C2C to “as usual” 

services in a heterogeneous population—two factors known to pose challenges to detecting 

significant group differences in intervention studies (Kazdin, 2015; Kravitz, Duan, and Braslow, 

2004; Varadhan and Seeger, 2013). In other words, “treatment as usual” very often performs just 

as well as evidence-based mental health interventions, even when those interventions have 

demonstrated efficacy in studies when compared with no-treatment or to waitlist control conditions 

(Kazdin, 2015); and interventions typically work better for certain individuals than for others—a 

phenomenon known as “treatment heterogeneity” (Kravitz, Duan, and Braslow, 2004; Varadhan 

and Seeger, 2013). The secondary research questions in Table 3.3 allowed us to explore treatment 

heterogeneity more carefully by examining the effect of C2C on different subgroups of participants 

for each set of outcomes. 

Cost Evaluation

The goal of the C2C cost evaluation was to quantify the resources required to implement and 

maintain the program, overall and average per program client. We defined C2C program client as 

any CBO client who received at least one C2C skill from staff: screening, MHFA, MI, or PE. 

The program cost analysis results, when combined with the program’s economic impacts, can 

inform government agencies regarding whether to continue investing in C2C and expand it to 

23 
 

 

 

 

other CBOs. CBOs thinking about implementing C2C may also find these results helpful as they 

weigh the program’s costs and determine how to financially support the model. Table 3.4 shows 

the cost evaluation research question. 

Table 3.4. Cost Evaluation Research Questions

Primary Secondary

What are the costs to CBOs associated with How did the average cost of serving a client change 

implementation of the C2C program, overall and over time?

average per program client?

Contribution to the Literature

Findings from the C2C evaluation will add to the existing evidence related to mental 

health task-shifting and its effect on outcomes for low-income populations. It will also 

provide information on implementation best practices, costs, and potential impacts for CBOs, 

policymakers, and others wishing to replicate C2C. In the future, findings from C2C may help to 

inform a variety of models intended to expand the mental health workforce and increase access 

to mental health services for high-risk and underserved communities. 

24 
 

References 

Aarons, G. A., M. Hurlburt, and S.M. Horwitz, “Advancing a Conceptual Model of EvidenceBased Practice Implementation in Public Service Sectors,” Administration and Policy in 

Mental Health, 38(1), 2011, pp. 4–23. 

Acri, M., S. Frank, S.S. Olin, G. Burton, J. L. Ball, J. Weaver, and K. E. Hoagwood, “Examining 

the Feasibility and Acceptability of a Screening and Outreach Model Developed for a Peer 

Workforce,” Journal of Child and Family Studies, 24(2), 2015, pp. 341–350. 

Acri, M., S. S. Olin, G. Burton, R. J. Herman, and K. E. Hoagwood, “Innovations in the 

Identification and Referral of Mothers at Risk for Depression: Development of a Peer-toPeer Model,” Journal of Child and Family Studies, 23(5), 2014, pp. 837–843. 

Aitken, C., D. Wain, D. I. Lubman, L. Hides, and M. Hellard, “Mental Health Screening Among 

Injecting Drug Users Outside Treatment Settings: Implications for Research and Health 

Services,” Mental Health and Substance Use, 1(2), 2008, pp. 99–103. 

Akincigil, A., Olfson, M., Siegel, M., Zurlo, K. A., Walkup, J. T., and S. Crystal, “Racial and 

Ethnic Disparities in Depression Care in Community-Dwelling Elderly in the United States,” 

American Journal of Public Health, 102(2), 2012, pp. 319–328. 

Allen, J., R. Balfour, R. Bell, and M. Marmot, “Social Determinants of Mental Health, International 

Review of Psychiatry, 26(4), 2014, pp. 392–407. 

American Hospital Association, “The State of the Behavioral Health Workforce: A Literature 

Review,” 2016. 

http://www.hpoe.org/Reports-HPOE/2016/aha_Behavioral_FINAL.pdf 

Anderson, C. M., C. S. Robins, C. G. Greeno, H. Cahalane, V. C. Copeland, and R. M. Andrews, 

“Why Lower Income Mothers Do Not Engage with the Formal Mental Health Care System: 

Perceived Barriers to Care,” Qualitative Health Research, 16(7), 2006, pp. 926–943. 

Anderson, J. K., E. Howarth, M. Vainre, P. B. Jones, and A. Humphrey, “A Scoping Literature 

Review of Service-Level Barriers for Access and Engagement with Mental Health Services 

for Children and Young People,” Children and Youth Services Review, 77, 2017, pp. 164–176. 

Ayangbayi, T., A. Okunade, M. Karakus, and T. Nianogo, “Characteristics of Hospital Emergency 

Room Visits for Mental and Substance Use Disorders,” Psychiatric Services, 68(4), 2017, 

pp. 408–410. 

Ayer, L., M. S. Dunbar, M. Martineau, C. Stevens, D. Schultz, W. Y. Chan, M. Abbott, R. Weir, 

H. H. Liu, D. Siconolfi, and V. L. Towe, Evaluation of the Connections to Care (C2C) 

Initiative: Interim Report, Santa Monica, Calif.: RAND Corporation, RR-2497-MFANYC, 

2018. 

https://www.rand.org/pubs/research_reports/RR2497.html 

25 
 

Barrowclough, C., G. Haddock, N. Tarrier, S. W. Lewis, J. Moring, R. O’Brien, N. Schofield, 

J. McGovern, “Randomized Controlled Trial of Motivational Interviewing, Cognitive 

Behavior Therapy, and Family Intervention for Patients with Comorbid Schizophrenia 

and Substance Use Disorders,” American Journal of Psychiatry, 158(10), 2001, 

pp. 1706–1713. 

Belkin, G. S., J. Unutzer, R. C. Kessler, H. Verdeli, G. J. Raviola, K. Sachs, C. Oswald, 

E. Eustache, “Scaling up for the “Bottom Billion”: “5 x 5” Implementation of 

Community Mental Health Care in Low-Income Regions,” Psychiatric Services, 62(12), 

2011, pp. 1494–1502. 

Biaggi, A., S. Conroy, S. Pawlby, and C. M. Pariante, “Identifying the Women at Risk of 

Antenatal Anxiety and Depression: A systematic Review,” Journal of Affective Disorders, 

191, 2016, pp. 62–77. 

Bisson, J. I., A. Ehlers, R. Matthews, S. Pilling, D. Richards, and S. Turner, “Psychological 

Treatments for Chronic Post-Traumatic Stress Disorder: Systematic Review and MetaAnalysis,” British Journal of Psychiatry, 190, 2007, pp. 97–104. 

Bolton, P., J. Bass, R. Neugebauer, H. Verdeli, K. F. Clougherty, P. Wickramaratne, 

L. Speelman, L. Ndogoni, and M. Weissman, “Group Interpersonal Psychotherapy for 

Depression in Rural Uganda: A Randomized Controlled Trial,” JAMA, 289(23), 2003, 

pp. 3117–3124. 

Brenes, G. A., Penninx, B. W., Judd, P. H., Rockwell, E., Sewell, D. D., and J. L. Wetherell, 

“Anxiety, Depression and Disability Across the Lifespan,” Aging Mental Health, 12(1), 

2008, pp. 158–163. 

Brummelte, S., and L. A. M. Galea, “Depression During Pregnancy and Postpartum: Contribution 

of Stress and Ovarian Hormones,” Progress in Neuro-Psychopharmacology and Biological 

Psychiatry, 34(5), 2010, pp. 766–776. 

Burke, B. L., H. Arkowitz, and M. Menchola, “The Efficacy of Motivational Interviewing: A 

Meta-Analysis of Controlled Clinical Trials,” Journal of Consulting Clinical Psychology, 

71(5), 2003, pp. 843–861. 

Butterworth, P., and B. Rodgers, “Mental Health Problems and Marital Disruption: Is It the 

Combination of Husbands and Wives’ Mental Health Problems That Predicts Later 

Divorce?,” Social Psychiatry and Psychiatric Epidemiology, 43(9), 2008, pp. 758–763. 

Cadigan, J. M., C. M. Lee, and M. E. Larimer, “Young Adult Mental Health: A Prospective 

Examination of Service Utilization, Perceived Unmet Service Needs, Attitudes, and Barriers 

to Service Use,” Prevention Science, 20(3), 2019, pp. 366–376. 

26 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Chang, C. K., R. D. Hayes, G. Perera, M. T. M. Broadbent, A. C. Fernandes, W. E. Lee, 

M. Hotopf, and R. Stewart, “Life Expectancy at Birth for People with Serious Mental Illness 

from a Secondary Mental Health Care Case Register in London, UK,” American Journal of 

Epidemiology, 173, 2011, pp. S311–S311. 

Chatterjee, S., S. Naik, S John, H. Dabholkar, M. Balaji, M. Koschorke, M. Varghese, 

T. Rangaswamy, H. A. Weiss, P. Williams, P. McCrone, V. Patel, G. Thornicroft, 

“Effectiveness of a Community-Based Intervention for People with Schizophrenia and 

Their Caregivers in India (COPSI): A Randomized Controlled Trial,” The Lancet, 383(9926), 

2014, pp. 1385–1394. 

Chibanda, D., P. Mesu, L. Kajawu, F. Cowan, R. Araya, and M. A. Abas, “Problem-Solving 

Therapy for Depression and Common Mental Disorders in Zimbabwe: Piloting a TaskShifting Primary Mental Health Care Intervention in a Population with a High Prevalence of 

People Living with HIV,” BMC Public Health, 11, 2011, p. 828. 

Chow, J. C., K. Jaffee, and L. Snowden, ” Racial/Ethnic Disparities in the Use of Mental Health 

Services in Poverty Areas,” American Journal of Public Health, 93(5), 2003, pp. 792–797. 

Chung, B., L. Jones, E. L. Dixon, J. Miranda, K. Wells, and Council Community Partners in 

Care Steering, “Using a Community Partnered Participatory Research Approach to Implement 

a Randomized Controlled Trial: Planning Community Partners in Care,” Journal of Health 

Care for the Poor and Underserved, 21(3), 2010, pp. 780–795. 

City of New York, Office of the Mayor, Report: Understanding New York City’s Mental Health 

Challenge, press release, New York, NY, 2015. As of September 9, 2020: 

https://www1.nyc.gov/assets/home/downloads/pdf/press-releases/2015/thriveNYC_white 

_paper.pdf 

Clement, S., E. Brohan, D. Jeffery, C. Henderson, S. L. Hatch, and G. Thornicroft, “Development 

and Psychometric Properties the Barriers to Access to Care Evaluation Scale (BACE) 

Related to People with Mental Ill Health,” BMC Psychiatry, 12, 2012, p. 36. 

Cunningham, M., and L. H. Zayas, “Reducing Depression in Pregnancy: Designing Multimodal 

Interventions. Social Work, 47(2), 2002, pp. 114–123. 

Cunningham, R. M., S. T. Chermack, M. A. Zimmerman, J. T. Shope, C. R. Bingham, 

F. C. Blow, and M. A. Walton, ” Brief Motivational Interviewing Intervention for Peer 

Violence and Alcohol Use in Teens: One-Year Follow-Up,” Pediatrics, 129(6), 2012, 

pp. 1083–1090. 

Cygan-Rehm, K., D. Kuehnle, and M. Oberfichtner, “Bounding the Causal Effect of 

Unemployment on Mental Health: Nonparametric Evidence from Four Countries,” 

Health Economics, 26(12), 2017, pp. 1844–1861. 

27 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

De Roten, Y., G. Zimmermann, D. Ortega, and J. N. Despland, “Meta-Analysis of the Effects of 

MI Training on Clinicians’ Behavior. Journal of Substance Abuse Treatment, 45(2), 2013, 

pp. 155–162. 

Donker, T., K. M. Griffiths, P. Cuijpers, and H. Christensen, “Psychoeducation for Depression, 

Anxiety and Psychological Distress: A Meta-Analysis. BMC Medicine, 7, 2009, p. 79. 

Dos Santos, P. F., M. L. Wainberg, J. M. Caldas-de-Almeida, B. Saraceno, and J. D. Mari, 

“Overview of the Mental Health System in Mozambique: Addressing the Treatment Gap 

with a Task-Shifting Strategy in Primary Care,” International Journal of Mental Health 

Systems, 10, 2016, p. 1. 

Duncan, G. J., Magnuson, K., and E. Votruba-Drzal, “Moving Beyond Correlations in Assessing 

the Consequences of Poverty,” Annual Review of Psychology, 68, 2017, pp. 41–434. 

Elliott, D. (2016). Two American experiences: The racial divide of poverty. 

https://www.urban.org/urban-wire/two-american-experiences-racial-divide-poverty 

Erickson, D., and N. Andrews, “Partnerships Among Community Development, Public Health, 

and Health Care Could Improve the Well-Being of Low-Income People,” Health Affairs, 

30(11), 2011, pp. 2056–2063. 

Essau, C. A., P. M. Lewinsohn, J. R. Seeley, and S. Sasagawa, “Gender Differences in the 

Developmental Course of Depression,” Journal of Affective Disorders, 127(1–3), 2010, 

pp. 185–190. 

Goldman-Mellor, S., A. Caspi, L. Arseneault, N. Ajala, A. Ambler, A. Danese, H. Fisher, 

A. Hucker, C. Odgers, T. Williams, C. Wong, T. E. Moffitt, “Committed to Work but 

Vulnerable: Self-Perceptions and Mental Health in NEET 18-Year Olds from a 

Contemporary British Cohort,” Journal of Child Psychology and Psychiatry, 57(2), 2016, 

pp. 196–203. 

Gopalan, G., L. Goldstein, K. Klingenstein, C. Sicher, C. Blake, and M. M. McKay, “Engaging 

Families into Child Mental Health Treatment: Updates and Special Considerations,” Journal 

of the Canadian Academy of Child and Adolescent Psychiatry, 19(3), 2010, pp. 182–196. 

Gournellis, R., K. Tournikioti, G. Touloumi, C. Thomadakis, P. G. Michalopoulou, 

C. Christodoulou, A. Papadopoulou, and A. Douzenis, “Psychotic (Delusional) Depression 

and Suicidal Attempts: A Systematic Review and Meta-Analysis,” Acta Psychiatrica 

Scandinavica, 137(1), 2018, pp. 18–29. 

Govindarajan, V., and R. Ramamurti, “Task Shifting Could Help Lower Costs in U.S. Health 

Care,” Harvard Business Review. 

https://hbr.org/2018/07/task-shifting-could-help-lower-costs-in-u-s-health-care?utm_medium 

=social&utm_source=twitter&utm_campaign=hbr 

28 
 

 

 

 

 

 

 

 

 

 

 

 

Hadfield, H., and A. Wittkowski, “Women’s Experiences of Seeking and Receiving 

Psychological and Psychosocial Interventions for Postpartum Depression: A Systematic 

Review and Thematic Synthesis of the Qualitative Literature,” Journal of Midwifery & 

Women’s Health, 62(6), 2017, pp. 723–736. 

Hahm, H. C., B. Cook, A. Ault-Brutus, and M. Alegria, “Intersection of Race-Ethnicity and 

Gender in Depression Care: Screening, Access, and Minimally Adequate Treatment,” 

Psychiatric Services, 66(3), 2015, pp. 258–264. 

Hingson, R., and J. Howland, “Alcohol as a Risk Factor for Injury or Death Resulting from 

Accidental Falls: A Review of the Literature,” Journal of Studies on Alcohol and Drugs, 

48(3), 1987, pp. 212–219. 

Hofmann, S. G., and J. A. Smits, “Cognitive-Behavioral Therapy for Adult Anxiety Disorders: 

A Meta-Analysis of Randomized Placebo-Controlled Trials,” Journal of Clinical Psychiatry, 

69(4), 2008, pp. 621–632. 

Hohmna, M., N. Doran, and I. Koutsenok, “Motivational Interviewing Training for Juvenile 

Correctional Staff in California: One Year Initial Outcomes,” Journal of Offender 

Rehabilitation, 48(7), 2009, pp. 635–648. 

Huang, K. Y., J. Nakigudde, E. Calzada, M. J. Boivin, G. Ogedegbe, and L. M. Brotman, 

“Implementing an Early Childhood School-Based Mental Health Promotion Intervention in 

Low-Resource Ugandan Schools: Study Protocol for a Cluster Randomized Controlled 

Trial,” Trials, 15, 2014, p. 471. 

Hunter, S. B., M. Chinman, P. Ebener, P. Imm, A. Wandersman, and G. W. Ryan, “Technical 

Assistance as a Prevention Capacity-Building Tool: A Demonstration Using the Getting to 

Outcomes I Framework,” Health Education & Behavior 36(5), 2009, pp. 810–828. 

Jensen, C. D., C. C. Cushing, B. S. Aylward, J. T. Craig, D. M. Sorell, and R. G. Steele, 

“Effectiveness of Motivational Interviewing Interventions for Adolescent Substance Use 

Behavior Change: A Meta-Analytic Review,” Journal of Consulting and Clinical Psychology, 

79(4), 2011, pp. 433–440. 

Kagee, A., A. C. Tsai, C. Lund, and M. Tomlinson, “Screening for Common Mental Disorders in 

Low Resource Settings: Reasons for Caution and a Way Forward,” International Health, 

5(1), 2013, pp. 11–14. 

Kakuma, R., H. Minas, N. van Ginneken, M. R. Dal Poz, K. Desiraju, J. E. Morris, S. Saxena, 

and R. M. Scheffler, “Human Resources for Mental Health Care: Current Situation and 

Strategies for Action,” The Lancet, 378(9803), 2011, pp. 1654–1663. 

29 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Kataoka, S. H., L. Zhang, and K. B. Wells, “Unmet Need for Mental Health Care Among 

U.S. Children: Variation by Ethnicity and Insurance Status,” American Journal of 

Psychiatry, 159(9), 2002, pp. 1548–1555. 

Kazdin, A. E., “Treatment as Usual and Routine Care in Research and Clinical Practice,” 

Clinical Psychology Review, 42, 2015, pp. 168–178. 

Kazdin, A. E. (2017). Addressing the treatment gap: A key challenge for extending evidencebased psychosocial interventions. Behavior Research and Therapy, 88, 7–18. 

Kazdin, A. E., and S. Rabbitt, “Novel Models for Delivering Mental Health Services and 

Reducing the Burdens of Mental Illness,” Clinical Psychological Science, 1(2), 2013, 

pp. 170–191. 

Kessler, R. C., M. Petukhova, N. A. Sampson, A. M. Zaslavsky, and H. U. Wittchen, “TwelveMonth and Lifetime Prevalence and Lifetime Morbid Risk of Anxiety and Mood Disorders in 

the United States,” International Journal of Methods in Psychiatric Research, 21(3), 2012, 

pp. 169–184. 

Kessler, R. C., S. Heeringa, M. D. Lakoma, M. Petukhova, A. E. Rupp, M. Schoenbaum, 

P. S. Wang, and A. M. Zaslavsky, “Individual and Societal Effects of Mental Disorders on 

Earnings in the United States: Results from the National Comorbidity Survey Replication,” 

American Journal of Psychiatry, 165(6), 2008, pp. 703–711. 

Kravitz, R. L., N. Duan, and J. Braslow, “Evidence-Based Medicine, Heterogeneity of 

Treatment Effects, and the Trouble with Averages,” Milbank Quarterly, 82(4), 2004, 

pp. 661–687. 

Krysinska, K., and D. Lester, “Post-Traumatic Stress Disorder and Suicide Risk: A Systematic 

Review,” Archives of Suicide Research, 14(1), 2010, pp. 1–23. 

Kvalevaag, A. L., P. G. Ramchandani, O. Hove, J. Assmus, M. Eberhard-Gran, and E. Biringer, 

“Paternal Mental Health and Socioemotional and Behavioral Development in Their Children,” 

Pediatrics, 131(2), 2013, pp. e463–e469. 

Lazear, K. J., S. A. Pires, M. R. Isaacs, P. Chaulk, and L. Huang, “Depression Among LowIncome Women of Color: Qualitative Findings from Cross-Cultural Focus Groups,” Journal 

of Immigrant and Minority Health, 10(2), 2008, pp. 127–133. 

Legha, R., E. Eustache, T. Therosme, K. Boyd, F. Reginald, G. Hilaire, S. Daimyo, G. Jerome, 

H. Verdeli, and G. Raviola, “Taskshifting: Translating Theory into Practice to Build a 

Community Based Mental Health Care System in Rural Haiti,” Intervention, 13(3), 2015, 

pp. 248–267. 

30 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Liem, J. H., K. Lustig, and C. Dillon, “Depressive Symptoms and Life Satisfaction Among 

Emerging Adults: A Comparison of High School Dropouts and Graduates,” Journal of Adult 

Development, 17(1), 2010, pp. 33–43. 

Lubman, D. I., L. Hides, A. Scaffidi, K. Elkins, M. Stevens, and R. Marks, “Implementing 

Mental Health Screening Within a Youth Alcohol and Other Drug Service,” Mental Health 

and Substance Use, 1(3), 2008, pp. 254–261. 

Lukens, E., and W. R. McFarlane, “Psychoeducation as Evidence-Based Practice: Considerations 

for Practice, Research, and Policy,” Brief Treatment and Crisis Intervention, 4(3), 2004, 

pp. 205–225. 

Lundahl, B., and B. L. Burke, “The Effectiveness and Applicability of Motivational Interviewing: 

A Practice-Friendly Review of Four Meta-Analyses,” Journal of Clinical Psychology, 

65(11), 2009, pp. 1232-1245. 

Matsuzaka, C. T., M. Wainberg, A. N. Pala, E. V. Hoffmann, B. M. Coimbra, R. F. Braga, 

A. C. Sweetland, and M. F. Mello, “Task Shifting Interpersonal Counseling for Depression: 

A Pragmatic Randomized Controlled Trial in Primary Care,” BMC Psychiatry, 17, 2017, 

p. 225. 

Maynard, B. R., C. P. Salas-Wright, and M. G. Vaughn, “High School Dropouts in Emerging 

Adulthood: Substance Use, Mental Health Problems, and Crime,” Community Mental Health 

Journal, 51(3), 2015, pp. 289–299. 

Mehta, P., A. Brown, B. Chung, F. Jones, L. Tang, J. Gilmore, J. Miranda, and K. Wells, 

“Community Partners in Care: 6-Month Outcomes of Two Quality Improvement Depression 

Care Interventions in Male Participants,” Ethnicity & Disease, 27(3), 2017, pp. 223–232. 

Milgrom, J., A. W. Gemmill, J. L. Bilszta, B. Hayes, B. Barnett, J. Brooks, J. Ericksen, 

D. Ellwood, and A. Buist, “Antenatal Risk Factors for Postnatal Depression: A Large 

Prospective Study,” Journal of Affective Disorders, Vol. 108, No. 1–2, 2008, pp. 147–157. 

Miller, W. R., and S. Rollnick, Motivational Interviewing: Helping People Change, New York, NY: 

The Guilford Press, 2013. 

Milner, A., Page, A., and A.D. Lamontagne, “Cause and Effect in Studies on Unemployment, 

Mental Health and Suicide: A Meta-Analytic and Conceptual Review,” Psychological 

Medicine, Vol. 44, No. 5, 2014, pp. 909. 

Miranda, J., J. Y. Chung, B. L. Green, J. Krupnick, J. Siddique, D. A. Revicki, and T. Belin, 

“Treating Depression in Predominantly Low-Income Young Minority Women: A Randomized 

Controlled Trial,” JAMA, Vol. 290, No. 1, 2003, pp. 57–65. 

31 
 

 

Olesen, S. C., Butterworth, P., Leach, L. S., Kelaher, M., and J. Pirkis, “Mental Health Affects 

Future Employment as Job Loss Affects Mental Health: Findings from a Longitudinal 

Population Study,” BMC Psychiatry, Vol. 13, No. 1, 2013, p. 144. 

Olsson, M. O., L. Bradvik, A. Ojehagen, and A. Hakansson, “Risk Factors for Unnatural Death: 

Fatal Accidental Intoxication, Undetermined Intent and Suicide: Register Follow-up in a 

Criminal Justice Population with Substance Use Problems,” Drug and Alcohol Dependence, 

Vol. 162, 2016, pp. 176–181. 

Pagoto, S. L., B. Spring, E. J. Coups, S. Mulvaney, M. F. Coutu, and G. Ozakinci, “Barriers and 

Facilitators of Evidence-Based Practice Perceived by Behavioral Science Health Professionals,” 

Journal of Clinical Psychology, Vol. 63, No. 7, 2007, pp. 695–705. 

Patel, V., G. S. Belkin, A. Chockalingam, J. Cooper, S. Saxena, and J. Unutzer. (2013). Grand 

challenges: Integrating mental health services into priority health care platforms. PLOS 

Medicine, Vol. 10, No. 5), e1001448. 

Paul, K. I., and K. Moser, “Unemployment Impairs Mental Health: Meta-Analyses,” Journal of 

Vocational Behavior, Vol. 74, No. 3, 2009, pp. 264–282. 

Pekkala, E., and L. Merinder, “Psychoeducation for Schizophrenia,” Cochrane Database of 

Systemic Reviews (2), 2002, CD002831. 

Penninx, B. W. J. H., S. Leveille, L. Ferrucci, J. T. M. van Eijk, and J. M. Guralnik, “Exploring 

the Effect of Depression on Physical Disability: Longitudinal Evidence from the Established 

Populations for Epidemiologic Studies of the Elderly,” American Journal of Public Health, 

Vol. 89, No. 9, 1999, pp. 1346–1352. 

Petrenko, C. L., S. E. Culhane, E. F. Garrido, and H. N. Taussig, “Do Youth in Out-of-Home 

Care Receive Recommended Mental Health and Educational Services Following Screening 

Evaluations?,” Child and Youth Services Review, Vol. 33, No. 10, 2011, pp. 1911–1918. 

Pharr, J. R., Moonie, S., and T. J. Bungum, “The Impact of Unemployment on Mental and 

Physical Health, Access to Health Care and Health Risk Behaviors,” ISRN Public Health, 

2012. 

Roll, J. M., J. Kennedy, M. Tran, and D. Howell, “Disparities in Unmet Need for Mental Health 

Services in the United States, 1997–2010,” Psychiatric Services, Vol. 64, No. 1), 2013, 

pp. 80–82. 

Roy-Byrne, P., M. G. Craske, G. Sullivan, R. D. Rose, M. J. Edlund, A. J. Lang, A. Bystritsky, 

S. S. Welch, D. A. Chavira, D. Golinelli, L. Campbell-Sills, C. D. Sherbourne, and M. B. Stein, 

“Delivery of Evidence-Based Treatment for Multiple Anxiety Disorders in Primary Care: A 

Randomized Controlled Rrial,” JAMA, Vol. 303, No. 19, 2010, pp. 1921–1928. 

32 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Saloner, B., and B. L. Cook, “Blacks and Hispanics Are Less Likely Than Whites to Complete 

Addiction Treatment, Largely Due to Socioeconomic Factors,” Health Affairs, Vol. 32, 

No. 1, 2013, pp. 135–145. 

Santiago, C. D., S. Kaltman, and J. Miranda, “Poverty and Mental Health: How Do Low-Income 

Adults and Children Fare in Psychotherapy?,” Journal of Clinical Psychology, Vol. 69, 

No. 2, 2013, pp. 115–126. 

Scheirer, M. A., and J. W. Dearing, “An Agenda for Research on the Sustainability of Public 

Health Programs,” American Journal of Public Health, Vol. 101, No. 11, 2011, pp. 2059–2067. 

Schwalbe, C. S., H. Y. Oh, and A. Zweben, “Sustaining Motivational Interviewing: A MetaAnalysis of Training Studies,” Addiction, Vol. 109, No. 8, 2014, pp. 1287–1294. 

Sentell, T., M. Shumway, and L. Snowden, “Access to Mental Health Treatment by English 

Language Proficiency and Race/Ethnicity,” Journal of General Internal Medicine, Vol. 22, 

No. 2, 2007, pp. 289–293. 

Shear, K., E. Frank, P. R. Houck, and C. F. Reynolds, 3rd, “Treatment of Complicated Grief: A 

Randomized Controlled Trial,” JAMA, Vol. 293, No. 21, 2005, pp. 2601–2608. 

Shippee, N. D., B. H. Rosen, K. B. Angstman, M. E. Fuentes, R. S. DeJesus, S. M. Bruce, and 

M. D. Williams, “Baseline Screening Tools as Indicators for Symptom Outcomes and Health 

Services Utilization in a Collaborative Care Model for Depression in Primary Care: A 

Practice-Based Observational Study,” General Hospital Psychiatry, Vol. 36, No. 6, 2014, 

pp. 563–569. 

Smith, J. L., K. M. Carpenter, P. C. Amrhein, A. C. Brooks, D. Levin, E. A. Schreiber, 

L. A. Travaglini, M. C. Hu, and E. V. Nunes, “Training Substance Abuse Clinicians in 

Motivational Interviewing Using Live Supervision via Teleconferencing,” Journal of 

Consulting Clinical Psychology, Vol. 80, No. 3, 2012, pp. 450–464. 

Stevens, C., E. Tosatti, L. Ayer, D. Barnes-Proby, G. Belkin, S. Lieff, and M. Martineau. 

Helpers in Plain Sight: A Guide to Implementing Mental Health Task Sharing in CommunityBased Organizations. Santa Monica, Calif.: RAND Corporation, TL-317-MFANYC, 2020. 

https://www.rand.org/pubs/tools/TL317.html 

Strandh, M., A. Winefield, K. Nilsson, and A. Hammarström, “Unemployment and Mental 

Health Scarring During the Life Course,” European Journal of Public Health, Vol. 24, 

No. 3, 2014, pp. 440–445. 

Stubbs, B., D. Vancampfort, N. Veronese, K. G. Kahl, A. J. Mitchell, P. Y. Lin, P. T. Tseng, 

J. Mugisha, M. Solmi, A. F. Carvalho, and A. Koyanagi, “Depression and Physical Health 

Multimorbidity: Primary Data and Country-Wide Meta-Analysis of Population Data from 

190,593 People Across 43 Low- and Middle-Income Countries,” Psychological Medicine, 

Vol. 47, No. 12, 2017, pp. 2107–2117. 

33 
 

 

 

 

 

 

 

 

 

Substance Abuse and Mental Health Services Administration. “Key Substance Use and Mental 

Health Indicators in the United States: Results from the 2017 National Survey on Drug Use 

and Health,” 2018. 

https://www.samhsa.gov/data/sites/default/files/cbhsq-reports/NSDUHFFR2017/ 

NSDUHFFR2017.pdf 

Terza, J. V., “Alcohol Abuse and Employment: A Second Look,” Journal of Applied Economics, 

Vol. 17, No. 4, 2002, pp. 393–404. 

Thomas, A. C., and P. K. Staiger, “Introducing Mental Health and Substance Use Screening into 

a Community-Based Health Service in Australia: Usefulness and Implications for Service 

Change,” Health & Social Care in the Community, Vol. 20, No. 6, 2012, pp. 635–644. 

Thomas, K. C., A. R. Ellis, T. R. Konrad, C. E. Holzer, and J. P. Morrissey, “County-Level 

Estimates of Mental Health Professional Shortage in the United States,” Psychiatric Services, 

Vol. 60, No. 10, 2009, pp. 1323–1328. 

ThriveNYC. (2019). “Connections to Care: Mental Health Integration in Community-Based 

Organizations. 

https://thrivenyc.cityofnewyork.us/program/connections-to-care-mental-health-integration 

-in-community-based-organizations 

Towe, V. L., L. Leviton, A. Chandra, J. C. Sloan, M. Tait, and T. Orleans, “Cross-Sector 

Collaborations and Partnerships: Essential Ingredients to Help Shape Health and WellBeing,” Health Affairs, Vol. 35, No. 11, 2016, pp. 1964–1969. 

Van Ginneken, N., P. Tharyan, S. Lewin, G. N. Rao, S. M. Meera, J. Pian, S. Chandrashekar, and 

V. Patel, “Non-Specialist Health Worker Interventions for the Care of Mental, Neurological 

and Substance-Abuse Disorders in Low- and Middle-Income Countries,” Cochrane Database 

of Systemic Reviews, Vol. 11, 2013, CD009149. 

Varadhan, R., and J. D. Seeger, “Estimation and Reporting of Heterogeneity of Treatment 

Effects,” in Velentgas P, Dreyer NA, Nourjah P, et al., editors, Developing a Protocol for 

Observational Comparative Effectiveness Research: User’s Guide. Rockville, MD: Agency 

for Healthcare Research and Quality (US); Chapter 3, 2013. 

Vigo, D., G. Thornicroft, and R. Atun, “Estimating the True Global Burden of Mental Illness,” 

Lancet Psychiatry, Vol. 3, No. 7, 2016, pp. 602–602. 

Vos, T., R. M. Barber, B. Bell, A. Bertozzi-Villa, S. Biryukov, I. Bolliger, et al., “Global, 

Regional, and National Incidence, Prevalence, and Years Lived with Disability for 301 

Acute and Chronic Diseases and Injuries in 188 Countries, 1990–2013: A Systematic 

Analysis for the Global Burden of Disease Study 2013,” The Lancet, Vol. 386, No. 9995, 

2015, pp. 743–800. 

34 
 

 

 

 

 

 

 

Wang, P. S., M. Lane, M. Olfson, H. A. Pincus, K. B. Wells, and R. C. Kessler, “Twelve-Month 

Use of Mental Health Services in the United States: Results from the National Comorbidity 

Survey Replication,” Archives of General Psychiatry, Vol. 62, No. 6, 2005, pp. 629–640. 

Watkins, K. E., S. B. Hunter, S. L. Wenzel, W. Tu, S. M. Paddock, A. Griffin, and P. Ebener, 

“Prevalence and Characteristics of Clients with Co-Occurring Disorders in Outpatient 

Substance Abuse Treatment,” American Journal of Drug and Alcohol Abuse, Vol. 30, No. 4, 

2004, pp. 749–764. 

Wong, E. C., R. L. Collins, and J. L. Cerully, Reviewing the Evidence Base for Mental Health 

First Aid: Is There Support for Its Use with Key Target Populations in California? Santa 

Monica, CA: RAND Corporation, RR-972-CMHSA, 2015. 

https://www.rand.org/pubs/research_reports/RR972.html 

Zalewski, M., S. H. Goodman, P. M. Cole, and K. A. McLaughlin, “Clinical Considerations 

When Treating Adults Who Are Parents, Clinical Psychology, Vol. 24, No. 4, 2017, 

pp. 370–388. 

35 
 

Part II. Implementation

The goals of the C2C implementation evaluation were to examine how C2C was implemented 

and whether C2C implementation changed CBO staff ability to address mental health issues and 

CBO client access to mental health services. 

Chapter 4 introduces the required components of the C2C program and the four phases of 

program implementation. It examines how CBOs and MHPs undertook planning, preparation, 

and decisions for early-stage implementation of C2C, using qualitative data from CBO and MHP 

leaders and staff and operational plans during the first two implementation phases, Exploration 

and Adoption Decision/Preparation. 

Chapter 5 continues examination of CBO-MHP decisionmaking for their C2C programs, and 

how individual programs evolved over time as organizations gained experience with integrating 

C2C into their workflows. This chapter focuses on the second two phases of implementation, 

termed Active Implementation and Sustainment. 

Chapter 6 focuses on understanding C2C’s capacity to transform CBOs, MHPs, and the 

community’s attitudes, behaviors, and culture surrounding mental health during the Active 

Implementation and Sustainment phases of C2C implementation. 

Appendix B provides further detail on the methods used for the implementation evaluation, 

including the primary data collection from key informants and focus groups with CBO and MHP 

leadership and staff, as well as CBO clients participating in C2C, the annual staff surveys, and 

the CBO quarterly reports. 

36 
 

 

4. Adaptation of C2C at Community-Based Organizations in 

Early-Stage Implementation

Michael Stephan Dunbar, Dana Schultz, Dionne Barnes-Proby, 

Michele Abbott, Clare Stevens, and Vivian L. Towe 

Key Findings

• On the launch of C2C, CBOs and MHPs were initially given limited implementation 

guidance and were expected to develop their own implementation plans to adapt the 

C2C model to a specific CBO setting. 

• In response to requests for clarity and additional support from CBOs/MHPs and 

observed challenges with respect to early implementation of some program components 

(e.g., MI training; continuous coaching, and supervision), the C2C Collaborative 

provided more detailed implementation guidance, trainings, and technical assistance 

activities to help support CBOs and MHPs develop plans for implementing C2C. Revised 

year 2 implementation guidance and training opportunities to address gaps in capacity 

(e.g., for MI) were seen as instrumental for developing and refining early implementation 

plans and building capacity to implement the C2C model. 

• CBOs and MHPs considered factors such as existing staffing structures, resources, 

organizational processes, and workflows when developing plans and implementing 

required C2C components in CBO settings. 

• Adapting the C2C model to specific CBO settings requires considerable upfront 

planning and consideration of site-specific staff and client needs (e.g., CBO and MHP 

capacity to train and supervise staff members in C2C skills, number of clients served, 

types of behavioral health problems experienced by clients), available resources, and 

organizational processes/workflows during early implementation. 

37 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Introduction

In this chapter, we describe specific considerations, challenges, and facilitators for tailoring 

C2C to fit the needs and culture of individual CBO settings, from the perspective of CBO 

and MHP leaders. We first review the required components of C2C and then describe the 

implementation process framework that guided our approach to assessing how CBOs adapted 

and modified C2C. 

Required Components of C2C

As described in Part 1, C2C involves integrating four core C2C skills into existing CBO 

programming: mental health screening, MHFA, MI, and PE. CBOs also develop and strengthen 

the pathways to clinical care (e.g., through their MHP partner) according to individual client needs, 

and CBOs may use other evidence-based skills in addition to the four core skills. In addition, 

C2C calls on both CBOs and MHPs to assume new roles that capitalize on their strengths and 

capabilities to increase capacity for and reduce barriers to providing mental health services. A 

close working relationship between these partnered entities is essential for ensuring the 

effectiveness of C2C: the MHP advises and trains CBO staff on effective, evidence-informed 

interventions, whereas the CBO tailors the implementation of those interventions to its own 

context. MHPs become not only service providers but also teachers who share their expertise and 

experience. CBO staff leverage trusted relationships with community members to deliver a range 

of mental health care supports as part of the everyday work they already do, making care more 

accessible. 

To build an intervention that operates through nonclinical settings, C2C integrated delivery 

of mental health skills into the programming of 15 social service CBOs located throughout NYC. 

As part of the C2C program, each CBO was asked to meet certain implementation requirements 

(Table 4.1). 

Table 4.1. Required Components of C2C Program

Requirement Description

Establishing a formal CBO-MHP All CBOs designated and contracted with MHP partners to directly assist in

relationship implementation of C2C program.

Staffing, training, and supervision CBOs and MHPs determined which staff members would fulfill specific roles

in the C2C program, and how CBO staff would be trained and supervised in 

the delivery of C2C services.

Engaging specific client CBOs and MHPs specified which CBO client populations would receive C2C

populations in C2C services services (e.g., all clients served by the CBO, clients who engaged in specific

types of CBO programming, etc.).

Integrating C2C into existing CBOs and MHPs integrated C2C services into their “regular” CBO services

CBO programming and workflows (see Chapter 1).

Establishing a process for mental CBOs and MHPs outlined specific referral procedures for linking CBO clients

health referrals with mental health needs to more intensive clinical services.

Continuous quality improvement CBOs and MHPs collected and reviewed programmatic data to meet program

targets and iteratively refine implementation of C2C services over time.

38 
 

 

 

All CBOs worked with MHPs to develop implementation plans that laid out how they 

would approach and enact these requirements during the implementation process. By design, the 

program allowed CBOs a great deal of flexibility in site-specific implementation. Each CBO had 

the latitude to navigate their relationship with an MHP, set up culturally relevant training and 

coaching, ramp-up staff readiness, and deliver the C2C skills to clients in a way that made sense 

for their CBO. Along with this latitude, CBOs were given limited specifications for the C2C model 

through the original Request for Proposals and additional implementation guidance distributed 

after C2C implementation began. The Collaborative provided more detailed guidance in year 2 to 

address programmatic requirements for which CBOs/MHPs had requested more clarity (e.g., 

PE curricula) and areas in which there was an observed need to improve implementation quality 

(e.g., continuous coaching and supervision). Further, all CBOs and MHPs received technical 

support in developing their operational plans—detailed documents that outlined specific plans 

for C2C staffing structure, training and supervision of CBO staff members, and how C2C would 

be enacted within the CBO setting—that required review and approval by the C2C Collaborative 

on-site implementation team. The C2C technical assistance team provided help from the beginning 

of the project through the third year of implementation to support CBOs to both meet the 

requirements and adapt C2C when necessary and appropriate. Key implementation events for 

C2C are summarized in Figure 4.1. 

Figure 4.1. Key C2C Program Implementation Events from 2016 Through 2019

Implementation Process Framework

To understand how CBOs adapted the C2C model and inform the design of the 

implementation study, we turned to an implementation process framework. The process of 

39 
 

implementing new evidence-based mental health services within community organizations can 

be described in terms of four broad phases, summarized in Figure 4.2 (Aarons, Hurlburt, and 

Horwitz, 2011). According to Aarons, Hurlburt, and Horwitz, 2011, the first phase, “Exploration,” 

is characterized by identifying some need, challenge, or approach that could be improved or 

addressed. The next phase is “Adoption Decision/Preparation,” in which organizations plan 

and prepare for implementing change (i.e., the C2C program), including how they intend to 

implement specific program components. The third phase is “Active Implementation,” in which 

organizations enact program strategies, monitor program metrics, consider various factors 

affecting implementation, and iteratively refine implementation approaches. Over time, 

organizations may transition from Active Implementation to “Sustainment,” in which program 

components become established and are integral to the organization’s workflow. For C2C, there 

is an iterative process where learning from early implementation resulted in adjustments to plans 

that were then implemented. 

Figure 4.2. Conceptual Model of C2C Program Implementation Phases in CBO Settings

SOURCE: Adapted from Aarons, Hurlburt, and Horwitz, 2011.

Within each phase, myriad contextual factors operating at external (e.g., sociopolitical 

context, policy environment) and internal levels (e.g., organizational characteristics such as size, 

leadership orientation, and organizational climate) affect decisionmaking for how specific 

program strategies will be implemented. Transitions between these phases are not necessarily 

40 
 

 

 

linear or fixed; movement into a new phase, such as Active Implementation, does not necessarily 

mark the “completion” of activities conducted in the Exploration or Adoption Decision/Preparation 

phases. Over time, organizations may move back and forth throughout these phases because they 

enact different program components, iterate on specific approaches, and engage in continuous 

quality improvement (CQI) practices. 

This chapter focuses on how the CBOs and MHPs handled the first two phases of C2C 

implementation (Exploration and Adoption Decision/Preparation), with an emphasis on factors 

that influenced how CBOs developed and refined plans for implementing the required C2C 

components within diverse CBO settings. As described elsewhere (Ayer et al., 2018), CBOs and 

MHPs conducted these processes in implementation years 1 and 2. Here we examine how CBOs 

and MHPs undertook planning, preparation, and decisions for early-stage implementation of 

C2C, using qualitative data from CBO and MHP leaders and staff and operational plans during 

these first two phases. 

Research Questions

For the primary research question regarding how C2C program strategies were implemented, 

and the key facilitators of and barriers to effective implementation of C2C program strategies 

within and across CBO and MHP partnerships, we assessed factors that influenced CBOs’ and 

MHPs’ decisionmaking regarding the six key required components of C2C described in Table 2.1. 

We broke this question into six parts corresponding to these components to better understand 

how C2C strategies could meet the needs of specific CBO settings, staff, and clients. 

• How did CBOs and MHPs 

- establish a formal CBO-MHP relationship 

- plan for staffing, training, and supervision 

- plan to engage specific client populations in C2C services 

- plan to integrate C2C into existing CBO programming 

- plan to establish a process for mental health referrals 

- plan for CQI of C2C? 

Methods

This section summarizes the data sources and analysis approach for our examination of how 

CBOs and MHPs approached the first two phases of C2C implementation (Exploration and 

Adoption Decision/Preparation phases of the Aarons model of the implementation process), 

including the factors that influenced the planning and early implementation of C2C. See 

Appendix B for complete details. 

Data Sources

Data on the Exploration and Adoption Decision/Preparation stages of C2C were drawn from 

the following data sources. 

41 
 

 

 

Operational Plans

CBOs submitted initial operational plans in their first year of funding (year 1) which were 

subject to multiple rounds of review and required final approval from the C2C Collaborative 

prior to initiating implementation of the C2C program at the CBO. In the second year of funding 

(year 2), after the C2C Collaborative distributed updated implementation guidance, CBOs 

submitted updated operational plans, which detailed any changes in implementation plans (e.g., 

based on experiences with the program in year 1 and updated implementation guidance.) (Note: 

The C2C Collaborative engaged in routine monitoring/program oversight activities to ensure that 

CBOs met implementation targets and adhered to operational plans. Although CBOs were 

contractually required to participate in some evaluation activities, RAND data collection efforts 

for the independent evaluation were separate from the program monitoring activities led by the 

C2C Collaborative.) 

From these operational plans for each CBO, the RAND research team abstracted data (e.g., 

staffing and supervision structure, CBO and MHP roles, integration of C2C skills into existing 

programming vs. stand-alone C2C service episodes, number and types of staff to be trained, 

types of PE curricula used), organized information by C2C implementation component, and 

identified themes related to key implementation decision points. 

Key Informant Interviews

During the summers of 2017 to 2019, RAND conducted key informant interviews with 

CBO leaders (e.g., executive directors of CBOs and C2C program directors), MHP leaders 

(e.g., clinical directors and counselors), CBO frontline staff (e.g., staff trained in and providing 

C2C skills to CBO clients), and CBO clients (e.g., people who received C2C skills) to collect 

qualitative data on program implementation (Table 4.2). The current chapter focuses on 

information collected during 2017 and 2018, which assessed key informant experiences during 

the Exploration and Adoption Decision/Preparation phases of implementation. 

Table 4.2. Number of Key Informants Interviewed by Year and Type of Informant

CBO Frontline

Year CBO Leaders MHP Leaders Staff CBO Clients

2017 35 29 80 38

2018 36 26 61 35

2019 24 16 – –

Total 95 71 141 73

Interviews in 2017 and 2018 were conducted in person, whereas those in 2019 were 

conducted over the phone. RAND worked with CBO leaders to identify and invite key 

informants to participate in interviews each year. In 2019, phone interviews were held with 

42 
 

 

 

 

CBO and MHP leadership mainly to assess any changes in the implementation model since the 

previous round of data collection and to obtain specific information on program sustainment and 

QI practices. 

The RAND research team developed interview protocols to address key evaluation questions. 

The interview protocols covered a range of topics, including overall program implementation, 

intervention fidelity, attitudes toward and experiences with C2C, job satisfaction, collaboration 

with partners, and implementation barriers and facilitators. Different interview protocols were 

developed for CBO leaders, MHP leaders, CBO staff, and CBO clients. Below are examples of 

questions included in each protocol (see Appendix B for more detailed descriptions). 

• CBO leaders: 

- How has C2C changed how your organization approaches client mental health or 

substance use issues? 

- Could you talk about ways in which the program has changed over the course of 

the past year? 

• MHP leaders: 

- What has been difficult about implementing the C2C program, if anything? 

- What is your (MHP) role in supporting CBO staff in delivering each of the C2C 

modalities? 

• CBO staff: 

- How has the C2C program changed the way that your organization serves clients? 

- What has your experience been with supervisors overseeing your delivery of C2C 

modalities to clients? 

• CBO clients: 

- Have you been offered any (mental health and wellness odds ratio [OR] C2C 

services)? What specific types of services have you been offered? 

- Could you tell me a little bit more about some of your reasons for accepting or not 

accepting some of the services that you were offered? 

C2C Model Summaries

CBO leadership provided documentation on specific components of their C2C model in 

year 4 of the implementation process, in the form of operational plans and quarterly narrative 

summaries. From these, the RAND research team abstracted the data and constructed narrative 

summaries and tables that describe C2C model components for each CBO. CBO leaders were 

asked to review and modify these, as needed. Appendix A presents the final version of these 

summaries. 

In addition, the research team reviewed documentation from technical assistance efforts (e.g., 

webinar presentation materials, notes from conference calls, handouts) provided to CBOs and 

MHPs over the course of the implementation process to identify common challenges encountered 

by CBOs in implementing required components of the C2C model in their settings. 

43 
 

 

 

Analysis and Synthesis of Findings

For the analyses presented in this chapter, we organized information from all of the data 

sources described above into categories defined by the six key components of the C2C model 

and implementation guidance provided by the C2C Collaborative. We organized data by 

category into spreadsheets, prioritizing data abstracted from operational plans, to identify key 

themes across CBOs for each of the categories. The team then reviewed and incorporated 

information from CBO-specific model summaries (Appendix A) to help contextualize differences 

in adaptation of model components across CBOs. We then integrated data from key informant 

interviews and technical assistance to compile site-specific considerations and variability in 

approaches to adapting the C2C model within settings. 

Results

Although CBOs were permitted and encouraged to tailor C2C to meet the needs of their 

organization, staff, and clients, all were subject to the same basic requirements of the C2C 

model. For each of the six requirements (Table 4.1), we describe the different approaches CBOs 

and MHPs considered when planning how they would implement these required components, 

factors that influenced planning and decisionmaking, and challenges faced in determining how 

to meet that requirement during the early stages of implementation (Exploration and Adoption 

Decision/Preparation phases of C2C). Chapters 5 and 6 provide details on later implementation 

of these requirements and changes made over the full course of implementation (Active 

Implementation and Sustainment phases of C2C). 

Primary Research Question 1, Part 1: How Did CBOs and MHPs Establish a Formal 

CBO-MHP Relationship?

A foundational requirement of the C2C model was that each CBO partner with an MHP to 

implement C2C. This requirement was grounded in the concept that, by establishing a formal 

linkage, CBOs and MHPs could capitalize on their respective capabilities and areas of expertise 

to help reduce barriers to providing mental health services. For example, CBOs could benefit 

from the mental health expertise of MHPs to better address client mental health needs in the 

community; MHPs could learn about client and community needs and establish a stronger 

presence within communities. CBOs considered what types of MHPs they needed as partners 

and how to operationalize those partnerships while still honoring the individual priorities of their 

organizations. Below, we outline key factors that affected how CBOs approached this requirement, 

including MHP characteristics and capabilities, structuring roles and responsibilities, and client 

data sharing. 

44 
 

 

MHP Characteristics and Capabilities

Approximately half of CBOs (47 percent; n = 7) selected MHPs with whom they had a 

preexisting working relationship or organizational affiliation (e.g., MHP and CBO were part of 

the same “parent” organization). The remaining eight CBOs forged new relationships with 

MHPs. 

Training, coaching, and supervision. All CBOs shared the responsibility of training and 

supporting CBO staff in delivering C2C skills with the MHPs. For CBOs, the decision to take a 

larger role in training, coaching, and supervision was often motivated by CBOs’ interest in 

sustainability of the activities. CBOs recognized that they would ultimately be responsible for 

training, coaching, and supervision, so they strove to adopt an active role early in implementation. 

From the beginning, MHPs typically had limited capacity to provide day-to-day coaching, 

supervision, and other activities (e.g., administrative oversight, coordination of trainings) to 

support the implementation of C2C components in the CBO setting. For example, as detailed 

below and in subsequent chapters, many MHPs did not initially plan to maintain a regular (e.g., 

daily) on-site presence within the CBO. In addition, many MHPs did not have expertise to train 

and supervise CBO staff in one or more of the C2C skills (e.g., MI). As such, all CBOs had to 

assume some responsibility for coordinating and implementing trainings and coaching and 

supervising their staff. Over time, many CBO-MHP partnerships increased their original 

estimates of MHP staff time needed for coaching and supervision (see Chapter 5). 

Staff capacity. To address capacity from the outset, all MHPs made sure some of their staff 

had protected time to see referred C2C clients or set up processes to ensure that C2C clients 

could be seen in a timely fashion. CBOs also partnered with MHPs with specialized skills to 

meet the needs of their clients. For example, several CBOs needed MHPs with the ability to 

provide services in languages other than English, whereas others needed MHPs with expertise 

working with adolescents and young adults. Some CBOs discovered unanticipated needs later in 

implementation, such as acute services to directly address trauma or substance misuse (discussed 

in Chapter 5). 

Accessibility of services. In partnering with the MHP, CBOs also considered a myriad of 

factors related to accessibility, including geographic proximity, hours of operation (e.g., whether 

the MHP offered weekend or evening hours to accommodate working clients), linguistic/cultural 

competency, and out-of-pocket costs for MHP services (e.g., whether the MHP would be willing 

to offer services at a reduced cost or on a sliding scale). Where possible, CBOs opted to have 

MHPs on-site to support staff in the form of case consultations and structured individual and/or 

group supervision sessions. Over time, as CBOs discovered that proximity was a primary 

driver of accessibility, many arranged for additional on-site MHP presence to increase client 

engagement during the Active Implementation and Sustainment phases of implementation 

(discussed in Chapters 5 and 6). 

Adaptability. Despite careful planning, many CBOs recognized that they could not 

anticipate all of their MHP needs up front and began to seek more flexible arrangements during 

45 
 

 

 

early implementation. Some asked their MHP for flexibility regarding intake procedures, 

scheduling, and/or policies regarding missed appointments. As one CBO leader reported, 

Our . . . [clients] understand that if they miss an appointment, they might have to 

wait, and so we had to negotiate with our MHP to let them know we understood 

their policy but that we wanted to make sure our clients were able to be seen 

more quickly and they were able to accommodate us, but . . . [we had] to 

negotiate with the systems and schedules. 

Other CBOs found that their original MHP could not be flexible in the ways needed to make 

C2C successful, so CBOs found new MHPs or partnered with multiple providers to meet their 

needs. For example, some CBOs opted to have one MHP provide training and supervision 

services whereas one or more additional MHPs served as referral sources for clients in need of 

more intensive services. 

Structuring Roles and Responsibilities

Coordination. For all partnerships, both the CBO and MHP designated a lead staff member 

who was responsible for internal coordination at their own organization and external coordination 

with their partner. Designating one lead at each organization streamlined communication and made 

it easier for both partners to identify whom to talk to when troubleshooting initial challenges 

and/or determining who was responsible for which tasks. Lead staff also served as primary 

contacts to the C2C Collaborative and technical assistance providers. A majority of CBOs 

(87 percent; n = 13) reported having a minimum of one CBO staff member with specialty mental 

health training present on-site at the CBO on at least some days during early implementation. 

CBOs and MHPs also had to work closely to determine who should be responsible for what 

tasks, including the division of labor for training, coaching, and supervising CBO staff on the 

core skills, and who would provide overall management and oversight of the partnership and 

C2C implementation more generally. 

Communication. During early implementation, most of the communication between 

CBOs and MHPs focused on clarifying roles, responsibilities, and specific details of program 

implementation. Partners used meetings for program planning, protocol development, and 

training plans. Aside from regularly scheduled in-person or phone meetings, CBOs and MHPs 

found it particularly helpful to set up methods and points of contact for ad hoc communication on 

urgent matters. As C2C implementation progressed, communication centered on other aspects of 

implementation such as how to ensure high-quality service delivery. CBOs and MHPs discussed 

staff comfort and competency in delivery of C2C skills and enhancement of skills through 

training, coaching, and supervision. CBOs and MHPs also used regular meetings to review data 

and identify gaps in service delivery, address challenges related to CBO and MHP capacity, and 

troubleshoot referrals. 

Integration of workflows. Overall, one of the greatest challenges facing CBOs—particularly 

for those choosing to have MHP staff on-site at the CBO—was how to integrate MHPs into 

46 
 

 

 

 

 

their culture and workflows. For example, some partnerships discussed challenges related to 

insufficient communication between the parties, which negatively affected program planning 

and implementation. One MHP described their experience with their CBO: 

It’s sometimes bumpy with the communication because we have conflicting 

priorities. A clinic has certain priorities. [The CBO] has certain priorities. C2C 

has certain priorities. So sometimes it’s like, “how are we going to do this so that 

we can satisfy all of our programmatic requirements and responsibilities?” But 

we figured it out, and I think we are stronger and more connected [as a result]. 

From the outset, frequency of MHP presence was primarily determined by MHP availability 

and C2C funds, and CBOs had to develop workflows that addressed client needs on days when 

MHPs were not on-site. Critical to these partnerships was flexibility and adaptability to the other 

organization’s needs and cultures. 

Client Data Sharing

From the beginning, both CBOs and MHPs focused on how to securely share client 

information. MHPs in particular needed to adhere to Health Insurance Portability and 

Accountability Act of 1996 (HIPAA) regulations to protect private patient health information. 

MHPs had to articulate what they could and could not share, and together the partnerships would 

come to an agreement about the types of information they could share with the appropriate 

protections and client consent in place. At a minimum, most partnerships developed information 

sharing or release forms and related procedures that allowed CBOs to share screening data with 

MHPs, and MHPs to share information with CBOs about client appointments kept or missed. One 

MHP discussed their data sharing protocol and how they worked to ensure the confidentiality of 

clients: 

I wouldn’t say [data sharing] is an issue. I would say it’s more of a topic of 

discussion. Family workers [at the CBO] are not bound by confidentiality but 

understand why therapists can’t share certain information. So, we really keep it 

limited to attendance, engagement with the clinic, compliance with treatment and 

if there is a crisis that supersedes confidentiality in any setting. So, once we had a 

clear definition of what types of information would be shared in a case 

conference, I think that helped. 

Primary Research Question 1, Part 2: How Did CBOs and MHPs Plan for Staffing, 

Training, and Supervision?

All partnerships were required to develop a staffing plan that outlined how C2C would be 

staffed by the CBO and the MHP, specified the types of program staff to be trained in each C2C 

core skill, and included plans for supervising CBO staff in the core skills. Below, we summarize 

how the partnerships approached staffing, training, and supervision and describe some of the 

challenges that arose. 

47 
 

 

Staffing

In operational plans, CBOs and MHPs were required to specify which staff members would 

occupy key leadership roles and to detail the expected contributions from other CBO and MHP 

staff (e.g., direct care staff to deliver C2C, support for data collection and reporting, senior team 

members to supervise and support CBO and MHP staff members). From the outset, CBOs 

needed to determine whether they had existing program staff with the requisite skills to fulfill 

C2C roles, such as the CBO C2C lead or other C2C management roles, or whether they needed 

to hire new personnel. In some cases, CBOs could use existing staff with some mental health 

experience or training already (e.g., job developers, shelter case managers, early child education 

family workers). In addition, a few CBOs leveraged graduate student interns from local social 

work and psychology programs to extend skills delivery beyond that of existing nonclinical 

staff. Depending on the CBO, graduate students could be integrated in every phase of 

implementation from intake/screening to making referrals to the MHP without major cost 

implications. 

CBOs also needed to consider whether they had resources to support a full-time, dedicated 

C2C coordinator at the CBO and a designated intake coordinator at the MHP. C2C coordinators 

had a range of responsibilities, including ensuring staff participation in initial and ongoing C2C 

training, tracking program implementation and client engagement, managing relations and 

communication with the MHP, and providing or facilitating coaching and supervision. All CBOs 

and MHPs specified designated coordinator/leadership roles for both the CBO and MHP side of 

the partnership in their operational plans. One CBO leader described how clear designation of 

coordinating positions or “point persons” at both the CBO and MHP were critical to successful 

implementation: 

Having someone designated specifically as our C2C coordinator and someone 

specifically as the designated intake person at the counseling center being the 

main connections has allowed much more fluid communication. From the MHP 

side, they really like getting to know more about the shelter and our clients, so it 

has provided them more context on our clients too, whereas before the C2C 

program, there wasn’t as much of an interface between the counseling center and 

our agency. It just feels like there is more communication around the clients’ 

referrals and whether or not they followed up. 

Some CBO staff initially voiced that they were uncomfortable, ill equipped, or reticent to 

adopt C2C, or otherwise indicated that C2C program requirements were not “in their job 

description” or “perceived this as extra work.” As discussed in more detail in Chapter 5, some 

staff members specifically expressed discomfort surrounding program elements such as mental 

health screening due to concerns about their ability to discuss and help clients with problems 

such as suicidal ideation. CBOs worked with staff to help them see the value of implementing 

C2C skills with their clients and learn efficient strategies for utilizing these skills. Many CBOs 

strived for seamless integration of C2C into existing programming and helped staff recognize 

48 
 

that C2C skills were already reflected in their current practice with clients. One MHP lead 

illustrated this point by saying: 

I think getting people to understand what C2C was about and not looking at it as 

one additional thing to do and helping them to connect it to the work they were 

already doing, making the work meaningful and showing them how that can be 

done– I feel like that made a difference. 

Training of CBO Staff

Training plans. MHPs were asked to develop a training plan in collaboration with CBO 

leads and to establish benchmarks to indicate when CBO staff had attained the requisite 

knowledge to deliver core C2C skills. As C2C was launching, it was thought that MHPs would 

be primarily responsible for all training of CBO staff in C2C skills, but in practice not all MHPs 

could fulfill those duties, particularly with respect to MI. Only three CBOs employed providers 

with intermediate to advanced MI skills and had capacity to provide MI trainings at the start of 

the project. Recognizing these training gaps, the C2C Collaborative encouraged CBO staff to 

participate in free MHFA trainings (including “train the trainer” courses) delivered by the NYC 

DOHMH during the first year of implementation (note: DOHMH-sponsored MHFA trainings 

were independent of C2C and were available prior to the launch of the C2C initiative). In 

addition, the C2C Collaborative convened specialists in MI to provide a free, in-depth training 

referred to as the MI Institute (see the interim report for a more in-depth description of these 

additional resources) (Ayer et al., 2018). 

To minimize staff and client perceptions of C2C as an additional and separate practice and to 

ensure sustainability, CBOs sought to integrate C2C skills training into the existing professional 

development structure during early implementation. Further, training plans also evolved to 

include program monitoring activities and service delivery. For example, because CBOs 

modified existing program forms or systems to incorporate new C2C skills (e.g., intake forms 

included a checkbox to indicate whether screening was completed, or case management fields in 

client management systems included options for C2C skills), staff were trained to use these 

options to capture C2C implementation. 

As implementation progressed, certain CBO staff received training to become trainers, and 

all CBOs revised their training plans to enhance precision and efficiency in training and skill 

delivery choices. These refinements were made in response to a better understanding about CBO 

staff roles and workflows. For example, for some CBOs, it made more sense to assign a smaller 

group of staff to deliver curriculum-based PE to clients and/or make referrals. 

Training CBO staff members. Each CBO aimed to train all participating C2C staff 

members in at least one C2C skill and train at least some client-facing staff members in multiple 

C2C skills to prepare them to address the range of client needs or situations that might arise. 

However, training on screening was often limited to staff who handled intakes (when many CBO 

protocols specified screenings take place). See Chapter 5 for a more comprehensive description 

of C2C staff training, including number of staff trained in specific C2C skills. 

49 
Training content and processes. Training content included information related to 

implementing each C2C skill with a focus on EBP requirements (e.g., 8 hours of MI training), 

as well as the process for making referrals to the MHP or other providers. Training formats 

included role-play, didactic strategies, scenarios/vignettes, interactive games, case studies, brain 

storming sessions, and worksheets. Though staff were also trained on when they should use their 

new skills, CBO staff were often uncertain about the settings and scenarios in which to use their 

new skills during early implementation (note: this was anticipated and was part of the rationale 

for requiring ongoing coaching in the C2C model). 

Ongoing training. CBOs planned to provide booster trainings to deepen or expand staff 

skills to meet staff and client needs. During early implementation, CBOs determined whether 

and what types of booster training modules were necessary through ongoing supervision and 

staff feedback. For example, CBOs held supplementary trainings on topics including suicide risk 

assessment and crisis intervention, trauma-informed care, and mindfulness-based stress reduction. 

CBOs also balanced the need for both service delivery and training, because too much training 

time could reduce opportunities to deliver other services and could be perceived as overly 

burdensome to staff. In addition, CBOs sought to ensure that ongoing training aligned with client 

needs. Staff at one CBO felt the trainings were too ambiguous and did not relate directly to staff 

ability to respond to the needs of the clients, so the CBO spent time researching and revising 

training to address these concerns. Another CBO conducted internal training satisfaction surveys 

to assess staff views of the trainings and address areas of concern. 

As implementation progressed, CBOs provided training sessions for newly hired staff, which 

sometimes proved challenging if a formal training session was not immediately available. Newly 

hired staff in these cases had to rely on their coworkers to bring them up to speed until they were 

able to participate in a more formal C2C training session (see Chapters 5 and 6 for more details 

surrounding challenges and facilitators of training). 

Supervision

CBO-MHP partnerships were given specific guidance for required coaching and supervision 

activities in year 2 of the implementation process. Prior to this guidance, not many CBOs were 

consistently carrying out coaching and supervision programs. The requirement included 

• At least once per quarter, CBO staff who had (a) been trained in one of the C2C skills 

and (b) used that skill in their work had to receive a coaching session. The coaching 

session was intended to help the staff further develop their use of this skill, ideally by 

having the supervisor directly observe the staff member using that skill and then 

providing feedback. 

• At least twice per quarter, all CBO staff implementing one or more C2C skills were 

required to participate in a supervision session that included reflective supervision 

50 
 

activities that focus on eliciting and processing staff members’ experiences, thoughts, and 

feelings surrounding use of C2C skills with clients.1 

• At least twice per year, CBO staff implementing C2C were required to complete a selfassessment of their own knowledge, skills, and practice related to C2C. 

In addition, MHP staff were expected to monitor implementation to ensure CBO staff 

received high-quality coaching and supervision. MHP staff were required to provide coaching 

and lead supervision activities with CBO staff unless the partnership designated other qualified 

supervisors to perform those tasks (e.g., trained CBO staff, qualified individuals from other MHP 

organizations). MHP staff could train and supervise CBO supervisors to perform new coaching 

and supervision tasks. As a part of continuous coaching and supervision, MHP staff were also 

tasked with choosing or developing self-assessment tools for CBO staff. 

Overall, partnerships took many different approaches to fulfilling these requirements. 

Because CBO staff needs shifted, most coaching and supervision programs evolved to include 

case consultation, individual and group reflective supervision, one-on-one coaching, and audio 

recording and review. One common approach to coaching and supervision involved group 

meetings in which CBO and MHP staff shared client progress and MHP staff provided 

consultation to CBO staff. 

Other adjustments that occurred over time included 

• finding a balance between individual support and group sessions to match the amount of 

time MHP staff could dedicate to coaching and supervision 

• training CBO staff who were highly skilled in C2C to perform some coaching duties to 

provide more feedback and practice opportunities, allow for more peer-to-peer learning, 

and free up MHP time to develop customized supervision activities 

• allowing coaching and supervision to be provided over the phone to eliminate the need 

for travel time and increase the number of consultations from MHP staff 

• seeking additional resources (e.g., through consulting arrangements with other clinical 

providers) to provide CBO staff with the ongoing feedback needed to attain C2C skill 

mastery. 

Partnerships took a very individualized approach to fidelity monitoring, depending on who 

was supervising CBO staff (i.e., MHP or trained CBO supervisors), the time supervisors had 

on-site at the CBO, and the supervisors’ familiarity and comfort with CBO clients and staff. 

Partnerships had to carefully consider not only CBO staff but also client comfort and 

preferences. For example, CBO staff shared that some clients did not feel comfortable being 

observed by a third party while discussing mental health concerns with CBO staff. Since direct 

1 Zero to Three, a nonprofit organization focused on early development and well-being, provides an overview of 

reflective supervision principles and links to additional resources (see Zero to Three, Three building blocks of 

reflective supervision, webpage, 2016). 

51 
 

 

 

 

observation of CBO staff delivering C2C skills was not feasible, many CBOs adopted role-play 

to assess fidelity. 

Initially, fidelity monitoring was not highly structured, primarily involving periodic supervisor 

check-ins without a standard protocol. Although some CBOs were interested in developing 

quantitative assessments, they did not have the expertise or resources to follow through on this. 

To address these challenges, as well as requests from CBOs and MHPs for support in cultivating 

more robust and standardized fidelity monitoring practices, the C2C Collaborative shared a 

standard fidelity monitoring tool among the partnerships in implementation year 2. The tool, 

developed by RAND and the McSilver Institute for Poverty Policy and Research at the New 

York University Silver School of Social Work (NYU McSilver) Institute, asked supervisors to 

rate their confidence in staff delivery of C2C skills and competency in each of the four C2C skills. 

For MI and MHFA, supervisors also completed a checklist to document whether CBO staff used 

all the core components of those skills during a direct observation or role-play. Although a 

majority of CBOs reported using the tool as part of required supervisory practices, actual 

documentation and tracking of fidelity monitoring varied widely across CBOs. Because C2C 

became more integrated and operational plans were solidified, some CBOs expanded the type of 

staff who delivered C2C skills. As C2C expanded, it was increasingly challenging for supervisors 

to find the time and resources to conduct individual fidelity monitoring with every staff member 

on a regular basis. 

Primary Research Question 1, Part 3: How Did CBOs and MHPs Plan to Engage 

Specific Client Populations in C2C Services?

CBOs were expected to serve one or more of the following vulnerable populations, as 

described in Part 1 (e.g., young adults between the ages of 16 and 24 who are not in school and 

are not employed, unemployed or underemployed adults age 18 or older, and parents/caregivers 

who are expecting or who have children 4 years of age or younger). 

In planning to reach these populations, CBOs made other, more nuanced decisions about 

whom to target based on the clients they served and programs they offered. For example, many 

CBOs focused on clients who resided in high-poverty areas or in public housing, received public 

assistance, had not completed high school or obtained a general equivalency diploma (GED), were 

unemployed, or were considered high-risk populations (e.g., formerly incarcerated individuals, 

runaway and homeless youth, survivors of violence or trauma) because those were the clients 

seeking their services. In addition, some CBOs (40 percent; n = 6) focused C2C service delivery 

on clients for whom they felt mental health issues might be a barrier to achieving the intended 

program outcomes, such as completing a GED. Other CBOs (60 percent; n = 9) opted to deliver 

some C2C skills organization wide to all clients because (as reported by CBO staff) they believed 

that C2C would benefit all staff/clients in the CBO and/or because staff and programs were 

intertwined, so serving only a subset was not feasible. 

52 
 

 

 

 

 

 

 

Primary Research Question 1, Part 4: How Did CBOs and MHPs Plan to Integrate C2C 

into Existing CBO Programming?

All CBOs were required to implement the four core C2C skills and establish a process for 

referring clients to the MHP, when needed, to ensure a smooth transition to more intensive 

mental health care. In practice, CBOs faced a number of decisions regarding how to best 

integrate delivery of specific C2C skills into existing staffing structures and workflows. CBOs 

varied with respect to whether C2C skills were typically delivered in “stand-alone” sessions (i.e., 

apart from typical CBO programming) or integrated into existing CBO services (see Table 4.3, 

which shows ultimate decisions [as of July 2019] surrounding how CBOs integrated C2C skills 

into programming). This distinction had implications for the extent to which implementing new 

C2C skills required additional resources (e.g., staffing, time, physical space) and/or modifications 

to existing workflows within the CBO. 

Table 4.3. CBOs with Integrated and Stand-Alone Delivery of C2C Skills

Skill

Integrateda

Percentage (n)

Stand-Alone

Percentage (n)

Screening 87 (13) 13 (2)

MHFA 100 (15) 0 (0)

MI 93 (14) 7 (1)

PE 60 (9) 40 (6)

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019. a Skills delivered both as integrated and stand-alone are counted within ”integrated.”

Screening

Screening was viewed as a foundational component of all C2C programs, playing a key role 

in determining client mental health needs. 

Screening tools. Program requirements indicated that all CBOs screen for depression, anxiety, 

and drug and alcohol use. Within these requirements, CBOs had the freedom to select the tools 

that best fit their program needs. When available, CBOs were required to use standard language 

translations of validated screening tools so that every participant was screened with the same 

version of the tool. In selecting tools, CBOs often relied on the expertise of their MHPs. For 

example, if MHP staff were already trained and experienced in using and interpreting certain 

screening tools (e.g., the nine-item Patient Health Questionnaire [PHQ-9] for depression), the 

CBO often chose to adopt and train staff in those same tools. In doing so, CBOs were able to 

simultaneously draw on MHP expertise and make a warm handoff referral from the CBO to the 

MHP go more smoothly. Screening tools and processes are described in more detail in Chapter 5 

of the report. 

Mode of screening. CBOs also had to make decisions surrounding the manner and mode 

of screenings, such as whether they should be administered one-on-one or in a group setting, 

whether clients would complete the screening tools themselves (i.e., by completing a self-report 

form alone) or during an interview with a trained staff member, and whether to administer 

53 
 

 

 

 

screenings on a computer or paper. Because of limited time, one CBO specifically chose to screen 

clients in groups to ensure they could maximize their reach while still ensuring a confidential 

experience. Other CBOs chose to do one-on-one screenings and had to find private spaces for 

CBO staff to conduct them. One CBO initially asked clients to self-administer the screenings 

but noticed lower-than-expected numbers of positive screens. Many CBOs opted for screening 

on paper. Although computer-based administration had the benefits of automatic scoring and 

easy transfer of scores to the CBO’s data management system, this option proved to be cost 

prohibitive for most. 

Screening capacity. CBOs also had to assess staff capacity for screenings. Over time, CBOs 

with more limited capacity found ways to expand screening by using staff adept at screening as a 

resource to train and coach other staff or obtaining additional financial/staffing resources for 

screening activities. 

Integrating screening into workflows. To reach more clients and quickly identify mental 

health needs, many CBOs decided to integrate screenings into their workflows by screening all 

clients as a part of intake into the programs where C2C was being implemented. During 

implementation, other CBOs learned that intake was not the optimal time for screening after 

staff reported that some clients felt overwhelmed by the intake paperwork or uncomfortable 

discussing mental health needs before having an established relationship with intake staff. In 

response, some CBOs shifted screening to a later point in their program workflows, allowing for 

conversations about mental health to stand alone and for staff and clients to have time to build 

rapport. For some CBOs, the decision on when to screen was an ongoing challenge, and they had 

to try many different options before finding the right balance between timing, staff capacity, and 

client needs. 

Screening thresholds. CBOs and MHPs worked together and solicited guidance from the 

C2C technical assistance providers (NYU McSilver)—and consulted guidance/manuals 

associated with validated screening tools—to determine the thresholds, or cut points, for 

screening scores that would prompt staff to offer a referral. Across CBOs, there was variation in 

the specific actions triggered by the threshold level. For example, although all CBOs adopted 

standard clinical cutoffs for severe symptoms, some chose to offer referrals even when screeners 

indicated moderate or mild symptoms. CBOs often used screening scores as one of several data 

points to determine when to offer a referral (e.g., a client may have only indicated mild symptoms 

on a screening tool but voiced more severe concerns in a follow-up conversation with staff and 

were therefore offered a referral). 

Sharing screening results. CBOs and MHPs also had to decide how to share screening 

results with clients. When screenings were delivered in a one-on-one manner and, in some cases 

in group settings, it was often possible for CBO staff to discuss screening results with participants 

immediately (as recommended). For group screenings, one CBO reviewed results in the days 

after the screening and then discussed results with clients individually on a staggered basis, 

beginning with clients who reported the most severe symptoms. Ensuring staff were comfortable 

54 
 

 

 

having follow-up conversations with clients was an ongoing area of focus and training for CBOs. 

Although many nonclinical CBO staff felt comfortable asking clients questions about their 

mental health, they were less comfortable about sharing results and managing client responses, 

which could be emotionally intense. To help staff, CBOs developed clear protocols that walked 

staff through how to describe screening results and discuss next steps with clients based on their 

scores. They also offered opportunities for CBO staff to shadow MHP staff sharing results with 

clients, and many arranged for on-call clinicians to discuss particularly difficult situations with 

CBO staff on demand. Because CBO staff became adept in other C2C skills such as PE and MI, 

they also used those skills when delivering screening results. 

Rescreening. Over time, some CBOs saw the need and opportunity to offer client rescreens. 

For clients whose first screen indicated problematic symptoms, rescreening offered the opportunity 

to determine if these symptoms had lessened and by how much. If symptoms were not improving, 

staff could have conversations with clients about additional options that might be a better fit for 

their needs. For clients who did not initially screen positive, rescreening offered the opportunity 

to monitor mental health symptoms and offer help if new needs arose. Whether and how a CBO 

chose to rescreen was influenced by staff capacity, CBO workflow, and client participation in 

programming (e.g., in some programs, clients were only engaged with the CBO for a short time, 

making rescreening difficult or impossible), among many other factors. As described in Chapter 5, 

the C2C Collaborative encouraged all programs to consider rescreening clients and fielded a 

pilot rescreening initiative in year 4 of implementation. 

Psychoeducation

C2C guidance specified a number of PE requirements. CBOs and MHPs would need to 

• choose empirically supported, curriculum-based PE protocols aligned with client needs 

• regularly review the needs of clients (e.g., by directly asking clients or staff who worked 

with clients regularly, or through review of C2C data already being collected, such as 

screening data) 

• use data on client needs to identify and then implement PE programs that could address 

those needs 

• work closely with MHPs to identify appropriate PE resources tailored to client needs and 

develop plans for implementing the PE curriculum within the CBO setting. 

Given these requirements, CBOs and MHPs worked closely with the technical assistance 

team to select appropriate PE curricula to best align with CBO workflow and client needs (e.g., 

most common mental health issues, priority concerns for clients and the community, empirically 

supported programs for use in target populations). As described by one CBO leader, technical 

assistance was instrumental in selecting appropriate PE materials that were adherent to C2C 

requirements prior to implementation, 

We are working with the technical assistance team to look at particular models in 

existence to know what we can use for psychoeducation . . . Since this “evidence55 
 

 

 

 

based” requirement came [in Year 2], we became a little confused about how to 

go about finding the appropriate evidence-based curriculum that we could use . . . 

Then we can talk about how to implement this in a structured way, but we are 

trying to figure out what is an acceptable way of providing psychoeducation that 

doesn’t reinvent the wheel. 

Examples of specific PE programs used by CBOs to address issues such as stress management, 

alcohol and other drug use, and trauma are described in Appendix A. CBOs typically chose to 

provide clients with PE in a range of settings (e.g., individual or group), using multiple formats 

(e.g., verbal, written), and with varying frequency (e.g., one-time, biweekly). For example, many 

CBOs provided PE in an individual setting to help clients understand screening results along 

with information about the mental health conditions, how to recognize and manage symptoms, 

and resources available to support symptom management and improvement (including referral 

and treatment options). In these cases, verbal PE was often accompanied by written materials 

clients could take with them and share with family, friends, or support networks. CBOs also 

offered PE in group settings to help maximize efficiency of delivery. Some CBOs with long-term 

client involvement (e.g., residential programs, adult education programs) chose to offer these 

PE groups weekly or biweekly over the course of months. Other CBOs with more limited time 

with clients offered one-time group sessions. Some CBOs integrated PE courses into their regular 

programming as a way to provide all clients with some basic information about mental health 

and mental health disorders. Others offered sessions as an option that clients could choose 

whether to attend. 

Mental Health First Aid

The C2C implementation guidance for MHFA acknowledged that its delivery involves 

multiple behaviors (e.g., identifying people experiencing a mental health crisis or distress, 

assessing risk of suicide or harm, listening nonjudgmentally, giving reassurance and 

information, encouraging appropriate professional help, and encouraging self-help and other 

support strategies) that can be applied differently in various situations. This presented challenges 

for identifying when MHFA might have been delivered. In recognition of this, year 2 guidance 

from the C2C Collaborative specified that CBOs were not required to track the number of 

program participants who received MHFA or the total number of MHFA sessions delivered to 

CBO clients. 

Given the emphasis of MHFA training on enhancing mental health knowledge (e.g., 

awareness of effective treatments, recognition of signs and symptoms of mental health problems, 

strategies for helping individuals with acute mental health problems), CBOs indicated that they 

viewed MHFA skills as “general” or foundational skills that CBO staff members could infuse 

into a wide range of client interactions for existing CBO services, as opposed to being delivered 

in stand-alone episodes. CBOs often explicitly considered how MHFA skills interacted with 

other C2C program components and how they might be used to help facilitate implementation of 

other C2C skills and program strategies (e.g., screening or client referrals). For example, CBOs 

56 
 

often chose to incorporate the MHFA “ALGEE” action plan (Assess for risk of suicide/harm; 

Listen nonjudgmentally; Give reassurance and information; Encourage appropriate professional 

help; Encourage self-help and other support strategies) into protocols for crisis management and 

referrals to more intensive mental health services. In addition, CBOs often used MHFA to 

supplement screenings to help determine client needs and level of risk, in lieu of or despite 

screening results. 

Motivational Interviewing

MI is a specific set of counseling skills that allow CBO staff members to be supportive in 

conversations with clients, resolve ambivalence about unhealthy behaviors, and help support 

positive behavior change. The C2C Collaborative guidance explicitly acknowledged that MI 

skills may be applied in a wide range of client interactions and encouraged CBOs to integrate 

MI practices into their daily work. As with MHFA, starting in year 2, CBOs were no longer 

required to report on the total number of participants who received MI, or the total number of 

MI sessions delivered because MI was being delivered as part of regular client interactions and 

not as a stand-alone. 

Nearly all CBOs (93 percent; n = 14) reported integrating MI into existing CBO services 

rather than implementing MI in stand-alone sessions. CBOs had many choices for determining 

when MI could be useful for staff and clients and how to best deliver these skills with quality in 

the CBO setting. For most CBOs, these decisions were complicated by the fact that most MHPs 

had limited expertise in MI implementation, training, and supervision. As a result, many CBOs 

and MHPs used the C2C Collaborative’s MI Institute to help support decisions regarding the 

integration of MI skills within CBO settings. MI was often explicitly described in operational 

plans as a tool for helping to build client motivation to engage in C2C programming (e.g., 

screening) and accept referrals. Because CBOs became more familiar with MI during early 

implementation, CBOs described using MI to help change behavior, overcome challenges, 

encourage engagement in programming, build rapport, problem solve, and gain a better 

understanding of clients. MI was also helpful in discussing screening results with clients to 

help them navigate ambivalence about the findings and the handoff to the MHP. Moreover, 

many CBOs emphasized the nonjudgmental, collaborative approach to client interaction and 

support and respect for client autonomy—frequently discussed by leaders as the spirit of MI— 

as being infused into all client interactions. 

Primary Research Question 1, Part 5: How Did CBOs and MHPs Plan to Establish a 

Process for Mental Health Referrals?

Connecting clients who wanted or needed a referral to more intensive mental health treatment 

(e.g., outpatient mental health treatment at an MHP or another clinical provider) was a critical 

component of the C2C model. All CBO-MHP partnerships were tasked with establishing 

57 
processes to identify clients who may benefit from more intensive services, as well as when and 

how to offer referrals to mental health treatment. Specific requirements included 

• offering referrals to clinical care for C2C clients whose screening results suggested a 

need for more intensive mental health treatment, as well as all C2C clients who requested 

one, regardless of their screening outcome (this latter point was clarified in the C2C year 

2 implementation guidance) 

• offering C2C clients a choice of clinical care providers, and identifying additional clinical 

care providers to meet needs an MHP could not address 

• achieving a performance target, namely that at least 70 percent of clients who received a 

referral to mental health services would complete the referral (i.e., attend at least one 

session). 

All CBO-MHP partnerships developed referral protocols, which specified the kind of referral 

to make depending on symptom level and the process for making referrals. Referral practices and 

associated barriers and facilitators to making referrals, as well as changes over the course of 

Active Implementation, are described in more detail in Chapters 5 and 6. 

Referrals for symptoms above the threshold. As noted earlier, all clients with severe 

symptoms were required to be offered a referral. CBOs worked with their MHPs to make 

decisions about how to refer individuals in crisis (e.g., clients who exhibit crisis behavior or 

express thoughts or intentions of self-harm or suicide). Depending on MHP capacity and 

proximity, as well as the severity of the situation, CBOs developed processes to make immediate 

referrals to an MHP provider in a crisis situation or used crisis lines and/or referrals to immediate 

inpatient care in cases of emergency. 

Referrals for subthreshold symptoms. CBOs took a much more individualized approach 

to deciding what actions to take if clients reported subthreshold symptoms on screening tools 

(e.g., mild or moderate depressive symptoms) or in instances when clients declined to complete 

screening. Decisions about what to offer these clients were based on a range of factors, including 

MHP capacity to accept referrals, CBO staff comfort level providing ongoing monitoring for 

clients, and types of follow-up services CBO staff could offer. 

Staff making referrals. Some CBOs chose to centralize the referral process, designating 

one or two staff to be trained in making and tracking referrals. Other CBOs trained all staff who 

administered screenings in referral protocols. Typically, staff making referrals were responsible 

for explaining what to expect from MHP services (i.e., both process and potential benefits), 

discussing individual preferences with the client, and determining whether the MHP was a good 

fit for the client. In cases where the MHP could not provide appropriate services for the client, or 

the client preferred not to access services from the MHP, CBO staff offered alternative options 

and connected clients to those services. 

Warm handoffs. CBOs received technical assistance in setting up processes for offering a 

referral to clinical care, including how to develop and implement a warm handoff approach in 

which CBO staff, MHP staff, and CBO clients all actively participated in the process of initiating 

clinical care. For most CBOs, this warm handoff process changed over time because they became 

58 
more familiar with how to describe the referral process to clients (including what to expect from 

MHP services) and what challenges clients faced in completing referrals. For CBOs with on-site 

MHPs or CBO staff members with specialty mental health training, a warm handoff often 

involved initiating a face-to-face connection between the client and the provider. Some CBOs 

without an on-site MHP also took this approach, even accompanying clients to the MHP clinic. 

Warm handoffs were also made via phone call by CBO staff on behalf of the CBO client, often 

with the client present and participating in the referral and scheduling conversation. Warm 

handoffs and referrals are discussed in more detail in Chapters 5 and 6. 

MHP point of contact. Some MHPs designated a single point of contact to receive C2C 

referrals. These MHP staff were familiar with C2C and any specific protocols in place for C2C 

referrals (e.g., flexible scheduling, follow-up communication with CBOs, etc.). Other CBOs 

made arrangements with their MHP to have clients complete initial MHP intake paperwork onsite at the CBO before attending their first appointment off-site, thus imparting to clients a level 

of familiarity with MHP staff and protocols before visiting the MHP clinic. One MHP reported 

strategies they had developed with their CBO partner to streamline the referral process and to 

reduce barriers for clients to complete the referral, 

We have one psychologist here at [MHP] who does all of the [CBO] referrals. 

Their referrals can cut the lines so to speak. They can jump to the front of the 

line. They get seen very quickly. [CBO] is in communication so that there’s no 

middle person involved other than the one who has to collect insurance 

information and demographic information. We design paperwork that facilitates 

that, and it gets either scanned and emailed or faxed. But the fact that we have 

someone who’s dedicated to the [CBO] referrals, I think, makes the biggest 

difference. 

Monitoring referral outcomes. Some CBO protocols included very specific guidance on 

first appointment timing (e.g., within 48 hours of referral offer), whereas others found the need 

to offer more flexibility, indicating the first appointment should occur as soon as possible, given 

MHP capacity. Once the initial connection to clinical care was made, CBOs and MHPs comonitored client referral outcomes. Referral protocols specified the follow-up steps both CBO 

and MHP staff would take to support successful client engagement in treatment. Many referral 

protocols specified that the MHP would reach out to the client if they missed an appointment 

and/or, with consent, the MHP would notify the CBO of the missed appointment so CBO staff 

could reach out to the client. CBOs and MHPs had to navigate individual privacy laws (HIPAA) 

and preferences when establishing follow-up steps, especially regarding information sharing 

between CBO and MHP. 

Primary Research Question 1, Part 6: How Did CBOs and MHPs Plan for Continuous 

Quality Improvement of C2C?

All CBOs were required to collect data about service delivery and solicit feedback from 

clients and staff about C2C implementation. CBOs were encouraged to use these data to 

59 
continuously monitor program performance, understand the impact services had on clients, and 

make changes to improve service delivery with the ultimate goal of improving individual client 

outcomes. 

Data Management Systems

CBOs received specific guidance on what types of quantitative data (e.g., on service outputs, 

client referrals) should be collected as part of the C2C initiative. One of the initial decisions 

CBOs had to make was how to best collect required quantitative data so that information could 

easily be compiled and reviewed. CBOs had to consider several trade-offs around data system 

cost, efficiency, and functionality. Many CBOs had existing data management systems to track 

programmatic outcomes (e.g., Salesforce, ClientTrack) and chose to add C2C-specific data 

elements to these existing systems. However, many existing data systems did not come with offthe-shelf data collection modules that fit C2C needs. Although some CBOs had staffing capacity 

to make their own changes and build new data tracking modules, other CBOs had to rely on 

outside developers for this task. Even with internal expertise to customize data systems, many 

CBOs found that limited software capacity led to tracking and storing at least some information 

(e.g., staff and client feedback) outside their main data management system. Another important 

consideration when selecting or modifying a data system was protecting sensitive client 

information. To do this, many CBOs decided to limit permissions to view or edit certain data 

(e.g., screening results, referrals to MHPs) to a small number of staff members who interacted 

regularly with clients or oversaw specific programs. 

Data Collection Processes

CBOs had to make decisions about how and when data would be collected and entered into 

their system. Although asking staff to directly input data into the system during or immediately 

after a client interaction might allow for the most accurate capture of information, this was not 

realistic in the workflow of many CBOs. Instead, some CBOs created paper forms that staff 

could use to check off or quickly summarize client interactions which would later be transferred 

into the system either by the person who completed the form or by a designated data entry 

person. A few CBOs were able to collect screening data on tablets that fed directly into the data 

management system. Most CBOs implemented a hybrid approach, in which some data were 

centrally entered (e.g., the CBO or MHP lead entered information about staff training, coaching, 

and supervision) while CBO staff tracked C2C skill delivery. 

Another important aspect of data collection was staff training on the data collection 

processes. Many staff were accustomed to entering information through case notes (where 

important information on service delivery and/or referral could get buried), whereas others had 

no experience entering information into a data management system. As a result, CBOs needed to 

train staff on the new data entry fields and processes. To ensure data quality, many CBOs found 

the need to institute regular data audits in which C2C leads or data management staff reviewed 

60 
 

 

 

 

 

data entry and followed up with staff who were not entering data appropriately. To ensure more 

complete data entry, some CBOs made specific data fields mandatory, such that staff could not 

advance in the system until the fields were completed. Because staff across CBOs expressed 

concern and frustration at the additional time that C2C data tracking required, many CBOs began 

regularly sharing performance metrics with staff to increase buy-in, demonstrate the effect of 

data collection efforts, and engage in QI efforts. Some CBOs dedicated a portion of staff 

meetings to reviewing C2C reports to celebrate successes in C2C skill delivery and to ask staff 

for feedback. This type of information sharing served the dual purpose of creating staff buy-in 

for the data collection process and generating practical solutions to identified challenges. 

Monitoring Program Performance

CBOs were required to use data to monitor progress toward targets such as number of staff 

members trained, number of clients receiving C2C skills, and number of referrals to MHPs. 

Only one target related to client referrals was standardized across all CBOs: 70 percent of clients 

attending a first appointment (“kept referral rate”) following referral to MHPs. Other targets 

were intended for sites to customize, such that CBOs and MHPs decided on and proposed 

numerical targets for approval by the C2C Collaborative. Target numbers varied widely 

depending on site-specific factors such as CBO size, client service structure and workflow, 

and target population. For example, the target number of new C2C clients that CBOs planned to 

serve in implementation year 1 ranged from 37 to 1,382. Over time, many of the annual sitespecific targets were revised to reflect changes in implementation plans and CBO experiences. 

Beyond quantitative data on these targets, CBOs leveraged a wide array of formal (e.g., client 

ratings of PE programming, staff ratings of C2C trainings) and informal feedback (e.g., input 

from CBO staff at meetings) to identify areas of focus and guide efforts to improve 

implementation of C2C components. 

In-house expertise with modifying data systems was particularly useful for creating highly 

customized reports to allow CBOs to maximize the utility of data they had collected. CBOs had to 

make important decisions about how to present data in a format that supported decisionmaking. 

Initially, CBOs focused on creating data reports that would allow them to meet reporting 

requirements and gain a basic understanding of what C2C implementation looked like. As 

implementation progressed, however, most CBOs found the need to customize reports to include 

sufficient detail on where service gaps might be occurring (e.g., whether service delivery varied 

widely by CBO program or individual staff member). This underscores the importance of CBO 

data management and analysis capacity to support the implementation of C2C. 

CBOs had different approaches to using the data to drive decisions about ongoing 

implementation. In the early stages of implementation, most CBOs used the data to monitor the 

extent to which C2C was being carried out as planned (e.g., the number of clients being reached 

by C2C, which services were being delivered and with what frequency, and whether training was 

occurring as planned). Typically, the CBO lead reviewed these data at least quarterly. Many CBOs 

61 
 

 

 

 

 

also shared some of their aggregated data in meetings with the MHP. In particular, CBOs found 

that sharing data on client screenings, referrals, and referral completions, as well as staff training 

and supervision, helped the partnership as a whole to understand if it was reaching targets and to 

jumpstart discussions on any improvements that might be needed. Moreover, efforts to collect 

and review qualitative data from staff and clients were helpful in contextualizing and making 

sense of quantitative programmatic data. This emphasizes the importance of supplementing 

quantitative data collection efforts with qualitative feedback to help guide data interpretation and 

support decisionmaking. 

Because CBOs collected more data and gained confidence in interpreting it, they began to 

expand how they used it and with whom they shared it. Some CBOs used screening data to make 

decisions about the kinds of PE workshops to offer. For example, one CBO found that many 

clients reported both past trauma and problematic alcohol use, so they implemented PE to address 

these two issues. Other CBOs gathered more than just C2C-required data to understand a problem 

and develop effective solutions. For example, one CBO with low rates of referral completion 

invited clients to participate in focus groups, interviews, or short surveys to better understand the 

challenges they faced to attending appointments. 

Continuous Quality Improvement Technical Assistance

Overall, CBOs varied widely with respect to existing resources to conduct rigorous CQI in 

the early stages of the implementation process. Because most found it challenging to determine 

the best process for structured CQI, all CBOs were given the opportunity to participate in a 

technical assistance program to support specific CQI projects during year 3. The C2C 

Collaborative sponsored two rounds of CQI workshops, which were led by an expert in the 

Getting To Outcomes (GTO®)

2 model for supporting quality implementation and improvement 

for prevention programming. For the first round, four CBOs that were more advanced in their 

CQI knowledge/practices and data collection capacity were invited to participate. For the second 

round, all remaining CBOs were invited to participate. Eight CBOs elected to participate in this 

program, which consisted of structured workshops designed to support them in the development 

and implementation of a Plan-Do-Study-Act (PDSA)3 approach to data-driven decisionmaking 

and CQI. The technical assistance support provided a set of structured activities to advance CBO 

capacity to identify areas of need, test changes to improve outcomes, develop process and 

outcome indicators to track impact, and make data-driven decisions. Participating CBOs varied 

widely with respect to existing CQI activities and areas of focus. As part of participation in this 

technical assistance program, each CBO formed a CQI team made up of CBO, and in some cases 

2 RAND Health Care, Getting to Outcomes® Improving Community-Based Prevention: A Toolkit to Help 

Communities Implement and Evaluate Their Prevention Programs, undated. 

3 Agency for Healthcare Research and Quality, Health Literacy Universal Precautions Toolkit, 2nd Edition: PlanDo-Study-Act (PDSA) Directions and Examples, undated. 

62 
 

 

 

 

 

MHP, staff and leadership. The team reviewed C2C and programmatic data to understand 

whether they were meeting their service delivery objectives and whether their services were 

having the desired effect on clients. Based on this data review, each of the CQI teams selected 

one area for improvement, developed and tested a change to service delivery, and collected 

process and outcome data to understand the effect of the change. All CBOs successfully 

completed their site-specific CQI projects. 

Discussion

Our assessment of how CBOs handled the first two phases of C2C implementation focused 

on the factors that influenced how CBOs developed and refined their plans. From CBO and 

MHP leaders, we learned that screening and PE (e.g., when, how, and to which clients screening/ 

PE required is delivered) required extensive time and effort early on to optimally match screening 

tools with PE curricula, meet population needs, and integrate these skills into existing CBO 

workflows. Technical assistance and detailed program guidance surrounding these C2C skills— 

provided in year 2 of implementation—helped to ease the burden associated with implementation 

of screening and PE. CBOs that consider implementing these components of the C2C model 

should devote considerable upfront time to assess site-specific client needs, available resources, 

and organizational processes/workflows prior to implementing screening and PE. With respect to 

funders, efforts to provide expert guidance to help CBOs and MHPs match screening and PE 

materials to client needs and technical assistance to help CBOs consider how to best integrate 

these skills into existing workflows earlier in the implementation process may help to mitigate 

these problems and expedite implementation of C2C. In contrast, MI and MHFA, which have 

more standardized training content, were seen as being more readily “infused” into existing 

workflows. However, particularly in the case of MI, limited capacity of MHPs to train and 

supervise staff members was a challenge in the early stages of the implementation process. 

This underscores the importance of ensuring appropriate resources to support MI training and 

supervision capacity prior to implementing the C2C model. 

We also learned about the importance of addressing the interrelatedness and interdependence 

of the different C2C components. Consistent with the multicomponent design of the C2C model, 

decisions regarding one component (e.g., screening) required consideration of how a given 

approach (e.g., group screenings) might affect implementation of other components (e.g., referral 

processes). Some decisions related to one requirement, such as selection of an MHP, are likely to 

have a cascading effect on decisions made for multiple other components. Implementation of any 

new EBP within service settings requires careful consideration of a variety of inner and outer 

contextual factors spanning multiple levels to ensure adequate planning and facilitate successful 

implementation (e.g., individual, organizational, community, system) (Aarons, Hurlburt, and 

Horwitz, 2011; Institute of Medicine, 2007). Similarly, decisions regarding data collection and 

data systems can significantly affect capacity to monitor outcomes and engage in targeted QI 

activities (Bakken, 2001; Institute of Medicine, 2007). More complex interventions involving 

63 
 

 

multiple components, such as C2C, may necessarily require careful attention to how different 

intervention components operate and interact with each other as well as with different contextual 

factors. Because decisions regarding C2C components are interrelated, CBOs should create 

detailed operational plans for implementing and monitoring all program components prior to 

enacting the C2C model. In addition, investments in early-stage technical assistance to assist 

CBOs in developing and refining operational plans (e.g., providing clear guidance on how 

selected strategies can or should be adapted to the CBO setting; drafting clear protocols and sitespecific practices for implementing each C2C skill) may be helpful in ensuring that organizations 

are well prepared to successfully implement all program components on launch and that plans 

align with known best practices. Future efforts to provide such guidance to CBOs throughout the 

Adoption/Decision and Preparation phases could help to mitigate challenges with adapting 

program components to diverse CBO settings and may be useful in avoiding delays in Active 

Implementation of the C2C model. 

CBOs viewed technical assistance and clear implementation guidance as instrumental to 

helping to develop and refine plans for adapting the C2C model to unique CBO settings. Lack of 

clarity in program guidance and additional needs for technical assistance in some domains (e.g., 

continuous coaching and supervision, selection of PE curricula) observed during year 1 of the 

implementation process were corrected in year 2. The technical assistance and support offered 

through the C2C Collaborative, combined with the emphasis on site-specific adaptations 

inherent to the C2C model, address critical implementation barriers identified in past research on 

the implementation of EBPs in CBOs. In their systemic review of evidence on barriers and 

facilitators to EBP implementation in “third sector” organizations (which include CBOs), BachMortensen, Lange, and Montgomery (2018) described challenges surrounding adaptation of EBPs 

to specific settings or populations as a significant theme. The authors specifically recommended 

clear guidelines and technical assistance surrounding how to adapt interventions and upfront 

investments to ensure appropriate program infrastructure and capability to implement EBPs on 

the part of organizations. Findings from this evaluation support these recommendations and 

suggest that future efforts to extend or replicate the C2C model in other CBO settings would 

benefit from robust support structures to aid in site-specific program adaption and investment in 

early-stage planning during the Exploration and Adoption Decision/Preparation phases to help 

ensure that CBOs have the capability to implement all components of the C2C model. In addition, 

the development of detailed CQI plans and careful assessment of data system requirements prior 

to Active Implementation may help CBOs and MHPs to organize and clarify decisionmaking 

processes and provide a strong foundation for QI during the Active Implementation and 

Sustainment phases of implementing the C2C model. 

As with the implementation of any new program or EBP (see Aarons, Hurlburt, and Horwitz, 

2011), implementing the C2C model may reveal critical insights that could alter decisionmaking 

regarding how to best implement C2C components in a given CBO. Moreover, inner and outer 

contextual factors and other circumstances that influence decisionmaking may change over time, 

requiring CBOs to alter initial plans. Use of routine stakeholder feedback (e.g., client, staff, 

64 
 

 

 

community), program data, and structured continuous QI practices throughout the implementation 

process can improve program flexibility and lead to more efficient and appropriate decisions 

regarding how to refine plans and adapt C2C components to best meet the needs of CBOs, 

clients, and other stakeholders as programs evolve. 

Limitations

It is important to consider this evaluation’s limitations when interpreting these findings. As 

described in the methods and noted throughout this report, data come from a mix of sources, 

including CBOs’ operational plans and key informant interviews that were conducted at a time 

when CBOs were in the early stages of the implementation process when implementation plans 

were still evolving. Data were also synthesized to address key themes surrounding how 

organizations approached plans for implementing the C2C model, and CBOs implementation 

plans changed over time (see Chapters 5 and 6). As such, the findings described here may not 

capture the full breadth of planning and decisionmaking that CBOs undertook during the C2C 

initiative. Other CBOs seeking to adopt the C2C model may encounter additional challenges or 

confront unique considerations when planning to enact and adapt the C2C model components 

within their organization. 

Summary

In this chapter, we reviewed the six required components of the C2C model and presented an 

array of critical factors that CBOs and MHPs considered when initially deciding how to implement 

C2C within CBO settings during the first two phases of C2C implementation (Exploration and 

Adoption Decision/Preparation). To implement the required components, CBOs and MHPs 

considered the needs of their client populations and existing organizational processes and 

workflow as well as C2C Collaborative’s detailed implementation guidance, training, and 

technical assistance. Clear implementation requirements for individual components of the C2C 

model, strong CBO-MHP linkages, and adaptive responses from CBOs and MHPs were critical 

to successful early implementation of C2C requirements across diverse CBO settings. Need for 

additional implementation guidance and technical assistance in implementation year 1 (e.g., with 

respect to requirements for PE curricula and continuous coaching and supervision), as well as 

limited capacity for MHPs to train and supervise CBO staff members in MI, contributed to early 

challenges in adapting the C2C model to CBO settings. However, updates to program guidance 

and enhanced technical assistance in implementation year 2 helped CBOs navigate challenges 

and iteratively refine aspects of the C2C model and set the stage for successful implementation 

of different C2C program components in the later phases. Nearly all (14 of 15) CBOs implemented 

the required components of C2C within their organizations and remained active participants 

through the duration of C2C implementation. In subsequent chapters, we describe how CBOs 

and MHPs implemented the C2C model in practice and how program components evolved over 

the course of the Active Implementation and Sustainment phases of the implementation process. 

65 
 

 

References

Aarons, G. A., M. Hurlburt, and S. M. Horwitz, “Advancing a Conceptual Model of EvidenceBased Practice Implementation in Public Service Sectors,” Administration and Policy in 

Mental Health, Vol. 38, No. 1, 2011, pp. 4–23. 

Agency for Healthcare Research and Quality, Health Literacy Universal Precautions Toolkit, 

2nd Edition: Plan-Do-Study-Act (PDSA) Directions and Examples, undated. As of 

September 9, 2020: 

https://www.ahrq.gov/health-literacy/quality-resources/tools/literacy-toolkit/healthlittoolkit2 

-tool2b.html 

Ayer, L., M. S. Dunbar, M. Martineau, C. Stevens, D. Schultz, W. Y. Chan, M. Abbott, R. Weir, 

H. H. Liu, D. Siconolfi, and V. L. Towe, Evaluation of the Connections to Care (C2C) 

Initiative: Interim Report, Santa Monica, Calif.: RAND Corporation, RR-2497-MFANYC, 

2018. As of February 4, 2020: 

https://www.rand.org/pubs/research_reports/RR2497.html 

Bach-Mortensen, A. M., Lange, B. C., and P. Montgomery, “Barriers and Facilitators to 

Implementing Evidence-Based Interventions Among Third Sector Organisations: A 

Systematic Review,” Implementation Science, Vol. 13, No. 1, 2018, p. 103. 

Bakken, S., “An Informatics Infrastructure Is Essential for Evidence-Based Practice,” Journal of 

the American Medical Informatics Association, Vol. 8, No. 3, 2001, pp. 199–201. 

Institute of Medicine, The State of Quality Improvement and Implementation Research: Expert 

Views, Washington, D.C.: The National Academies Press, 2007. 

RAND Health Care, Getting to Outcomes® Improving Community-Based Prevention: A Toolkit 

to Help Communities Implement and Evaluate Their Prevention Programs, undated. As of 

September 9, 2020: 

https://www.rand.org/health-care/projects/getting-to-outcomes.html 

66 
 

 

5. Evolution of C2C Implementation

Michael Stephan Dunbar, Dana Schultz, Vivian L. Towe, Wing Yi Chan, and Michele Abbott 

Key Findings

• By implementation year 4, 14 of 15 CBOs had implemented the required components of 

C2C within their organizations and remained active participants in the C2C initiative. 

• From March 2016 through December 2019, CBOs and MHPs trained more than 1,700 CBO 

staff members and provided C2C services to more than 41,000 unique clients. 

• Slightly more than 60 percent of all staff members at participating CBOs had been trained in 

at least one C2C skill by implementation year 4, with MHFA as the most commonly 

reported training. 

• Training and service delivery metrics showed steep increases early on, followed by 

stabilization into year 2 and beyond, which suggests that ramp-up of training and initial 

rollout of C2C skills in CBO settings took slightly more than a year to achieve. Coaching 

and supervision for CBO staff members increased considerably from year 2 to 3, following 

clarifying guidance in year 2 for coaching/supervision requirements. 

• CBOs successfully incorporated C2C skills into CBO workflows and refined implementation 

strategies in response to site-specific factors (e.g., client flow through established CBO 

programs), staff feedback, and community needs. 

• CBOs and MHPs worked diligently to improve referral processes and these efforts led to 

very high rates of kept first appointments for clients referred to MHPs: rates surpassed 

70 percent by year 3 and exceeded 80 percent in year 4 of implementation. 

• By year 4, organizational leaders reported a commitment to sustaining some elements of 

C2C into the future. 

67 
 

 

 

 

Introduction

As described in Chapter 4, the process of adopting and implementing C2C in CBO contexts 

can be described in terms of four broad phases: Exploration, Adoption Decision/Preparation, Active 

Implementation, and Sustainment (Aarons, Hurlburt, and Horwitz, 2011). Chapter 4 focused on 

the first two phases for C2C, during which CBOs were expected to work collaboratively with 

their selected MHPs to develop and enact plans for integrating C2C services and other required 

components of the C2C model (e.g., establishing referral mechanisms) within each CBO setting. 

This chapter focuses on changes in implementation of C2C during the Active Implementation 

phase, spanning 4 years of program implementation, and progression into the Sustainment phase 

(year 4 of C2C). 

During Active Implementation, organizations put implementation plans into practice and 

iteratively refined program components to move toward stable, sustainable processes. Consistent 

with the emphasis in C2C Collaborative guidelines on site-specific adaptations of the C2C model 

to meet the unique needs of each CBO/community setting, CBOs varied considerably in the ways 

that they implemented the C2C model. These variations manifested in timelines for initiating 

specific requirements (e.g., training for specific core mental health skills, such as PE or MI; 

continuous coaching and supervision of CBO staff) and specific modifications to components of 

their site-specific C2C programs during the Active Implementation phase. Variations in timelines 

for initiating specific parts of the C2C model may also reflect program management challenges, 

capacity constraints within CBO-MHP partnerships (e.g., capacity to train and supervise MI), and 

gaps in guidance and technical assistance in the early stages of implementation (see Chapter 4). 

In this chapter, we present aggregate findings and common themes across CBOs and, where 

appropriate, describe differences in relation to key CBO characteristics (organization size and 

type), rather than comparing and contrasting individual CBOs. Specifically, it focuses on the ways 

that C2C training, coaching, and supervision evolved over time; how CBO-MHP partnerships 

addressed referral barriers; how partnerships modified practices to address sustainability issues; 

and how C2C was delivered in practice. 

Research Questions

For the primary research questions related to how C2C program strategies were implemented 

and key facilitators of and barriers to effective implementation of C2C program strategies within 

and across CBO and MHP partnerships, we used both quantitative and qualitative data sources to 

explore 

1. how CBOs and MHPs provided training, coaching, and supervision to support staff in 

implementing C2C 

2. how C2C services were provided to clients. 

For the primary research question related to the extent to which CBOs identified clients with 

mental health or substance use issues as a result of C2C implementation, and the key facilitators 

68 
 

 

 

 

 

of and barriers to effective implementation of C2C program strategies within and across CBO 

and MHP partnerships, we used using both quantitative and qualitative data sources to explore 

1. how CBOs and MHPs altered processes for client mental health referrals and improved 

referral completion rates 

2. how CBOs and MHPs adapted to accommodate and sustain the C2C program. 

Methods

This section summarizes the data sources and analysis approach for our examination of how 

C2C implementation changed over time during the Active Implementation and Sustainment 

phases of the Aarons model of the implementation process. See Appendix B for complete details. 

Data Sources

Data on the Active Implementation and Sustainment stages of C2C were drawn from the 

following data sources. Site visits and calls were also conducted; these are described in 

Appendix B. 

Key Informant Interviews

As detailed in Chapter 4 of this report, RAND conducted key informant interviews with CBO 

leaders (e.g., CBO executive directors and C2C program directors), MHP leaders (e.g., clinical 

directors and counselors), CBO frontline staff (e.g., staff trained in and providing C2C skills to 

CBO clients), and CBO clients (e.g., people who received C2C skills). These interviews took 

place during the summers of 2017 to 2019 with the purpose of collecting qualitative data on 

program implementation (see Table 4.2 for information on the number of people interviewed, 

by informant type, in each year). This chapter focuses on information collected during 2018 

and 2019, which assessed key informant experiences during the Active Implementation and 

Sustainment phases of implementation. Interviews in 2018 occurred in person, whereas 

interviews in 2019 were conducted over the phone. 

CBO Quarterly Reports

All CBOs provided to the Mayor’s Fund aggregate quarterly data on the clients they served 

and C2C services they delivered. Data used for this final report came from years 1 through 4 of 

program implementation (March 2016–July 2019) and included information on key program 

services and outcomes, program performance against targets, and other aspects of program and 

contract management. 

Annual Staff Survey

RAND conducted the three waves of annual staff surveys in the summers (May–September) 

of 2017 to 2019 to gain a broader view on program implementation from the perspective of CBO 

program staff who were trained in C2C skills and provided services to clients. CBO staff who 

69 
 

 

 

 

 

had ever received training in any of the four core mental health skills and who were still actively 

working at the CBO with a valid email address were eligible to participate. A total of 140 CBO 

staff members responded to the 2017 survey (34 percent of response rate). Results from the first 

wave of the survey (2017) were reported in the interim evaluation report (Ayer et al., 2018). A 

total of 252 staff members completed surveys in wave 2 (52 percent of response rate), and a total 

of 320 staff members completed surveys in wave 3 (50 percent of response rate). 

Staff surveys covered topics such as staff experiences with C2C training and service delivery, 

confidence in one’s ability to administer C2C skills, knowledge about mental health issues, 

organizational climate, perceptions of the C2C program within the CBO (e.g., organizational 

support for the C2C mission, communication), and staff use of specific resources and strategies 

during client interactions. Valid, reliable existing measures were used when available to assess 

key constructs of interest. When existing measures were not available, RAND worked with the 

C2C Collaborative to develop novel assessment items. Appendix B presents a comprehensive 

description of survey content. 

C2C Model Summaries

As described in Chapter 4, the RAND research team developed narrative summaries and tables 

describing the specific components of each CBO’s C2C model. CBO leaders reviewed and edited 

the summaries for accuracy, as needed. Appendix A presents the final version of these summaries. 

Analysis and Synthesis of Findings

For the analyses presented in this chapter, we categorized CBOs into the following 

broad categories, based on the main types of services provided by the CBO: job training and 

employment (n = 5); youth development (n = 3); parent/caregiver-serving (i.e., organizations 

that serve parents and caregivers of young children) (n = 3);1 and other (n = 4) (Table 5.1). 

For the quantitative data, we conducted univariate analyses (e.g., means, percentages, 

counts) using SAS version 9.4 and Excel to describe implementation measures, including 

frequency and type of C2C training and coaching, delivery of C2C skills to clients, and staff 

behaviors toward clients with mental health issues. We also assessed variability (e.g., range, 

standard deviations) in the metrics across CBOs and performed subanalyses by CBO type. 

For qualitative data, we used a mixed-method software environment (Dedoose) to conduct 

thematic analysis and identify recurring patterns, or themes, in the interview data. A team of four 

coders, all of whom participated in interview data collection, engaged in iterative rounds of data 

analysis to inform the development of a hierarchical code tree consisting of key themes. These 

coders held frequent coding reconciliation meetings to establish a robust shared sense of how the 

code tree represented the data and to ensure that coding was consistent. 

1 One parent/caregiver-serving was not able to implement the program and the funder terminated their participation 

in C2C at the end of implementation year 3. 

70 
 

 

 

 

 

 

 

Table 5.1. CBOs by Type

CBO Type CBOs

Job training and employment The HOPE Program

Bedford-Stuyvesant Restoration Corporation

Northern Manhattan Improvement Corporation

STRIVE

Center for Employment Opportunities

Youth development The Door

Red Hook Initiative

Hetrick-Martin Institute

Parent/caregiver-serving Hudson Guild

Sheltering Arms

Committee for Hispanic Children and Families

Other Safe Horizon

CAMBA

Voces Latinas

Arab American Association of New York

The research team then reviewed and organized quantitative and qualitative data into 

categories based on key evaluation questions. We then reviewed and integrated findings from 

CBO model summaries and other data sources to help contextualize site-specific considerations 

and variability across CBOs. 

Because of the variability with which CBOs implemented and altered site-specific C2C 

program components, we present aggregate findings over time and summarize common themes 

across CBOs rather than comparing and contrasting individual CBOs. For key implementation 

metrics for training, coaching/supervision, and C2C service provision, we present descriptive 

statistics (e.g., range, standard deviation). Where relevant, we describe differences across CBOs 

in relation to program size and main CBO service type to help illustrate how and where the 

process for implementing C2C varied or did not. 

Results

In this section, we summarize the refinements made throughout the Active Implementation 

and Sustainment phases and describe how CBOs varied with respect to the type, timing, extent 

of—and reasons for—modifications to components of their C2C programs. We focus on how 

training, coaching, and supervision supported implementation, how C2C services were delivered, 

how referral processes evolved, and how sustainability was addressed. 

Consistent with the conceptual model outlined in Aarons, Hurlburt, and Horwitz (2011) and 

described in Chapter 4, CBOs’ experiences with implementing C2C led them to change and 

refine many of their procedures over time. Many CBOs modified approaches to training CBO 

staff members (e.g., by incorporating role-plays and concrete examples of staff/client interactions) 

71 
 

in C2C to ensure that trainings aligned with staff and client experiences. In addition, CBOs and 

MHPs iteratively refined referral processes (e.g., by modifying MHP scheduling practices to 

expand available hours, by conducting portions of the intake process on site at the MHP to 

reduce burden for referred clients) to address common barriers to accessing clinical care. Some 

of these changes were intended to improve client outcomes, whereas others were made to 

improve procedural logistics and efficiencies and to make C2C easier to navigate overall. The 

most significant areas of focus for almost all CBOs were making referrals to the MHP, increasing 

the rate of client visits with the MHP, and ongoing engagement in mental health care. 

Overall, 14 of 15 CBOs implemented all of the required components of the C2C program 

outlined in Chapter 4. After continued program issues in its first 3 years, the performance of one 

provider did not meet the program performance criteria for continuation funding beyond year 3. 

They never reached implementation of all required components, and the funder ended their 

participation in C2C after year 3. Where appropriate, we include data from this CBO in 

aggregate results for implementation years 1–3; we also note instances in which data from this 

CBO was excluded and how overall patterns of findings may or may not be affected by inclusion 

of these data. The 14 CBOs that continued to participate in C2C delivered, at minimum, all four 

core C2C mental health skills to CBO clients over the course of all phases of implementation. 

CBOs varied considerably with respect to timelines for initiating specific activities, how they 

implemented specific components, and the types of challenges that CBOs experienced when 

bringing operational plans into action within the CBO setting. 

Question 1. How Did CBOs and MHPs Provide Training, Coaching, and Supervision to 

Support Staff in Implementing C2C?

During early implementation, CBOs and MHPs selected and designed training materials, 

and they worked together to implement C2C training for CBO staff members, coaching and 

supervision activities, and monitor how staff members delivered C2C skills to clients. This 

section describes how these activities progressed during the later phases of implementation. 

Training

Given their relative expertise in delivering mental health services to clients, MHPs typically 

assumed a lead role in overseeing development of training protocols and materials and led C2C 

trainings in the early stages of program rollout. Some MHPs shared that C2C training required 

more time and effort than anticipated, which meant that they did not have enough time or 

resources to provide optimal support to CBO staff. 

In looking at who provided the C2C trainings during the first 3 years of implementation, the 

ratio of MHP- to CBO-provided training hours varied over time. 

• Year 1: CBOs reported an average of 42.2 hours of training from MHP providers 

(median: 30; range: 0–168) and 26.6 hours of C2C training provided by CBO staff 

(median: 12; range: 0–79), for a ratio of 1.6. 

72 
 

 

 

 

 

 

 

 

• Year 2: CBOs and MHPs spent roughly equivalent numbers of hours on training CBO 

staff with an average of 24.9 hours from MHP providers (median: 15; range: 0–168) and 

21.1 hours from CBO staff (median: 6.5; range: 0–109), for a ratio of 1.2. 

• Year 3: CBOs received an average of 39.2 hours of training from MHP providers 

(median: 24; range: 0–154) and 27.1 hours from CBO staff (median: 17; range: 0–93), 

for a ratio of 1.5. 

Overall amounts of training may have dropped off somewhat in year 2 due to a focus on 

building up coaching and supervision processes following clarifications in implementation 

guidance for continuous coaching and supervision and updated operational plans (see Chapter 4). 

In year 4, when operational plans had been finalized and CBOs were well into the Active 

Implementation phase of C2C, most CBO leaders reported that both CBO and MHP staff 

members contributed to training delivery for each of the four C2C skills (Table 5.2). 

Table 5.2. Distribution of Training Provided During Year 4 (2019)

CBO Provided

Training

MHP Provided

Training

Both CBO and 

MHP Provided

Training

Training Provided

by Non-MHP 

External Partner

Screening 3 4 7 0

MHFA 3 6 4 5

MI 3 4 6 9a

PE 3 7 4 4

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019.

NOTE: Categories are not mutually exclusive; as such, not all results sum to the total number of CBOs (n = 14).

For CBOs that undertook some or all training responsibilities, early decisions to implement 

training “in house” were driven by a number of factors, including limited MHP trainer availability 

and development of CBO trainer expertise. For example, one CBO leader in the early stages of 

the implementation process (summer 2017) reported, 

I mean the [MHP], they really developed our psychoeducation [training] . . . but 

because they were limited to being here only once a week, I had to make sure I 

was also trained in all this material. So that’s why the bulk of training fell on us 

since [MHP] was only here once a week. 

In later stages of implementation, CBOs reported that building up in-house training capacity 

allowed for more effective resource utilization and greater independence with regard to when and 

where trainings occurred, which supported flexibility and contributed to sustainability of C2C 

trainings. This change was particularly notable in the case of MI training. As one MHP leader 

reported, 

When we started the MI Institute at [CBO], we didn’t have to travel anywhere, 

we could kind of use our own internal resources to train ourselves. We have 

had staff turnover, so folks who were in the MI Institute left, then we filled the 

spots with who haven’t had any MI training . . . . Using our MI trainers and 

MI supervisors to retrain new staff to kind of [enhanced] the sustainability. 

73 
 

 

 

Staff training, overall and by C2C skill. As of December 2019, a total of 1,769 CBO staff 

members and supervisors had received training in C2C skills. The number of staff members 

trained increased most rapidly during the first year of implementation, after which the number of 

trainees showed relatively stable, linear increases across time (Figure 5.1). Over the course of 

implementation, an average of 117.7 staff members were trained per CBO (median: 88; range: 

31–371). 

Figure 5.1. CBO Staff and Supervisors Trained in C2C Skills

887	

1,215	

1,636	

1,769	

0	

200	

400	

600	

800	

1,000	

1,200	

1,400	

1,600	

1,800	

2,000	

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3

Year	1 Year	2 Year	3 Year	4

Number of Staff and	Supervisors Trained

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

As noted earlier, the first year of implementation coincided with the availability of C2C 

Collaborative-sponsored MHFA trainings, which allowed for a large number of CBO staff 

members to be trained in a short period of time with minimal effect on CBO or MHP resources. 

Further, MHFA was seen by many CBOs and MHPs as foundational for introducing CBO staff 

members to basic mental health concepts; this was viewed as particularly important for those 

staff members who may have had little formal education or training in mental health. As a result, 

of the four C2C skills, MHFA was the most common skill in which CBO staff members were 

trained in year 1 (Figure 5.2). Following year 1, there was a shift toward emphasizing training in 

other C2C skills—in particular, MI. 

As implementation progressed, most trained staff reported having been trained in more than 

one C2C skill (Figure 5.3). Overall, the proportion of staff trained in all four core C2C skills rose 

from 31 percent in year 2 to nearly half (48 percent) in year 4. 

Further, the penetration of C2C training within the CBO—or percentage of all staff members 

within a CBO who had received training in a C2C skill—was considerable (Figure 5.4). In early 

implementation, CBOs typically focused on training a smaller set of core staff, but over time they 

74 
	 	 	 	 	 	

	 	 	 	

	 	 	 	

	 	 	 	

	 	 	 	

100%

90%

80%

26.7%

19.3%

23.0%

31.1%

17.9%

17.2%

18.6%

46.2%

22.7%

17.0%

12.3%

48.0%

Trained in 1 C2C Skill 70%

60% Trained in 2 C2C Skills

50%

40% Trained in 3 C2C Skills

30%

Trained in 4 C2C Skills

20%

10%

0%

Year 2 Year 3 Year 4

(N = 135) (N = 290) (N = 300)

800

Number of Staff Trained

700

600

500

400

300

200

100

0

Year	1 Year	2 Year	3 Year	4	(Q1–Q3)

MHFA

Number of Staff Trained	in	... Screening

Psychoeducation

 

 

 

Figure 5.2. CBO Staff Trained by C2C Skills

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

NOTE: Individual staff may have been trained in more than one modality per quarter.

Figure 5.3. C2C Skills Trainings Received by CBO Staff

SOURCE: Data from CBO staff survey, summer 2017 to 2019.

broadened training availability to include a larger number of staff. By year 4, the time at which 

CBOs had by and large attained “steady-state” implementation, on average nearly 60 percent of 

staff members within CBOs had received training in at least one C2C skill, with MHFA being 

the most common skill. However, there was considerable variability in penetration of C2C 

trainings across CBOs and with respect to specific C2C skills. 

75 
120.0%

100.0% 97.6% 96.4%

80.0% 76.2%

82.1%

60.0%

40.0%

20.0%

0.0%

13.5%

7.5% 2.6%

20.0%

2.1%

MHFA MI Psychoeducation Screening

61.1%

26.9% 26.1%

 

 

Figure 5.4. CBO Staff Trained in Each C2C Skill (Year 4)

Max Min Median

SOURCE: Data from quarterly CBO reports provided to RAND staff, 2019 (year 4).

Training approach. CBOs continued to vary training approaches by the specific C2C skill 

throughout implementation. For example, nearly all CBOs (n = 14) trained a subset of staff 

members in screening and PE but provided MHFA to a broader range of staff members. This 

approach is consistent with the manner in which most CBOs integrated the different C2C skills 

into existing programming. CBOs that incorporated screening into intake procedures typically 

opted to train only individuals who conducted those activities in the course of their typical CBO 

responsibilities (e.g., intake specialists). Similarly, because PE was often integrated into existing 

CBO programming or delivered in conjunction with screening, most CBOs opted to train only a 

subset of staff members in PE. For example, PE groups targeting specific skills or behaviors 

were often delivered as part of CBO programming that included adult education classes, career 

development classes, support groups for young people, and support groups for parents. However, 

at least one organization chose to train a larger segment of staff members (all direct service staff) 

in PE to ensure that staff members who had direct client interactions could deliver this C2C skill 

(see Appendix A). In contrast, MHFA skills were viewed as more foundational, generalized 

skills that could be implemented in a wide range of client interactions. In the context of C2C, MI 

was typically described as being “infused” into all elements of CBO client work. As such, many 

CBOs (n = 6 for MI; n = 13 for MHFA) opted to train a broader set of staff members in these 

C2C skills beginning in years 2 and 3. For example, as described by one CBO leader, 

I’ve noticed from year 2, it’s heavily focused on implementation [as opposed to 

program development in year 1], and motivational interviewing as well has had a 

big impact on that. We have a huge MI initiative at [CBO], which just kind of gets 

everyone across the board, and we opened it up to the [whole] organization. . . . 

76 
 

 

So, it kind of really expanded our C2C services in year 2 going into year 3, which 

was huge for us. 

In addition to training CBO staff members, one CBO also reported training some clients in 

C2C skills, specifically MHFA and MI, to better equip them in peer support roles within the 

CBO. Most clients who participated in C2C skills trainings were highly engaged and acted in 

peer support roles. C2C trainings were seen as a means of augmenting these clients’ capacity to 

serve in these peer support roles within the CBO. As described by one CBO staff member, 

We have not only trained all of our program staff in youth MHFA but we adapted 

the youth mental health first aid curriculum and trained youth members, who are 

our peer leaders’ They’re part of an internship [program], an’ they’re seen as the 

leadership amongst the youth membership community. I actually trained a cohort 

of young people in mental health first aid, and that was really, really cool. 

Such efforts went above and beyond the requirements of the C2C initiative and represented 

changes from the ways in which many CBOs had originally envisioned providing skills trainings. 

Additional Skill Training

Although not explicitly required to do so, many CBOs incorporated additional content or 

skill trainings into their C2C programs that went beyond the four core C2C skills. Specifically, 

several CBOs incorporated content related to trauma-informed care, which aligned with a shift 

over the course of C2C implementation toward increased screenings for traumatic experiences 

and PTSD. Other CBOs incorporated additional trainings in and integrated its principles into 

PE content. Such additions were seen as beneficial for both staff trainees and clients. In response 

to CBO staff members’ concerns and discomfort with the prospect of discussing suicide 

prevention with clients, at least one CBO reported incorporating crisis management and suicide 

prevent screenings into their existing C2C skills trainings (typically PE and screening). As 

described by one CBO leader during summer 2017, 

We are going to introduce a suicide training. We have to. Based on the conversations 

that staff have expressed discomfort. A large part of the discomfort with the 

screening has been around the question of [suicide ideation]. 

Some CBOs, based on their experiences conducting C2C trainings and feedback obtained 

from staff members and clients, identified specific content gaps and adapted training plans 

and protocols accordingly to meet the needs of their CBOs and their clients. 

Staff Buy-In and Tailoring C2C Trainings

As detailed in the interim evaluation report, trainings for specific skills varied considerably 

with respect to protocols, content, and degree of standardization across CBOs (Ayer et al., 2018). 

For example, MHFA trainings follow a standard curriculum that focuses on imparting mental 

health knowledge and specific steps for managing mental health crisis situations. In contrast, 

training and materials for administering screenings were heavily tailored to CBO staff needs and 

existing workflow. Because of this variability, CBOs and MHPs collaborated closely and iterated 

77 
 

 

 

 

extensively on training processes to ensure trainings for the core screening skill were appropriate 

for the CBO setting. 

Because CBOs gained experience with C2C, they devoted resources to ensuring that 

trainings aligned with the needs of CBO staff members and addressed how to apply the skills 

directly to client interactions. To this end, many CBOs and MHPs reported taking steps to ensure 

that trainings were interactive (vs. purely didactic) and addressed real-world CBO situations. 

Strategies such as role-plays and breakout discussions continued to be incorporated into trainings 

to concretely demonstrate the application of specific skills to real-life scenarios that trainees had 

encountered. As one MHP leader reported, 

[We ask trainees] to pick an example, work through for themselves with a 

modality [C2C skill] they might use and then employ it [in role plays] when we 

went to those break out groups and then at the end we had them [CBO staff] 

complete just a brief evaluation survey, on the other side of that worksheet that 

mostly it was like, what was this training like for you? Did it feel helpful enrolled 

into your work? What modality feels the most relevant to what you do on a 

consistent basis? How much did you learn today given what you already learned 

previously and how can we support you? How can we support you moving 

forward and how can we adapt these changes to suit your needs? 

Incorporating examples from CBO staff members also helped them feel more connected to 

trainings and related the training content to their day-to-day experiences with clients. As 

described by one CBO leader, 

Well, we encourage staff to bring real life scenarios . . . The staff members feel 

more empowered, more invested, and their takeaways for them are more 

meaningful. 

Trainers also took time to connect with CBO staff trainees and to explicitly link training 

materials to on-the-job challenges. As reported by one MHP trainer, 

In the beginning, I was really focusing on MI skills and are you really understanding 

the MI concepts. And I began to see the glazed, dazed over look. And I was like, 

“oh, okay. I have to really, really take it slow just like we’re asking them to with 

clients.” And so now it’s really organic in the sense of sometimes it’s really 

focused on self-care. How has your week been going? What have you been 

dealing with? And then I try my best to identify, help them see how what’s up for 

them is connected to the themes that we’ve been talking about, secondary posttraumatic stress, ambiguous loss and grief, and the trainings that we’ve had. 

CBO and MHP trainers also regularly solicited feedback from CBO staff trainees 

surrounding efficacy and relevancy of training. As reported by one CBO leader, refining the 

approach to delivering C2C trainings was an iterative process that involved both formal and 

informal feedback from trainees: 

I’m really always trying to evolve and try new things based on the feedback that I 

get from [trainees]. Whether it’s verbal or I do the [feedback] surveys at the end, 

[we’re] trying to incorporate that. 

78 
 

 

 

 

Other CBO leaders described efforts to continuously improve C2C skills trainings by soliciting 

written feedback and conducting focus groups with trainees to ensure that trainings met their 

needs. For example, when discussing efforts to refine trainings, one CBO leader reported, 

We knew we wanted to do an exit survey and we started to roll it out and then we 

have modified it. We wrote questions and we realized, “oh this question actually 

isn’t giving us the answer that we’re looking for.” We modified that. We wanted 

to do staff focus groups, to really understand. 

One CBO leader reflected on how efforts to incorporate staff experiences into C2C trainings 

helped underscore how of C2C applied to staff members’ day-to-day work: 

I think [training is] pretty well received. . . . One thing we always want to do is 

try to relate the context to something experiential and touch on something that’s 

happened that week or concern the staff has been talking about to make it more 

easily applied. So it’s in everyone’s best interest to be there. 

Over time, efforts to ensure that C2C skills trainings were appropriately tailored to the 

CBO helped align with staff and client needs in diverse CBO environments. 

Staff Discomfort with Addressing Serious Mental Health Issues

As described in the C2C interim evaluation report (Ayer et al., 2018), many CBOs 

experienced challenges with trainings in the early stages of implementation, such as concerns 

regarding additional workload. Some of these concerns lessened over time because CBO staff 

members recognized improvements in their client interactions after the trainings. As reported by 

one CBO leader, 

What is the main reason staff are resistant? Really it’s because they feel it’s 

additional work. Where we are now, people understand it’s a training opportunity 

and a personal development opportunity to engage better with clients and to 

solicit more information from clients to help them better. Everybody understands 

that now. 

However, some challenges persisted throughout implementation. For example, many CBOs 

identified ongoing staff members discomfort with addressing mental health concerns with 

clients, particularly with issues of suicide and serious mental illness. As reported by one CBO 

staff member, 

I’m still not comfortable going through the suicide prevention and the depression 

and anxiety check list. And the reason why I’m not comfortable it’s not because 

of those terminology but because I’m afraid I’ll make a mistake. I’m afraid I’ll 

trigger the client even more. So, I would prefer to do away with that checklist 

and having the social worker handle that and not me. I just feel like I don’t have 

the degree in social work or mental health, I’m not licensed, and I just don’t feel 

comfortable doing that. The mental health first aid and motivational interviewing, 

I’m comfortable. But the suicide prevention I’m not and I would prefer when we 

refer the client and we talk to the client about services and if they’re willing and 

if they sign a consent form and all that stuff, and just refer them to the social 

worker. 

79 
 

 

 

 

 

Although such challenges were routinely endorsed with respect to initial trainings for CBO 

staff members, providing follow-up trainings and ongoing support for C2C staff trainees paid 

dividends in improving staff members’ comfort and confidence in delivering C2C skills over time. 

Staff Turnover and Other Organizational Issues

Some CBO leaders specifically reported challenges related to frequent staff turnover, which 

led to interruptions in training and a need to hold additional trainings for newly hired staff. As 

described by one CBO leader, 

Once we have people trained, we have a hard time keeping it going because we 

have high turnover. So, we are working on how to keep the quality going with 

the staff turnover. 

Similar concerns were echoed by other CBO leaders, who viewed extant challenges with 

turnover and the ongoing need to train new hires in C2C as a persistent “big lift” throughout the 

Active Implementation stage. For example, one CBO leader said, 

We also have at any given time, which is not unheard of at all in the social 

service world, a 30 to 40 percent turnover rate . . . . So you are not just training 

people, it wasn’t just the first year and we got everybody through the training, we 

are now in year 3 and we are still training new staff that are coming on board so I 

think that there is this big lift in the beginning to get everybody trained, but there 

is always going to be that turnover. 

Starting in implementation year 3, all CBOs began reporting on departures of C2C staff 

members. Although totals varied by quarter, the percentage of C2C staff member departures 

relative to all C2C staff members was fairly low, averaging approximately 6 percent (range: 

1–13 percent). Despite this, more than 40 percent of CBO staff perceived staff turnover to be a 

problem in years 2 through 4 of implementation. Generally, CBO leadership did not attribute 

turnover challenges directly to C2C; rather, turnover was seen as a preexisting and perennial 

challenge in some CBOs. However, efforts to shift training capacity “in house” were seen as 

helpful in ensuring continuity and sustainability of C2C to allow CBOs to deal with such 

challenges over the course of implementation. 

Along with turnover, CBO staff reported organizational readiness challenges such as 

communication issues and staff burnout. CBO staff perceptions of C2C implementation varied 

by CBO type. Overall, staff from youth development organizations reported experiencing more 

challenges and stressors than staff from parent/caregiver-serving and job training and employment 

organizations. For example, among staff at youth development CBOs, 60 percent agreed that 

frequent staff turnover was a problem and 68 percent agreed that staff members often show signs 

of stress and strain (Figure 5.5). These percentages for staff at youth development CBOs are 

much higher than the combined average rates of other CBO types. Although CBO staff reported 

some staff-related challenges, most reported that the goals of C2C were clearly communicated 

and they understood how C2C fits within their work. More than 60 percent of all staff survey 

respondents agreed that C2C operated with clear goals and objectives. 

80 
	 	 	 	 	 	

	 	 	 	 	 	

	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	

	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	

	 	 	

	 	

Percent Agree or	Strongly Agree

0% 25% 50% 75% 100%

This program operates with clear goals and

objectives.

Your duties are clearly related to the goals of this

program.

Program staff understand how this program fits as

part	of the treatment	system in your community.

Frequent	staff turnover is a problem for this

program.

Staff members often	show signs of stress and	

strain.

Program staff are always kept well informed.

61%

70%

63%

65%

58%

62%

69%

62%

53%

60%

58%

53%

47%

43%

60%

47%

43%

68%

45%

55%

51%

40%

50%

42%

Parent/Caregiver-Serving Job	Training & Employment Youth	Development Other

Figure 5.5. Perceptions of C2C Program Functioning by CBO Type

SOURCE: Data from CBO staff survey, summer 2017 to 2019.

Coaching and Supervision

CBOs and MHPs continued to devote considerable resources to ensuring that, once trained, 

CBO staff members had appropriate support to use new C2C skills effectively with clients. 

During implementation, some CBO staff members expressed a desire for more extensive followup coaching and supervision to aid them in delivering high-quality mental health support to 

clients, including additional booster sessions to help reinforce knowledge and skills: 

I think if there was more contact and more practice, I would feel better about it 

but, I think even once again, it’s hard because we are so busy all of the time and 

of course it would be beneficial to have more trainings, more booster sessions, 

more refreshers. 

I thought the trainings themselves were good. But as I complain with all 

training, if they are a one-day training or two-hour trainings, a few weeks later it 

81 
Number of Coaching and	Supervision	

Hours Provided

MHP

CBO

MHP

CBO

MHP

CBO

MHP

CBO

MHP

CBO

MHP

CBO

MHP

CBO

MHP

CBO

MHP

CBO

MHP

CBO

MHP

CBO

600 Y2 Implementation

MHP

Guidance Distributed:	

CBO

500

MHP

CBO

MHP

400

300

CBO

MHP

200

CBO

100

0

Y1	Q1 Y1	Q2 Y1	Q3 Y1	Q4 Y2	Q1 Y2	Q2 Y2	Q3 Y2	Q4 Y3	Q1 Y3	Q2 Y3	Q3 Y3	Q4 Y4	Q1 Y4	Q2 Y4	Q3

Screening Psychoeducation Motivational Interviewing Other

 

 

 

 

may not be firmly planted in my head because there’s not an easy way to follow 

up and practice these things. 

The material is good to go back to review. But I really feel that booster 

courses would keep us on our toes. 

I think I need a follow-up training on the screenings, but that’s just myself. I 

think there are a lot of people who aren’t comfortable with it. 

In response to such feedback, as well as the updated guidance from the C2C Collaborative 

surrounding coaching and supervision, many CBOs and MHPs dramatically expanded their 

delivery of follow-up supports to staff. In year 2, both the quantity (i.e., number of hours) and 

frequency of coaching and supervision increased substantially (Figure 5.6). Similar to the initial 

C2C skills training, MHPs provided most of the coaching and supervision in the early stages of 

implementation (i.e., years 1 and 2). However, as plans solidified, and CBO supervisors accrued 

more experience and expertise in C2C skills training, CBO staff members took on a more 

substantial role in coaching and supervision activities, essentially matching MHPs by the end of 

year 2 and into year 3. In year 4, overall CBO coaching and supervision hours decreased again 

relative to the hours provided by MHPs. This may be attributable to a range of factors, including 

streamlining of coaching and supervision practices and increased emphasis on more intensive 

fidelity monitoring practices in years 3 and 4, which involved more intensive input (and time) 

from MHP clinical supervisors. 

Figure 5.6. Coaching and Supervision Hours by Core C2C Skill

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

NOTE: “Other” includes coaching and supervision for MHFA and any other C2C topics or skills (e.g., trauma-informed 

care) outside screening, PE, or MI.

82 
 

 

 

 

Early Challenges and Improvements

MHPs described several early challenges to providing coaching and supervision. One 

challenge concerned an apparent lack of buy-in from CBO leadership. Because CBO staff were 

not supervised by MHPs as part of their regular CBO employment, MHPs had to rely on CBO 

leadership to encourage and compel CBO staff to attend coaching and supervision sessions. 

MHPs shared that CBO staff did not attend these sessions because they were not mandatory, 

were not valued by CBO management, and their importance was not clearly communicated. As 

one MHP explained in 2017, 

They [CBO staff] don’t really get clear communication from their supervisors or 

the leadership that they needed to be there. So, they don’t really think they need 

to go, and they do get overwhelmed. You know, it’s very demanding work. There 

are a number of unexpected events that happen regularly, and they find it very 

difficult to keep a schedule. 

Some CBO staff viewed C2C as additional work, which meant that coaching and supervision 

sessions were simply another task added to their workload. In discussing challenges with CBO 

staff engagement in coaching and supervision sessions, one MHP said, 

It’s hard to get into mental health. They [CBO staff] have to have buy-in and 

interest and investment in learning more and it’s a lot of added work. 

Relatedly, some CBO staff missed coaching and supervision because they felt overworked 

or viewed the nature of their work as incompatible with structured meetings. Other CBO staff 

needed to respond to unexpected events during their workday, making a highly structured 

approach infeasible. For MHPs, concerns like these meant that they had to be flexible with their 

approach by having individual sessions or group sessions depending on the circumstance or 

using unstructured encounters to provide coaching and supervision. Some MHPs also provided 

incentives such as food and metro cards to encourage staff to attend coaching and supervision. 

When asked about their experience with coaching and supervision, one MHP reported, 

I think the nature of working at [CBO] is that things happen on an unscheduled 

basis. So, then it’s very hard to plan out even though it’s an hour before your 

group you have no idea what is going to be going on that day . . . And it’s hard 

for [CBO staff], it’s a cultural shift almost for us to say you have a meeting and 

why don’t you go to it? 

In implementation year 2, clarifying guidance issued by the C2C Collaborative outlined 

specific requirements for continuous coaching and supervision of CBO staff members (see 

Chapter 4). This updated guidance, in conjunction with targeted technical assistance, led to an 

increased emphasis on mitigating challenges to meet quarterly coaching and supervision targets 

in years 2 to 4 of implementation. By the first quarter of year 4, CBOs reported providing 

quarterly coaching and supervision to 101 percent of the targeted number of staff supervisees on 

average (range: 54–161 percent) with most CBOs meeting or exceeding the coaching and 

83 
 

 

 

 

supervision targets. Although ten CBOs exceeded their coaching and supervision targets in the 

first quarter of year 4, four CBOs did not reach coaching and supervision targets in this quarter. 

CBO Staff Buy-In over Time

Even with the challenges described above, CBO staff buy-in to the C2C program improved 

over the course of implementation. CBO staff increasingly felt empowered by their ability to 

help clients with mental health issues and attributed this to their C2C coaching and supervision. 

One CBO leader reported, 

[Staff members] feel more self-actualized. From my perspective, as someone 

who runs a program, you want staff who feel better about what they’re doing. I 

like to know they’re feeling fulfilled. I’ve observed the family workers feel much 

more fulfilled. I think it’s brought up chances to ask meaningful questions. 

Another CBO leader described the importance of empowering CBO staff through coaching 

and supervision, because they were the ones with the most direct and trusting client relationships 

that would afford the opportunity to address client mental health issues: 

I think that it’s been about empowering staff because they really are the ones who 

are meeting and speaking to participants every single day . . . So, providing them 

with the proper tools to have conversations with participants, I think that’s been a 

key component to engage participants into services. 

CBO staff buy-in improved when staff members felt supported by their coworkers and 

supervisors to perform C2C-related tasks. One CBO leader reported their approach to messaging 

C2C training as a time for staff to build teamwork: 

Staff training during lunch works well since our staff doesn’t have common 

break times, that was a good time to get them all and it feels good and supportive. 

C2C has been a lot for staff, so we had to shorten some of those trainings, the 

idea of having lunch together supports that, it is not common here that we have 

opportunities to sit back and connect with our peers. 

Similarly, CBO leaders reported that C2C coaching and supervision allowed for ongoing, 

in-depth discussions about staff members’ challenges with C2C, and opportunities to further 

address mental health stigma. One CBO leader stated, 

I think one of the things that we found helpful is just providing space for staff to 

be able to talk about the struggles that they’re sitting with, and saying, ‘this is not 

my job to do this.’ And being able to talk about where that clinical line is . . . 

reassuring them of the supports that are around them, and being really clear and 

saying, ‘we want you to have this knowledge and we want you to have these 

tools because even though you haven’t had full clinical training there are tools 

that you can begin to use that are very helpful in providing mental health 

supports to young people.’ And the more adults in our organization that have 

those tools, the more well-supported you feel and less stigmatized the idea of 

talking about mental health issues is. 

84 
 

 

 

As with initial C2C trainings, supervisors incorporated real-world experiences from staff 

members to maximize the perceived benefit and relevance of C2C skills coaching and 

supervision for staff members. For example, one CBO leader reported, 

They weren’t getting the support that they needed either in terms of in-depth 

support around difficult situations with clients nor being helped from a self-care 

perspective and helping them understand everything from helping them have a 

trauma frame, helping them have a culturally informed frame, helping them even 

understand why we’re thinking about mental health beyond, “oh yeah, that’s a 

good idea.” But like really the specificities of each case. And then helping them 

integrate that into thinking about how does this affect me, and what am I bringing 

to the process: that reflective piece. 

Fidelity Monitoring

During coaching and supervision sessions, supervisors implemented a range of strategies to 

ensure that CBO staff members were delivering C2C skills to clients as intended and consistent 

with best practice. These strategies included direct observation (when possible), modeling of 

best-practice skills delivery, and role-plays in which CBO staff members demonstrated use of 

C2C skills with a supervisor based on an actual or plausible real-life scenario. Throughout 

implementation, the use of specific fidelity monitoring strategies varied depending on factors 

such as the CBO environment, the nature of programming (e.g., whether services were typically 

delivered in one-on-one vs. group settings), and supervisor/supervisee availability. One MHP 

supervisor said, 

Unfortunately, I don’t get an opportunity to see the staff in action just because 

I’m in a different building and, also, they’re not always here with their clients. 

Through the roleplaying and through the coaching in terms of us meeting and 

them talking about some of the challenges that they are having, that’s how I’m 

starting to gauge on some of the skills. Also, we just had our first coaching 

session for all of them so with time I’ll be able to gauge in terms of how they’re 

using some of the stuff and how they are able to integrate the process within how 

they do things overall, so with time I’ll have a better understanding. 

Moreover, some CBOs began to integrate aspects of C2C fidelity monitoring into other 

dimensions of existing CBO supervisory practices for non-C2C skills (i.e., for standard CBO job 

performance supervision). As described by one CBO leader, 

[MHP] is supervising and coaching on implementation of C2C modalities but we 

[CBO] are also trying to integrate MI compliant supervision for all of our managers 

and also asking them to assume some responsibility as a minimum checking in 

with their staff members to just kind of reinforce and trigger a memory of using 

those modalities in their interactions with clients. 

Differing Approaches and Varying Degrees of Quality

Although recognizing the value of fidelity monitoring, partnerships took a very individualized 

approach depending on who was supervising CBO staff (e.g., MHP or trained CBO supervisor), 

the time supervisors had on-site at the CBO, and their familiarity and comfort with CBO clients 

85 
 

 

 

 

and staff. For most CBOs, fidelity monitoring processes grew and strengthened over time 

because their focus shifted from training staff to ensuring high-quality implementation of C2C 

skills. 

Before beginning to monitor fidelity, C2C leaders at the CBO and MHP first had to 

normalize the process with CBO staff. This included helping staff understand that fidelity 

monitoring was intended to support skill development, assess the strengths in individual staff 

practice, and identify concrete strategies to improve their skills. Because this type of feedback 

was new for many CBO staff, C2C supervisors needed time to build trust and develop ways to 

deliver and help staff members process this kind of feedback. 

In determining approaches to fidelity monitoring, partnerships also had to carefully consider 

both CBO staff and client comfort and preferences. Implementation guidance specified that fidelity 

monitoring would ideally involve direct observation of CBO staff interacting with clients and 

delivering C2C skills. However, there are many situations where having a stranger (i.e., the MHP 

or CBO supervisor) in a sensitive CBO staff-client conversation could negatively affect the client. 

To circumvent this challenge, supervisors routinely observed trained CBO staff members in a 

role-play with each other (i.e., with one acting as the client and the other as the service provider) 

and offered observational feedback. Supervisors also routinely engaged in modeling C2C skills 

in the context of role-plays with staff trainees to demonstrate use of C2C skills with fidelity. 

Over the first 3 years of C2C implementation, CBOs encountered significant challenges with 

developing and implementing plans for monitoring and quantifying fidelity of C2C skill delivery. 

One MHP leader discussed the importance of fidelity monitoring and described concerns with 

the rollout and maintenance of fidelity monitoring procedures at the CBO: 

I think fidelity is going to be, over time, the most challenging thing. When 

people, of course, get comfortable in a skill set, they make it their own. Right, 

which is a good thing. [But] we also do want to maintain the fidelity. So, I think, 

we’ll just have to keep doing supervisions and overseeing people working and 

making sure that they are keeping to the fidelity. I think that will definitely be an 

ongoing effort. Especially as more staff comes on and we have to make sure that 

they are maintaining fidelity. 

In response to these challenges, and requests from CBOs and MHPs for support in cultivating 

more robust and standardized fidelity monitoring practices, the C2C Collaborative shared a 

standard fidelity monitoring tool, developed in partnership with RAND, to collect quantitative 

assessments of staff that could be compiled and tracked at each CBO. Rating forms were designed 

to be completed based on direct observations of staff members delivering C2C skills to clients. 

Although a majority of CBOs reported utilizing this or other tools as part of required 

supervisory practices, documentation and tracking of fidelity monitoring information (to allow 

supervisors or evaluators to review improvements over time) was inconsistent and data quality 

varied widely across CBOs. Key challenges contributing to this inconsistency included, among 

other factors, variability in approaches to supervision practices (e.g., use of role-play vs. inperson observation), lack of opportunity for direct observation of skills in the context of CBO 

86 
 

workflows (e.g., due to the unpredictable timing of staff-client interactions), staff turnover, 

and differences across CBOs with respect to the number and types of supervisors tasked with 

monitoring fidelity to particular C2C skills. Further, collecting consistent fidelity data was 

difficult when staff turnover was high, and CBOs had to focus on training staff instead of 

monitoring staff progress and use of C2C skills. 

Variation in supervision practices across CBOs was another challenge to collecting uniform 

fidelity data. For example, one CBO had a single MHP supervisor who was charged with 

monitoring fidelity for all four core C2C skills for all staff members while the remaining 

13 CBOs divided supervisory responsibilities for all C2C skills across two or more supervisors. 

In this latter group, CBOs varied with respect to which types of C2C skills (e.g., PE, MI) 

individual supervisors were charged with overseeing. As such, although CBOs reported 

increased utilization of more standardized fidelity monitoring tools over time, comparison of 

fidelity across CBOs was not possible. 

Question 2: How Were C2C Services Provided to Clients?

The protocols that CBOs used for deciding when and with whom to deliver the core C2C 

skills evolved over time, and these changes were highly specific to the individual CBO setting. 

Screening and PE were delivered as distinct activities, with some variation across CBOs. The 

screening tools that many CBOs chose at the beginning of the project changed over time based 

on staff and leadership perceptions of their clients’ evolving needs. Screening procedures also 

changed due to observed inefficiencies in practice, client feedback, or lack of client uptake. PE 

was implemented differently across CBOs in terms of content, timing, and settings. Some CBOs 

used PE as a way to introduce the C2C program to clients, whereas others offered more formal 

workshops on topics such as stress management, relationship issues, and harm reduction for 

people using drugs or alcohol. Some CBOs changed their PE topics and delivery mechanisms 

over time by routinely collecting and reviewing feedback from clients, but not all CBOs did so. 

As described earlier, the implementation of MHFA and MI involved infusing these skills into 

all client interactions rather than discrete episodes of service delivery. With MHFA, some CBO 

protocols specified applying MHFA to clients perceived to be in a crisis situation and emphasized 

the importance of training staff to recognize mental distress. Most CBOs did not report major 

changes to MHFA protocols over time. CBOs and MHPs had significant challenges with 

implementing MI trainings in the early stages of program implementation, which led to the 

creation of the C2C MI Institute in implementation year 2 and delayed the initiation of MI 

delivery to clients. Following the rollout of MI trainings, most CBOs described MI as being 

“infused” into virtually all client interactions rather than being delivered in discrete service 

episodes (see Chapter 4). 

This section first focuses on overall C2C skill delivery and then on implementation of 

screening and PE as distinct activities. A detailed summary of each CBO’s unique approach to 

C2C program implementation, as of year 4 of the C2C initiative, is provided in Appendix A. 

87 
4,000	

Average Number of New C2C Clients per

Quarter

3,500	

3,000	

2,500	

2,000	

1,500	

1,000	

500	

-

Year	1 Year	2 Year	3 Year	4

(Q1–Q3)

 

 

 

Overall C2C Skill Delivery

From March 2016 through December 2019, CBOs provided C2C services to more than 

41,000 unique clients. The number of new clients receiving C2C services increased dramatically 

from the end of year 1 (March 2017) to the end of year 4, quarter 3 (December 2019), rising 

from 5,100 to 41,294 clients served through the C2C initiative. Across all CBOs, the average 

proportion of the eligible pool of CBO clients receiving at least one C2C skill also increased 

from 34 percent in year 1, quarter 4 (range: 3–100 percent) to 72 percent in year 4, quarter 3 

(range: 12–100 percent). (Figure 5.10 shows the quarterly trend in this proportion across the full 

period of program implementation.) 

The actual number of clients served in year 1 fell slightly below the target of 8,662. This is 

consistent with feedback from CBO and MHP leaders that organizations placed considerable 

emphasis on developing and refining training and service delivery protocols in year 1 prior to 

rolling out C2C services to clients. In year 2, when most sites had established robust training and 

support programs for CBO staff members who delivered C2C skills to clients, CBOs met the 

target number of participants served, and the number of clients who received C2C services in 

year 3 exceeded the target of 11,011. Across the full project period, approximately 2,751 clients 

were served with at least one C2C skill per CBO (median: 2,327), with the number of clients 

served ranging from 296 to 8,067 across CBOs. The average number of new C2C clients served 

per quarter also increased over time (Figure 5.7). By year 3, CBOs served about 3,500 new 

C2C clients each quarter. 

Figure 5.7. New C2C Program Clients Served per Quarter, by Year

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

88 
 

 

The average number of C2C clients served per CBO (e.g., clients that received at least one 

C2C skill) during implementation varied by CBO type (Figure 5.8). On average, the five CBOs 

focused on job training and employment served 3,067 clients per CBO (median: 2,748; range: 

1,495–5,644), whereas the three youth development CBOs served an average of 4,270 clients per 

CBO (range: 710–8,067), the three parent/caregiver-serving CBOs served an average of 1,071 

clients per CBO (range: 296–2,327), and the four other CBOs served an average of 2,477 clients 

per CBO (range: 802–4,268). 

The lower average number clients served by parent/caregiver-serving CBOs compared with 

the other CBO types may be attributable to a number of factors. First, one of the three parent/ 

caregiver-serving CBOs was discontinued from participation in the C2C initiative by the funder 

at the end of year 3. As described earlier, the CBO’s inability to implement the program was due 

to a range of factors that affected service delivery capacity and, in turn, resulted in lower numbers 

of clients served at this site over implementation years 1 through 3. However, this outlier does 

not appear to fully explain the observed pattern of findings; after excluding data from this CBO, 

the pattern persists with parent/caregiver-serving CBOs, averaging 1,459 clients per CBO. 

Another potential factor is that parent/caregiver-serving CBOs tended to enroll families in CBO 

services to capacity during a brief window each year (e.g., in summer/fall) and did not enroll 

again until the next year. Moreover, it was common for the same families to enroll in CBO 

services each year, which created limited opportunities to enroll new C2C clients from year to 

year (note: this was reflected in CBOs’ annual programmatic targets for new C2C clients). This 

is in stark contrast to other types of CBOs, for which enrollment in CBO services occurred on 

more of a rolling basis. 

Figure 5.8. Average C2C Program Clients Served per CBO, by CBO Type

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

89 
 

 

 

 

 

 

This pattern may also be related to the extent to which a CBO’s C2C target population was 

directly engaged with CBO activities, which varied by CBO type. Parent/caregiver-serving 

CBOs often provided more direct services to children (e.g., preschool, day care, or afterschool 

programs) than to parents/caregivers; as such, the parents/caregivers of young children, who 

were the C2C target populations, were not necessarily directly engaged in CBO programming 

themselves. Although the parent/caregiver-serving CBOs offered some types of services for 

parents (e.g., skills trainings and educational courses; social activities), parents/caregivers varied 

with respect to the level of engagement and the frequency of contacts with CBO staff. In 

contrast, most other CBOs aimed to provide services to CBO clients who were directly engaged 

in CBO services, which may have increased opportunities for C2C service provision. As such, 

parent/caregiver-serving organizations aimed to serve fewer C2C clients per year compared with 

other organizations, as shown in Table 5.3. These organizations also reduced their site-specific 

annual targets for number of C2C clients served from year 1 to year 2 and beyond, whereas other 

types of organizations showed stable or increasing targets for annual number of new clients 

served over the course of the implementation process. 

Table 5.3. Annual New C2C Client Targets, by Year and CBO Type

CBO Type

Y1 (Mean

[Min, Max])

Y2 (Mean

[Min, Max])

Y3 (Mean

[Min, Max])

Y4 (Mean

[Min, Max])

Job training and

employment

809 (500–1,500) 815 (500–1,500) 833 (575–1,500) 833 (577–1,500)

Youth development 817 (200–1,250) 817 (200–1,250) 1,233 (200–2,000) 1,233 (200–2,000)

Parent/caregiverserving289 (250–317) 176 (150–209) 218 (150–334) 235 (120–350)

Other 325 (100–600) 350 (100–600) 613 (150–1,500) 613 (100–1,500)

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

NOTE: Years 1 through 3 include data from 15 CBOs; year 4 includes data from 14 CBOs.

Relative to their site-specific program targets, on average, all types of CBOs fell below their 

anticipated targets in implementation year 1. As implementation progressed, all CBO types came 

close to or exceeded their targets the average number of clients served (see Figure 5.9). This 

suggests that, with experience and a better understanding of how CBO services would be provided 

to clients in practices, organizations were largely successful in achieving their programmatic 

goals with respect to number of clients served, regardless of CBO service type. 

C2C Service Delivery Stabilization

Across all CBOs, the percentage of CBO clients eligible to receive C2C services who 

received C2C services each quarter stabilized in year 2 and remained fairly steady through later 

stages of implementation (Figure 5.10). This suggests that it takes about 1 year to establish the 

core components of C2C leading to routine skill delivery, which is consistent with stabilization 

90 
100%

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3

Year	1 Year	2 Year	3 Year	4

0%

25%

50%

75%

Percent of Eligible CBO Clients

Who Received at Least One	C2C

Support

 

 

 

Figure 5.9. Average Percent of Annual New Client Targets Met per CBO, by CBO Type

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

NOTE: Years 1 through 3 include data from 15 CBOs; year 4 includes data from 14 CBOs. Figure shows average of

program-specific percentages within each CBO type.

of training and service delivery protocols and efforts to integrate C2C services into existing CBO 

programming and workflows early in implementation. 

At the end of year 4 quarter 3, job training and employment CBOs provided at least one C2C 

skill to an average of 71 percent of total CBO clients (range: 12–100 percent), youth development 

CBOs provided C2C skills to an average of 86 percent of total CBO clients (range: 59–100 percent), 

parent/caregiver-serving CBOs an average of 62 percent (range: 24–100 percent), and other 

CBOs an average of 96 percent (range: 85–100 percent). 

Figure 5.10. Eligible CBO Clients Who Received at Least One C2C Skill

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

91 
	 	 	 	 	 	

	 	 	 	 	 	

	 	 	 	 	

	 	

	 	 	 	

100%

15.3%

10.9%

29.9%

24.8%

19.0%

Missing 90%

80%

6.8%

7.2%

29.7%

34.8%

21.5%

None of the Clients I Have 70%

4.2%

12.2%

34.0%

30.9%

18.8%

Seen

60%

Some (Less	Than Half of the 50% Clients	I	Have	Seen)

40%

Many	(More	Than	Half	of 30% the	Clients I Have	Seen)

20%

All	Or Nearly All	of the

10% Clients	I	Have	Seen

0%

Year 2 Year 3 Year 4

(N = 137) (N = 293) (N = 288)

 

 

 

 

 

Among staff members trained in C2C, most reported using C2C with more than half of their 

clients across the three survey waves (Figure 5.11). In implementation year 2, slightly more than 

one-tenth of trainees (11 percent) reported never using C2C skills with clients with this percentage 

declining to about 7 percent in year 3 and rising to 12 percent in year 4. 

Figure 5.11. Use of C2C Skills by CBO Program Staff

SOURCE: Data from CBO staff survey, summer 2017 to 2019.

Skill Delivery by Nonspecialist CBO Staff

Over time, most sites relied primarily on nonspecialist, or lay, CBO staff to deliver C2C 

skills to clients (Table 5.4). However, some CBOs (27 percent; n = 4) indicated that screening 

and PE were delivered by someone other than CBO lay staff. Although these CBOs varied in the 

size and types of services delivered, they were similar with respect to having CBO staff members 

Table 5.4. Delivery of Specific C2C Skills by Lay Staff

Delivered by CBO Lay Staff

C2C Skill Percentage (n)

Screening 73 (11)

MHFA 100 (15)

MI 100 (15)

PE 73 (11)

SOURCE: Data from CBO model summaries provided to RAND

staff by CBO leaders, July 2019.

92 
2,500	

500	

1,000	

1,500	

2,000	

Number of Clients Received Screening

Avg/Quarter = 1,486

0	

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3

Year	1 Year	2 Year	3 Year	4

 

 

 

with specialized mental health training embedded within the CBO setting prior to implementing 

C2C. Thus, although all CBOs leveraged lay staff to deliver some C2C skills to clients, the 

extent to which specific C2C skills were task-shifted away from staff with specialized mental 

health trainings and onto lay staff varied. 

This lack of task-shifting for screening and PE at some CBOs was attributable in part to 

challenges surrounding integration of C2C skills into existing CBO workflows. As described by 

a leader from one CBO that relied on counseling staff (LCSWs) rather than lay staff to conduct 

screening, 

So when we initially thought that all staff members would have time to do 

screenings within their work, and all the task shifting would happen in reality, we 

realized that that wasn’t something that was actually compatible with, you know 

to staff work burdens and things like that and we were going to have to rely much 

more strongly on referrals to our social workers, to our MHP rather than staff 

doing screenings themselves. 

Although lay staff members received training in screening, the CBO relied primarily on 

clinical staff to deliver this particular skill to clients. This suggests that although lay staff 

may have been trained in certain C2C skills, not all lay staff trainees routinely utilized these 

skills with CBO clients. 

Screening

CBOs ramped up screenings considerably over the course of year 1 (Figure 5.12). At the end 

of year 1, CBOs screened an average of 228 clients per CBO (median: 96; range: 5–1,442). 

CBOs screened an average of 414 clients in year 2 (median: 207; range: 24–1,656), 445 in year 3 

(median: 248; range: 20–1,583), and 436 in the first three quarters of year 4 (median: 26; range: 

48–1,967). By the end of year 2, the average number of client screenings per quarter had largely 

Figure 5.12. Clients Screened for Mental Health Symptoms per Quarter

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

93 
	 	

2,500	

0	

500	

1,000	

1,500	

2,000	

Median Number	of	Clients	Screened per	CBO

Job	Training & Youth Parent/Caregiver Other All CBOs

Employment Development -Serving

Year	3 Year	4	(Q1–Q3) Full Project

 

 

stabilized, with more than 1,300 clients being screened per quarter. By December 2019, CBOs 

were screening more than 1,600 clients per quarter and had screened a cumulative total of 

22,284 clients. 

At the CBO level, the number of clients screened varied in relation to CBO type (Figure 5.13). 

By year 4, when CBOs had fully implemented C2C screenings, the three parent/caregiverserving CBOs screened an average of 237 clients (range: 177–295), compared with the average 

of 2,432 clients screened by the five job training and employment CBOs (median: 2,010; range: 

919–5,718), 2,290 clients screened by the three youth development CBOs (range: 286–5,129), 

and 637 clients screened by the four CBOs in the other category (median: 641; range: 416–851). 

Figure 5.13. Clients Screened per CBO, by CBO Type

SOURCE: Data from quarterly CBO reports provided to RAND staff, year 3, March 2016 to December 2019.

CBOs also varied with respect to the proportion of C2C clients (i.e., individuals who 

received any C2C service) who were administered screening, which was particularly evident in 

later implementation years after CBOs had rolled out all four core C2C skills. As shown in 

Table 5.5, job training and employment organizations screened a majority (more than 70 percent) 

of clients who received any C2C service. In contrast, on average, less than one-third of C2C 

clients in parent/caregiver-serving organizations received screening by year 4 of implementation. 

94 
 

 

Table 5.5. Percent of Clients Who Received Any C2C Skill That Received at Least a Screening, by 

CBO Type and Project Year

CBO Type Y1 (%) Y2 (%) Y3 (%) Y4 (%) Total (%)

Job training and employment 81 84 80 71 79

Youth development 45 48 51 65 54

Parent/caregiver-serving 77 13 8 28 18

Other 46 22 18 33 26

Total 67 53 48 56 54

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

The lower screening rates for parent/caregiver-serving CBOs may be due to differences in 

the nature and frequency with which the clients engaged with CBO staff and their programming 

generally. By and large, encounters between clients at parent/caregiver-serving CBOs and CBO 

staff members were typically brief and occurred during transitional periods (e.g., picking up a 

child from the CBO). As such, parent/caregiver-serving CBOs may have had systematically fewer 

(and/or shorter duration) contacts between CBO/MHP staff and the CBO target population, and 

thus fewer opportunities to provide screening, compared with other types of CBOs. For example, 

as reported by one staff member, juggling both child and parent concerns in the context of 

providing C2C services was a challenge: 

The time, you know, with us to do [screenings], both her and the family worker 

to do it is just not conducive . . . by the time you get back to parents to follow up, 

they don’t want to come in, they’re busy. 

In addition, parental expectations surrounding the nature of interactions with CBO staff and 

the focus of the CBO on child well-being may reduce some parents’ willingness to accept a 

screening from CBO/MHP staff and/or may heighten discomfort with introducing screening 

materials on the part of CBO staff. As reported by some CBO staff members, 

Where parents are expecting this to just be an educational setting. “I want to 

enroll my child. I want to pick up, I wanna drop off.” And so being able to 

successfully crossover that line and invite them into share personal information. I 

think that’s been challenging . . . the initial resistance that some parents might. 

There have been people who’ve declined screening due to concerns about 

confidentiality and privacy and scheduling. It’s [also] been challenging like I 

mentioned to screen two caregivers, even one. People are here to pick up and 

drop off and don’t have time to sit and be screened. 

Most CBOs viewed implementation of formal screenings for mental health issues as 

foundational to the C2C program, with many prioritizing implementation of this core C2C skill 

in year 1. This was attributable to a number of factors, including the critical role of screening 

results in client referral processes, as well as the perception among CBO and MHP leaders that 

screening was a concrete, well-defined skill that could be readily incorporated into existing CBO 

95 
	 	 	 	 	 	 	 	

10,000	

9,000	

8,000	

7,000	

6,000	

5,000	

4,000	

3,000	

Number of Clients

2,000	

1,000	

0	 88% 87% 40% 34% 32% 32% 21% 26% 25% 30% 18% 25% 33% 38%

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3

Year	1 Year	2 Year	3 Year	4

Received at Least a Screening Received Any Other C2C Skill

 

 

workflows. Screening was the most commonly reported C2C skill delivered to clients in the first 

year of implementation based on quarterly report data (Figure 5.14). By the end of year 2, most 

CBOs began providing all four core C2C skills to clients. By year 4, screening accounted for 

about one-quarter of all C2C skills provided to CBO clients. 

Figure 5.14. Screenings and Other C2C Services Delivered to CBO Clients

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

Screening Approach over Time

As C2C implementation progressed, some CBOs modified their use of specific screening 

tools to better align with clients’ needs. In the early stages of implementation, depression was 

the most common condition assessed by CBO screening and remained so for the duration of 

implementation; however, by year 4 of C2C implementation, the types of screenings provided to 

CBO clients became increasingly diverse (Figure 5.15). For example, many CBOs (n = 8) began 

to incorporate screenings for PTSD in later stages of implementation. 

CBOs also evolved with respect to the ways in which screenings were integrated into regular 

CBO services and workflows. For example, as reported by CBO and MHP leaders, many CBO 

staff members reported resistance to administering screenings to clients due to concerns about 

interfering with client relationship-building. This was a particularly notable concern because the 

CBO staff-client trust was seen as foundational to the successful implementation of the C2C 

initiative. As reported by one CBO leader, 

Despite the fact that we’ve trained all the program staff in screening, nobody 

seems to want to try to administer one. They just feel maybe as awkward as I do 

sometimes. Where it’s like this piece of paper that kind of interjects itself into the 

discussion . . . they scream because it just seems like you are interrupting a really 

nice flow. And I’ve certainly felt that way too. 

96 
	 	

	 	

Number of Screenings

4,500

4,000

3,500

3,000

2,500

2,000

1,500

1,000

500

0

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3

Year	1 Year	2 Year	3 Year	4

Depression	Screenings Anxiety Screenings

PTSD Screenings Substance Use Screenings

Other Screenings Total	Participants Received Screening

Figure 5.15. Number of Screenings for Mental Health Symptoms, Overall and by Type

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

Similarly, a number of CBO leaders voiced concerns about the effect of integrating screening 

into intake procedures on underreporting of symptoms, particularly when screenings were offered 

prior to the formation of a strong bond with the CBO. As reported by one CBO leader, 

The screenings are not as positive in that people are not indicating that they have 

these mental health conditions and because we’re not referring as many people as 

I had targeted. So now we’re rescreening at certain milestones to see if that yields 

more openness. Our current practice is to do it only at intake and our thought is 

that they might not be as open. . . . But that has been a surprise to me because I 

thought at this point things would bump up significantly with referrals. 

These sentiments were also echoed by CBO staff members: 

I feel like initially many clients may not be willing as soon as you do the 

assessment but then a few weeks or months later they come back and say that 

they talked to the social worker and now they are ready. And I think that in our 

first initial meeting, they are not always forthcoming with what’s going on, so I 

think it is good that the screeners are done at that point with the family assessment 

then along the line they might want to seek the services. 

Consequently, some CBOs modified the ways in which screening materials were integrated 

into standard client flow within the CBO system by delaying implementation of screenings to a 

point at which clients had developed a stronger relationship with CBO staff members and/or by 

implementing less intensive, informal prescreening questions early on and following up later 

97 
 

 

 

 

 

with full screening batteries once an individual had sufficiently integrated into CBO services. As 

described by one CBO leader, 

I think at the beginning of year 2, we decided [to] use the pre-screen more as a 

conversational tool to better engage clients. I think that has helped us to increase 

the enrollment for Connections to Care in addition to also further engaging of 

staff and shifting from initial training to more coaching and supervision has also 

helped to get staff more comfortable with starting a conversation, asking certain 

questions that they normally are not used to asking. 

Rescreening

During year 4, the C2C Collaborative introduced a pilot rescreening initiative that required 

CBOs to systematically rescreen C2C clients for depression and anxiety. CBOs rolled out the 

pilot rescreening effort on a phased schedule, based on their readiness, and received customized 

technical assistance for this pilot. CBO and MHP leaders shared their perceptions of rescreening 

at the beginning stages of this pilot process during the final site visit. Additional details on the 

pilot rescreening process, including preliminary outcomes, are reported by the C2C Collaborative 

elsewhere. 

One CBO opted to incorporate rescreening protocols during the earlier stages of the 

implementation process (i.e., years 1 and 2) after feedback from MHPs and considerations 

related to typical client flow and retention in CBO programming. As described by this CBO 

leader of a job training and employment CBO, 

And then one of the things that [MHP] is working on in terms of coaching is 

with our [staff member who administers screenings]. We were advising her to 

administer screens when needed but now we want her to do a rescreen around 

that first paycheck. It’s a couple weeks after getting their jobs. That will bring 

with it new job feelings and if you’re in recovery, how will you navigate? And 

then maybe after 90 days of employment, do another screen. 

Those CBOs not already conducting rescreens expressed concerns about feasibility at the 

outset of the pilot for reasons such as differences in client flow through standard CBO 

programming and the nature and duration of client engagement in CBO services. In some 

instances, CBOs implemented structured multisession programming (e.g., training programs 

with classes spread across several weeks or months) that included systematic follow-up contacts, 

which would lend themselves to periodic rescreening. As reported by one CBO leader, 

My thinking right now is that we would continue to do our individual screenings, 

for the most part, their individual screenings in week 3, then in the final week of 

work wellness class, which would be maybe five weeks later, in our last class, I 

would just do the rescreen as a group. But I don’t know if that’s too soon. . . Then 

we are moving to a five-week model. We’re piloting that, so I will have to get 

creative about how we are going to make that work and capture those rescreens. 

For CBOs that had repeated, structured contacts with clients, rescreening was typically 

viewed as an opportunity to further engage with clients on mental health issues in a structured 

98 
 

 

 

way and to break down stigma surrounding mental health issues over the course of a client’s 

progression through CBO programming. One MHP reported, 

We call it [rescreening] emotional check-in week. We’ve done it more in a larger 

campaign kind of way...focused on alcohol and substance use. It generates nice 

conversations between CBO staff and young people and it sometimes rises to the 

level of pulling in counseling staff for a warm handoff kind of referral. I feel like 

we will keep doing it in that way because it’s an awareness building thing, and I 

think it does a lot to reduce stigma and create relationship opportunities between 

youth and staff. 

In other instances, client engagement with the CBO had a more limited timescale with 

minimal follow-up, which made the prospect of rescreening clients less feasible. For example, 

one CBO leader described that the sporadic and variable nature of client contacts with the CBOs 

made rescreening clients a challenge if not impossible in the context of their C2C programs: 

It’s extremely difficult to rescreen clients who—let’s just say they came for 

immigration assistance. We help them out, we refer them somewhere. . . . I don’t 

see it working for the majority of clients who commonly leave at times and then 

may come back a year later. But how can we do it on a timely basis that’s 

structured if it’s not one on one? Even for the women’s groups or young 

women’s groups, they are not the same. . . . Rescreening works for individual 

clients that a social worker has. But in like community-wise. . . . I don’t think 

that’s possible. 

Thus, factors such as CBO workflow and the extent to which clients engaged in repeated 

interactions with CBO staff members were seen as critical considerations in determining 

whether and how rescreening procedures could be effectively implemented in CBO settings. 

Psychoeducation

Starting in year 2, CBOs were asked to implement evidence-based PE. Some CBOs and 

MHP partners discussed some initial challenges related to identifying an evidence-based 

curriculum. For example, CBOs and MHPs that served non-English-speaking and/or racial/ 

ethnic minority clients shared that it was difficult to find evidence-based curriculum available in 

multiple languages (e.g., Spanish, Arabic) or culturally relevant to their clients. Further, some 

CBOs and MHPs felt that they did not have the time or resources to identify, assess fit, and 

select an appropriate curriculum. Although CBOs and MHPs largely understood the importance 

of finding a curriculum that met their needs, some felt burdened by the decision and preferred 

receiving a list of approved curricula to choose from. One CBO leader reported their 

experience: 

Since this “evidence-based requirement” came in, we became a little confused 

about how to go about finding the appropriate evidence-based curriculum that we 

could use . . . for psychoeducation we were, like, you were telling our little 

underbudgeted organization to find the evidence-based model? 

99 
 

 

 

 

 

 

 

 

 

One MHP reported similar feedback: 

In regard to the psychoeducation component of the model, it is a little too 

burdensome on the MHP. . . . To ask us to research it and find them and come up 

with them on our [own] I personally think it is too burdensome, I don’t have time 

to do that and then adapt them to our setting and implement it, I just don’t have 

the time to research models that may not even exist. 

In response to these concerns, the C2C Collaborative provided extensive guidance and 

technical assistance to assist CBOs and MHPs in identifying appropriate PE curricula. For 

example, the Collaborative engaged CBOs and MHPs in one-on-one technical assistance calls 

and provided an online resource bank to help guide decisionmaking. When deciding on the type 

of PE, many CBOs decided to use materials that met the specific needs of their clients instead of 

general mental health knowledge curricula. For example, all parent/caregiver-serving CBOs 

decided to use PE curricula that focused on parenting and parental wellness (e.g., multiple 

structured sessions focused on parenting skills, parent-child communication, and conflict 

resolution). Parents could also learn about self-care and mental well-being during these sessions. 

In contrast, the CBOs that served survivors and homeless and/or lesbian, gay, bisexual, 

transgender, and questioning (LGBTQ) youth delivered trauma-focused PE curriculum because 

of the significant level of trauma experienced by their clients. Finally, some job training and 

employment CBOs delivered stress reduction and coping-skills workshops in a group setting. 

Specific PE curricula implemented by CBOs are described in detail in Appendix A. 

Regardless of the type of PE, most CBOs and MHPs indicated that PE was successful at 

reducing stigma, normalizing mental health illnesses, enhancing coping skills, and increasing 

acceptance of mental health services among clients. After attending PE workshops, CBO staff 

shared that some clients seemed more comfortable discussing mental health issues with CBO staff, 

requested more PE materials, and came back to the CBO specifically to attend stress management 

workshops. The group discussion and learning format that many CBOs used appeared to be 

effective for many clients since hearing about other people’s experience provided some comfort 

and clients were able to share resources with each other. As one CBO leader reported, 

Because we’ve implemented it [psychoeducation] in working with the youth, 

working with seniors, and also working with just the general population. People feel 

a lot more comfortable when they are in a group setting. It’s less intimidating, and it 

normalizes the things they are dealing with. They also build connections with others. 

Other CBO leaders reported that they chose to provide PE with clients before screening 

because it was perceived as being useful in educating individuals about mental health problems, 

destigmatizing mental health services, and obtaining more accurate reporting from clients. As 

one CBO leader explained, 

We do psychoeducation before we do screening – we think that this helps destigmatize mental health services before they are screened. We heard that this 

was a barrier during our coaching and supervision meeting, so now we do 

psychoeducation before just to normalize what the clients will be receiving. 

100 
 

 

	 	 	 	 	 

	 	

Question 3: How Did CBOs and MHPs Alter Processes for Client Mental Health 

Referrals and Improve Referral Completion Rates?

All CBOs set up referral mechanisms during the early phases, many of which were altered 

over time to address lower-than-anticipated rates of referred clients completing a visit with an 

MHP. In this section, we describe the ways CBOs and MHPs revised their referral processes and 

referral tracking efforts to address the specific client barriers identified by staff and increase 

referrals during the Active Implementation and Sustainment phases. A more detailed description 

of CBO and MHP efforts to address referral barriers and modify approaches to care coordination 

throughout implementation follows in Chapter 6. 

Streamlining and Solidifying the Process

Through December of 2019, CBOs reported 5,019 client referrals to mental health treatment, 

4,551 to the MHPs, and 468 to external mental or behavioral health providers. These represent 

formal referrals that were accepted by CBO clients and documented in CBO reporting systems, 

and may not capture the full scope of mental health referrals that CBO staff offered to CBO 

clients. CBOs collectively averaged 335 referrals per quarter, with the peak referrals reaching 

more than 450 per quarter (Figure 5.16). Cumulatively, each CBO made an average of 

Figure 5.16. Client Accepted Referrals and Kept Referrals by Type

Accepted Referrals to External	Mental/Behavioral Health Provider

Kept External Provider	Referrals

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

101 
	 	

1,600 1,482

1,400 1,310

Number	of Referrals

1,200

74%

1,056

1,000

800 703

600

400

200

0

63% 83%

62%

Year	1 Year	2 Year	3 Year	4	(Q1–Q3)

At	Least	One Kept	Appointment Total Accepted Referrals

 

 

357 accepted referrals through December 2019 (median: 186; range: 55–1,241). At some 

CBOs, existing (pre-C2C) referral processes lacked specificity on how to determine who needed 

mental health treatment and who would make this determination. Over time, through C2C, these 

consistency issues were addressed with better protocols, more decision support from leadership, 

and technical assistance that helped CBOs simplify and clarify the referral process. For example, 

referral protocols were updated to more clearly tie referrals to screening results by specifying 

score ranges for screeners and explicit steps to take based on scores or providing more detail by 

assigning specific CBO staff to coordinate referrals and make sure counselors contact clients 

within 24 hours of a referral. 

CBOs reported on the number of referrals made as well as the number of referrals that 

resulted in a completed appointment based on attendance data provided by MHPs or external 

providers. The referral completion rate, defined as attending at least one clinic appointment 

following a referral, for both MHPs and external MHPs (i.e., clinical providers who did not 

formally partner with CBOs as part of C2C) generally increased over time particularly for MHPs 

(Figures 5.16 and 5.17). For year 3, nearly three quarters (74 percent) of referrals to MHPs 

resulted in a kept appointment, exceeding the target of 70 percent. For the first three quarters 

of year 4, 83 percent of referrals to MHPs resulted in a kept appointment. This increase was 

anticipated given the formal relationship established between CBOs and MHPs and the emphasis 

on streamlining referral pathways in the C2C model. 

Figure 5.17. Percentage of Kept Referrals to MHPs

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

102 
 

 

 

 

 

Notably, percentages of kept referrals ranged from 11 to 97 percent across individual CBOs. 

The number of clients referred also varied substantially by CBO type (Figure 5.18). Five job 

training and employment CBOs referred an average of 639 clients (median: 639; range: 114–1,241), 

and three youth development CBOs referred an average of 341 clients (range: 103–555). Three 

parent/caregiver-serving CBOs referred many fewer clients with an average of 59 clients (range: 

55–63), as did the four other CBOs type with an average of 159 clients (median: 160; range: 

112–203). 

Figure 5.18. Client Referrals by CBO Type

SOURCE: Data from quarterly CBO reports provided to RAND staff, March 2016 to December 2019.

This pattern of variation across CBO type aligns with findings for client screening discussed 

above and may be attributable to factors such as lower screening rates at parent/caregiver-serving 

CBOs, likely leading to lower referral rates. As one CBO staff member reported, 

The way it’s designed is that the families will get as much in-house as possible. 

So, referrals are difficult for our families to embrace. They would much rather do 

whatever needs to be done in-house . . . . And really the identified client would 

be the child. And so “I’m only here because of my child. So, if a child is safe and 

well in my perspective, then I don’t really understand why I would need to go out 

and get services.” So that’s something that we struggle with a little bit. 

In contrast, the comparatively high number of referrals observed among job training and 

employment CBOs may be attributable to a range of factors. First, several of these CBOs had 

one or more on-site MHP counselors, which allowed for efficient “warm handoffs” from CBO 

staff to MHP counselors. These organizations also tended to have clear, standardized screening 

and referral protocols and coaching/supervision practices, which may have led to more consistent 

103 
 

 

 

 

referral practices. In addition, these CBOs tended to engage clients in more structured and 

intensive programming that involved repeated direct contacts between clients and different CBO 

staff members (e.g., trainers/educators, case managers, etc.). The more frequent contact may 

have facilitated development of stronger staff-client relationships, increased opportunities for 

making referrals, and motivated clients to pursue treatment if needed. Moreover, clients engaged 

with the job training and employment CBOs may have pursued treatment for reasons related to 

obtaining employment, such as addressing mental health symptoms that could interfere with job 

performance, which may have increased openness to accepting referrals. 

Referral Tracking Procedures

As noted in Chapter 4, many CBOs used case conferences during supervision to monitor 

client progress and coach CBO staff. During the case conferences, CBO staff and leaders 

would meet with the MHP to share updates and discuss how to improve service delivery. As 

implementation progressed, CBOs followed up with efforts such as having CBO staff checking 

in with the MHP about client visits or using a data system to track client mental health progress. 

CBO leaders reported some examples: 

When [a client] is referred for services, they are contacted within two weeks by 

the therapist. After the first two weeks, the therapist is contacted once a month 

for three months and we have a folder for each center that says the screening date 

and we are doing a screening three months after the first to see if there is a 

change in their [clients’] emotional wellness. 

We track across the board: when did the screening take place, what modalities 

have been applied, outcomes of screening, is person being referred to MHP, who 

is MHP, date of appointment, did they [clients] show up to appointment. So, the 

site coordinator, with the aid of customer service staff, follows up with the MHP 

regardless of who it is on a regular basis to see if the client showed up. 

Not all CBOs started out with data systems to track and monitor client progress, but some 

made significant improvement in later years of implementation. For example, one CBO leader 

reported, 

So comparing ourselves with some of the other sites, seeing the infrastructure 

that was already developed . . . and also having a robust database and information 

structures, those were all things that we really had to scale up quite quickly and 

didn’t necessarily have enough support with our typical system to be able to do 

so because they were so unique to our organization. 

When I first came to [CBO] six years ago there was no . . . the data system set for 

tracking services to young people and youth attendance, participation, and 

outcomes was really just a . . . we were relying on government issued data 

systems through our government grant, but nothing proprietary that was really 

customizable to the work that we were doing. It’s been a long road but –and 

we’re not at the end– but we’ve been in a process of moving our system to 

Salesforce. 

104 
 

 

 

 

Other CBOs had existing data system capacity from the outset enabling them to set up a 

referral tracking system more readily, which helped to streamline and automate client referral 

and follow-up processes. As one CBO leader reported, 

We have also improved our system. Staff fill out a [form] in Salesforce that 

sends, it’s called logging a need, that’s our language. You log a need like the 

kid needs an eval or needs medication management. Then that pushes to the 

psychiatrist, the member navigator, and the director of counseling. Then they 

start coordinating to get the appointment and to get the young person there. I feel 

like all that stuff has improved. 

Question 4: How Did CBOs and MHPs Adapt to Accommodate and Sustain the 

C2C Program?

Because capacity building is such a critical component of ensuring long-term success of 

mental health task-shifting interventions (Belkin et al., 2011), CBOs and MHPs also addressed 

plans for sustainability of C2C programming within the CBO setting. In the interim evaluation 

report, we described the process of C2C integration into existing CBO programming; that is, 

how C2C operationally became a routine part of CBO and MHP staff roles and responsibilities 

that can be carried out independently of direct funding (Ayer et al., 2018). Within CBOs, the 

successful integration of C2C into their operations, programming, and staff approaches to clients 

depended on factors such as workflows, training, and staff and leadership buy-in. In this section, 

we highlight ways in which CBOs created an environment that encouraged the sustainable 

integration of C2C and provide evidence on its effectiveness. 

Commitment to Sustaining Components of C2C

Much of the first several years of C2C implementation were focused on building foundational 

processes, such as integrating screening and other C2C skills into existing CBO procedures, 

coaching and supervising CBO staff, and establishing referral protocols. However, CBOs also 

directed efforts toward ensuring that their staff understood the lasting nature of C2C programming 

and the need to adopt it holistically. For example, one CBO used a written agreement to encourage 

staff to commit to and maintain C2C programming: 

[The CBO coordinator] created a C2C agreement that all staff who are working 

with students . . . understood what participation in the [C2C program] and with the 

wellness team would look like, so they are aware of all their responsibilities. Then, 

individual directors on other teams are able to build in elements of that agreement 

into staff annual goals. I think that’s key to making it not just seem like something 

new that you have to do on top of your work, but core to your daily practice. 

Another CBO used incentives to encourage staff to practice and deliver the C2C skills, while 

CBOs began expanding C2C to other parts of the CBO once they saw the effect on both staff and 

clients. For example, one CBO leader reported, 

Our big change . . . is we are expanding [C2C services to other into CBO 

programs and client populations]. We originally started in the education and 

105 
 

 

career services population and then we noticed other departments, mainly the 

domestic violence department, and also the legal department which has 

immigration and also eviction prevention services, other staff members really 

wanting the support that C2C provides, in terms of case conceptualization, the 

vicarious trauma trainings, the continued coaching and supervision. Other staff in 

the agency wanting the same continued coaching. Also, the clients needing some 

form of extra support and some kind of counseling. It doesn’t feel right to clients 

leave the building without saying, we have C2C here. 

Quality Improvement

CBOs were on a spectrum with regard to their capabilities to implement QI processes, but 

over time all CBOs changed practices in an effort to strengthen implementation. For example, 

some CBOs were able to set up formal data-driven processes for measuring progress toward 

specific goals (e.g., number of clients receiving specific C2C services; number of referrals made 

to MHPs and external providers) early in the implementation process by leveraging existing data 

systems and integrating C2C metrics into QI practices that were already in place within the CBO. 

Early QI efforts were often tied to contract-specific targets and were used to help develop and 

refine service delivery processes in years 1 and 2. Because CBOs became more comfortable with 

the nuts and bolts of C2C implementation, they began revisiting the decisions about program 

structure and responsibilities, more closely monitoring their indicators of progress, and refining 

their approach to move the needle on those indicators. 

After CBOs had established stable training, supervision, and service delivery processes, 

many began to utilize program data and specific metrics to monitor service delivery and identify 

areas for improvement. All CBOs used program data to inform choices around service delivery, 

training, and supervision. For example, as described by one CBO lead, 

We did a quality improvement project where we tried to align the client’s 

schedule with meetings their job coaches, job developer, retention specialist on 

the day that the mental health provider was here. We have been attempting to 

change scheduling so that it’s more convenient for the individual when they 

come here on the same day to meet with the mental health provider and the 

community-based organization staff member that they came to see. 

In addition to the CQI Immersion Workshop, some CBOs participated in highly structured 

PDSA cycles supported by technical assistance and peer learning during year 3 of implementation. 

Through these PDSA cycles, teams not only tested changes to service delivery but also tracked 

specific metrics to understand the effect of the change. Overall, CBOs varied with respect to 

their specific areas of focus for QI depending on site-specific priorities and client flow. For 

example, one CBO reported, 

It’s through the CQI implementation that we did with RAND. We identified the 

need to implement support through one of our program tracks, through the hard 

skills portion. We designed a peer support group to occur with the clients to 

provide them with a little bit of additional support through their transition from 

106 
 

 

 

 

 

 

 

 

 

the soft skills portion to the hard skills portion and then getting them ready to 

move towards employment. 

Overall, these efforts helped to reinforce the importance of providing structured QI support 

to CBOs to help ensure iterative, purposeful improvements to specific components of sitespecific C2C programs, which can increase their effectiveness and longevity in the CBO 

setting. 

Financial Choices

There was a mix of perspectives on whether CBOs would be able to maintain C2C service 

delivery after current funding ends. In implementation year 4, some CBO leaders reported that 

they were considering obtaining external funds to help sustain ongoing implementation of 

C2C, and many shared that they were thinking carefully about what activities they could keep 

and which they could not without dedicated funding support. For example, one CBO leader 

described thoughts surrounding potential changes to staffing and roles that would allow C2C 

to continue: 

I think also as far as financial sustainability, we are also looking about bringing 

on an LMSW . . . this added capacity and added staff person . . . . Over time, that 

might not be sustainable because at some point, the grant will end, and we still 

want to continue to provide these services. So we are also revisiting the job 

description of the staff here, so meaning that if you bring on an LMSW, it means 

that person might also have to do benefits screenings and enrollments, which is 

currently now done by our research coordinators who are not LMSWs, or they 

might have to do intake. 

Another CBO leader wanted to change how they presented their CBO and obtained external 

funding to focus mainly on the integration of mental health into their services: 

We’ve been pushing to shift our approach to fundraising to be about the [C2C] 

model, which is about the integration of mental health services with every other 

programmatic track and the service of the whole youth and one stop shop. I hope 

that what that means is that the funding for the program as a whole that there will 

be more sustainability because it won’t be siloed. 

Another CBO leader acknowledged that funding constraints would likely prevent them from 

sustaining all elements of C2C moving forward, although they expressed interest in retaining 

some components of the C2C model. One C2C leader reported, 

So, we did a financial analysis of the components we would want to keep and 

what that would cost. We wouldn’t be able to keep everything. We probably 

would . . . keep the screenings and the psychoeducation. The relationship with 

our clinic we could easily continue. 

107 
 

 

Another CBO reported efforts to actively seek funding to support continuation of the C2C 

program into the future: 

We secured one substantial multi-year grant, that will help transition the 

operation of C2C, and we are working on a number of others . . . we don’t have it 

all nailed down yet, but we do have some sort of promising pieces in place. 

Overall, funding constraints for CBOs, which traditionally operate with very austere budgets 

to deliver a wide range of non-C2C services to clients, were seen as a major barrier to sustaining 

all elements of the C2C model after funding for the C2C initiative ends. Despite this, most CBOs 

expressed intentions to retain some elements of the C2C model moving forward, and many had 

begun to seek out alternative sources of funding to sustain the C2C program into the future. 

Discussion

In this chapter, we reviewed in detail information about the extent to which CBOs 

implemented C2C and highlighted differences in implementation by CBO type and overall 

changes over time. 

Training, Coaching, and Supervision

During the first 2 years of implementation, there was a ramp-up period during which 

CBO-MHP partnerships focused on preparatory activities, refining plans, and implementing 

initial C2C trainings. Nearly all CBOs reported early challenges with staff buy-in when initially 

implementing C2C trainings, which are consistent with findings from prior studies examining 

novel implementation of EBPs (Kimber, Barwick, and Fearing, 2012). However, ongoing 

efforts to maximize the relevancy and perceived value of C2C trainings helped to mitigate these 

challenges over time. For CBOs considering future adoption of the C2C model, early efforts to 

tailor trainings and maximize the extent to which the C2C program aligns with staff members 

needs and the organization’s mission (Bach-Mortensen, Lange, and Montgomery, 2018) may be 

useful in cultivating staff buy-in and ramping up program capacity. 

Following the initial rollout of C2C trainings, CBOs and MHPs began to implement routine 

coaching and supervision of CBO staff to support implementation of C2C skills in practice. 

Coaching and supervision ramped up considerably during implementation year 2, following 

clarifications in implementation guidance from the C2C Collaborative surrounding coaching and 

supervision requirements. In addition, increases in coaching and supervision provided by CBO 

staff in years 2 through 4 may reflect efforts to ensure in-house sustainability of C2C programming 

within CBOs. However, monitoring CBO staff members’ fidelity to implementing C2C skills was 

a persistent challenge for CBO and MHP supervisors. Technical assistance efforts surrounding 

fidelity monitoring (e.g., providing a fidelity checklist) were seen as helpful in moving toward a 

standardized approach to routinely assess fidelity during supervision sessions. But because 

organizations varied widely in terms of staffing structures and their approaches to implementing 

108 
 

 

C2C skills in the CBO setting, CBOs and MHPs approached fidelity monitoring in different 

ways, which led to challenges with collecting and reported fidelity metrics in a standardized way. 

In addition, supervisors experienced significant challenges (e.g., due to the ad hoc nature of skills 

delivery in the CBO setting; MHP scheduling limitations) with routinely observing CBO staff 

members delivering C2C skills to clients in action. Monitoring and ensuring fidelity associated 

with implementation of EBPs has been identified as a key implementation barrier in previous 

studies (Bach-Mortensen, Lange, and Montgomery, 2018), and efforts to clearly identify core 

intervention components and competencies for both practitioners and supervisors are recognized 

as critical factors in ensuring intervention fidelity (Gearing et al., 2011). Yet, ensuring fidelity 

can be particularly challenging when attempting to adapt intervention components to meet the 

unique needs of a specific setting or target population (e.g., Kegeles et al., 2015). For programs 

that are designed to be adapted to diverse CBO settings, like C2C, extensive technical assistance 

surrounding fidelity monitoring—combined with clear guidelines surrounding core program 

components and fidelity assessment practices—may be critical to ensure that CBOs have both 

the capacity and capability to implement program components (Bach-Mortensen, Lange, and 

Montgomery, 2018). As such, to ensure that the model components are implemented with 

fidelity, as intended, efforts to replicate C2C at other CBOs in the future would benefit from 

technical assistance and clear guidelines that specifically address how organizations should 

develop and tailor fidelity monitoring plans. 

Nearly all CBOs reported some degree of concern and/or pushback from their staff during 

initial implementation. Yet, CBOs and MHPs invested considerable time and effort into 

responding to their staff members’ confusion and concerns about training. The CBO-MHP 

partnerships involved CBO staff in these efforts to improve training content and processes to 

optimize trainings and follow-up support for CBO staff members. Overall, CBOs recognized the 

value of investing in training, coaching, and supervision, because it led to more staff buy-in and 

commitment to using C2C skills in practice. These factors are consistent with several key 

facilitators observed in prior studies examining implementation of EBPs in CBOs, such as 

organizational support/prioritization and flexibility surrounding program implementation and 

efforts to align program implementation with the mission of the organization (Bach-Mortensen, 

Lange, and Montgomery, 2018). For CBOs seeking to adopt the C2C model in the future, such 

efforts to incorporate staff feedback may be critical for expediting improvements to training 

processes and ensuring that the C2C model is optimally aligned with staff needs and workflow. 

C2C Service Provision to Clients

Routine reporting by CBOs captured the scope and scale of C2C in reaching New Yorkers. 

Overall, C2C reached more than 41,000 unique clients and more than 5,000 referrals to more 

intensive mental health treatment were made. Most C2C services provided to these clients were 

delivered by nonspecialist CBO staff members. Integrating and scaling up new evidence-based 

mental health practices in community settings often requires a multiphased approach to build 

109 
 

partnerships, infrastructure, and capacity (e.g., Grant, Simmons, and Davey, 2018; Saldana and 

Chamberlain, 2012); and the time required to launch new evidence-based services can vary 

widely, depending on a range of factors including existing organizational infrastructure, 

organizational readiness, and intervention complexity (Hurlburt et al., 2014). The numbers of 

clients served by C2C suggest clear inflection points and steeper increasing curves early on, 

followed by stabilization during the Active Implementation phase. The number of new clients 

who received CBO services rose dramatically during years 1 and 2, and the average number of 

clients served each quarter began to stabilize in year 2 and beyond. By December 2019, CBOs 

were screening well more than 1,600 clients per quarter and had screened a cumulative total of 

more than 22,000 clients. This suggests that, on average, the ramp-up of training and staff 

development in C2C skills—that is, building up organizational capacity to transition to Active 

Implementation of the C2C model (Aarons, Hurlburt, and Horwitz, 2011; Bach-Mortensen, 

Lange, and Montgomery, 2018)—took slightly over a year to achieve. 

CBOs devoted considerable resources toward integrating C2C skills into “typical” CBO 

workflows and iteratively refined these implementation approaches over the Active 

Implementation and Sustainment phases to best meet the needs of CBO clients and staff 

members. CBOs varied with respect to how they implemented the core C2C skills in practice, 

and some components carried unique challenges—and benefits—with respect to implementation 

in a given CBO setting. Careful consideration of workflow and the target client population may 

be critical for setting programmatic goals and assessing and refining implementation of C2C 

in different types of service settings. For example, parent/caregiver-serving CBOs showed 

systematically lower output with respect to service delivery and referrals compared with 

other CBO types. However, parent/caregiver-serving CBOs that remained in C2C through 

implementation year 4 were successful in achieving site-specific service delivery targets in 

implementation years 2 through 4 (e.g., number of total C2C clients served); this was attributable 

in part to careful consideration of CBO client/family volume and enrollment windows when 

determining annual service targets, which were lower than those for CBOs that enrolled new 

clients on a rolling basis. Similarly, client flow and engagement with CBO programming over 

time was seen as a critical consideration for whether and how efforts to rescreen C2C clients 

were implemented in the rescreening pilot. This is consistent with findings from past studies, 

which have identified client/community factors related to engagement and retention as key 

barriers to implementation of EBPs in nonclinical settings (Bach-Mortensen, Lange, and 

Montgomery, 2018). As a whole, these findings suggest that careful attention to client 

engagement in CBO services is a critical factor for determining how best to implement specific 

components of the C2C model in a given CBO setting. 

Mental Health Referrals and Referral Completion Rates

Improving linkages to clinical services for clients with needs for more intensive mental 

health treatment was a major focus of C2C. Through December 2019, CBOs referred more than 

110 
 

5,000 clients to mental health treatment. CBOs and MHPs experienced many challenges related 

to client referrals to more intensive mental health treatment, including but not limited to client 

stigma and practical barriers such as scheduling challenges and provider availability. CBOs and 

MHPs worked diligently to overcome these challenges and streamline referral pathways with 

referrals nearly doubling in year 2 and continuing to rise over subsequent years. Efforts to 

institute clear, standardized referral protocols and increasing capacity for MHPs to accept “warm 

handoff” referrals on-site appeared to be instrumental in driving these improvements. Completion 

rates for referrals to MHPs surpassed the C2C program’s overall target goal of 70 percent in 

year 3 and exceeded 80 percent in year 4. 

Given the uniqueness of the C2C task-shifting model and the range of CBO contexts for C2C 

implementation, it is difficult to find established benchmarks for assessing the extent to which 

C2C referral rates compare with rates observed in similar studies. However, there is an extensive 

literature examining attendance and referral completion rates for clinical mental health services 

in a range of settings. For example, multiple studies examining attendance rates (i.e., attending a 

first appointment following referral to clinical services) for outpatient behavioral services have 

found initial attendance rates below 60 percent (Gallucci, Swartz, and Hackerman, 2005; Orne 

and Boswell, 1991; Sparr, Moffitt, and Ward, 1993; Zivin et al., 2009). One study by Alvidrez 

and colleagues (2009) examined a small community sample of Black adults (n = 42) in San 

Francisco who had been referred to outpatient mental health treatment. This study tested the 

effect of a culturally informed PE intervention (i.e., a psychoeducational booklet about stigma 

based on experiences of Black mental health consumers) versus general information about 

mental health treatment on participants’ attitudes toward and subsequent entry into outpatient 

mental health treatment. Although the report had limited power to detect effects of the 

intervention (compared with a control group that received only general information), 

approximately 74 percent of participants attended treatment within 3 months, which was higher 

than rates observed among individuals receiving no materials in the previous year (59 percent). 

This is comparable to rates observed for C2C. 

Other approaches to streamlining referral pathways to mental health treatment, such as the 

integration of mental health treatment into primary care settings, have found referral completion 

rates of around 81 percent (Auxier et al., 2012). This is particularly notable because the average 

rates of referrals to MHPs that resulted in a kept appointment in years 3 and 4 of C2C 

implementation approached or exceeded those observed in studies of integrated primary care 

clinics, which typically provide on-site mental health specialty services and often accommodate 

same-day appointments (Auxier et al., 2012). These C2C rates of kept referrals are also in line 

with previous reports from “best practice” mental health service providers in NYC, which indicate 

that use of extensive outreach and client engagement practices can lead to 70 to 80 percent of 

clients attending an initially scheduled session (Smith et al., 2010). Although comparisons to 

C2C should be made with caution, given considerable differences in settings, model components, 

and populations, this suggests that CBOs were successful with respect to enhancing referral 

111 
 

 

 

 

pathways through the C2C initiative. In the context of high rates of unmet treatment need 

observed among the general NYC population, with less than 40 percent of adults with mental 

health problems accessing appropriate treatment, rates of kept referrals to MHPs in years 3 and 4 

represent a considerable achievement. 

Adaptations to Accommodate and Sustain the C2C Program

Overall, organizational leaders reported a strong commitment to ensuring that key changes 

brought about by C2C would persist within their organizations. Over the course of Active 

Implementation, the focus was on QI efforts because it was increasingly important in driving 

iterative improvements to specific components of site-specific C2C programs. These activities 

helped CBOs to assess and optimize the effectiveness of specific elements of C2C programming, 

which in turn helped CBOs to identify whether and how aspects of the C2C initiative could be 

sustained in the CBO setting in the future. As they moved into the Sustainment phase, some 

CBOs had begun to explore how specific elements of C2C could be financially supported 

following the end of the C2C contract period. 

Upfront investments of time, effort, and perhaps funding are to be expected to establish 

partner relationships, referral mechanisms, etc. (e.g., Aarons, Hurlburt, and Horwitz, 2011; 

Bach-Mortensen, Lange, and Montgomery, 2018; Saldana and Chamberlain, 2012). But it may 

be difficult to fund and sustain, for example, an on-site mental health counselor indefinitely, 

unless CBOs are already co-located with an MHP. Many of these program components were 

seen as having high value for staff and clients, and a number of CBO and MHP leaders expressed 

optimism that elements of C2C would have a lasting effect on their organizational cultures, 

practices, and ways in which they served their communities. The subsequent chapter (Chapter 6) 

provides a detailed description of how CBOs and MHPs underwent cultural changes through 

participating in the C2C initiative. 

Limitations

As described in Chapter 4, it is important to consider this evaluation’s limitations. These data 

come from a mix of quantitative and qualitative sources, including staff surveys and key informant 

interviews that were conducted over 1 year prior to this report. Readers should also note that the 

site visit and staff and client survey data are all based on self-report, and that the data come from 

a subsample of individuals who volunteered to participate in those portions of the evaluation. In 

addition, the implementation processes, challenges, and successes described as of implementation 

year 4 are based on information from the 14 CBOs that are currently participating in the program. 

Although some of the lessons learned may be generalizable to other organizations, some 

implementation experiences likely are unique, hinging on individual CBO characteristics, the 

current policy and funding environment, and the novelty of the C2C program. Other CBOs 

112 
 

 

 

 

seeking to replicate the C2C model in the future may encounter additional challenges (or successes) 

that are specific to their organizations and the communities they serve. 

Summary

During the first 2 years of implementation, there was a ramp-up period during which 

CBO-MHP partnerships focused on preparatory activities, refining plans, and implementing 

initial C2C trainings. Nearly all CBOs reported early challenges with staff buy-in when initially 

implementing C2C trainings, but ongoing efforts to maximize the relevancy and perceived value 

of C2C trainings helped to mitigate these challenges over time. Through December 2019, C2C 

reached more than 41,000 unique clients, and CBOs successfully integrated the core C2C 

skills—screening, MHFA, MI, and PE—into existing CBO programming. In addition, CBOs 

made more than 5,000 referrals to more intensive mental health treatment. As implementation 

progressed, CBOs and MHPs made efforts to address referral barriers and streamline and solidify 

referral pathways between CBOs and MHPs. By year 4, rates of kept referrals exceeded 80 percent. 

Toward the end of implementation year 4, CBO leaders were in the early stages of considering 

future plans for C2C program sustainability and described challenges—and opportunities—for 

carrying C2C program elements forward into 2020 and beyond. 

113 
 

 

 

 

References

Aarons, G. A., M. Hurlburt, and S. M. Horwitz, “Advancing a Conceptual Model of EvidenceBased Practice Implementation in Public Service Sectors,” Administration and Policy in 

Mental Health, Vol. 38, No. 1, 2011, pp. 4–23. 

Alvidrez, J., L. R. Snowden, S. M. Rao, and A. Boccellari, “Psychoeducation to Address Stigma 

in Black Adults Referred for Mental Health Treatment: A Randomized Pilot Study,” 

Community Mental Health Journal, Vol. 45, 2009, pp. 127–136. 

Auxier, A., C. Runyan, D. Mullin, T. Mendenhall, J. Young, and R. Kessler, “Behavioral 

Health Referrals and Treatment Initiation Rates in Integrated Primary Care: A Collaborative 

Care Research Network Study,” Translational Behavioral Medicine, Vol. 2, No. 3, 2012, 

pp. 337–344. 

Ayer, L., M. S. Dunbar, M. Martineau, C. Stevens, D. Schultz, W. Y. Chan, M. Abbott, R. Weir, 

H. H. Liu, D. Siconolfi, and V. L. Towe, Evaluation of the Connections to Care (C2C) 

Initiative: Interim Report, Santa Monica, Calif.: RAND Corporation, RR-2497-MFANYC, 

2018. As of February 4, 2020: 

https://www.rand.org/pubs/research_reports/RR2497.html 

Bach-Mortensen, A. M., B. C. Lange, and P. Montgomery, “Barriers and Facilitators to 

Implementing Evidence-Based Interventions Among Third Sector Organisations: A 

Systematic Review,” Implementation Science, Vol. 13, No. 1, 2018, p. 103. 

Belkin, G. S., J. Unutzer, R. C. Kessler, H. Verdeli, G. J. Raviola, K. Sachs, C. Oswald, and 

E. Eustache, “Scaling Up for the ‘Bottom Billion’: ‘5 x 5’ Implementation of Community 

Mental Health Care in Low-Income Regions,” Psychiatric Services, Vol. 62, No. 12, 2011, 

pp. 1494–1502. 

Gallucci, G., W. Swartz, and F. Hackerman, “Impact of the Wait for an Initial Appointment on 

the Rate of Kept Appointments at a Mental Health Center,” Psychiatric Services, Vol. 56, 

2005, pp. 344–346. 

Gearing, R. E., N. El-Bassel, A. Ghesquier, S. Baldwin, J. Gillies, and E. Ngeow, “Major 

Ingredients of Fidelity: A Review and Scientific Guide to Improving Quality of Intervention 

Research Implementation,” Clinical Psychology Review, Vol. 31, 2011, pp. 79–88. 

Grant, K. L., M. B. Simmons, and C. G. Davey, “Three Nontraditional Approaches to Improving 

the Capacity, Accessibility, and Quality of Mental Health Services: An Overview,” Psychiatric 

Services, Vol. 69, No. 5, 2018, pp. 508–516. 

Hurlburt, M., G. A. Aarons, D. Fettes, C. Willging, L. Gunderson, and M. J. Chaffin, “Interagency 

Collaborative Team Model for Capacity Building to Scale-Up Evidence-Based Practice,” 

Children and Youth Services Review, Vol. 39, 2014, pp. 160–168. 

114 
Kegeles, S. M., G. Rebchook, S. Tebbetts, and E. Arnold, “Facilitators and Barriers to Effective 

Scale-Up of an Evidence-Based Multilevel HIV Prevention Intervention,” Implementation 

Science, Vol. 10, 2015, p. 50. 

Kimber, M., M. Barwick, and G. Fearing, “Becoming an Evidence-Based Service Provider: Staff 

Perceptions and Experiences of Organizational Change,” Journal of Behavioral Health 

Services Research, Vol. 39, 2012, pp. 314–332. 

Orme, D. R., and D. Boswell, “The Pre-Intake Drop-Out at a Community Mental Health Center,” 

Community Mental Health Journal, Vol. 27, 1991, pp. 375–379. 

Saldana, L., and P. Chamberlain, “Supporting Implementation: The Role of Community 

Development Teams to Build Infrastructure,” American Journal of Community Psychology, 

Vol. 50, No. 4, 2012, pp. 334–346. 

Smith, T. E., J. Burgos, V. Dexter, J. Norcott, S. V. Pappas, E. Shuman, A. Appel, M. E. Harrison, 

I. R. Nossel, S. M. Essock, “Best Practices for Improving Engagement of Clients in Clinic 

Care,” Psychiatric Services, Vol. 61, 2010, pp. 343–345. 

Sparr, L. F., M. C. Moffitt, and M. F. Ward, “Missed Psychiatric Appointments: Who Returns 

and Who Stays Away,” American Journal of Psychiatry, Vol. 150, 1993, pp. 801–805. 

Zivin, K., P. N. Pfeiffer, R. J. McCammon, J. S. Kavanagh, H. Walters, D. E. Welsh, D. J. 

Difranco, M. M. Brown, and M. Valenstein, “‘No-Shows’: Who Fails to Follow Up with 

Initial Behavioral Health Treatment?” American Journal of Managed Care, Vol. 15, 2009, 

pp. 105–112. 

115 
 

 

 

6. How C2C Transformed Organizational Culture and Approaches 

to Client Care Coordination

Michael Stephan Dunbar, Dana Schultz, Wing Yi Chan, Michele Abbott, and Vivian L. Towe 

Key Findings

• CBOs underwent a cultural shift with respect to orientations toward client mental health, 

as evidenced by observations about how CBO staff and leadership adopted a “common 

language” with each other and clients who were more conscious about mental health and 

well-being in general. 

• Seventy-seven percent of CBO staff reported feeling like C2C training helped them 

address their clients’ mental health–related problems, and 64 percent felt satisfied with 

the continuing supervision they received to maintain their C2C skills. 

• Of the four C2C skills, MI was highlighted as the most effective tool to address clients’ 

mental health needs and provide better services to clients. 

• CBO staff perceived community mental health stigma to be relatively high over the 

course of implementation, and it was thought to be a primary barrier to clients accepting 

referrals to seek mental health services outside CBOs. 

• Other client barriers to accepting referrals included practical barriers (e.g., lacking the 

time, transportation, and childcare to see an off-site provider, inability to pay or lack of 

insurance), previous negative experience with mental health services, and general 

discomfort with the idea of receiving mental health services. 

• CBOs and MHPs worked collaboratively to develop solutions to minimize these client 

barriers, such as giving metro cards to clients or MHPs adding walk-in hours for C2C 

clients specifically. 

• Over time, CBOs and MHPs made adjustments to their organizational structures and 

policies to improve coordination surrounding clients’ mental health needs. 

• In terms of C2C diffusion, CBOs shared best practices with each other and many reported 

plans to extend the formal CBO-MHP relationship beyond the life of the C2C contract. 

116 
 

Introduction

In addition to increasing access to mental health care, C2C was also designed to transform 

the culture and approach to providing mental health services. Through engagement with C2C, 

CBOs, MHPs, and community members worked collaboratively to learn about mental health and 

change their attitudes toward mental health care. Together, CBO-MHP partnerships identified 

clients’ needs, selected and adapted interventions, and shared both learning and QI approaches. 

Staff from both types of organizations were also challenged through training and practice to 

confront or evolve their attitudes toward mental health issues (e.g., mental health stigma), the 

way mental health services are typically delivered, and who delivers them. The hope was that 

C2C’s efforts would begin influencing community well-being more broadly, first by the 

diffusion of knowledge about mental health by trained CBO lay staff, who are often highly 

engaged in the communities they serve, and then through the potential downstream effects of 

CBO clients who have benefited from C2C serving as supporters or resources for families and 

neighbors on where to get mental health support. 

In this report, we conceptualize the process of adopting and implementing C2C program 

strategies in CBO contexts in terms of four broad phases, based on the conceptual model outline 

by Aarons, Hurlburt, and Horwitz (2011) (see Chapter 4). The current chapter focuses on 

understanding C2C’s capacity to transform CBOs, MHPs, and the community’s attitudes, 

behaviors, and culture surrounding mental health during the Active Implementation and 

Sustainment phases of C2C implementation. 

First, we share findings on how C2C may have changed CBOs’ and MHPs’ approaches to 

providing care to clients with mental health needs. Next, we describe the organizational changes 

in referral and care coordination that CBOs and MHPs implemented to improve clients’ access to 

care. Finally, we explore the broader diffusion of C2C model in the city and beyond. 

Research Questions

For the primary research question related to how CBO staff improved knowledge of mental 

health and C2C skills, as well as attitudes, and behaviors about mental health issues and services, 

we explored the following specific research question using both quantitative and qualitative data 

sources: 

1. How did C2C affect the role of mental health in addressing CBO clients’ needs and 

achieving CBO goals? 

For the primary research questions related to the extent to which CBOs identified clients with 

mental health or substance use/misuse issues and key facilitators of and barriers to effective 

implementation of C2C program strategies (specifically linkages to clinical care), we explored 

the following specific questions using both quantitative and qualitative data sources: 

1. How did C2C address community- and individual-level barriers to using more intensive 

clinical mental health care? 

117 
 

 

 

 

 

 

 

2. How did C2C affect CBO and MHP approaches to client care coordination and 

referrals? 

3. How did C2C facilitate partnerships among CBOs within and outside the C2C network? 

Methods

This section summarizes the data sources and analysis approach for our examination of 

how C2C transformed CBOs, MHPs, and the community’s attitudes, behaviors, and culture 

surrounding mental health during the Active Implementation and Sustainment phases of the 

Aarons model of the implementation process. 

Data Sources

Data on the Active Implementation and Sustainment stages of C2C are drawn from the 

following data sources. For a more comprehensive description of data sources, please see 

Chapters 4 and 5, and Appendix B. 

Key Informant Interviews

As described in Chapter 4 of this report, RAND conducted key informant interviews with 

CBO leaders (e.g., CBO executive directors and C2C program directors), MHP leaders (e.g., 

clinical directors and counselors), CBO frontline staff (e.g., staff trained in and providing C2C 

skills to CBO clients), and CBO clients (e.g., people who received C2C skills) during the 

summers of 2017 to 2019 to collect qualitative data on program implementation (Table 4.2). 

The current chapter focuses on information collected during 2018 and 2019, which assessed 

key informant experiences during the Active Implementation and Sustainment phases of 

implementation. 

CBO Quarterly Reports

As described in Chapter 5, all CBOs provided aggregate quarterly data to the Mayor’s Fund 

on the clients they served and C2C services they delivered. Data used for this final report came 

from years 1 through 4 of program implementation (March 2016–July 2019). 

Annual Staff Survey

As described in Chapter 5, RAND conducted the three waves of annual staff surveys in the 

summers (May–September) of 2017 to 2019 to gain a broader view on program implementation 

from the perspective of CBO program staff who were trained in C2C skills and provided services 

to clients. Staff surveys covered topics such as staff experiences with C2C training and service 

delivery, confidence in their ability to administer C2C skills, knowledge about mental health 

issues, organizational climate, perceptions of the C2C program within the organization (e.g., 

organizational support for the C2C mission; communication), and staff use of specific resources 

118 
 

 

 

and strategies during client interactions. A comprehensive description of survey content is 

presented in Appendix B. 

C2C Model Summaries

As described in Chapter 4, the RAND research team developed narrative summaries and 

tables describing the specific components of each CBO’s C2C model (Appendix A). CBO 

leaders reviewed and edited the summaries for accuracy, as needed. 

Analysis and Synthesis of Findings

For these analyses, we integrated the quantitative and qualitative findings throughout the 

chapter. As in Chapter 5, we also categorized CBOs into the following broad categories or types 

for analytic purposes, based on the main types of services provided by the CBO: job training and 

employment (n = 5), youth development (n = 3), parent/caregiver-serving (i.e., organizations 

that serve parents and caregivers of young children) (n = 3),1 and other (n = 4) (Table 5.1). 

Similar to the approach described in Chapter 5 of this report, the research team then reviewed 

and integrated quantitative and qualitative data to identify key themes and cross-site variability 

with respect to shifts in organizational culture and approach to client care coordination among 

CBOs and MHPs. We then reviewed and integrated findings from CBO model summaries and 

other data sources to help contextualize site-specific considerations and variability across CBOs. 

Results

Although CBO-MHP partnerships faced numerous challenges throughout these phases of 

implementation, the primary actors, the CBOs and MHPs, appeared to change their orientation 

toward mental health needs and care through their engagement with C2C. Whether as a trainer, a 

service provider, or a recipient of C2C services, experiences with C2C seemingly changed the 

mindsets and behaviors of those involved, bringing about a culture shift in the way people 

conducted their day-to-day work over time and altering the way in which these organizations 

worked together to address clients’ needs. 

Question 1: How Did C2C Affect the Role of Mental Health in Addressing CBO Clients’ 

Needs and Achieving CBO Goals?

During C2C’s early implementation, many CBO staff expressed concerns about C2C 

implementation, specifically task-shifting, in part because C2C was seen as an effort that would 

require staff members to provide additional services and supports to clients in a different way 

1 One parent/caregiver-serving CBO was not able to implement the program and the funder terminated their 

participation in C2C at the end of implementation year 3. 

119 
	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	

	 	 	

	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	

Percent	 	 	 Agree or	Strongly Agree

0% 25% 50% 75% 100%

I feel like my training in C2C skills has improved my

ability to help my clients with mental	health

problems.

I am satisfied with the supervision I have received in

C2C skills.

I could	use additional training, coaching, and/or

supervision in C2C skills.

I am satisfied with the training I have received in

C2C skills. 72%

67%

64%

77%

 

(Ayer et al., 2018). Some of these concerns involved not knowing enough about mental health 

conditions and treatment options, feeling anxious about engaging with clients on a topic that was 

potentially stigmatizing and/or uncomfortable, or feeling overwhelmed by the idea of taking on 

additional responsibilities, including the time for training and delivering new skills in addition to 

existing tasks. CBO leaders, with the help of the MHP, other external resources, and feedback 

from staff about their concerns, were able to guide the CBOs toward a more comfortable and 

positive orientation to C2C, which eventually meant fuller integration of C2C into everyday 

practice. In this section, we describe how C2C moved CBOs toward an acknowledgment that 

addressing mental health was integral to the overall approach to providing CBO services and 

helping clients achieve their goals. 

Training, Coaching, and Supervision

Training, supervision, and coaching in the four core C2C mental health skills was a critical 

part of C2C that—over time—CBO staff came to see as valuable instead of burdensome. This 

was evidenced by staff expressing interest in receiving even more training and coaching and 

supervision (67 percent) and feeling like the training improved their ability to work with clients 

(77 percent) during year 4 (Figure 6.1). 

Figure 6.1. CBO Staff Satisfaction with C2C Training (Year 4)

SOURCE: Data from CBO staff survey, summer 2019.

As described in Chapter 5, over the course of implementation, most CBO staff members 

received training in more than one C2C mental health skill. By implementation year 4, nearly 

half of trainees (48 percent) reported being trained in all four core C2C skills. The intention of 

the training was to equip CBO staff with a more extensive “toolbox” that increased their capacity 

120 
 

 

and comfort level with handling a wide array of client mental health concerns. One CBO staff 

member described how they used different C2C skills to help clients: 

With every client, it can be a mixture. If the client comes in angry, screaming that 

“nobody is helping me,” I know [CBO’s psychoeducation program focused on 

trauma-informed care]. Or, they might come in hyperemotional and crying so I 

know MHFA. Or, they might come in and seem like everything is ok but when 

we are really speaking about it they talk about the symptoms they are feeling like 

not eating or not sleeping, then I know that I need to use psychoeducation. So, I 

think it is knowing what the client is presenting in this way and having this 

toolbox to meet the client where they are and knowing how to respond in every 

situation. 

C2C coaching and supervision also helped CBO staff learn to assess clients’ mental health 

concerns and make informed decisions about referring clients to receive additional support and 

services. CBO staff reported that consistent and supportive supervision sessions created an 

environment in which they felt comfortable asking questions and seeking advice. As such, CBO 

staff often used their regular supervision sessions to seek consultation from MHPs or their CBO 

supervisors. At some CBOs, MHP providers were on-site (n = 9 on-site full-time; n = 5 on-site 

part-time), which made consultation easier. For example, one CBO leader reported, 

And also, another thing that I’ve watched with staff is being able to know when 

they’re over their head. And being comfortable with that. Being comfortable 

enough to now say, because they have developed these skills, but they also have 

a support team. There’s someone else there on that site. 

As implementation progressed, MHPs described how working closely with CBO staff 

through C2C had helped to reconceptualize the role of lay staff in addressing client mental health 

needs through task-shifting. As described by one MHP leader, 

I think the fact that it’s so comprehensive and everyone is involved, and it’s 

really a team effort, is what makes it so successful. . . . I’ll tell you, in the 

beginning, I struggled with the task-shifting idea. Because I was like, I went to 

grad school and I got a license and I did all this training . . . and what do you 

mean – you want other people to do my job? But now, here we are a year later 

and it’s like, oh yeah, this really does work. So, I think that, that’s been 

terrifically successful. 

Staff Buy-In

Starting in year 2 of implementation, CBO staff buy-in began to improve considerably after 

CBO staff overcame concerns about delivering C2C services because of the stigma associated 

with mental health. Training in the four C2C skills appears to have helped CBO staff develop a 

better understanding about how mental health issues could affect client progress toward CBOspecific program goals. With more confidence in their C2C skills, instead of feeling frustrated or 

internalizing their clients’ lack of progress in their program, CBO staff used their skills to 

identify and address their clients’ mental health issues. CBO staff also began to apply C2C core 

skills in their regular work with their clients with a growing recognition that C2C skills “made 

121 
 

 

 

 

 

 

their job easier.” CBO leaders commented on how CBO staff began to recognize the need to 

provide mental health services to clients as their engagement in C2C deepened. Specifically, one 

CBO leader said, 

For me, I would say [the biggest change has been] task-shifting and allowing 

everyone to realize that you don’t have to have an LMSW or LCSW to provide 

crisis intervention or to help clients who are in need of mental health first aid, 

you understand? A lot of clients are not saying, “I want to see a therapist right 

now.” A lot of clients are saying, “Can you help me?” How can we use the tools 

that we have, the training that we have to help clients in immediate crisis?. . . 

That also goes towards supporting the staff. A lot of times staff just need to feel 

empowered and feel as though they are able to meet the clients’ needs. 

Another CBO leader described a “tipping point” for C2C as implementation entered its 

second year, such that as staff members became more familiar with C2C skills, they recognized 

its utility for working with clients. This leader stated, 

[In] year 2, the staff really evolved. It felt like a changing or tipping point, turning 

of the tides, where people were like, “Okay, is this going to be an additional task in 

year 1?” By year 2 it was like, “I see that there’s a need. This fits.” 

A CBO staff member described their own experience of learning how to better navigate 

client interactions more effectively as a result of C2C: 

It’s been for me, like a light. Like turning on a light in a dark room. It opened my 

eyes and allowed me to better understand the individuals I work with and 

understand when they really need assistance and what kind they might need. And 

not just for our clients we work with but for myself. It allows me when I’m 

frustrated dealing with a person, to evaluate how I handle myself and I know if 

it’s time to remove myself. 

Equally important was how CBO staff saw the positive effect of C2C on their clients. CBO 

staff relayed how they had observed improvements in clients’ successful use of CBO services 

after beginning to address their mental health needs. As described by one CBO leader, witnessing 

clients thrive after engaging in mental health treatment provided powerful examples of positive 

impacts: 

At this stage, I think the staff has bought into it. I think they’ve seen the positive 

impact in that we have several [clients] who’ve been engaged in mental health 

treatment for over a year and they’ve seen the positive impact it’s had on the 

[client] and the family. 

One CBO leader specifically discussed how their organization leveraged program data to 

help visualize the connection between engagement in C2C mental health–related services and 

client successes. Routinely sharing this data with CBO staff helped greatly in cultivating staff 

buy-in for C2C. As one CBO leader reported, 

To be honest, once we started data sharing, I think that’s when staff buy-in 

became really. . . it grew exponentially, because now they have this tangible 

122 
 

 

 

 

thing that they can see, ‘Okay, this is how this person is being affected, this is the 

result of this.’ I think that really gave them a sense of . . . That was necessary, 

and that they really needed to address these things [mental health concerns]. 

C2C also helped empower staff by providing a structured manner in which to directly 

confront and lessen the effect of some of the logistical, financial, and emotional barriers to 

receiving mental health care. These barriers were a persistent source of frustration for some 

CBO staff who felt like past attempts to link clients in need to mental health services were often 

stymied by “bigger picture” factors beyond their control. When CBO staff began to recognize the 

strength of C2C skills and how these skills helped meet the needs of their clients, they became 

more invested in C2C and more willing to deliver C2C. As reported by one staff member, 

C2C has been very helpful because especially anything that has to do with mental 

health, it’s very delicate, it’s very fragile. A lot of people are nervous to go 

through that journey on their own. And having a team to kind of support you and 

kind of walk you through it, have those conversations with you and kind of just 

check in. And really, that doesn’t happen anywhere else. If you go look for 

mental health services, there’s no one checking in and saying, how was it? And 

that’s what C2C has, I think, really been successful at. 

CBO Leadership Buy-In and Messaging

CBO leadership buy-in and active participation in C2C was an important facilitator to 

encouraging staff buy-in to the program. When CBO leaders and management teams actively 

participated in C2C training and meetings, CBO staff were more likely to participate and 

perceive participation as important. One MHP reported the involvement of one CBO director: 

[CBO director] also wants to find ways to implement MI at the executive level so 

we’re going to meet to talk about some of the challenges she’s having and how 

we can think of ideas for her to address certain issues. And I feel like it’s also 

great for staff to see that their director isn’t above it. She’s asking for this, she’s 

[in] meetings, she’s in the trainings, she does the role-playing with us. 

Sometimes if the frontline staff are not seeing the higher level of executive 

engaged, they are not going to necessarily buy-in, but since everybody is on the 

same level it just makes it easier for them to feel more motivated. 

In addition to active participation, messaging from CBO leadership was also important. 

CBO staff appeared to be more invested in C2C services when CBO leaders emphasized the 

significance of client mental health to the mission of the CBO. Some CBO leaders consistently 

communicated to their staff that supporting clients’ mental health was a critical component of 

their programming. When asked about facilitators to C2C implementation, one MHP said, 

They [CBO leadership] really think mental health is very important to fuse into 

the culture at [CBO] and they’ve seen the value of it. I think everybody is 

onboard with the importance of this project. That’s everyone from the 

management down to the staff. 

123 
 

 

 

 

In contrast, when CBO leadership was not supportive or consistent in their approach to C2C, 

staff felt confused and C2C implementation suffered. One MHP reported their experience 

working with CBO leaders who were not supportive of C2C: 

One of the [organization leaders] never really seemed to understand C2C, nor 

committed to it . . . Would state that she was supportive of it, but never came to 

any of the group coaching sessions, left psychoeducation training early, often 

interacted with clients [in a manner] that is the complete opposite of MI, doesn’t 

encourage supervisors to move the initiative forward. One staff said to me, “All 

this stuff sounds good, but you know this is a joke because you say this when we 

come to meet, but then when I leave here, when [this individual] calls me and she 

doesn’t care about what you suggest or your recommendations, so what am I 

supposed to do?” I think that says everything. 

Motivational Interviewing

Across CBOs, CBO staff embraced MI as a particularly useful tool in working with clients, 

suggesting that its use will continue into the future. Of the four C2C skills, CBO staff highlighted 

MI as the most effective tool to address client mental health needs and provide better services. In 

particular, the “MI spirit” and nonjudgmental, client-centered approach resonated deeply with 

many CBO staff members who consistently shared feedback about the positive effect of MI on 

their interactions with clients throughout all 4 years of implementation. As described by one 

CBO staff member, 

It started to become second nature for me to use MI, for me to think about mental 

health when we are sitting in a circle talking about safety and I think it’s definitely 

changed the way the organization responds to crisis and also the way that we deal 

with clients one-on-one. 

Several CBO leaders and staff members described how using nonjudgmental language and 

setting realistic expectations supported their work with clients. Further, MI was seen as 

particularly helpful for providing the primary programmatic services and facilitating client 

progress toward specific goals. One CBO leader reported how staff members applied MI skills 

while working with clients to resolve their housing issues: 

[S]taff worked with a family with very unrealistic expectations about what kind 

of housing they should be given – this was a hard family to work with but staff 

really applied what they learned at C2C training to talk things through with the 

family and I am not sure the family would have taken the apartment they did had 

the staff not been trained and taught to talk to the family the way they did. I think 

mostly the MI skills played a key role in how our staff could use the language 

and the approach that took the client through this process rather than them just 

becoming frustrated and not wanting to help the family if the family wasn’t 

willing to compromise. 

Over time, and through ongoing training, coaching, and supervision, MI provided an 

organizing framework or “shared language” through which CBO staff members could view 

client interactions and discuss client progress within CBO programming more generally. All 

124 
 

 

 

 

 

14 CBOs that continued with the C2C program through year 4 described MI as being integrated 

and infused into nearly all interactions with clients. For example, one CBO leader reported, 

I’d say with MI, we are developing a shared language around and through our 

programming. It comes up in common conversations now, it is becoming a part 

of us—people that have gone through the training—and something we often refer 

to. People go through continuous coaching on MI, and that is something we’ve 

been able to establish better, so now there is more depth and it is better established. 

Cultural Shift in Mental Health Approach and Service Delivery

As implementation progressed, multiple CBO leaders and staff identified changes to the way 

in which the CBO conceptualized the role of mental health in their overall programmatic service 

delivery. For example, CBO staff began to use a different language that was much more conscious 

about mental health and well-being in general. One CBO leader described the importance of 

MHFA trainings in driving a more standardized approach for how staff addressed client mental 

health issues: 

[MHFA] is a way of being. Staff implements those steps a lot. They were already 

doing it, but it was enhanced and now they have the language for it. 

Another CBO leader echoed the way in which C2C more generally had helped to establish 

a common language for how staff and clients talked about mental health within the CBO 

setting, stating, 

I think we kind of like had to change our language...it’s almost like learning a 

new language, like trying to make that shift happen consistently, when I’m 

working especially around a young person talking about goals and change. I’ve 

tried to make it almost ingrained into what I do and how I speak. 

These changes went beyond how CBO staff spoke to clients to encompass interactions across 

all levels of the organization. As described by another CBO leader, 

I think [C2C has] been transformative and has really pushed the envelope in 

regards to us understanding where our clients are at and what service they need, 

what support they need, and then what lens are we actually looking at that, right? 

It’s kind of like really integrated itself, not just in client engagement but in the 

way also that we are engaging with each other, top down, even in supervision 

meetings. 

A different CBO leader expanded on this concept, noting how MI training specifically 

resulted in more global changes in approaches to staff management within the CBO setting: 

I think that training the staff on MI has been extremely rewarding for them not 

only on how they address their clients . . . but also in the way that we manage 

people. Like I think about managing folks in a compassionate way using MI so 

then our managers are also using MI structures to manage other frontline staff. 

So it’s not just a way how we handle clients, it’s a way that we handle ourselves 

internally with each other and like the strategies of supervising and managing. 

Cultural changes in the CBO environment related to mental health also had apparent benefits 

for CBO staff. For example, the opportunity to deliver C2C skills helped some CBO staff 

125 
 

 

 

unlearn prior negative perceptions of mental health and apply their new knowledge to their own 

mental health status. CBO leaders and staff discussed trauma exposure frequently and found that 

C2C training helped them to understand their own trauma history (in addition to their clients’) 

and acknowledge the consequences of experiencing trauma. One MHP leader described their 

perception of how training had a positive effect on CBO staff: 

With the trainings there is a change in how they learn to see clients’ trauma, 

or their own trauma histories . . . . The training challenges their own trauma 

history . . . staff have approached trainers and told them that the training has had 

a profound effect. 

C2C also empowered staff members with critical knowledge and coping skills to more 

effectively navigate challenges and stressors associated with their day-to-day work with clients. 

For example, staff learned skills to better serve clients with mental health issues without feeling 

overwhelmed. One staff reported the benefits of C2C training: 

When I did the vicarious trauma training, it was very eye opening because I had 

worked at another domestic violence shelter before this for five years but I didn’t 

know what vicarious trauma was and I didn’t know I was at the verge of a 

burnout and so recognizing all those things and how that impacted my work life 

and my home life, and trying to separate it and take care of myself, that way I 

don’t impact my personal life. 

Many CBOs also placed an emphasis on self-care during supervision. In addition to discussing 

client progress, CBO supervisors asked staff to reflect on their work and the effect of their work 

on their mental health. Some implemented self-care activities such as meditation and yoga as a 

result of the C2C initiative. 

Organizational Support to Address Client Mental Health Needs

When asked about organizational support for addressing client mental health needs, CBO 

staff overwhelmingly endorsed that they had access to the educational or resource materials that 

they or their client needed (Figure 6.2). Moreover, this was evident from the early stages of 

implementation (year 2) and remained high throughout implementation. For example, more than 

70 percent of CBO staff respondents agreed with all statements about organizational support for 

addressing client mental health during each wave of the survey. 

Reorientation of Client Perspectives on Mental Health and Well-Being

C2C’s role in introducing clients to information about mental health and well-being was 

perceived as critical to clients’ willingness to engage with C2C. CBO leaders viewed the 

integration of C2C into CBO workflows as a way to help destigmatize mental health by 

facilitating more open and constructive conversations about mental health across staff and 

clients. Some C2C components, such as PE, were seen as particularly helpful in breaking down 

mental health stigma among clients. For example, curricula that oriented clients to symptoms of 

126 
	 	 	 	 	 	 	 	

	 	 	

	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	

	 	

	 	 	 	 	

	 	

Percent	 	 	 agree or	strongly agree

0% 25% 50% 75% 100%

My organization encouragesme to	ask	clients about

behavioral health difficulties.

I can	identify	the places or people where I should	

refer clients experiencing behavioral	health

difficulties.

My organization has access to	an adequate number

of resources or	people to whom I could refer	clients

experiencing behavioral	health difficulties.

I have easy access to	the educational or resource

materialsI need	to learn	about	behavioral health.

Year	2 Year	3 Year	4

 

 

Figure 6.2. Organizational Support for Addressing Client Mental Health

SOURCE: Data from CBO staff survey, summer 2017 (year 2), 2018 (year 3), and 2019 (year 4).

common mental health conditions such as depression, trauma, and anxiety were perceived to 

help normalize mental health symptoms and provide clients—and staff—with a common 

language to discuss mental health and well-being. One CBO leader reported observations of 

these changes, 

I feel like part of the mission or the goal was to destigmatize and normalize 

and I’m starting to feel like there’s more normalization, at least within the 

population here at [CBO], which hopefully it will continue with our mission of 

destigmatizing mental health and the need for services. 

Indeed, CBO clients reported having a new understanding of mental health and mental health 

services, which they attributed to C2C, especially PE. In learning about mental health symptoms 

and treatment for mental health problems, some clients described feeling less isolated with their 

mental health issues after learning that others shared similar challenges. Some became aware of 

their own symptoms. One client reported, 

Sometimes you don’t even know [you are] dealing with suffering from anxieties. 

I didn’t know I was suffering from anxieties either until I took this [stress 

reduction course]...Check! Check! Check! I actually check, checked more. I 

know I’m worried, but I didn’t associate that as being anxious and anxieties and 

all that. 

Providing information about mental health was also seen as helpful in increasing some 

clients’ willingness to seek treatment. For example, some CBO clients agreed to seek help 

because they wanted to have better relationships with their family and provide better care to their 

children. Other CBO clients shared that they learned how to communicate with their children 

127 
 

 

 

 

 

more effectively from their counselors. CBO clients also shared how they went to parenting 

classes delivered as part of C2C or counseling with their children and/or their spouse. When 

asked about their motivation to start counseling, one client reported, 

I refused first because of my situation, I thought, “I don’t want to discuss it, I 

want to leave it behind and not think about it” but my daughter she was having 

bad dreams and couldn’t sleep she was having nightmares so I went with her and 

it is started to help her and we all go together now because it’s a family thing but 

we each have our own therapist. 

Delivering PE in a nonjudgmental, client-affirming style—consistent with MI—appeared to 

make PE content resonate more deeply with clients. As described by one client, who participated 

in a group PE program that included a module on signs and symptoms of depression, 

She also lets you know that it’s okay not to be embarrassed about what you’re 

suffering from. Because a lot of people feel like they don’t need to get help 

because people will call them crazy. That’s major because a lot of times you can 

get to a point where you can harm yourself because you’re stuck. A lot of people 

don’t understand what depression can cause. They don’t recognize the signs and 

a lot of people don’t understand why they suffer from depression like that. It’s 

really good because she breaks everything down and takes the time to explain it 

to you. She really takes the time out to let the class know it’s okay. 

CBO clients reported on positive experiences with the C2C program and specifically 

highlighted the critical role that CBO staff members played in shaping favorable impressions 

toward C2C. As reported by one client, 

They met me down where I was . . . and helped me build my confidence . . . deal 

mentally with my emotions and what was going on at home and in my personal 

life. And then they referred me to counseling, so I feel like this is great—and the 

people are awesome! 

Question 2: How Did C2C Address Community- and Individual-Level Barriers to Using 

More Intensive Clinical Mental Health Care?

Despite CBO and MHP efforts to reduce barriers and facilitate access to mental health 

services, some challenges persisted throughout implementation. In this section, we describe how 

C2C worked to address different types of barriers to engaging with more intensive mental health 

treatment. As described in Chapter 7, the impact study examined the effectiveness of C2C in 

reducing individual-level barriers to mental health care. 

Community Mental Health Stigma

Although C2C helped to destigmatize mental health among CBO staff and clients, staff 

perceived mental health stigma to be relatively high in the community over the course of 

implementation and an ongoing barrier to receiving mental health services outside of CBOs. 

128 
 

CBO staff members provided ratings of community stigma surrounding mental health on the 

Social Distance Scale (Link, 1989) (see Appendix B) in all three staff surveys.2 For this scale, 

items were summed and averaged to generate a total score with scores (range: 1–6) with higher 

scores indicating greater perceived community acceptance of individuals with mental health 

conditions. The mean score in year 4 of implementation was significantly higher than year 2 

(p<.001), suggesting a slight improvement in perceptions of community attitudes over time. 

However, direct comparisons across the survey waves are not possible because the same 

respondents were not followed over time; any differences across waves should be interpreted 

with caution. 

In addition, perceptions of community stigma remained relatively high throughout 

implementation (Figure 6.3). Approximately half of CBO staff respondents in years 3 and 4 

agreed with the statement “Most people think less of someone with a history of behavioral health 

problems,” suggesting that negative views on mental health persisted in the community. 

However, there was some indication of progress in community perceptions, with the percentage 

of respondents who agreed with the statement “Most people would accept someone with a history 

of behavioral health problems as a teacher of young children in a public school” increasing from 

16 percent in year 2 to around 80 percent in years 3 and 4. One possible driver of item-level 

differences over time is that the year 2 survey occurred at a time when many CBOs had just 

begun to implement C2C skills trainings. By implementation 3, CBOs had ramped up both 

training and implementation of all four C2C mental health skills, most notably PE, which may 

have had an effect on staff members’ attitudes. As noted above, any difference over time should 

be interpreted with caution because the same respondents were not followed over time. 

As shared by CBO staff on the wave 3 survey (fielded summer 2019), community stigma 

surrounding mental health was viewed as a significant challenge with respect to referrals to more 

intensive mental health treatment. Nearly two-thirds (65 percent) of CBO staff believed that 

clients declined referrals because of concerns about what their friends, family, and community 

would think of them if they found out about their use of mental health treatment (Figure 6.4). 

Between 33 and 56 percent of staff endorsed other barriers, including high cost, lack of insurance 

coverage, logistical issues (e.g., transportation, scheduling appointments, etc.), and mistrust of 

the mental health treatment system. 

2 Twelve stigma survey questions were adapted from the devaluation-discrimination measures to assess the extent to 

which respondents believe that most people will devalue or discriminate against a person with a history of mental 

health treatment. Items were answered with a 6-point Likert scale from 1 (strongly disagree) to 6 (strongly agree). 

Six items were reverse scored. 

129 
 

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	

	 	 	 	 	 	

	 	 	 	 	 	 	 	 	

	 	

	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	

	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	

	 	 	 	

	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	

	 	 	 	 	 	 	 	 	

	 	 	 	

	 	 	 	 	 	 	 	 	

	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	

%	 	 	 	 Agree or Strongly Agree

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Most people would willingly acceptsomeone with a 36%

62% history	of behavioral health problems as a	close friend. 56%

Most people believe that someone with a history of 43%

behavioral health problems	is	just as	intelligent as	the 57%

average	person. 55%

Most people believe that someone with a history of 28%

behavioral health problems	is	just as	trustworthy as	the 73%

average	person. 64%

Most people would acceptsomeone with a history of 16%

behavioral health problems as a	teacher of young children 82%

in a public	school. 78%

Most people feel that entering	a treatmentfacility for 52%

behavioral health problemsis a	sign of personal failure 59%

(R). 62%

Most people would not hire someone with a history of 47%

behavioral health problemsto take care of their children, 45%

even if they had been in recovery (R). 44%

Most people think less of someone with a history of

behavioral health problems(R).

40%

51%

53%

Most employers will	hire someone with a history of

behavioral health problems	if he or she is	qualified for the

job.

41%

55%

64%

Most employers will	pass over the application of

someone with a history of behavioral health problemsin

favor of another applicant(R).

56%

45%

47%

Most people in	my community would	treatsomeone with	

a history of behavioral health problemsjust	as they would

treat anyone.

28%

64%

61%

Most young people would	be reluctant to date someone

with a history of behavioral	health problems(R).

56%

60%

53%

Once they know a	person has a	history of behavioral

health problems, most people will take his	or her opinions	

less	seriously (R).

Year	2 Year	3 Year	4

43%

54%

56%

 

 

Figure 6.3. Staff Perceptions of Community Mental Health Stigma by Wave

SOURCE: Data from CBO staff survey, summer 2017 (wave 1), 2018 (wave 2), and 2019 (wave 3).

NOTE: R = item is reverse scored during measure construction.

Community mental health stigma was also viewed as particularly pervasive and challenging 

for clients who came from specific communities and/or subgroups, such as first-generation 

immigrants. For example, as described by one CBO staff member, 

That’s the other layer, which is the stigma within our communities . . . We’re 

dealing with people who are just directly coming here. So, I think that’s just 

layers and layers of complex cases we have, so I think the referral process is 

difficult; I think that’s the biggest barrier – for folks to come and actually see a 

social worker. People are just not really used to mental health services. That’s 

just not in their vocab. 

130 
	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	

	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	

	 	

	 	 	 	 	 	 	 	 	

	 	 	 	 	

	 	 	 	 	 	

	 	 	 	 	 	 	

	 	 	 	

	 	 	 	 	

	 	 	 	 	 	 	

	 	 	 	 	 	

	 	 	 	 	

	 	 	 	 	 	 	 	

Which	of	these	 	 	 	 	 	 	 	 statements explain	why your clients do not accept

behavioral	health referrals?

Percent	Endorsing

0% 25% 50% 75% 100%

Concerned that getting behavioral health services	might

cause friends, family, or community to have a	negative... 65%

Cannot	afford	the	cost 56%

Concerned that	they might	be committed to a psychiatric 54% hospital or might have to take medicine

Too difficultto arrange for childcare 49%

Too	difficultto	get time off work 49%

Health insurance does not	cover any behavioral health 46% services

Concerned that getting behavioral health services	might 45% have a negative effect on their job

Too difficultto find a behavioral	health provider who is 44% culturally/linguistically competent

Do not feel comfortable leaving the CBO to get 43% behavioral health services

Too difficultto schedule an	appointmentfor behavioral 41% health services

Concerned that the information they give the counselor 41% might not be kept confidential

Health insurance does not pay enough for behavioral 39% health services

Too	difficultto	arrange transportation 33%

Other	barrier 12%

 

 

Figure 6.4. Staff Perception of Why Clients Decline Mental Health Referrals

SOURCE: Data from CBO staff survey, summer 2019.

This highlights the importance of widespread novel, culturally informed efforts to address 

mental health stigma more broadly within communities, particularly those in which mental 

health problems and/or stigma are subject to high levels of social disapproval, in helping to 

reduce barriers to mental health service utilization and maximize the potential effects of efforts 

like C2C. However, preexisting mental health stigma was not seen as a significant barrier to 

service use for all client populations. As described by some CBO staff members, there was a 

131 
 

 

sense that some clients, specifically young people in some CBO settings, were more open to 

discussing mental health concerns than C2C program leaders had anticipated. One CBO leader 

reported, 

Young adults who are coming in who don’t really have that stigma attached to it, 

and they’re open to these kinds of services, which surprised us when they started 

to say, ‘Yeah, sure, I could use someone to help, because I know I have some 

anger issues.’ So, I think that I’m just going to say it’s a millennial thing, maybe, 

but they’re starting to become a little bit more aware of it, so we can see how on 

that side, when the stigma doesn’t exist, kind of coming in they’re more open to 

those resources. 

Client Barriers to Completing Referrals

Both CBOs and their clients reported ongoing barriers related to following through with 

referrals to mental health services. These CBO-reported barriers to scheduling and attending an 

appointment with the MHP after the referral included practical barriers (e.g., lacking the time, 

transportation, and childcare to see an off-site provider, inability to pay or lack of insurance), 

previous negative experience with mental health services, and general discomfort with and 

insufficient education about mental health care. When asked about their experiences with 

counseling at MHPs, clients also discussed a number of practical barriers such as lack of 

childcare and scheduling conflicts. These barriers made it difficult to start and to continue 

counseling. As one client reported, 

I actually was supposed to have mental health services as well. I was supposed to 

have them last year, but I got a job and it didn’t work out. And I tried to get in the 

beginning of this year, same thing, I wanted something that was after work. And 

it seemed the person that I had felt like it was an imposition, like I was keeping 

her from living her life. 

Another client reported on the difficulties associated with seeking treatment as a parent 

with limited childcare options: 

I take myself and my kids and when I go sometimes it’s hard for me to talk 

because I’m with my kids and I’m too busy parenting to have a full in-depth 

conversation with them. It is hard to take care of my kids while I am trying to 

have the session. 

Some clients described a hesitation to receive counseling services because of previous 

negative experiences. For example, a client might not trust MHP staff based on issues with 

previous counselors violating confidentiality. Further, CBO and MHP staff sometimes reinforced 

the clients’ negative perception of mental health care. For example, some clients shared that 

scheduling appointments was difficult because—in their past experiences—other (non-C2C) 

MHPs were nonresponsive, and that during appointments MHPs were not considerate or 

compassionate about the multiple demands in their life. However, some clients viewed the 

132 
services at CBOs and MHPs as different, describing their trust of CBO and MHP staff and 

comfort level receiving care. A client described this experience: 

I had therapy before, but it was from the government. And the majority of the 

staff—from the receptionist to all of them, they are cold people, they don’t 

even treat you like a person. I prefer to come here because they never treat you 

that way. 

In addition, CBO staff found that some clients appeared to be uncomfortable with counseling 

because of insufficient knowledge about mental illnesses and mental health care. For example, 

clients from some cultural backgrounds had a different understanding about mental health in 

which they attributed mental illnesses to personal weaknesses or religious causes, which further 

reduced clients’ willingness or comfort with seeking mental health treatment (e.g., from a 

counselor or psychiatrist). Further, CBOs shared that some clients did not think that mental 

health services would be effective for them because of limited knowledge about mental health 

care. CBO staff also reported that clients were intimidated by mental health services because 

they were not familiar with psychotherapy and were worried that MHPs would require them to 

take medication. 

CBOs and MHPs often worked collaboratively to develop solutions to minimize some of the 

logistical and other barriers. For example, CBOs gave metro cards to clients who needed to use 

public transportation to get to the MHP. Some CBOs secured additional funding to help pay 

for clients’ mental health services, and some MHPs changed the structure of their services to 

meet the needs of the clients. For example, some MHPs added walk-in hours for C2C clients 

specifically. 

Question 3: How Did C2C Affect CBO and MHP Approaches to Client Care 

Coordination and Referrals?

Care coordination is a practice in which the specific needs of a client and the services 

planned for and received by the client are communicated across multiple service providers 

(Berry et al., 2013; Brophy et al., 2014). Each participating provider is typically assigned a 

specific role in a client’s care and asked to report on their activities to the other providers on a 

routine basis. In this section, we describe the ways that CBOs and MHPs implemented care 

coordination for C2C clients, challenges that arose, and whether and how they were addressed. 

We also discuss the ways CBOs and MHPs changed their referral processes to increase clients’ 

access to mental health services. 

On-Site Counselors and Warm Handoffs

One of the most significant changes adopted by some CBO-MHP partnerships to increase 

referral completion rates was bringing MHP counselors on-site at CBOs. Partnerships also 

increased the frequency of on-site presence of MHP and CBO staff counselors (i.e., individuals 

with formal mental health training) over time after observing high need among clients and 

attrition when the counselor was not immediately available on referral. Some CBOs had to change 

133 
	 	 	 	 	 	 	 	

	 	 	

	 	 	 	 	 	 	

	 	 	 	 	 	

	 	 	 	 	 	 	

	 	 	 	 	 	

	 	 	

Percent	 	 	 Agree or	Strongly Agree

0% 25% 50% 75% 100%

My clients do	or would feel comfortable accepting a	

referral	to a behavioral	health provider outside my

organization (i.e., external referral).

My clients do	or would feel comfortable accepting a	

referral	to a behavioral	health provider inside my

organization.

My clients do	or would feel comfortable talking about

their behavioral	health issues with me.

50%

72%

73%

 

 

 

 

MHP partners because their original partner was unable to provide any or enough on-site presence 

at the CBO. 

By implementation year 4, all 14 CBOs reported having a trained MHP on-site (or 

immediately adjacent) at the CBO either part-time (n = 5) or full-time (n = 9; note: one CBO 

had an MHP located immediately adjacent to the CBO, and one was counted as full-time). One 

CBO experienced persistent challenges with providing on-site counseling due to a long-standing 

position vacancy but was able to provide on-site counseling full-time beginning in year 4. CBO 

leaders shared that C2C funding increased capacity to provide co-located counseling support 

within the CBO, which was seen as extremely helpful in terms of addressing practical barriers to 

getting clients into more intensive mental health treatment. As described by one CBO leader, 

We never had a mental health provider onsite or a social worker. That’s never been 

in a budget. . . . So, this is something that’s radical. And the fact that we can now 

start offering short-term therapy to participants who mention interests in some type 

of mental health assistance and weren’t quite getting to the clinic. We, kind of, 

broke down that barrier and started offering to them here onsite. . . . So, that was 

really the biggest change with the short-term therapy is the fact that it’s onsite. 

According to CBO staff, the availability of on-site counselors at CBOs helped clients feel 

more comfortable accepting referrals when offered (Figure 6.5). Overall, CBO staff perceived 

that clients were more comfortable accepting a referral to an MHP within the CBO (72 percent) 

than with leaving the CBO for an appointment (50 percent). 

Some CBOs and MHPs sought out novel, more cost-effective approaches to expand 

on-site MHP support by leveraging clinical masters or doctoral trainees (e.g., in social work, 

Figure 6.5. Staff Perception of Client Comfort with Accepting Referrals

SOURCE: Data from CBO staff survey, summer 2019.

134 
 

 

 

 

 

 

 

psychology, counseling) affiliated with the MHP, and who received supervision from MHP 

clinicians, to bolster the on-site MHP presence. As described by one CBO leader, 

I think if we’re seeing increased utilization and people needing counseling inhouse, it expands beyond what we have as funded staff positions—relying more 

heavily on our funded [licensed counseling] staff being supervisors—so we have 

more interns. So, it would be another pathway to having our longevity and being 

able to meet the growing need. 

One CBO leader also reported on efforts to expand the presence of on-site MHP interns to 

expand capacity as the C2C program evolved, 

We were really looking at what was really successful and making it bigger . . . in 

working with our MHP, we noticed that counseling sessions are happening, and 

at a large rate, and we needed to expand. We needed to scale it, and we went 

from two to . . . Year 1, we had two interns. Year 2, we had three interns, and in 

year 3, we’ll have five interns. 

In addition to having counselors on-site, CBO staff consistently mentioned the warm handoff 

approach as an important facilitator to successful referrals. For CBOs, this included activities 

such as accompanying clients to the MHP to sign up for services, calling the MHP to schedule 

an appointment for clients, and explaining the process of psychotherapy to clients prior to 

scheduling an appointment. MHPs also recognized the importance of warm handoffs and took 

ownership of the referral process to follow up with CBOs and engage CBO staff actively during 

the process. One MHP described their referral process that included counselors reaching out to 

CBO staff: 

We try to do a warm hand-off on those referrals so it’s not just that we get 

something in the system that says, “this person has been referred.” But if this is 

a young person who has a close relationship with a member of [CBO], then 

whichever counselor gets assigned to that young person will try to touch base 

with [the CBO staff member] to be able to do an in-person introduction because 

we find it very helpful for young people to be introduced by the person that they 

already have a close relationship with, in order to be able to increase the 

likelihood that they will engage in counseling. 

MHP Policies and Practices

To address barriers that limited client access to mental health care at MHPs, some MHPs 

changed their policies and practices to accommodate client needs. For example, one MHP 

reported working with their CBO to secure external funding to pay for services when clients did 

not have any or adequate health insurance. Another MHP explained how they modified their 

payment options to allow clients to pay in installments. MHPs also expanded their flexible dropin hours to accommodate clients who could not take time off from work to receive mental health 

services. One MHP leader reported the ways in which clinical care in the C2C initiative differed 

from traditional outpatient counseling arrangements with respect to scheduling: 

I mean, all of my clients from [CBO] . . . It’s not like a regular psychotherapy 

office where they come in once a week, at the very same time. If they need 

135 
 

 

different appointment times on different days, we accommodate them. If they 

want to come twice a month, or once a month even, just for medicine 

management, we’ll do that. 

To specifically address stigma as a barrier to receiving mental health care, some CBOs and 

MHPs implemented additional outreach services to destigmatize mental health and increase 

client comfort with mental health services. These outreach services included peer escorts to 

accompany clients to their first MHP appointment, educational materials to explain different 

types of mental health services provided by psychologists and psychiatrists, and patient panels to 

discuss mental health issues. In addition, one MHP reported increasing the number of bilingual 

therapists to meet the needs of clients. Several MHPs reported engaging in cultural competency 

training with clinic staff to increase their cultural awareness and to improve their ability to work 

with clients from diverse backgrounds. 

Challenges Navigating MHP Practices and Policies

Many CBOs found it challenging to navigate their MHPs’ practices and policies for 

provision of services to C2C clients, which may also reflect broader barriers to many New 

Yorkers seeking mental health care. For example, some CBOs shared that the lengthy MHP 

intake process was often a barrier for clients. Instead of seeing a therapist during the first 

appointment, many MHPs, adhering to a traditional psychotherapy model, required clients to 

have an intake appointment first. One CBO staff reported, 

That’s the challenging way. Yeah, because then they have to go and do the intake 

and then you have to make it to the next appointments and at that point I think 

from what I’ve heard from [a CBO staff member] we lose some folks. 

Some MHPs had strict attendance policies that were challenging for some C2C clients. CBOs 

reported that clients would miss appointments because of work, childcare, or other obligations. 

Because rescheduling was not easy at some MHPs, CBOs indicated that clients would terminate 

treatment. A number of MHPs referred clients to other service providers for treatment that they 

did not offer, such as psychiatric medication management services. Some CBOs expressed 

frustration with navigating multiple referrals for clients, particularly with respect to clients who 

needed more intensive psychiatric services. One CBO leader reported, 

It was just a gap that kept continuing . . . once somebody had an identified 

psychiatry meeting and we knew that they needed more intensive care, we were 

running through the barriers of connecting them to [MHP] and going to [an 

external psychiatrist], it being a little bit farther and required at least two or three 

modes of transit to get there. [The psychiatry provider has] a very rigid approach 

to appointments because their hospital system . . . our young people would engage 

in one appointment, two appointments, and then be dropped off their case load. 

Referrals to Other Providers

Because CBOs began providing the full complement of C2C services during implementation 

year 2 and beyond, they learned more about the different needs of their clients. CBOs found that 

136 
some clients needed more comprehensive care, such as medical treatment or services related to 

surviving domestic violence, in addition to mental health services. When MHPs had a long wait 

list or were otherwise unable to provide services for a given client (e.g., due to insurance 

requirements, cultural/language competencies), CBOs would refer their clients to other 

behavioral health providers and utilize other city-sponsored initiatives such as NYC Well. As 

described by one CBO leader, specific external referral practices were driven by the needs of 

individual clients, 

It also depends on the client and other like specific needs they might have. Like if 

they have like other physical health concerns as well, we might refer them to the 

floating hospital because they’re all inclusive. 

MHP Referrals to CBOs

As implementation progressed, MHPs appeared to gain a deeper understanding and 

appreciation of the CBO environment, including the population being served by CBOs. The 

partnership with CBOs also helped MHPs learn more about the services available at CBOs and 

how they might be valuable to their clients. For example, one MHP referred their clients to 

receive workforce training with their CBO partner. Other MHPs sent their clients to CBOs for 

exposure to the C2C core skills (e.g., attend PE, receive MHFA training). Through engagement 

with C2C, MHPs developed more effective collaboration with CBOs, which allowed MHPs to 

connect their clients to needed social services provided by CBOs. One MHP stated, 

I would say [among the] biggest changes . . . reverse referrals. We have referred 

a lot of patients who are interested in working and need that support. The 

feedback from the patients has been amazing. 

New Care Coordination Approaches

Many CBO-MHP partnerships reported on efforts to initiate new kinds of meetings at the 

organizational level to coordinate care for clients, described by some as “case conferences” or 

“collaborative tracking meetings.” In these meetings, CBO and MHP staff would have group 

discussions about clients who were not yet in care, clients who were already linked to mental 

health care, and specific client treatment plans. Outside of conversations about clients, CBOs and 

MHPs also discussed some of the structural and logistical barriers to mental health care and 

possible solutions at these meetings. In some cases, measures of the effectiveness of care 

coordination efforts were reviewed, such as the number of referrals or completed appointments. 

For MHPs who had agreed to prioritize clients from their CBO partner, the care coordination 

meetings also provided an opportunity to ensure that CBO clients were really being prioritized. 

All CBOs and MHPs were required to identify staff members to fulfill key leadership/ 

management and service roles for each organization (see Chapter 4). In addition, clarifying 

guidance surrounding care coordination issued in year 2 required CBOs and MHPs to designate a 

person or team at both agencies to facilitate and simplify communication about clients. This care 

coordinator role typically involved connecting with clients to schedule an appointment, following 

137 
up on whether a client has kept an appointment, and answering questions about logistics that 

would support client linkage and retention in mental health care. One coordinator spoke about 

the specifics of their role at their CBO: 

[I]t would be primarily my role as a C2C coordinator to interface with their 

[clients’] treatment planning, to make sure the treatment plan is in place. And 

then there is a communication once a week about whether there is adherence, and 

then there is a touch up once a month to talk about the treatment planning. . . 

[there] will be a psychiatrist who sees our clients, and they will be able to just 

communicate. So, it will be their treatment worker and myself interfacing, and 

then the psychiatrist being in direct communication with me if any follow-up is 

needed. 

Challenges with Exchanging Client Information

Throughout implementation, CBOs and MHPs addressed the challenges related to sharing 

clinical information across organizations, including the need to comply with HIPAA regulations. 

Partnerships that were sharing clinical information as part of their care coordination efforts set up 

procedures to obtain client permission for the sharing of clinical information. For example, 

CBOs would include signed consent forms with the information sent to MHPs when a referral 

was made. Several CBOs described the intricacies of working with MHPs that were part of a 

hospital system and having to work out special processes and exemptions for client data sharing. 

For example, one hospital required that all mental health care happen on hospital grounds so 

there was no existing policy for sharing client data with a CBO. On the MHP side, some reported 

that they did not always share all the client information that CBOs requested to protect client 

confidentiality. One CBO reported a specific client data request made to an MHP: 

[W]e’ve requested having access to the medical record number in the hospital 

system, and that’s being worked out so that we’re just then communicating, not 

just based on name and date of birth, but also just having actual medical record 

number. And then, we’re in closer phone communication with the scheduling, the 

intake director on the hospital side, and she’s been a part of all our shared patient 

model meetings. She is a really high up level stakeholder that’s able to 

understand the concept, and help us navigate getting quicker appointments, when 

that’s needed, or streamlining communication. 

Question 4: How Did C2C Facilitate Partnerships Among CBOs Within and Outside the 

C2C Network?

Because C2C was designed to expand mental health task-shifting in NYC, it was critical for 

CBOs within the C2C network to work closely with each other and share processes, challenges, 

and solutions. It was also important for CBOs to share mental health care practices and resources 

with organizations outside the C2C network. Future guidance for task-shifting efforts like C2C 

can benefit from understanding how these relationships worked to advance C2C implementation 

and promote task-shifting more broadly. In this section, we describe the ways in which C2C 

facilitated partnerships within and beyond the C2C network. 

138 
 

 

Sharing Within the C2C Network

Overall, CBOs described developing positive relationships with the other CBOs in the C2C 

network and collaborating with them to share best practices related to the C2C skills or other 

mental health services. These connections were facilitated in part by regular C2C program 

convenings, organized by the C2C Collaborative, that brought participating CBOs and MHPs 

together to engage in collaborative learning events and to exchange ideas. In addition to these 

structured activities, CBO and MHP staff members routinely engaged in more informal 

networking through email, phone calls, and in-person “site visits” to learn from one another. 

The collaboration within the C2C network started during the first year and remained strong 

throughout implementation. For example, one CBO shared how they learned about group 

screening during a visit to another CBO. Another CBO hosted an open house in collaboration 

with their MHP after learning from another CBO how this approach had helped clients become 

more familiar and comfortable with mental health services. CBOs also co-led training sessions 

on specific C2C skills. Overall, CBOs shared that the opportunity to learn from each other saved 

time and resources and prevented them from repeating mistakes. 

C2C also brought together CBOs that served similar clients and provided opportunities 

for shared learning. For example, a subset of CBOs that focused on workforce development 

formed a workgroup during the early stages of implementation that continued to meet regularly 

throughout implementation to share best practices for working with their clients. As one CBO 

leader reported, 

We’re all meeting at [a CBO] to talk about the process and what issues we’re 

having and what materials we have. So, it’s about coming together and seeing 

what we can do to help each other out as we prepare for year 2 implementation. 

Many CBOs also viewed the C2C meetings for all participants at “convenings” organized by 

the C2C Collaborative as valuable to the learning process. These meetings of the C2C network 

promoted a sense of community and facilitated collaboration and learning among CBOs. For 

example, one CBO presented their art-based PE curriculum at a C2C meeting to share this 

innovative practice. When asked about the collaboration among CBOs, many CBO leaders 

agreed that the relationships and information sharing would continue beyond C2C. As described 

by one CBO leader, 

The grantee meeting was great. They’ve just been connecting me to really 

important folks out there doing similar work in other agencies. We’ve already 

cross-pollinated, if you will, in some ways. We had some other contract 

deliverable that had to do with education for bilingual parents of LGBTQ youth 

and we offered it to another site where a colleague of mine—I met her through 

C2C—so like going over there, the curriculum supports her clients and one of our 

deliverables is to deliver this bilingual education to parents and we don’t have 

parents but we have youth. So, it’s like we’re really able to help each other. 

139 
 

 

 

 

 

 

Sharing with Organizations Outside the C2C Network

CBOs described their role in bringing the C2C model to other organizations interested in 

providing mental health services. In interviews with CBO and MHP leaders conducted in year 4, 

some interviewees reported that, on learning about the mental health services available at CBOs, 

other organizations in the community requested information about C2C and its approach to 

integrating mental health care into CBOs. In response, one CBO sent staff members to these 

other organizations to discuss mental health issues with staff. Over time, external organizations 

began to see some CBOs as “experts” in providing mental health services to underserved 

populations. As reported by another leader, the reach of C2C went beyond NYC, with 

organizations from other countries also seeking out CBOs for consultation, 

We have had some global implications...I have been studied by a fellow in the 

U.K. who wanted to learn about C2C to replicate the model at a women crisis 

center . . . I have also been studied by an Australian-based non-governmental 

organization. They will take some pieces of C2C and implement it at the program 

in Australia. This is transatlantic peer mentoring, and best practices from C2C 

that have global implications. 

CBOs also expanded their reach and began to provide mental health and wellness services to 

clients from other organizations. One CBO leader shared that they felt competent to provide PE 

services to other organizations after delivering the curriculum to three cohorts of clients. Some 

CBOs viewed the extension of mental health services to other organizations as a way to expose 

more organizations to the C2C model and thus disseminate the model. A CBO leader discussed 

how they shared C2C practices with CBOs outside the C2C network: 

So that was really helpful and seeing that shift was important for us. Also 

...another organization in Brooklyn reached out to us, and we sent our social 

worker up to their organization to do pretty much the same thing she did with our 

clients here [as part of the C2C program]. 

One MHP also described their assessment of C2C and its influence in the broader community 

during the third year of implementation, and specifically highlighted the importance of support 

for collaboration and exchange of best practices amongst providers within the C2C initiative: 

[C2C] has built a larger community in New York City of practitioners who are 

speaking a common language and exchanging ideas and practices that can 

become best practices. 

Discussion

In this chapter, we discussed the extent to which C2C transformed CBOs’, MHPs’, and the 

community’s attitudes, behaviors, and culture surrounding mental health. 

Role of Mental Health

CBO leaders described how C2C trainings helped to cultivate a common language with 

which staff members and clients could discuss mental health and well-being, as well as a more 

140 
 

standardized framework for addressing client mental health needs. Over time, CBO staff 

began to recognize the strength of C2C skills in meeting client needs and supporting clients in 

achieving their goals. This observation of the benefits of C2C programming for their clients may 

have helped CBO staff become more invested in C2C and more willing to deliver C2C skills. 

Overall, consistent with previous reviews of EBP implementation in nonclinical settings (BachMortensen, Lange, and Montgomery, 2018), effective messaging, high staff buy-in, and 

perceived alignment of C2C with organizational mission were viewed as important facilitators to 

the implementation of C2C. Such factors were also critical in bringing about broader changes 

with respect to how CBOs prioritized client mental health over time. 

PE was seen as particularly useful in breaking down mental health stigma among clients 

when delivered using an MI-centric, nonjudgmental approach. Such an approach may be 

instrumental in addressing common barriers to seeking clinical care for clients who may benefit 

from more intensive mental health treatment. Anecdotal reports from client interviewees suggest 

that PE in the CBO context may help to correct misperceptions surrounding common mental 

health conditions (e.g., anxiety) and treatment options. In combination with mental health 

screening, C2C PE may help to increase downstream utilization of mental health services by 

enhancing clients’ ability to recognize whether/when they might be experiencing symptoms that 

could benefit from treatment. This is potentially important given that lack of perceived need for 

treatment is among the most commonly cited reasons for forgoing mental health treatment 

among individuals who experience mental health symptoms but have not received treatment (i.e., 

“I didn’t think I needed [treatment]”) (see Mojtabai et al., 2011). Chapter 7 provides more 

information regarding the effectiveness of C2C in reducing client-reported treatment barriers. 

CBO leadership also reported observing the benefits of staff using their C2C skills with 

clients to help them more successfully navigate CBO programs. In terms of training and 

supervision in C2C skills received by CBO staff, more than three-quarters of staff survey 

respondents reported feeling like the training helped them address their clients’ mental health– 

related problems. This suggests that CBO staff began to recognize the strength of C2C services 

and how these services met client needs and became more invested in C2C and more willing to 

deliver C2C services. As such, careful consideration of how C2C is expected to address specific 

staff and client issues or needs, and how C2C aligns with organizational mission (Bach-Mortensen, 

Lange, and Montgomery, 2018), are critical components of effectively messaging C2C to staff 

and clients to enhance buy-in. 

Of the four core C2C skills, MI was seen as having the greatest and most widespread effect 

on organizational culture within CBOs. CBO leaders and staff members credited MI training 

with establishing a common language for discussing client interactions and efforts to enhance 

client motivation and progress toward specific goals within diverse CBOs. Moreover, MI was 

seen as having a broader effect on communication styles and interpersonal interactions among 

staff members. These findings suggest that, of the core C2C skills, MI—specifically the “MI 

spirit” and orientation toward client interactions—may be an appealing strategy for CBOs with 

141 
 

 

respect to improving communication, supporting changes in organizational climate, and moving 

the orientation toward client well-being within a range of different settings. However, MI is more 

than an orientation toward client interactions (Miller and Rollnick, 2008; Miller and Rollnick, 

2013) that encompasses a broad array of clinical skills and requires extensive training, practice, 

and supervision to implement with fidelity (Moyers et al., 2005). As such, although MI was seen 

by CBOs as an important and appealing component of the C2C model, integration of MI skills 

within CBOs may require extensive planning and investment to yield the transformative effects on 

organizational culture cited by CBOs in the C2C initiative. Despite this, even brief introductory 

training in basic MI skills may pay dividends with respect to widespread effects on staff-client 

and staff-staff interactions within CBO settings. 

Because the C2C skills were primarily implemented within the CBO setting, the bulk of 

organizational change was expected to occur within CBOs (Aarons, Hurlburt, and Horwitz et al., 

2011). However, MHPs also reported undergoing organizational changes as a consequence of 

their participation in C2C. As described above, many MHPs expanded their clinical presence 

(including the presence of the clinical interns who they trained) on-site at CBOs to allow for 

client intakes and brief counseling outside their home clinics. In addition, through efforts to 

reduce practical barriers to client referrals through C2C, many MHPs modified their clinic hours 

and/or rules surrounding appointment no-shows and clinic wait lists. Further, by virtue of 

learning more about the scope of services provided by CBOs, MHPs engaged in reverse referrals 

to CBO services that could address a wide range of challenges confronted by their clients. This 

link to CBO services was seen as promoting more holistic treatment within the MHP. These 

findings suggest that MHPs experienced a number of organizational changes and experienced 

tangible—and in some cases unexpected—benefits from their participation in C2C. Establishing 

formal arrangements with CBOs through programs such as C2C may be an appealing way to 

increase community presence and promote access to a more comprehensive set of social services 

for community mental health clinics who serve vulnerable populations. 

Barriers to Using Mental Health Care

Client barriers to referrals included practical barriers (e.g., scheduling difficulties, lacking 

the time, transportation, and childcare to see an off-site provider, inability to pay or lack of 

insurance), previous negative experience with mental health services, and general discomfort 

with the idea of receiving mental health services. In addition, although C2C helped to 

destigmatize mental health among CBO staff and clients, staff perceived community mental 

health stigma to be relatively high over the course of implementation and a primary and ongoing 

barrier to clients accepting referrals to seek mental health services outside CBOs. CBOs and 

MHPs worked collaboratively to develop solutions to minimize client barriers, such as giving 

metro cards to clients or MHPs adding walk-in hours for C2C clients specifically. Findings from 

our analysis of C2C’s effectiveness in addressing client-reported barriers to service utilization 

are described in detail in Chapter 7. 

142 
The relatively high levels of community stigma reported by staff members are consistent 

with prior community-based research. A number of studies have found that stigma associated 

with mental health is persistent in many communities, especially in racial/ethnic minority 

communities (Brown et al., 2010; Conner, Koeske, and Brown, 2009). A recent review found 

that stigma is even found among health care professionals who provide care to patients with 

mental health illnesses (Mestdagh and Hansen, 2014). Although nearly all CBOs endorsed 

stigma as a barrier to referrals, there was some anecdotal evidence relayed by CBO staff 

interviewees that some CBO clients tended to show less stigma (i.e., adolescents and young 

adults, for whom mental health treatment seeking was viewed as more normative), which may 

have increased the willingness to accept mental health referrals in some client populations. In 

the context of high community stigma toward mental health and mental health services (e.g., 

perceptions that seeking treatment is a sign of personal weakness), additional strategies beyond 

stigma-reduction efforts and PE-based interventions may be needed to appropriately address 

client mental health needs. For example, ongoing efforts to target stigma within historically 

underserved communities (e.g., through public education campaigns, partnerships with 

community leaders) may be needed to optimize the effect of C2C and similar programs on 

reducing unmet mental health treatment need. Chapter 7 provides a more detailed examination 

of whether C2C reduced stigma and other barriers to mental health care. 

Care Coordination and Referrals

Over time, CBOs and MHPs made adjustments to their organizational structures and policies 

to address challenges and improve coordination surrounding clients’ mental health needs. CBOs 

reported that solutions such as having on-site counselors at CBOs, warm handoffs to MHPs, and 

changing MHP practices and policies all helped to minimize some client barriers to receiving 

mental health care. CBO leaders shared that clients had to overcome many barriers before they 

could actually see an MHP, which often required CBOs and MHPs to work collaboratively to 

make structural changes in their practices. 

The integration of MHPs within more comprehensive “treatment teams” in non-MHP service 

settings (e.g., hospitals, primary care clinics) is foundational to “integrated care” and related 

models (Kodner, 2009) aimed at reducing barriers to accessing mental health treatment. This 

type of integration has been shown to increase mental health service use in a range of different 

settings and for different populations (e.g., Fuller et al., 2011). Integration of mental health 

services on-site at the CBO may represent a particularly important approach to increasing 

referral completion rates for clients with a need for more intensive mental health treatment. 

However, integration of MHPs within CBOs can also pose challenges for organizations, 

including but not limited to practical issues such as clinician availability and the cost of 

maintaining an MHP presence within the CBO. Some CBOs overcame these challenges by 

leveraging MHP trainees, typically master’s or doctoral-level clinicians in training, who were 

able to provide some specialty mental health services within the CBO setting and serve as on-site 

143 
 

 

“faces” of the MHP, which facilitated warm handoffs and smoothed the referral process. 

Through establishing such arrangements, many CBOs and MHPs were able to demystify the 

MHP (i.e., because the on-site intern became a “known entity” within the CBO), increase their 

capacity to address more intensive client mental health needs, and augment referral processes in 

a resource-efficient way that was seen as beneficial for all parties. 

Partnerships Among CBOs Within and Outside the C2C Network

C2C was designed to increase NYC’s capacity to provide mental health care to vulnerable 

NYC residents. All organizations lauded the extensive guidance, technical assistance, and 

supplementary trainings (e.g., MI Institute, CQI workshop) that were provided by the 

C2C Collaborative as part of C2C and perceived these supports as essential for successful 

implementation. This is consistent with past research on the implementation of EBPs (BachMortensen, Lange, and Montgomery, 2018) and underscores the importance of technical 

assistance networks and collaborative learning in adapting and implementing behavioral health 

interventions in CBO settings. CBOs and MHPs developed positive relationships within the 

C2C network and learned from one another’s experiences throughout implementation. Through 

sharing common implementation challenges and collaborating on solutions to address these 

issues, organizations were able to support one another in coping with the stress of bringing C2C 

to life. This underscores the importance of establishing and facilitating information exchange 

among organizations who may be considering adopting C2C or similar task-shifting approaches 

to address client mental health needs. 

By implementation year 4, several CBO and MHP leaders reported anecdotal experiences 

surrounding the diffusion of C2C model beyond the C2C network. CBOs and MHPs shared their 

expertise gained from the C2C initiative with local organizations (i.e., within NYC) as well as 

those based in other countries. Because CBOs became more confident in delivering C2C, other 

organizations in the community began to come to the CBOs for consultation and advice. CBOs 

shared their experiences with C2C model with other organizations in NYC and beyond in an 

effort to help those organizations better address issues surrounding client mental health. This 

finding is of particular importance because it suggests that C2C had a broader effect in NYC 

with other organizations potentially adopting some elements of the C2C model. The experiences 

of the CBOs and MHPs who participated in the initial C2C implementation represent a valuable 

resource that may help to guide implementation decisions for other organizations in the future 

(see Stevens et al., 2020). It is also important because spread of the C2C model outside the 

network may have affected our ability to detect interventions effects in the impact study. 

Limitations

As in previous chapters, readers should consider the limitations of the evaluation when 

interpreting these findings. Key findings presented in this chapter were drawn from a mix of 

144 
 

quantitative and qualitative data sources and may not capture the full range of experiences from 

all stakeholders involved in the implementation of C2C. In addition, as described in Chapter 5, 

findings from later stages of the implementation process (i.e., implementation year 4) are based 

on information from the 14 CBOs that are currently participating in the program. Although 

some of the lessons learned may be generalizable to other organizations, some implementation 

experiences likely are unique to individual CBOs, MHPs, and clients. Organizations seeking to 

replicate the program in the future may have different experiences, including challenges and 

successes, over the course of the implementation process. 

Summary

CBO leaders and staff reported that C2C helped to reconceptualize the role of client mental 

health in the CBO setting and helped to establish a common language through which staff and 

clients could communicate about mental health. In addition, a majority of CBO staff survey 

respondents (more than three-quarters) indicated that C2C training helped them address their 

clients’ mental health-related problems. Clients also reported having a new understanding about 

mental health and mental health services, which they learned from receiving C2C services. 

Throughout implementation, CBOs and MHPs described numerous barriers related to client 

referrals to more intensive clinical services at MHPs and other clinical providers. In particular, 

despite perceptions that C2C helped to break down stigma within the CBO setting, CBO staff 

members viewed the stigma in the community around mental health and mental health services 

as a significant barrier to accessing mental health treatment. Over time, CBOs and MHPs worked 

collaboratively to address these barriers. CBOs and MHPs also changed their approach to care 

coordination over the course of implementation and took steps to standardize and streamline 

information-sharing practices. Routinely connecting and exchanging ideas with other CBOs and 

MHPs participating in C2C was a commonly cited facilitator of successful implementation. 

Moreover, many CBO and MHP leaders reported sharing lessons learned from C2C with 

organizations outside the C2C network. 

145 
 

 

 

 

References

Aarons, G. A., M. Hurlburt, and S. M. Horwitz. (2011). Advancing a conceptual model of 

evidence-based practice implementation in public service sectors. Administration and Policy 

in Mental Health, 38(1), 4–23. 

Ayer, L., M. S. Dunbar, M. Martineau, C. Stevens, D. Schultz, W. Y. Chan, M. Abbott, R. Weir, 

H. H. Liu, D. Siconolfi, and V. L. Towe, Evaluation of the Connections to Care (C2C) 

Initiative: Interim Report, Santa Monica, Calif.: RAND Corporation, RR-2497-MFANYC, 

2018. As of February 4, 2020: 

https://www.rand.org/pubs/research_reports/RR2497.html 

Bach-Mortensen, A. M., B. C. Lange, and P. Montgomery, “Barriers and Facilitators to 

Implementing Evidence-Based Interventions Among Third Sector Organisations: A 

Systematic Review,” Implementation Science, Vol. 13, No. 1, 2018, p. 103. 

Berry, L. L., B. L. Rock, B. S. Houskamp, J. Brueggeman, and L. Tucker, “Care Coordination 

for Patients with Complex Health Profiles in Inpatient and Outpatient Settings,” Mayo Clinic 

Proceedings, Vol. 88, No. 2, 2013, pp. 184–194. 

Brophy, L., C. Hodges, K. Halloran, M. Grigg, and M. Swift, “Impact of Care Coordination on 

Australia’s Mental Health Service Delivery System,” Australian Health Review, Vol. 38, 

No. 4, 2014, pp. 396–400. 

Brown, C., K. O. Conner, V. C. Copeland, N. Grote, S. Beach, D. Battista, and C. F. Reynolds, 

“Depression Stigma, Race, and Treatment Seeking Behavior and Attitudes,” Journal of 

Community Psychology, Vol. 38, No. 3, 2010, pp. 350–368. 

Conner, K. O., G. Koeske, and C. Brown, “Racial Differences in Attitudes Toward Professional 

Mental Health Treatment: The Mediating Effect of Stigma,” Journal of Gerontological 

Social Work, Vol. 52, No. 7, 2009, pp. 695–712. 

Fuller, J. D., D. Perkins, S. Parker, L. Holdsworth, B. Kelly, R. Roberts, L. Martinez, and 

L. Fragar, “Effectiveness of Service Linkages in Primary Mental Health Care: A Narrative 

Review Part 1,” BMC Health Services Research, Vol. 11, No. 1, 2011, p. 72. 

Kodner, D. L., “All Together Now: A Conceptual Exploration of Integrated Care,” Healthcare 

Quarterly (Toronto, Ontario), Vol. 13, 2009, pp. 6–15. 

Link, B. G., F. T. Cullen, A. Struening, P. E. Shrout, and B. P. Dohrenwend, “A Modified 

Labelling Theory Approach to Mental Disorders: An Empirical Assessment,” American 

Sociological Review, Vol. 54, No. 3, 1989, pp. 400–423. 

Mestdagh, A., and B. Hansen, “Stigma in Patients with Schizophrenia Receiving Community 

Mental Health Care: A Review of Qualitative Studies,” Social Psychiatry and Psychiatric 

Epidemiology, Vol. 49, No. 1, 2014, pp. 79–87. 

146 
 

 

Miller, W. R., and S. Rollnick, Motivational Interviewing in Health Care, New York: Guilford 

Press, 2008. 

Miller, W. R., and S. Rollnick, Motivational Interviewing, Helping People Change, 3rd ed., 

New York: Guilford Press, 2013. 

Mojtabai, R., M. Olfson, N. A. Sampson, R. Jin, B. Druss, P. S. Wang, K. B. Wells, H. A. Pincus, 

and R. C. Kessler, “Barriers to Mental Health Treatment: Results from the National 

Comorbidity Survey Replication,” Psychological Medicine, Vol. 41, No. 8, 2011, 

pp 1751–1761. 

Moyers, T. B., T. Martin, J. K. Manuel, S. M. L. Hendrickson, and W. R. Miller, “Assessing 

Competence in the Use of Motivational Interviewing,” Journal of Substance Abuse 

Treatment, Vol. 28, No. 1, 2005, pp. 19–26. 

Shippee, N. D., B. H. Rosen, K. B. Angstman, M. E. Fuentes, R. S. DeJesus, S. M. Bruce, and 

M. D. Williams, “Baseline Screening Tools as Indicators for Symptom Outcomes and Health 

Services Utilization in a Collaborative Care Model for Depression in Primary Care: A 

Practice-Based Observational Study,” General Hospital Psychiatry, Vol. 36, No. 6, 2014, 

pp. 563–569. 

Stevens, C., E. Tosatti, L. Ayer, D. Barnes-Proby, G. Belkin, S. Lieff, and M. Martineau. (2020). 

Helpers in Plain Sight: A Guide to Implementing Mental Health Task Sharing in CommunityBased Organizations. Santa Monica, Calif.: RAND Corporation, TL-317-MFANYC. 

https://www.rand.org/pubs/tools/TL317.html 

147 
 

 

Part III. Impact

The overarching goal of the C2C impact evaluation was to examine the effect of C2C’s 

task-shifting approach of integrated mental health skills and support into the work of CBOs on 

participants’ access to and utilization of mental health care, mental health and functioning, and 

outcomes targeted by the programs and services offered by the CBOs. 

Chapter 7 examines whether C2C reduced barriers to mental health care, and whether C2C 

increased various forms of mental health care utilization, relative to the comparison group (i.e., 

clients of CBOs who did not participate in C2C). 

Chapter 8 examines whether C2C improved mental health symptoms over time relative to the 

comparison group. 

Chapter 9 examines whether C2C improved outcomes in the domains of employment, housing, 

education, and incarceration, relative to the comparison group. 

Appendix C provides extensive detail on the methods used to execute the impact analyses, as 

well as additional data related to that provided in the chapters of this section. 

148 
 

 

 

 

7. Impact of C2C on Mental Health Care Access and Utilization

Daniel Siconolfi, Dana Schultz, Lynsay Ayer, Joshua Snoke, and Elie Ohana 

Key Findings

• From baseline to follow-up, both C2C participants and those in the comparison group 

reported fewer logistical, attitudinal, and stigma barriers to mental health care, more 

utilization of clinical outpatient care, and less utilization of inpatient and emergency care. 

• In the overall sample, C2C did not result in greater improvement on measures of access 

to and utilization of mental health services compared with usual CBO services. 

• C2C was more effective for certain subgroups, showing an effect in the following areas: 

- reduced attitudinal barriers to mental health care in youth and young adults 

- increased use of outpatient mental health services at youth development CBOs 

- reduced emergency department use among parents and caregivers and youth and 

young adults 

- reduced residential treatment use at job training and employment CBOs and 

among unemployed/underemployed adults and parents and caregivers. 

149 
 

 

 

Introduction

C2C was designed to improve access to evidence-informed mental health services for at-risk 

populations in NYC through its task-shifting approach. For this segment of the impact evaluation, 

we investigated the extent to which access to mental health care and utilization of mental health 

care services changed for C2C participants, relative to comparison participants (i.e., clients of 

CBOs who did not participate in C2C). We expected to find that barriers to mental health care 

decreased more for C2C participants than for comparison participants. We also expected to find 

that, among persons with mental health need at baseline, C2C participants would report greater 

utilization of mental health services at follow-up, relative to comparison participants. 

This chapter examines whether C2C reduced barriers to mental health care, and whether C2C 

increased various forms of mental health care utilization among New Yorkers seeking services 

from CBOs. Because of variability in the types of CBOs participating and the clients being 

served, these analyses first examined effects among the entire sample of participants and then 

looked at effects for different populations or settings within the sample. 

Background

For low-income adults and youth, such as those targeted by the C2C program, prior research 

has documented factors that impede access to mental health services and those that may increase 

mental health care utilization. Here we briefly review that literature, because it applies to our 

approach to evaluating the effect of C2C. 

Barriers to Mental Health Care

There are many barriers to accessing mental health services (Clement et al., 2012; Gulliver, 

Griffiths, and Christensen, 2010; Priester et al., 2016), which can be broadly categorized into 

three groups: logistical barriers, non-stigma-related attitudes, and stigma (Clement et al., 2012). 

Logistical barriers to mental health care include factors such as high cost for services, lack of 

health insurance, shortages of MHPs, inflexibility in scheduling, lack of culturally inclusive 

and/or linguistically appropriate care, prior negative experience with mental health services, and 

mistrust of the health care system (Santiago, Kaltman, and Miranda, 2013; Substance Abuse and 

Mental Health Services Administration, 2019a). Examples of non-stigma attitudinal barriers 

include concerns about treatment side effects or the belief that the problem would resolve naturally. 

Mental health treatment stigma refers to “the stigma and discrimination that individuals believe 

to be associated with receiving care for a mental health problem” (Clement et al., 2012). 

Stigma-related barriers include individuals’ concerns about the potential negative 

consequences associated with seeking treatment for a mental health problem. Mental health 

stigma can interfere with help seeking (Clement et al., 2015; Schnyder et al., 2017). Internalized 

stigma, or the inward shame attributed to mental health problems, is a common impediment to 

help seeking (Clement et al., 2015). For example, someone in need of mental health care may 

150 
 

 

avoid seeking treatment due to shame, embarrassment, or fear that others would find out about 

their mental health problem (Hadfield and Wittkowski, 2017; Lazear et al., 2008). 

Unmet Need for Mental Health Services and Mental Health Care Utilization

Unmet need for mental health services is prevalent in NYC. For example, estimates indicate 

that less than 40 percent of New Yorkers with major depressive disorder report receiving mental 

health treatment (City of New York, Office of the Mayor, 2015), despite evidence that depression 

can be treated effectively (Cipriani et al., 2018; Lopez-Lopez et al., 2019; Siu et al., 2016). 

Nationally, about one in four persons (24 percent) with any mental illness report unmet mental 

health needs, and this increases to nearly one in two persons (45 percent) among those with 

serious mental illness (Substance Abuse and Mental Health Services Administration, 2019b). 

Mental health care utilization is even lower among historically underserved individuals, 

including racial/ethnic minority and other socioeconomically disadvantaged groups, which may 

contribute to or exacerbate mental health problems and other disparities, such as economic and 

educational inequalities (Ault-Brutus and Alegria, 2018; Breslau et al., 2017; Burnett-Zeigler et 

al., 2012; Santiago, Kaltman, and Miranda, 2013). 

Early detection of mental health problems is one way to address issues related to mental 

health care utilization. Recent studies have demonstrated that a positive screening for mental 

health problems is related to greater likelihood of mental health service utilization, such as 

outpatient services and inpatient care (Petrenko et al., 2011; Shippee et al., 2014; Thomas and 

Staiger, 2012). However, there are potential racial and ethnic disparities in screening in clinical 

settings (Alegria et al., 2016; Hahm et al., 2015; Thomas and Staiger, 2012). Mental health 

screenings in community settings, therefore, have the potential to reduce such disparities 

(Thomas and Staiger, 2012). In addition, individuals who engage in substance use treatment 

do not always receive mental health screening, despite the fact that they are vulnerable to 

cooccurring mental health problems. With adequate training, substance use treatment programs 

can successfully implement mental health screening for youth and adults (Lee et al., 2010; 

Lincoln et al., 2006; Lubman et al., 2008). Although C2C was envisioned as an approach to 

address the unmet need for mental health services, the specific combined package of the four 

C2C skills has not been studied previously. Further, evidence of the effectiveness of the individual 

skills for reducing barriers or increasing utilization is limited. As a result, any effect of C2C, 

whether delivered individually or in combination, on improving these outcomes would be 

expected to be small. 

Research Questions

At the individual level, C2C aimed to assess whether integration of C2C skills into CBO 

workflows could reduce barriers to mental health care and increase utilization of mental health 

services. For the primary research question of whether C2C participants showed greater reductions 

151 
 

in barriers to mental health care relative to the comparison group, we tested whether the following 

were lower among C2C participants: 

• logistical barriers to mental health care 

• attitudinal barriers to mental health care 

• stigma-related barriers to mental health care 

• internalized stigma regarding mental health care. 

For the primary research question of whether C2C participants reported greater utilization of 

mental health services relative to the comparison group, we tested the following hypotheses: 

• Among persons with unmet mental health needs (one or more baseline mental health 

scores at or above the moderate symptom threshold), C2C participants have greater 

utilization of the following than comparison group participants over time: 

- mental health services in outpatient clinical settings 

- nonclinical mental health supports in community settings. 

• Among persons with unmet mental health needs, C2C participants have less utilization of 

the following than comparison groups participants over time: 

- inpatient care for emotional, mental health, or substance use problems 

- emergency department/urgent care settings for any reason. 

We also examined one secondary research question related to our hypotheses: 

• Does program effectiveness vary for the specific populations targeted by C2C (i.e., adults 

age 18 or older who are unemployed or underemployed, young adults ages 16–24 who 

are not in school and are not employed, and parents/primary caregivers who are expecting 

or who have children up to the age of 4) or CBO service types (e.g., job training and 

employment program, youth development program)? 

Methods

At the individual level, C2C aimed to demonstrate that implementation of C2C program 

strategies into CBO workflows can reduce perceived barriers to mental health care for all 

participants and increase mental health services utilization among those with unmet mental 

health needs. The impact evaluation tested the extent to which C2C improves participants’ access 

to mental health care and well-being on a variety of dimensions by comparing C2C clients with 

clients who receive similar social services, but not C2C mental health services, from similar local 

CBOs (see Appendix C, section C.1 for more information on design of the impact evaluation). 

This quasi-experimental evaluation design allowed us to examine the effect of C2C on 

barriers and utilization for both the primary and secondary research questions: (1) for the primary 

research questions, we examined the pooled effect of C2C across all treatment participants (i.e., 

C2C clients) compared with all comparison participants (clients of CBOs who did not participate 

in C2C), and (2) for the secondary research question, we examined the effect of C2C on specific 

subgroups of participants, including the three C2C target populations (i.e., adults age 18 or older 

152 
 

 

 

who are unemployed or underemployed, young adults ages 16–24 who are not in school and are 

not employed, and parents/primary caregivers who are expecting or who have children up to the 

age of 4) and the two CBO service types with sufficient sample sizes (e.g., job training and 

employment programs and youth development programs). 

Information on data collection and data analyses are summarized briefly below. See 

Appendix C for complete details. 

Data Collection

C2C group. Thirteen of the 15 C2C CBOs participated in the impact evaluation. It was not 

possible to include two organizations in the impact component of the evaluation: one CBO was 

excluded from the analysis because they were not able to implement the program, and the funder 

terminated their participation in C2C; the other launched implementation with an intervention 

model that was substantially different from the other CBOs in C2C and offered its program in a 

language that would have required extensive translation resources for study recruitment and data 

collection materials. The 13 remaining C2C CBOs included five job training and employment 

programs, three youth development programs, a homeless shelter, two parent/caregiver-serving 

organizations, a domestic violence organization, and an immigrant-serving organization. 

Comparison group. We successfully recruited ten comparison group CBOs to participate in 

the impact evaluation. Comparison group participants were recruited from CBOs in NYC that 

served similar populations (e.g., geographic location, race/ethnicity, primary language, age) and 

provided similar services as C2C CBOs (e.g., three job training and employment programs, three 

youth development programs, a homeless shelter, a parent/caregiver-serving organization, a 

domestic violence organization with multiple locations, and an immigrant-serving organization) 

but were not implementing C2C or providing other mental health services. 

Procedures. For both C2C and comparison CBOs, RAND data collection staff worked 

closely with CBO staff to integrate the study data collection procedures into their client 

workflow. At C2C CBOs, staff administered their CBO-specific C2C screenings to clients, 

explained the opportunity to participate in the C2C impact evaluation, and provided recruitment 

materials. Clients who were interested in participating in the study were referred by CBO staff 

to RAND data collection staff, who obtained informed consent to administer the eligibility 

screening and baseline assessment for those eligible for the study. For the comparison CBOs, 

CBO staff supported the early stages of study recruitment (e.g., through recruitment events, 

referring clients to the study during program intake, posting or distributing recruitment flyers), 

and RAND data collection staff administered the study screening and baseline assessment for 

those eligible for the study. 

Eligibility. The primary inclusion criterion for the impact evaluation was meeting a 

minimum threshold on one or more of the eligibility screening measures for one of five mental 

health conditions (depression, anxiety, PTSD, alcohol use, other substance use [i.e., drug use]). 

We set eligibility thresholds lower than clinical thresholds (“cut points”) typically used to inform 

153 
 

 

 

 

 

 

 

 

further screening diagnosis to be inclusive of persons with subclinical levels of mental health 

symptoms in addition to those with clinically significant symptoms. 

Main study sample. For the analyses for the primary research questions, we used baseline 

survey data collected as part of study enrollment and follow-up client survey data collected from 

June 2017 through February 2020. This includes baseline data on 1,838 study participants 

(1,232 participants in the C2C group, 606 participants in the comparison group), 6-month data on 

688 participants at 6 months postbaseline (443 C2C group participants, 245 comparison group 

participants), and 12-month postbaseline data on 732 participants retained at 12 months (464 C2C 

group participants, 268 comparison group participants). The retention rate at 6 months was 

37 percent (36 percent of C2C group, 40 percent of comparison group); the retention rate at 

12 months was 40 percent (38 percent of C2C group, 44 percent of comparison group). For 

mental health care barriers, we analyzed the 6- and 12-month follow-up time points separately. 

For mental health care utilization, we used pooled 6- and 12-month data (i.e., participants with 

one or both follow-up time points were included in analysis). For this pooled follow-up data, 

the retention rate was 49 percent (47 percent of C2C group, 53 percent of comparison group). 

Baseline sample characteristics are summarized in Table 7.1 below with complete details in 

Appendix C, section C7. 

Unmet need subsample. For the utilization analyses, we focused on the subgroup of both 

samples (C2C and comparison) whose baseline data indicated unmet mental health need at 

baseline. Unmet mental health need at baseline was defined as having one or more screening 

scores above the moderate or intermediate threshold for the baseline mental health indicators 

(depression, generalized anxiety, psychological distress, PTSD, alcohol use, and other substance 

use), regardless of recent mental health service utilization. At baseline, 81.3 percent of 

participants had unmet mental health need (n = 1,494). Of the pooled 6- and 12-month retained 

sample, 83.5 percent (n = 756) had unmet need at baseline. The unmet need subsample is 

described in Table 7.1. Detailed subsample characteristics for the unmet need group are provided 

in Appendix C, section C7.2. 

Table 7.1. Baseline Characteristics

Main Sample (n = 1,838) Unmet Need Subsample (n = 1,494)

Characteristics C2C Comparison C2C Comparison

Gender (% female)

Age (mean)

Race/ethnicity (% Black or 

Hispanic)

Education (% high school,

GED, or less)

Employment (% unemployed)

Income (% <$5,000)

Housing (% stably housed)

49

29.8

90

69

66

60

41

66

30.7

90

72

48

57

48

49

30.0

89

68

66

60

41

67

30.0

89

65

50

58

46

154 
Subgroup samples. For the analyses for the secondary research questions of whether C2C 

program effectiveness varied for different target populations (e.g., adults age 18 or older who are 

unemployed or underemployed, young adults ages 16–24 who are not in school and are not 

employed, and parents/primary caregivers who are expecting or who have children up to the age 

of 4) or CBO service types (e.g., job training and employment program, youth development 

program), Table 7.2 provides details on the sample size for each subgroup examined. These 

groups are not mutually exclusive. 

155 
 

 

 

 

 

 

Table 7.2. Sample Sizes for Subgroup Analysis

Subgroup 6 Months 12 Months 1 Year (Pooled 6 and 12 Months)

Full C2C Comparison Full C2C Comparison Full C2C Comparison 

Sample Group Group Sample Group Group Sample Group Group

CBO service type

Job training and employment 422 296 126 448 308 140 565 392 173

programs

Youth development programs 158 84 74 160 80 80 196 102 94

C2C target population

Adults age 18 or older who are 463 318 145 488 336 152 603 422 181

unemployed or underemployed

Young adults ages 16 to 24 145 95 50 156 104 52 200 135 65

who are not in school and are 

not employed

Parents/primary caregivers who 160 100 60 171 108 63 166 104 62

are expecting or have children 

up to age 4

156 
 

 

 

 

 

Power analysis. For the client survey, we retained a total of 689 participants in the study 

at 6 months and 732 at 12 months. Given this sample size for the main sample, there was an 

80-percent power to detect an intervention effect of size 0.223 at 6 months and 0.215 at 12 months, 

which are small-sized effects according to Cohen’s effect size classification (Cohen, 1988). For 

the pooled utilization analysis that combined 6- and 12-month data, there was sufficient power to 

detect a small intervention effect size (0.215). 

Measures. The specific outcome measures for these analyses included self-reported measures 

of mental health care barriers and mental health care utilization. For barriers, we used the three 

subscales of the Barriers to Access to Care Evaluation (BACEv2) (Clement et al., 2012) to 

measure logistical barriers (e.g., service cost), attitudinal barriers (e.g., concern about medication 

side effects), and stigma barriers (e.g., concern about how others will react). Because of the 

salience of internalized stigma to C2C’s goals, we also separately examined the scale’s single 

item for internalized stigma, which is also contained in the stigma barriers subscale. For outcomes 

analysis, we used the BACE scoring approach of averaging responses instead of treating each 

item as a binary (yes or no) response. 

For utilization, we used items adapted from the Community Partners in Care (CPIC) 

evaluation (Chung et al., 2014) and asked respondents about different services or supports 

they might have received from the CBO and about their exposure to the different C2C skills. 

Six- and 12-month follow-up data were pooled for these variables; for example, affirmative use 

of a given service at either 6 or 12 months was coded as “utilization over the past year.” 

Data Analyses

We used propensity scores that estimate the probability that each individual received C2C 

(vs. services as usual) based on their baseline characteristics to adjust for any differences 

between the C2C and comparison groups at baseline. The resulting propensity score weights 

balance the two groups on baseline demographic variables (age, gender, race/ethnicity, income 

level, education level, employment status, housing status, incarceration status, target population, 

and CBO service type) and baseline outcome variables (depression, anxiety, PTSD, alcohol use, 

other substance use, psychological distress, logistical barriers, attitudinal barriers, and stigma 

barriers) at both 6 and 12 months. Pooled 6- and 12-month data for utilization analysis also used 

propensity scores generated for those specific analyses. 

For the pooled 1-year utilization outcomes in the main analysis comparing the overall C2C 

group to the overall comparison group, we imputed missing data at either time point for each 

individual. Fully conditional specification, using the R package mice, was used to create ten 

imputations across all variables. The imputed outcome variables were then collapsed to create 

pooled outcomes for the year. We did not impute data for the secondary research question related 

to subgroup differences (e.g., among job training and employment program clients) due to 

smaller sample sizes for these secondary analyses. 

157 
 

 

 

For the primary research question of whether C2C participants showed greater reductions in 

barriers to care and greater increases in mental health services utilization, we conducted the 

following series of analyses for each outcome using propensity score weights (1) at both 6 and 

12 months (barriers to care) and (2) over the pooled 1-year follow-up (utilization of services): 

• Differences within groups over time comparing each participant retained at each 

follow-up (i.e., 6 or 12 months) with that participant’s score at the baseline assessment. 

• Intervention effects over time using an intent-to-treat approach in which we compared 

all individuals in the C2C group with all those in the comparison group, regardless of the 

actual amount of C2C skills received. With this method, we examined the average change 

from baseline to follow-up for C2C participants compared with the average change from 

baseline to follow-up for comparison group participants to assess the program’s effect. 

- Propensity score weighted models adjusted the comparison group participants 

to be similar to the C2C group on key demographic factors (see Appendix C for 

more details) and also controlled for the baseline value of the relevant outcome in 

the regression models. 

- Doubly robust models, called “doubly robust,” because they adjusted for potential 

group differences both with the propensity score weights and with the inclusion 

of covariates in the regression models, used the propensity score weights and 

controlled for demographic characteristics (age, gender, race/ethnicity, income, 

employment status, education level, housing status, recent homelessness, and 

incarceration status), subgroup (target population, CBO service type), and the 

baseline outcome in the regression models. 

For continuous outcomes measuring the number of times something happened, we report 

Cohen’s d as the standardized effect size. For binary outcomes measuring whether or not 

something happened, we report the OR as the effect size. The standardized effect size was 

always derived from the doubly robust model. 

In this chapter, we present the results of the 12-month barriers to care analyses and describe 

any notable findings from the 6-month analyses. The tables with 6-month analysis results for 

barriers to care can be found in Appendix C, section C8.1. The pooled 1-year analyses of mental 

health care utilization are presented entirely in this chapter. 

Because we conducted large numbers of simultaneous hypothesis tests, it was necessary to 

adjust for multiple comparisons to account for the possibility that some results will achieve 

statistical significance simply by chance. We made this adjustment by setting cut scores used for 

determining significance based on the number of tests conducted for the set of analyses described 

in this chapter. Appendix C, section C6.2 provides information about the p-value used to determine 

significance for these analyses. In the results section, we present information about the trends 

observed for intervention effects before the multiple comparison adjustment to indicate results 

that are approaching statistical significance. For multiple comparisons corrections in this chapter, 

we grouped analysis into two categories— access and barriers, and utilization—and only 

adjusted for multiple comparisons within those categories. 

158 
 

 

 

 

 

 

 

For the secondary research question of whether C2C program effectiveness varied for 

different target populations (e.g., adults age 18 or older who are unemployed or underemployed, 

young adults ages 16–24 who are not in school and are not employed, and parents/primary 

caregivers who are expecting or who have children up to the age of 4) or CBO service types 

(e.g., job training and employment program, youth development program), we conducted this 

same series of analyses using propensity score weighting that was recomputed for each subgroup, 

including the difference within groups over time and intervention effects over time. 

Results

We started our analyses by examining mental health care access and utilization at baseline to 

better understand the experiences of study participants prior to C2C. We did this by examining 

three domains of barriers (logistical, attitudinal, and stigma) and the stand-alone internalized 

stigma item. 

Overall, mental health care barriers across all three domains were common for our study 

sample of individuals seeking services at a range of CBOs in NYC. Among clients at C2C and 

comparison CBOs in NYC, the attitudinal barrier domain was the most strongly endorsed both 

overall (mean of 1.00) and for each group (Table 7.3). Strength of barriers was reported on a range 

where 0 indicated no interference with receiving care and 3 indicated “a lot” of interference 

with receiving care. The barrier strength score for each domain reflects the average strength 

endorsement for barriers in that domain. Both across and within groups, the next most-strongly 

endorsed barrier domain was the internalized stigma component of the stigma-related barriers 

Table 7.3. Strength of Baseline Barriers to Seeking Professional Care for a Mental Health 

Problem—Domain Scores (Weighted)

Strength of Endorsement

Full Sample

Mean (SD)

C2C Group

Mean (SD)

Comparison

Group

Mean (SD)

pValue

Logistical barriers to mental health care 0.72 (0.67) 0.71 (0.64) 0.73 (0.71) 0.757

Attitudinal barriers to mental health care 1.00 (0.62) 1.01 (0.62) 0.98 (0.63) 0.399

Stigma barriers to mental health care 0.81 (0.77) 0.83 (0.76) 0.78 (0.79) 0.218

Feeling embarrassed or ashamed (internalized stigma) 0.88 (1.08) 0.91 (1.08) 0.85 (1.08) 0.323

NOTE: SD = standard deviation.

(feeling embarrassed or ashamed; mean of 0.88), followed by the stigma-related barriers domain 

overall (mean of 0.81). The logistical barriers to care domain was the least strongly endorsed 

(mean of 0.72). Overall, there were no statistically significant differences between groups in how 

strongly each barrier domain was endorsed. 

We also looked at the number of specific barriers to mental health care endorsed within each 

domain and found no statistically significant differences between groups (Table 7.4). 

159 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 7.4. Number of Endorsed Baseline Barriers to Seeking Professional Care for a Mental Health

Problem—Individual Barrier Endorsement (Weighted)

Full Sample C2C Group Comparison Group

Number of Barriers Endorsed Mean (Range) Mean (Range) Mean (Range) p-Value

Logistical barriers to mental health care 2.86 (0–8) 2.87 (0–8) 2.85 (0–8) 0.886

Attitudinal barriers to mental health care 5.18 (0–10) 5.26 (0–10) 5.10 (0–10) 0.272

Stigma barriers to mental health care 4.65 (0–12) 4.73 (0–12) 4.55 (0–12) 0.398

Although our outcomes analysis focused on the changes in barrier scores, we also looked at 

details for the barriers endorsed within each domain for descriptive purposes. Overall, there were 

no statistically significant differences between groups in the numbers of barriers endorsed within 

a given domain. A detailed table describing the prevalence of endorsement for each item in the 

scale is shown in Appendix C, section C7.2. Below, we describe the most commonly endorsed 

barriers in each domain. 

• Logistical barriers: Study participants at baseline most often reported an inability to 

afford the financial costs involved (55.7 percent), uncertainty about where to go for 

professional help (46.6 percent), and difficulty taking time off work to seek mental health 

care (41.1 percent). There were no item-level statistically significant differences by 

group. 

• Attitudinal barriers: Two-thirds or more of study participants at baseline identified as 

barriers wanting to solve the problem on one’s own (76.7 percent); a dislike for talking 

about one’s feelings, emotions, or thoughts (66.4 percent); and believing the problem 

would get better by itself (63.0 percent). The only significant difference was for the 

barrier regarding thinking the problem would get better by itself, with the C2C group 

more likely than the comparison group to endorse this barrier (65.7 vs. 59.9 percent). 

• Stigma barriers: Study participants most often had concerns for the reaction from one’s 

family (49.9 percent), felt embarrassed or ashamed (49.3 percent), had concerns that it 

might harm employment opportunities (46.9 percent), and did not want a mental health 

problem on one’s medical records (47.0 percent). There were no item-level statistically 

significant differences by group. 

Our examination of baseline mental health care utilization focused on persons with unmet 

need at baseline, defined as having one or more scores above the moderate or intermediate 

threshold for the baseline mental health measures (depression, generalized anxiety, psychological 

distress, PTSD, alcohol use, and other substance use), and without regard to current or recent 

mental health care utilization. About 81.3 percent of participants who had unmet mental health 

need at baseline reported on mental health service use over the prior 6 months in outpatient 

settings, nonclinical settings, inpatient settings, and emergency settings, and lifetime use for 

overnight stays in a hospital for emotional, mental health, alcohol, or drug problems (Table 7.5). 

160 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 7.5. Baseline Mental Health Care Utilization Among Persons with Baseline Unmet Need

(Weighted)

Unmet Need C2C Unmet

Comparison 

Unmet Need 

Group Need Group Group

Percent Percent Percent

Mental Health Services Endorsed Endorsed Endorsed p-Value

Went to any outpatient MHPa,b 31.8 35.6 27.3 0.008c

Used any nonclinical settings or resourcesa,d 35.6 39.7 30.6 0.004c

Stayed in inpatient settinge 30.6 34.2 26.3 0.009c

Used emergency settingsa,f 43.2 43.4 43.0 0.915

Mean (SD) Mean (SD) Mean (SD) p-Value

Number of times went to any MHPa,b 3.6 (9.6) 4.4 (11.3) 2.7 (6.9) <0.001c

Number of times went to emergency settinga,f 1.2 (2.4) 1.2 (2.4) 1.2 (2.5) 0.743

Number of nights in residential treatment program for 4.6 (24.8) 6.5 (29.2) 2.2 (18.1) 0.107

alcohol or drug problemsa

NOTE: SD = standard deviation. a Past 6 months. b Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. c Statistically significant at the p<.05 level. d Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended

religious/spiritual places, parks and recreation, or community centers for mental health needs. e One or more nights in residential treatment program for alcohol/drug problems (past 6 months) or overnight stay in a 

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up). f

Hospital emergency room or an urgent care facility for any health reason.

At baseline, nearly one-third of study participants (31.8 percent) had received mental health 

services in an outpatient clinical setting (i.e., with a psychiatrist, psychologist, social worker, 

psychiatric nurse, or counselor) within the past 6 months. The difference between the groups was 

statistically significant (35.6 percent for C2C group vs. 27.3 percent for comparison group). C2C 

participants also reported significantly more mental health treatment visits than comparison 

participants (mean of 4.4 vs. 2.7). About one-third of participants had used nonclinical settings 

and sources of support in the past 6 months (35.6 percent) with a significantly higher proportion 

of C2C participants using any of these services at baseline (39.7 percent) relative to comparison 

participants (30.6 percent). About one-third of participants had used inpatient settings at 

baseline (30.6 percent) with a statistically significant difference by group (34.2 percent for C2C 

group vs. 26.3 percent for comparison group). This relatively high baseline prevalence may 

reflect the inclusion of lifetime use of inpatient hospitalization for mental health or substance use 

needs. For emergency settings, approximately 43 percent of participants had gone to a hospital 

emergency room or urgent care facility for any reason in the past 6 months, with an average of 

1.2 visits, and no difference by group for either utilization measure. 

161 
 

 

 

 

 

Primary Research Question 1: Do C2C Participants Show Greater Reductions in 

Barriers to Mental Health Care?

To assess whether C2C reduced barriers to mental health care, we first examined differences 

in the endorsement of mental health care barriers within each group over time. In this comparison, 

we consider the within-individual change between baseline and the follow-up time points and 

report the average change and its direction across participants in each group. Participants were 

asked to report whether each barrier “stopped, delayed or discouraged” them “not at all” (0) to “a 

lot” (3). 

Both groups evidenced statistically significant decreases in the strength of mental health care 

barriers between baseline and 12 months (Table 7.6). The largest mean difference for both groups 

emerged for the strength of internalized stigma, with average mean decreases of 0.57 and 0.63, 

respectively. Results were similar at 6 months, with statistically significant within-group decreases 

for all indicators (see Appendix C, section C8.1). However, the magnitude of decreases from 

baseline was generally larger at 12 months compared with 6 months. 

Table 7.6. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 438 −0.425 0.031 <.001b

Comparison 257 −0.439 0.050 <.001b

Attitudinal barriers C2C 443 −0.526 0.036 <.001b

Comparison 257 −0.443 0.048 <.001b

Stigma barriers C2C 439 −0.486 0.035 <.001b

Comparison 255 −0.490 0.052 <.001b

Internalized stigma C2C 420 −0.573 0.062 <.001b

Comparison 242 −0.632 0.092 <.001b

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately. b Statistically significant at the p<.05 level.

We next examined the intervention effect over time using an intent-to-treat approach that 

compared C2C participants with comparison participants regardless of the actual intervention 

dosage (exposure) that the C2C group received (Table 7.7). There was one statistically 

significant intervention effect for barriers to care at 12 months. In the propensity score weighted 

model at 12 months, the C2C group had a smaller decrease in internalized stigma relative to the 

comparison group. This is evidenced by the within-group reduction noted earlier, combined with 

the positive coefficient in Table 7.7. In the doubly robust models at 12 months, the treatment 

effect sizes were very small (Cohen’s d ranging from −0.05 to 0.18). At 6 months, there were no 

162 
 

 

 

 

 

 

 

 

Table 7.7. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Outcome

Group-Level Comparison of Mean Changes

Propensity Score Weighted Doubly 

Modela Robust Modelb

Estimate SE p-Value Estimate SE p-Value

Standardized 

Effect Size

for Doubly 

Robust Model

Logistical barriers 0.014 0.058 0.813 −0.022 0.037 0.545 −0.034

Attitudinal barriers −0.083 0.060 0.164 −0.038 0.045 0.401 −0.054

Stigma barriers

Internalized stigma

0.004

0.059

0.062

0.110

0.948

0.592

0.048

0.227

0.042

0.069

0.248

0.001c

0.066

0.179

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Statistically significant at the p<.05 level.

statistically significant intervention effects, and effect sizes were very small, ranging from 0.02 

to 0.07 (see Appendix C, section C8.1). 

Primary Research Question 2: Do C2C Participants Show Greater Increases in Mental 

Health Care Utilization?

We assessed whether C2C increased mental health care utilization among persons with 

unmet need at baseline who completed follow-up surveys at either 6 or 12 months (83.5 percent 

of the retained baseline sample). Unmet need was defined as meeting the moderate or middle 

symptom threshold for one of the six mental health measures at baseline (which are described in 

greater detail in Chapter 8) without regard for recent or current mental health service utilization. 

We imputed utilization data for within-group change and between-group analysis for intervention 

effects. 

We first examined changes in utilization among persons with unmet need at baseline within 

each group with a binary (yes/no) indicator to compare the percentage of study participants who 

reported receiving the mental health service at baseline with the percentage who reported 

receiving the service over the 1-year period of follow-up, pooled for each group (Table 7.8). For 

the clinical outpatient and nonclinical settings, improvement in the outcome measures would be 

indicated by an increase in utilization from baseline to 12 months, because these questions are 

positively framed (e.g., calling a hotline for support or a referral). For the inpatient and 

emergency settings, improvement in the outcome measures would be indicated by a decrease in 

utilization, because these are questions on emergency sources of care and reflect accumulated 

unmet need or potential crises (e.g., stayed overnight in a hospital for emotional, mental health, 

alcohol, or drug problems). 

163 
 

 

 

 

 

 

 

 

 

 

 

 

 

Table 7.8. Within-Group Mental Health Care Utilization over 1 Year Among Persons with Need at

Baseline—Binary (Weighted)

Utilization Among Persons with Baseline 

Unmet Need (Imputed)

Baseline Percentage

Binary (Yes or No) Outcome (Retained Sample) 1 Year (%) Difference p-Value

Went to any outpatient MHPa,b C2C 37.68 48.33 +10.66 <.001c

Comparison 31.98 42.71 +10.73 0.001c

Used any nonclinical settings or C2C 40.39 36.32 −4.06 0.181

a,d resources Comparison 32.91 30.24 −2.66 0.558

Stayed in inpatient settinge C2C 32.54 8.86 −23.69 0.001c

Comparison 24.15 6.33 −17.82 0.005c

Used emergency settingsa,f C2C 41.82 47.93 +6.11 0.160

Comparison 40.66 50.48 +9.82 0.036c,g

a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. b Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. c Statistically significant at the p<.05 level. d Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or

attended religious/spiritual places, parks and recreation, or community centers for mental health needs. e One or more nights in residential treatment program for alcohol/drug problems (past 6 months) or overnight stay in a 

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up) f

Hospital emergency room or an urgent care facility for any health reason. g Estimate was not statistically significant at p <.05 after adjusting for multiple comparisons.

For clinical outpatient settings, both groups had statistically significant increases in 

utilization of about 11 percentage points over 1 year of follow-up. For nonclinical settings, 

there were no statistically significant within-group changes in utilization. For inpatient settings, 

both groups had statistically significant decreases in any utilization: a 24-percentage-point 

decrease for the C2C group and an 18-percentage-point decrease for the comparison group. This 

latter finding may reflect the different recall periods for baseline (lifetime) and pooled follow-up 

data (past 6 months). 

For emergency settings, only the comparison group had a statistically significant increase 

in utilization over 1 year (about 10 percentage points), but this decrease was not statistically 

significant after adjusting for multiple comparisons to account for the possibility that some 

results will achieve statistical significance simply by chance because of the large number of 

hypothesis tests. 

We also examined utilization with measures indicating the number of visits (outpatient) or 

nights (inpatient) (Table 7.9). For clinical outpatient settings, both groups had statistically 

significant increases in the number of mental health visits over the past year. The C2C group 

had approximately three additional visits, and the comparison group had approximately two 

additional visits. Neither group had a statistically significant change in the number of visits for 

inpatient settings or emergency settings. 

164 
 

 

 

 

 

 

 

 

 

 

 

 

 

Table 7.9. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Continuous (Weighted)

Within-Group Mean Change

over 1 Year (Imputed)

Continuous (No. of Times) Outcome N Estimate SE p-Value

Clinical outpatient settings

Number of times went to any outpatient MHPa C2C 498 2.98 0.876 0.001b

Comparison 258 2.47 1.00 0.011b

Inpatient settings

Number of nights in a residential treatment

program for alcohol or drug problemsa C2C

Comparison

498

258

−1.99

0.067

1.91

1.79

0.836

0.492

Emergency settings

Number of times went to a hospital, emergency

room, urgent care facility for any health reasona C2C

Comparison

498

258

0.215

0.497

0.258

0.308

0.215

0.065

NOTE: SE = standard error. a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. b Statistically significant at the p<.05 level.

We next examined the intervention effect over time using the same intent-to-treat approach 

described earlier, in which all individuals in the C2C unmet need group were compared with 

all those in the comparison unmet need group regardless of the actual intervention dosage 

(exposure) that the C2C group received. For continuous outcomes, we report standardized effect 

sizes. For binary outcomes, we report OR (which represents the likelihood of C2C participants 

indicating a given outcome), relative to the comparison group. An OR of 1.0 means that the groups 

were equally likely to report the outcome, and a 95% confidence interval (95% CI) that includes 

the value of 1.0 means that we cannot rule out that there is no group difference. We do not show 

model results when the sample size is less than ten participants per group for the propensity score 

weighted model and less than 20 per group for the doubly robust models (noted by gray shading 

in the tables). 

Over 1 year, there were no statistically significant intervention effects for utilization of 

services across all three settings (Table 7.10). The 95% CIs all include the value of 1.0, further 

indicating a lack of effect for these outcomes. 

We also examined the between-group changes for the three continuous measures of 

utilization among persons with baseline unmet needs (Table 7.11). There were no statistically 

significant intervention effects in the doubly robust models, where the standardized effect sizes 

were very small (Cohen’s d ranging from 0.002 to −0.11). 

165 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 7.10. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Binary (Weighted)

Group-Level Comparison of Utilization over 1 Year (Imputed)

Propensity Score Weighted 

Modela Doubly Robust Modelb

Binary (Yes or No) Outcome OR 95% CI p-Value OR 95% CI p-Value

Went to any outpatient MHPa,b 1.26 [0.54–2.91] 0.297 1.18 [0.46–2.95] 0.370

Used any nonclinical settings or

b,c resources

1.32 [0.54–3.20] 0.271 1.08 [0.41–2.82] 0.445

Stayed in inpatient settingd 1.48 [0.41–5.15] 0.278

Used emergency settingsa,e 0.91 [0.39–2.10] 0.594 0.79 [0.31–1.99] 0.696

NOTE: Areas in gray indicate that the sample size is too small to run models. a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. b Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. c Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended

religious/spiritual places, parks and recreation, or community centers for mental health needs. d One or more nights in residential treatment program for alcohol/drug problems (past 6 months) or overnight stay in a 

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up). e Hospital emergency room or an urgent care facility for any health reason.

Table 7.11. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Continuous (Weighted)

Group-Level Comparison of Utilization over 1 Year (Imputed)

Propensity Score Weighted Doubly Robust Standardized 

Modela Modelb Effect Size for

Doubly Robust

Continuous (No. of Times) Outcome Estimate SE p-Value Estimate SE p-Value Model

No. of times went to any outpatient 0.503 1.22 0.342 0.025 1.28 0.492 0.002

MHPc,d

No. of nights in a residential treatment −2.06 2.40 0.789 0.497 2.25 0.419 0.022

program for alcohol or drug problems

c

No. of times went to a hospital, −0.282 0.518 0.707 −0.343 0.466 0.769 −0.111

emergency room, urgent care facility for

any health reasonc

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. d Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors.

Secondary Research Question: Does Program Effectiveness Vary for the Different 

Target Populations or CBO Service Types?

We sought to understand whether C2C program effectiveness varied for different target 

populations (adults age 18 or older who are unemployed or underemployed, young adults ages 

16–24 who are not in school and are not employed, and parents/primary caregivers who are 

166 
 

 

 

 

 

 

 

 

 

 

expecting or who have children ages 4 and younger) or CBO service types (job training and 

employment program, youth development program). Note that these groups are not mutually 

exclusive. We examined barriers to mental health care and mental health care utilization using 

the same approach as used in the main analysis. 

Barriers to Mental Health Care

Changes over Time Within Each Group

For each subgroup, we first examined differences within each group over time. In this 

comparison, we considered the within-individual change between baseline and the follow-up 

time points and examined the average change and its direction across participants in each group. 

Across all subgroups, the C2C and comparison groups evidenced statistically significant decreases 

between baseline and 6 months, and between baseline and 12 months, in each of the barrier 

domains (see Appendix C, section C8.3). 

C2C Intervention Effects

For each subgroup, we then examined the intervention effect over time using the intent-to-treat 

approach described earlier. Below, we summarize the intervention effects across the subgroups 

with statistically significant findings at 12 months, all of which support of the effectiveness of 

C2C (see Appendix C, section C8.3). None of the other subgroup findings were statistically 

significant. At 6 months, there were no intervention effects for any of the barrier categories 

across any of the subgroups in either propensity score weighted or doubly robust models. 

CBO service type

• Youth development program: 

- Logistical barriers at 12 months had a smaller decrease in the C2C group (in 

the propensity score weighted model before adjusting for multiple comparisons). 

The standardized effect size in the nonsignificant doubly robust model was small 

(d = 0.15). 

C2C target population

• Adults age 18 or older who are unemployed or underemployed: 

- Internalized stigma at 12 months had a smaller decrease in the C2C group (in 

the doubly robust model before adjusting for multiple comparisons) with a small 

standardized effect size of d = 0.17. 

• Young adults ages 16 to 24 who are not in school and are not employed: 

- Attitudinal barriers at 12 months had a greater decrease in the C2C group (in 

the doubly robust model before adjusting for multiple comparisons) with a small 

standardized effect size of d = −0.28. 

167 
 

 

 

 

 

Mental Health Care Utilization

Changes over Time Within Each Group

For mental health care utilization for each of the subgroups, we used the pooled data 

combining 6- and 12-month utilization to examine differences within each group over time. 

Below, we summarize the statistically significant within-group differences at 12 months for 

mental health care utilization across the different subgroups (see Appendix C, section C8.3). 

None of the other subgroup findings were statistically significant. 

CBO service type

• Job training and employment program: 

- C2C participants decreased utilization of any nonclinical services and inpatient 

services and the number of nights in residential treatment, and increased the 

number of outpatient visits. 

- Comparison participants decreased utilization of any nonclinical services, before 

the multiple comparisons correction, and also decreased utilization of any 

inpatient services. 

• Youth development program: 

- C2C participants decreased utilization of any nonclinical services, before the 

multiple comparison adjustment. 

C2C target population

• Adults age 18 or older who are unemployed or underemployed: 

- C2C participants decreased utilization of any nonclinical services, inpatient 

services, and the number of residential treatment nights, and increased the 

number of outpatient visits. 

- Comparison participants decreased utilization of both nonclinical services and 

inpatient services. 

• Young adults ages 16 to 24 who are not in school and are not employed: 

- C2C participants decreased utilization of any outpatient services, nonclinical 

services, and before the multiple comparison adjustment, emergency setting use. 

- Comparison participants decreased utilization of any nonclinical services. 

• Parents/primary caregivers who are expecting or who have children up to age 4: 

- C2C participants decreased utilization of any inpatient services, emergency 

settings, number of visits to emergency settings, and before multiple comparisons 

adjustment, number of residential treatment nights. 

- Comparison participants decreased utilization of any nonclinical services before 

the multiple comparisons adjustment. 

C2C Intervention Effects

For each subgroup, we also examined the intervention effect over time using the intent-totreat approach described earlier. Below, we summarize the statistically significant intervention 

168 
 

 

 

 

 

 

 

 

effects across the subgroups at 12 months (see Appendix C, section C8.3). None of the other 

subgroup findings were statistically significant. 

CBO service type

• Job training and employment program: 

- C2C participants had a greater decrease in residential treatment nights (in the 

propensity score weighted model before the multiple comparisons adjustment). 

The standardized effect size for the nonsignificant doubly robust model was very 

small (d = −0.001). 

• Youth development program: 

- C2C participants were three times more likely to have used clinical outpatient 

services (OR = 3.10; 95% CI: 1.54–6.26) in the propensity score weighted model. 

C2C target population

• Adults age 18 or older who are unemployed or underemployed: 

- C2C participants had a greater decrease in residential treatment nights (in the 

propensity score weighted model before the multiple comparisons adjustment). 

The standardized effect size for the nonsignificant doubly robust model was very 

small (d = 0.02). 

• Young adults ages 16 to 24 who are not in school and are not employed: 

- C2C participants were half as likely to have used emergency settings (OR = 0.47, 

95% CI: 0.22–0.99) in the propensity score weighted model before the multiple 

comparisons adjustment. 

• Parents/primary caregivers who are expecting or who have children up to age 4: 

- C2C participants were one-third as likely to have used emergency settings 

(OR = 0.34; 95% CI: 0.16–0.72) in the propensity score weighted model. 

- C2C participants had a greater decrease in the number of emergency setting 

visits (in both propensity score weighted and doubly robust models). The 

standardized effect size for the doubly robust model was medium (d = −0.48). 

- C2C participants had a greater decrease in residential treatment nights (in the 

propensity score weighted model before the multiple comparisons adjustment). 

The standardized effect size for the nonsignificant doubly robust model was very 

small (d = −0.01). 

Discussion

For this segment of the impact evaluation, we tested hypotheses that there would be evidence 

of greater improvements among the C2C group than the comparison group on measures of access 

to and utilization of mental health services. Overall, we found few statistically significant group 

differences in barriers to care and utilization (Table 7.12). In the main analysis, there was one 

intervention effect for barriers to care at 12 months favoring the comparison group, an 

unexpected finding which is discussed below. 

169 
 

 

 

 

 

 

Table 7.12. Summary of Mental Health Care Barriers and Utilization Outcomes in the Overall

Sample

Outcome

Difference Within Groups

over Time Intervention Effect

Barriers to mental health care

Logistical barriers

Attitudinal barriers

Stigma barriers

Decrease, both groups

Decrease, both groups

Decrease, both groups

Smaller decrease in C2C group 

vs. comparison group (12 months

only)

Internalized stigma Decrease, both groups

Utilization—over 1 year

Binary (yes or no)

Outpatient MHP Increase, both groups

Nonclinical settings

Inpatient settingsa Decrease, both groups

Emergency settings Decrease in comparison group only

(only before MCC)

Continuous (no. of times)

Went to any outpatient MHP Increase, both groups

Nights in a residential alcohol/drug 

treatment program

Went to emergency settings for any

health reason

NOTE: MCC = multiple comparisons correction. a One indicator in this category assessed lifetime utilization at baseline, which may partially explain decreases seen 

at follow-up, which used a pooled recall period of the past year.

Our findings should be interpreted in light of the limitation that to be eligible for the study 

C2C clients needed to reach a very low mental health symptom threshold that was not a 

requirement for participating in C2C or for CBO services in general. Thus, the study sample 

is not representative of all clients served by C2C or by the comparison CBOs. Further, we 

encountered problems with the quality of C2C dosage data that prevented us from being able to 

examine whether C2C effect differed depending on the “amount” of C2C that participants 

received. The evaluation sought to include a diverse sample of participants across CBOs and 

across the implementation timeline. Although we used propensity weighting to help balance 

potential differences across groups in study enrollment, these results may not represent the 

effects of C2C on all CBO clients. 

Overall Group Findings

In this section, we discuss findings from the overall sample where we compared the entire 

C2C sample to the entire comparison sample on barriers to care and service utilization. 

170 
 

 

 

Barriers to Mental Health Care

At baseline, we found that internalized stigma (feeling embarrassed or ashamed) was the 

most strongly endorsed barrier to mental health care. Other key barriers identified by both groups 

were the cost of mental health care, not knowing where or how to seek services, and stigma from 

others. These barriers are also common in the general population among adults who felt they 

needed substance use treatment but had not received it in the past year (Substance Abuse and 

Mental Health Services Administration, 2019b). For example, among these adults, 42 percent 

identified cost as a barrier to their care, and one-quarter said they did not know where to go for 

care (Substance Abuse and Mental Health Services Administration, 2019b). Internalized stigma 

is also a prevalent barrier to care. For example, a review found that internalized stigma was more 

often associated with impeded treatment seeking behavior, compared with other forms of stigma 

(Clement et al., 2015). Other stigma barriers are also salient: across studies, concerns about 

treatment confidentiality are a salient barrier for one in three persons, and concerns about effects 

to employment are a barrier for one in four persons (Clement et al., 2015). Both the C2C and 

comparison groups endorsed similar concerns at baseline, for example, concerns over the 

reactions of family members, or harms to employment opportunities. In their efforts to reduce 

mental health stigma or counter beliefs that mental health problems tend to resolve naturally 

without treatment, C2C and other concurrent ThriveNYC initiatives may have driven the 

significant within-group decreases in attitudinal and stigma barriers we saw in both groups. 

Intervention Effects

There were no intervention effects at 6 months, but we did find a small but significant group 

difference in internalized stigma at 12 months. Specifically, although both groups’ internalized 

stigma levels decreased significantly over time, the C2C group appeared to do so at a slower rate. 

One explanation for this counterintuitive finding is that C2C did not affect internalized stigma as 

well as usual or other mental health programming at the comparison CBOs. As noted in the 

introduction chapters, the ThriveNYC initiative heavily invested in mental health programming 

throughout the city, and we heard from many comparison CBOs about ThriveNYC initiatives 

that were underway at their organization. It is possible that some of these other initiatives were 

more effective than C2C for decreasing internalized stigma. It is also worth noting that the 

comparison group’s greater decrease in internalized stigma did not appear to translate into more 

favorable changes in mental health care utilization patterns (discussed in detail in the next 

section). 

Overall, the findings suggest that C2C did not have an effect on reducing barriers to mental 

health care. Several factors may explain this lack of intervention effects, including additional 

mental health programming being implemented throughout NYC during the study period, 

anecdotal evidence of diffusion of the model beyond the C2C network (see Chapter Six), and 

the addition of mental health services by some comparison CBOs. 

171 
 

 

 

 

 

 

Mental Health Care Utilization

C2C participants reported more mental health care utilization than comparison group 

participants at baseline, including outpatient clinical services, nonclinical settings, and inpatient 

settings. About one-third of C2C participants with unmet need and one-quarter of comparison 

participants with unmet need had utilized outpatient mental health care in the past 6 months at 

baseline. 

These rates of outpatient utilization were lower than those found in a trial of depression 

collaborative care implementation approaches in low-income communities in Los Angeles, 

where about half of participants in either arm of the study had received care from an outpatient 

clinical MHP at baseline and each follow-up (Chung et al., 2014). In the general population, 

past-year treatment rates (defined as “seeing or talking to a health professional or taking 

prescribed medication”) were 65 percent for adults with depression and 43 percent for adults 

with any mental illness (Substance Abuse and Mental Health Services Administration, 2019a), 

and only about one in ten in a nationally representative sample of adults 18 years or older reports 

recent mental health treatment utilization (De Luca et al., 2016; Dhingra et al., 2011). Outpatient 

settings and prescription medication were the most common forms of this treatment for any 

mental illness (26 and 36 percent) (Substance Abuse and Mental Health Services Administration, 

2019b). 

One explanation for the lower-than-anticipated use of outpatient mental health services in the 

C2C evaluation may be the population itself: C2C was intentionally designed to serve communities 

without easy access to mental health care, and the participating CBOs were selected with that in 

mind (Chapter 3). Further, in this analysis we relied on self-report of utilization that may not 

align with other data sources; the rate of kept referrals to the MHPs reported by CBOs in their 

quarterly reports (Chapter 5) was higher than the self-report numbers of overall outpatient 

utilization. (It is important to note that the quarterly report data were based on the entire C2C 

client population of which this sample is not representative.) 

We also observed a very high prevalence of inpatient utilization at baseline (31 percent), 

although it is worth noting that the baseline version included any lifetime hospitalization for 

mental health needs, whereas the follow-up only included 1 year. Indeed, at 1-year follow-up, 

about 9 percent of C2C participants and 6 percent of comparison participants had stayed in an 

inpatient treatment setting in the past year. This is higher than national estimates, where about 

3 percent of adults with any mental illness report receiving inpatient mental health services in 

the past year (Substance Abuse and Mental Health Services Administration, 2019b). 

At baseline, we also found high rates of past 6-month emergency care utilization in this 

sample (43 percent). Because this question referred to “any health reason” (not only mental 

health needs), it is likely that this high utilization of emergency settings at baseline represents 

unmet health needs more generally, including both physical and mental health. This prevalence 

is dramatically higher than national estimates of past-year emergency department utilization 

(18 percent) in double the recall period of the C2C survey (Gindi, Black, and Cohen, 2016). 

172 
 

 

 

These patterns of utilization—low outpatient mental health care utilization and high inpatient 

and emergency department utilization—are not surprising, given C2C’s focus on underserved 

populations. Reflecting the three target populations served by C2C, the evaluation sample had 

high rates of unemployment and very low incomes. Use of emergency care settings is higher 

among populations of low socioeconomic status and persons with behavioral health needs 

(Giannouchos et al., 2019; Krieg et al., 2016; Schmidt et al., 2018). These utilization patterns are 

precisely what C2C aimed to reverse by identifying mental health problems early and helping 

people find and engage in care, therefore moving utilization into outpatient settings and away 

from emergency and inpatient settings. 

Intervention Effects

For mental health care utilization, we did not find any formal intervention effects in the main 

analysis, and effect sizes were small, suggesting that C2C did not lead to changes in mental 

health care utilization. Other explanations for the lack of intervention effects include concurrent 

ThriveNYC citywide initiatives that may have reached comparison participants, ThriveNYCrelated and nonrelated mental health services that some comparison CBOs began providing 

during the study, anecdotal evidence from the implementation study of diffusion of the model 

beyond the C2C network, and the null findings for barriers to care. 

Variation in C2C Effectiveness for Different Subgroups

We also examined within-group changes and intervention effects over time separately for the 

three C2C target populations and two of the CBO types. 

Barriers to Care—Subgroups

Across all subgroups, C2C had no effect on barriers to mental health care. This lack of 

statistically significant intervention effects may be explained in part by reductions in all four 

barriers types (logistical, attitudinal, stigma, and internalized stigma) for both C2C and comparison 

groups. However, before correcting for multiple comparisons, we did observe three trends that 

warrant discussion. 

First, among the youth and young adult target population, C2C clients had a stronger decrease 

in attitudinal barriers from baseline to 12 months relative to their comparison peers—a difference 

that was no longer statistically significant after adjusting for multiple comparisons. However, 

this finding is notable because the small-to-medium effect size (d = −0.28) was larger than most 

of the other group differences identified in this study and therefore potentially more meaningful. 

Second, among clients of youth development CBOs, logistical barriers at 12 months had a 

smaller decrease for C2C participants relative to comparison peers, a difference that was small in 

size (d = 0.15) and did not remain significant after adjusting for multiple comparisons. Third, 

among adults 18 and older who were unemployed or underemployed, internalized stigma at 

12 months had a smaller decrease among C2C participants, relative to comparison group peers. 

173 
 

 

 

 

 

 

 

 

This finding did not hold after adjusting for multiple comparisons and was small in size (d = 0.17), 

though it was consistent with a finding in the main analysis at 12 months, discussed earlier. 

Mental Health Care Utilization—Subgroups

Within-group (C2C or comparison) changes in utilization trended in the same direction as 

those observed in the overall sample. Thus, although the strength of within-group change varied 

across subgroups, the directionality was consistent (e.g., decreased or increased over time). For 

example, all significant changes in nonclinical service use were reductions in use over time. 

We found three intervention effects in either propensity score weighted or doubly robust 

models, and these findings held after the multiple comparisons adjustment. It is also especially 

notable that these differences emerged within much smaller sample sizes than the main analyses, 

with several effect sizes larger than those in the main analysis. Still, we caution that, because we 

did not impute missing data for subgroup analyses, it is possible that these significant effects in 

subgroups could be attributed to potential bias of the sample (e.g., if those who responded to the 

follow-up surveys were different from those who did not). 

First, C2C clients at youth development CBOs were three times more likely to have used 

outpatient clinical services than their comparison peers, a much larger effect than the 

(nonsignificant) one identified in the main analysis. This is consistent with our hypothesis that 

C2C would increase use of outpatient care, in large part through the formal relationships CBOs 

formed with MHPs through C2C. It is interesting that this finding emerged specifically in the 

youth development programs suggesting that the youth development programs that implemented 

C2C were uniquely effective in linking youth with need to outpatient mental health care. They 

may have had particularly strong relationships with their MHPs and on-site mental health 

services which facilitated these connections (see Appendix A). Staff at C2C CBOs also 

perceived that youth were potentially more amenable to mental health services and referrals 

(Chapter 6). As described earlier, youth may also have more adaptivity and malleability in their 

attitudes toward mental health and treatment seeking. 

We also found two intervention effects among parents and primary caregivers. Parents and 

caregivers in the C2C group were one-third less likely to have used emergency settings over 

1 year in the propensity score model. This group also had a greater reduction in the number of 

emergency visits relative to their comparison peers, even in the doubly robust model, giving 

credence to the strength of the finding. The effect size for this subgroup was medium 

(d = −0.48), which is four times the effect size found in the main analysis. These findings are 

consistent with our original hypothesis that C2C reduced emergency setting utilization. It is 

possible that C2C better addressed the unmet behavioral health needs of parents, resulting in 

fewer crises, less hypervigilance, and/or improved stress coping skills. In turn, this might reduce 

emergency setting visits for their own needs or the needs of their children. Another potential 

explanation is that parents and caregivers in C2C had some of their previously unmet health and 

mental health needs met through C2C programming and services, resulting in fewer emergency 

setting visits (as described earlier, the emergency setting indicator included all health reasons for 

174 
 

 

 

visits, not only mental health). It is also possible that, by nature of parenthood, this subgroup had 

access to a wider safety net of programs including those addressing the needs of children. For 

example, in NYC, undocumented pregnant women and children qualify for insurance benefits 

that other undocumented immigrants do not (City of New York, 2020). C2C skills may have 

fostered a relationship between CBO staff and clients that allowed CBOs to more easily identify 

and address a range of parents’ needs and/or refer them to other services to meet these needs, 

including but not limited to mental health needs. It is also important to note that the wording of 

the survey question for emergency setting visits (“how many different times did you go to 

a hospital emergency room or an urgent care facility for any health reason?”) may have 

inadvertently captured visits to emergency settings by parents/caregivers for their children’s 

needs; if C2C parents’ and caregivers’ child health needs were met directly or indirectly through 

C2C, then an accompanying reduction in the family unit’s visits to emergency is plausible. 

Limitations

There are limitations to consider in interpreting these findings. First, the evaluation sample is 

not representative of the entire C2C and comparison client populations for several reasons. Study 

participants represent only a fraction of those involved in C2C services across the C2C CBOs. 

Also, eligibility for the impact evaluation required that participants were experiencing at least 

mild mental health symptoms at baseline. Although we used thresholds below established 

screening thresholds to include a wider range of symptom severity, more than 80 percent of 

clients who completed an eligibility screening were eligible. Second, participating in the 

evaluation was optional for C2C and comparison clients, and among those who did participate, 

the retention rate was low overall (less than 50 percent). This means that there may be bias 

among those who opted to participate in the study or differences among those who continued 

with the study that hindered our ability to detect intervention effects. There may also have been 

barriers to retention in the study, including those related to homelessness and incarceration, that 

differed between the groups. Third, all data were self-reported and therefore are subject to social 

desirability and recall bias in responses. Although we attempted to obtain administrative data 

from New York State (NYS) and NYC agencies to obtain more objective estimates of mental 

health care utilization, bureaucratic and COVID-19-related delays prevented us from obtaining 

those data in time for them to be used in these analyses. It is possible that self-reported data are 

under- or overestimates of actual utilization. However, the evaluation employed data collectors 

with experience and training in sensitive data collection to try to minimize this bias. Fourth, it 

was not feasible for us to include measures of fidelity or C2C implementation, or to measure the 

types and quality of other care that participants were receiving in addition to C2C, including 

CBO programming as well as mental health and substance use treatment. The implementation 

timeline may also be a factor because full C2C implementation took up to 3 years whereas study 

enrollment began halfway through year 2. For comparison group participants, there was also 

possible contamination with ThriveNYC or other efforts to integrate mental health services into 

175 
CBO programming. We cannot determine whether the patterns of results we saw were influenced 

by these other types of services that many clients received. Finally, as intended, the evaluation 

sample included individuals with very low incomes; high rates of baseline unemployment, 

housing instability, and homelessness; and recent incarceration. These factors can present 

challenges for study retention, including competing demands (e.g., meeting basic needs for food 

and shelter), residential instability, and lack of continuity in phone service (Begun, Berger, and 

Otto-Salaj, 2018; Gul and Ali, 2010). However, those challenges may make this study even more 

impactful, by shedding light on how an innovative task-shifting intervention could meet some of 

the needs of this underserved and often marginalized population. 

Summary

In the overall sample, we did not find support for our hypothesis that C2C participants would 

report reduced barriers to care and greater changes in utilization patterns than comparison group 

participants. This suggests that C2C was not effective in reducing barriers to mental health care 

and increasing utilization of mental health care. This may be attributable to the concurrent 

ThriveNYC and non-ThriveNYC initiatives that were underway in NYC and at comparison 

CBOs during the C2C study, which could have contributed to the observed improvement in these 

domains within the comparison group. The improvement observed in both groups may also be 

related to the degree of engagement both groups had with their CBOs. A substantial proportion 

of clients across both groups reported using a range of CBO services at least weekly, lending 

further weight to the notion that one reason for the lack of intervention effects is that both C2C 

and comparison participants had ongoing relationships and frequent contact with their CBOs, 

which in and of itself may have helped address barriers to care and influenced utilization patterns. 

Although C2C may not have had an effect on reducing barriers to mental health care or 

increasing mental health care utilization in the overall sample, the subgroup findings suggest 

that C2C may have had a greater effect on certain subpopulations. For example, C2C appeared 

to have benefits for youth and young adults regarding attitudes about mental health symptoms 

and help seeking in ways that were not apparent for other target populations. Similarly, C2C 

appeared to reduce emergency service utilization for parents and caregivers but not for other 

subgroups. Finally, C2C in the context of youth development programs may be especially potent 

for increasing outpatient treatment utilization among these clients. 

Our evaluation was focused primarily on detecting any intervention effects; we provided 

hypotheses to aid interpretation but future research is needed to more precisely identify the 

mechanisms that underlie potential intervention effects. Further, this is the first study, to our 

knowledge, to examine the effect of a mental health task-shifting intervention on these outcomes 

in the United States. Future research, with less heterogeneous samples, is needed to more fully 

explore C2C’s effectiveness on reducing barriers to mental health care and increasing utilization 

of mental health care. 

176 
 

 

 

 

 

References

Alegria, M., K. Alvarez, R. Z. Ishikawa, K. DiMarzio, and S. McPeck, “Removing Obstacles 

to Eliminating Racial and Ethnic Disparities in Behavioral Health Care,” Health Affairs 

(Millwood), Vol. 35, No. 6, 2016, pp. 991–999. 

Ault-Brutus, A., and M. Alegria, “Racial/Ethnic Differences in Perceived Need for Mental 

Health Care and Disparities in Use of Care Among Those with Perceived Need in 

1990–1992 and 2001–2003,” Ethnic Health, Vol. 23, No. 2, 2018, pp. 142–157. 

Begun, A. L., L. K. Berger, and L. L. Otto-Salaj, Participant Recruitment and Retention in 

Intervention and Evaluation Research, New York, NY: Oxford University Press, 2018. 

Breslau, J., M. Cefalu, E. C. Wong, M. A. Burnam, G. P. Hunter, K. R. Florez, and R. L. Collins, 

“Racial/Ethnic Differences in Perception of Need for Mental Health Treatment in a 

US National Sample,” Social Psychiatry and Psychiatric Epidemiology, Vol. 52, No. 8, 

2017, pp. 929–937. 

Burnett-Zeigler, I., K. Zivin, K. Islam, and M. A. Ilgen, “Longitudinal Predictors of First Time 

Depression Treatment Utilization Among Adults with Depressive Disorders,” Social 

Psychiatry Psychiatric Epidemiology, Vol. 47, No. 10, 2012, pp. 1617–1625. 

Chung, B., M. Ong, S. L. Ettner, F. Jones, J. Gilmore, M. McCreary, C. Sherbourne, V. Ngo, 

P. Koegel, L. Tang, E. Dixon, J. Miranda, T. R. Belin, and K. B. Wells, “12-Month Outcomes 

of Community Engagement Versus Technical Assistance to Implement Depression 

Collaborative Care: A Partnered, Cluster, Randomized, Comparative Effectiveness Trial,” 

Annals of Internal Medicine, Vol. 161, No. 10, 2014, pp. S23–S34. 

Cipriani, A., T. A. Furukawa, G. Salanti, A. Chaimani, L. Z. Atkinson, Y. Ogawa, S. Levcht, 

H. G. Ruhe, E. H. Turner, J. P. T. Higgins, M. Egger, N. Takeshima, Y. Hayasaka, H. Imai, 

K. Shinohara, A. Tajika, J. P. A. Ioannidis, and J. R. Geddes, “Comparative efficacy and 

acceptability of 21 antidepressant drugs for the acute treatment of adults with major 

depressive disorder: A systematic review and network meta-analysis,” Lancet, Vol. 391, 

No. 10128, 2018, pp. 1357–1366. 

City of New York, Office of the Mayor, Report: Understanding New York City’s Mental Health 

Challenge, press release, New York, NY, 2015. As of September 9, 2020: 

https://www1.nyc.gov/assets/home/downloads/pdf/press-releases/2015/thriveNYC_white 

_paper.pdf 

City of New York, “Health Insurance Link: Find What Fits: Immigrants,” 2020. As of May 20, 

2020: 

https://www1.nyc.gov/site/ochia/find-what-fits/immigrants.page 

177 
 

 

 

 

 

 

 

Clement, S., E. Brohan, D. Jeffery, C. Henderson, S. L. Hatch, and G. Thornicroft, “Development 

and psychometric properties the Barriers to Access to Care Evaluation scale (BACE) related 

to people with mental ill health,” BMC Psychiatry, Vol. 12, 2012, p. 36. 

Clement, S., O. Schauman, T. Graham, F. Maggioni, S. Evans-Lacko, N. Bezborodovs, 

C. Morgan, N. Rusch, J. S. Brown, and G. Thornicroft, “What Is the Impact of Mental 

Health-Related Stigma on Help-Seeking? A Systematic Review of Quantitative and 

Qualitative Studies,” Psychological Medicine, Vol. 45, No. 1, 2015, pp. 11–27. 

Cohen, J. Statistical Power Analysis for the Behavioural Sciences, 2nd ed., Hillsdale, NJ: 

Erlbaum, 1988. 

De Luca, S. M., J. R. Blosnich, E. A. Hentschel, E. King, and S. Amen, “Mental Health Care 

Utilization: How Race, Ethnicity and Veteran Status Are Associated with Seeking Help,” 

Community Mental Health Journal, Vol. 52, No. 2, 2016, pp. 174–179. 

Dhingra, S. S., M. M. Zack, T. W. Strine, B. G. Druss, J. T. Berry, and L. S. Balluz, 

“Psychological Distress Severity of Adults Reporting Receipt of Treatment for Mental 

Health Problems in the BRFSS,” Psychiatric Services, Vol. 62, No. 4, 2011, pp. 396–403. 

Giannouchos, T. V., H. C. Kum, M. J. Foster, and R. L. Ohsfeldt, “Characteristics and Predictors 

of Adult Frequent Emergency Department Users in the United States: A Systematic 

Literature Review,” Journal of Evaluation in Clinical Practice, Vol. 25, No. 3, 2019, 

pp. 420–433. 

Gindi, R. M., L. I. Black, and R. A. Cohen, Reasons for Emergency Room Use Among U.S. Adults 

Aged 18–64: National Health Interview Survey, 2013 and 2014, Hyattsville, MD: National 

Center for Health Statistics, 2016. As of September 9, 2020: 

https://www.cdc.gov/nchs/data/nhsr/nhsr090.pdf 

Gul, R. B., and P. A. Ali, “Clinical Trials: the Challenge of Recruitment and Retention of 

Participants,” Journal of Clinical Nursing, Vol. 19, No. 1–2, 2010, pp. 227–233. 

Gulliver, A., K. M. Griffiths, and H. Christensen, “Perceived Barriers and Facilitators to Mental 

Health Help-Seeking in Young People: A Systematic Review,” BMC Psychiatry, Vol. 10, 

2010, p. 113. 

Hadfield, H., and A. Wittkowski, “Women’s Experiences of Seeking and Receiving 

Psychological and Psychosocial Interventions for Postpartum Depression: A Systematic 

Review and Thematic Synthesis of the Qualitative Literature,” Journal of Midwifery and 

Women’s Health, Vol. 62, No. 6, 2017, pp. 723–736. 

Hahm, H. C., B. L. Cook, A. Ault-Brutus, and M. Alegria, “Intersection of Race-Ethnicity and 

Gender in Depression Care: Screening, Access, and Minimally Adequate Treatment,” 

Psychiatric Services, Vol. 66, No. 3, 2015, pp. 258–264. 

178 
 

 

 

 

 

 

 

 

 

 

Krieg, C., C. Hudon, M. C. Chouinard, and I. Dufour, “Individual Predictors of Frequent 

Emergency Department Use: A Scoping Review,” BMC Health Services Research, Vol. 16, 

No. 1, 2016, p. 594. 

Lazear, K. J., S. A. Pires, M. R. Isaacs, P. Chaulk, and L. Huang, “Depression Among LowIncome Women of Color: Qualitative Findings from Cross-Cultural Focus Groups,” Journal 

of Immigrant Minority Health, Vol. 10, No. 2, 2008, pp. 127–133. 

Lee, N., L. Jenner, A. Baker, A. Ritter, L. Hides, J. Norman, F. Kay-Lambkin, K. Hall, F. Dann, 

and J. Cameron, “Screening and Intervention for Mental Health Problems in Alcohol and 

Other Drug Settings: Can Training Change Practitioner Behaviour?” Drugs Education, 

Prevention and Policy, Vol. 18, No. 2, 2010, pp. 157–160. 

Lincoln, A. K., J. M. Liebschutz, M. Chernoff, D. Nguyen, and H. Amaro, “Brief Screening for 

Co-Occurring Disorders Among Women Entering Substance Abuse Treatment,” Substance 

Abuse Treat Prevention and Policy, Vol. 1, 2006, p. 26. 

Lopez-Lopez, J. A., S. R. Davies, D. M. Caldwell, R. Churchill, T. J. Peters, D. Tallon, 

S. Dawson, Q. Wu, J. Li, A. Taylor, G. Lewis, D. S. Kessler, N. Wiles, and N. J. Welton, 

“The Process and Delivery of CBT for Depression in Adults: A Systematic Review and 

Network Meta-Analysis,” Psychological Medicine, Vol. 49, No. 12, 2019, pp. 1937–1947. 

Lubman, D. I., L. Hides, A. Scaffidi, K. Elkins, M. Stevens, and R. Marks, “Implementing 

Mental Health Screening Within a Youth Alcohol and Other Drugs Service,” Mental Health 

Substance Use, Vol. 1, No. 3, 2008, pp. 254–261. 

Petrenko, C. L., S. E. Culhane, E. F. Garrido, and H. N. Taussig, “Do Youth in Out-of-Home 

Care Receive Recommended Mental Health and Educational Services Following Screening 

Evaluations?” Child Youth Services Review, Vol. 33, No. 10, 2011, pp. 1911–1918. 

Priester, M. A., T. Browne, A. Iachini, S. Clone, D. DeHart, and K. D. Seay, “Treatment Access 

Barriers and Disparities Among Individuals with Co-Occurring Mental Health and Substance 

Use Disorders: An Integrative Literature Review,” Journal of Substance Abuse and Treatment, 

Vol. 61, 2016, pp. 47–59. 

Santiago, C. D., S. Kaltman, and J. Miranda, “Poverty and Mental Health: How Do Low-Income 

Adults and Children Fare in Psychotherapy?” Journal of Clinical Psychology, Vol. 69, No. 2, 

2013, pp. 115–126. 

Schmidt, E. M., S. Behar, A. Barrera, M. Cordova, and L. Beckum, “Potentially Preventable 

Medical Hospitalizations and Emergency Department Visits by the Behavioral Health 

Population,” Journal of Behavioral Health Services & Research, Vol. 45, No. 3, 2018, 

pp. 370–388. 

179 
 

 

 

Schnyder, N., R. Panczak, N. Groth, and F. Schultze-Lutter, “Association Between Mental 

Health-Related Stigma and Active Help-Seeking: Systematic Review and Meta-Analysis,” 

British Journal of Psychiatry, Vol. 210, No. 4, 2017, pp. 261–268. 

Shippee, N. D., B. H. Rosen, K. B. Angstman, M. E. Fuentes, R. S. DeJesus, S. M. Bruce, and 

M. D. Williams, “Baseline Screening Tools as Indicators for Symptom Outcomes and Health 

Services Utilization in a Collaborative Care Model for Depression in Primary Care: A 

Practice-Based Observational Study,” General Hospital Psychiatry, Vol. 36, No. 6, 2014, 

pp. 563–569. 

Siu, A. L., U. S. Preventive Services Task Force, K. Bibbins-Domingo, D. C. Grossman, L. C. 

Baumann, K. W. Davidson, M. Ebell, F. A. Garcia, M. Gillman, J. Herzstein, A. R. Kemper, 

A. H. Krist, A. E. Kurth, D. K. Owens, W. R. Phillips, M. G. Phipps, and M. P. Pignone, 

“Screening for depression in Adults: US Preventive Services Task Force Recommendation 

Statement,” JAMA, Vol. 315, No. 4, 2016, pp. 380–387. 

Substance Abuse and Mental Health Services Administration, Key Substance Use and Mental 

Health Indicators in the United States: Results from the 2018 National Survey on Drug Use 

and Health, Rockville, MD, HHS Publication No. PEP19-5068, NSDUH Series H-54, 2019a. 

As of September 9, 2020: 

https://www.samhsa.gov/data/sites/default/files/cbhsq-reports/NSDUHNationalFindings 

Report2018/NSDUHNationalFindingsReport2018.pdf 

———, Results from the 2018 National Survey on Drug Use and Health: Detailed Tables, 

Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and 

Mental Health Services Administration, 2019b. As of September 9, 2020: 

https://www.samhsa.gov/data/report/2018-nsduh-detailed-tables 

Thomas, A. C., and P. K. Staiger, “Introducing Mental Health and Substance Use Screening into 

a Community-Based Health Service in Australia: Usefulness and Implications for Service 

Change,” Health & Social Care in the Community, Vol. 20, No. 6, 2012, pp. 635–644. 

180 
 

 

 

 

8. Impact of C2C on Mental Health Symptoms

Dana Schultz, Daniel Siconolfi, Lynsay Ayer, Joshua Snoke, and Elie Ohana 

Key Findings

• From baseline to follow-up, both C2C participants and those in the comparison group 

experienced statistically significant decreases in symptoms across all of the mental health 

conditions. These across-the-board symptom reductions may reflect a natural recovery 

process and the receipt of “usual” or other ThriveNYC services in the comparison group. 

• At 12 months, C2C did not result in greater improvements in any of the mental health 

symptoms compared with usual CBO services. At 6 months, there was evidence that C2C 

may have been more effective in decreasing substance use symptoms, although the effect 

size was very small. 

• C2C had no effect on mental health symptoms in the overall sample. 

• There was some evidence that C2C may have been more effective for certain subgroups, 

showing an effect in the following areas: 

- reduced alcohol use at job training and employment CBOs (with a small effect size) 

- decreased depressive symptoms at youth development CBOs (with a small-tomedium effect size) 

- decreased psychological distress for youth and young adults (with a small-to-medium 

effect size). 

181 
 

 

 

 

Introduction

By improving access to and utilization of mental health care services, C2C aimed to reduce 

symptoms of mental health problems among its participants. Thus, building on the access and 

utilization analyses described in Chapter 7, this chapter investigates whether C2C improved 

mental health, functioning, and well-being among New Yorkers seeking services from CBOs by 

examining their depression, generalized anxiety, PTSD, alcohol use, other substance use, and 

general psychological distress. For this segment of the impact evaluation, we expected to find 

that C2C participants would report greater reductions in symptoms across all of the mental health 

outcome measures, relative to comparison participants. 

In this chapter, we examine mental health symptoms over time to understand the nature of 

any changes in symptom status and the effectiveness of C2C skills on different mental health 

symptoms. Because of the wide variability in types of CBOs participating, and thus in the clients 

they serve, we examine effects among the entire sample of participants as well as effects for 

specific populations or settings within the sample. 

Background

Prior reports on the prevalence of mental health symptoms and diagnoses support our earlier 

assertion that there are disparities in mental health problems among different populations in the 

United States. At a macro level, such disparities may not be easy to detect. For instance, an 

estimated 18.3 percent of adults nationally, and a similar percentage (17.6 percent) of adults in 

NYS, have had one or more mental illness in the past year (Center for Behavioral Health 

Statistics and Quality, 2018; Substance Abuse and Mental Health Services Administration, 

2018). But closer inspection by locality or population segment may tell a different story. In 

NYC, for instance, the estimate of past-year mental illness exceeded 20 percent (Belkin et al., 

2016). 

The availability and granularity of prevalence data for specific mental health issues varies 

widely: 

• Estimates of the prevalence of mood disorders, such as depression, range from 9.5 percent 

in the general population to 20 percent among low-income populations (Muntaner et al., 

1998). It is estimated that more than half a million adult New Yorkers have depression at 

any given time, representing about 8 percent of the city’s adult population (Belkin et al., 

2016; McCray, Buery, and Bassett, 2015). 

• An estimated 18 percent of the general population has anxiety disorders such as generalized 

anxiety disorder (GAD) (Kessler et al., 2012). 

• Estimates of PTSD range from 3 to 4 percent with rates far higher for disadvantaged 

groups, such as Black Americans with a history of incarceration (Anderson, Geier, and 

Cahill, 2016; Kessler et al., 2005). 

• Nationally, past-year alcohol use disorder is estimated at 10 percent for 18 to 25 year 

olds and 5 percent for adults age 26 or older (SAMHSA, 2019). Among New Yorkers, 

182 
 

 

estimates of heavy or binge drinking in the last month range from 20 to 29 percent 

(Capua, Tuazon, and Paone, 2016). 

• According to the latest national estimate, about 8 percent of young adults ages 18 to 25 

and 2 percent of adults ages 26 or older had past-year illicit drug use disorder (SAMHSA, 

2019). In NYC, it has been estimated that about 10 percent of adults used illicit drugs in 

the last month (SAMHSA, 2019). Drug overdose deaths have leveled off at about 20 per 

100,000 over the last few years (Nolan et al., 2019). 

Mental health problems such as depression, anxiety, and PTSD often co-occur with alcohol 

and drug use particularly among young adults between the ages of 18 and 25 (Mericle et al., 

2012). The high rates of co-occurring disorders often result when individuals with symptoms of 

mental health issues turn to drugs and alcohol to help them cope with negative emotions (e.g., 

anger, sadness, anxiety, fear), leading to drug and alcohol addiction for some of these individuals 

(Metrik et al., 2016). Nationally, estimates of co-occurring disorders range from 3.7 to 11.8 percent 

of adults (Kessler et al., 2005; SAMHSA, 2019). 

Despite the prevalence of mental health problems, many individuals, particularly those from 

low-income, disadvantaged populations, do not receive treatment. Evidence-based interventions 

such as cognitive behavioral therapy (CBT), are generally effective in treating a wide range of 

mental health problems, including depression, anxiety, and PTSD (Bisson et al., 2007; Hofmann 

and Smits, 2008; Miranda et al., 2003; Roy-Byrne et al., 2010; Shear et al., 2005). Unfortunately, 

these interventions are not often accessible, particularly in community-based settings and for 

low-income, disadvantaged populations such as the groups targeted by the C2C program. 

Further, historically underserved individuals, including racial/ethnic minorities and other 

socioeconomically disadvantaged groups, are much less likely to utilize treatment for mental 

health issues for reasons such as perceived discrimination or perceived negative experiences, 

which may contribute to or exacerbate mental health problems and other disparities, such as 

economic and educational inequalities (Kataoka, Zhang, and Wells, 2002). 

C2C was envisioned as an approach to address the unmet need for mental health services in 

NYC by integrating mental health skills into the work of CBOs working with individuals who 

may be the least likely to have access to mental health support. Although the specific combined 

package of the four C2C skills has not been studied previously, the individual skills (screening 

for detection of mental health conditions, MI, MHFA, and PE) have been examined to determine 

their effectiveness on a range of outcomes (see Chapter 2 for more details). 

Evidence on the effectiveness of these four skills for improving mental health outcomes in 

community settings is limited, though this evaluation will add to the evidence base. When the 

C2C skills are delivered individually or in combination in community-based settings, the size of 

their effect on mental health symptoms would be expected to be very small, whereas referral to 

MHPs and services delivering evidence-based treatments would likely produce larger effects 

because these interventions provide more intensive focus on symptom relief. The “light touch” 

nature of the four C2C skills, possible variations in the quality of delivery, and relative 

183 
 

 

 

 

disadvantage of the individuals being served all contribute to this estimated small effect size. 

Further, small effect sizes like those expected here require very large sample sizes, which was 

not feasible in this community-based study. In addition, we should note that, because of the large 

citywide Thrive NYC mental health initiative, many New Yorkers have been exposed to mental 

health services and supports in recent years, beyond what is typical (City of New York, 2018). 

Therefore, even in this quasi-experimental study, we knew from the start that would be challenging 

to identify improvements in mental health outcomes attributable solely to C2C skills integrated at 

C2C CBOs. 

Research Questions

At the individual level, C2C aimed to assess whether integration of the C2C skills into CBO 

workflows could improve mental health outcomes for participants. For the primary research 

question of whether C2C participants showed greater positive improvement in depression, 

generalized anxiety, PTSD, alcohol use, substance use, and general psychological distress 

relative to comparison group participants over time, we tested whether C2C participants had 

• fewer mental health symptoms (depression, generalized anxiety, PTSD, psychological 

distress) 

• lower levels of alcohol and substance use 

• clinically significant improvements or reliable decreases in mental health symptoms. 

We also examined one secondary research question related to mental health symptoms: 

• Does program effectiveness vary for the specific populations targeted by C2C (i.e., adults 

age 18 or older who are unemployed or underemployed, young adults ages 16–24 who 

are not in school and are not employed, and parents/primary caregivers who are expecting 

or who have children up to the age of 4) or CBO service types (e.g., job training and 

employment programs, youth development programs)? 

Methods

Information on data collection and data analyses was provided in the methods section of 

Chapter 7. Below, we provide only information on data collection and analyses that differs from 

what was presented in Chapter 7. See Appendix C for complete details. 

Data Collection

The specific mental health outcome measures for these analyses included self-reported 

measures of depression (the eight-item Patient Health Questionnaire [PHQ-8]) (Kroenke et al., 

2010), generalized anxiety (Generalized Anxiety Disorder 7-Item Scale [GAD-7]) (Spitzer et al., 

2006), PTSD (PTSD Checklist for Diagnostic and Statistical Manual of Mental Disorders, Fifth 

Edition [PCL-5]) (Blevins et al., 2015), alcohol use (AUDIT) (Berner et al., 2007), and substance 

184 
 

 

 

 

 

 

 

 

 

 

 

 

use (DAST) (Yudko, Lozhkina, and Fouts, 2007) on the study eligibility screener, and general 

psychological distress (K6) (Kessler et al., 2005) on the baseline survey. 

Data Analyses

In addition to examining differences within groups and intervention effects over time, we 

also examined whether C2C resulted in clinically significant or reliable improvements in mental 

health outcome measures, controlling for the outcome at baseline. Table 8.1 summarizes how we 

defined whether there was clinically significant or reliable improvement for each measure using 

improvement thresholds found in the literature. 

Table 8.1. Clinically Significant and Reliable Improvement for Mental Health Outcomes

Clinically Significant Clinically Reliable 

Mental Health Outcome Improvement Improvement Citation(s)

Depression 5-point or greater decrease in 

score

Generalized anxiety 10-point or greater decrease in 

score

PTSD 10-point or greater decrease in 

score

Alcohol misuse Moving from above threshold 

for positive screen (≥8) to

below threshold for positive 

screen (<8)

Substance misuse Moving from above threshold 

for positive screen (≥3) to

below threshold for positive 

screen (<3)

Psychological distress Moving from above threshold 

for positive screen (≥13) to

below threshold for positive 

screen (<13)

Not applicable

5-point or greater

decrease in score

5-point or greater

decrease in score

Not applicable

Not applicable

Moving from above 

threshold for positive

screen (≥5) to below

threshold for positive

screen (<5)

Kroenke et al., 2010; 

Kroenke and Spitzer, 

2002; Löwe et al., 2004; 

Wells et al., 2013

Richards and Borglin, 

2011; Spitzer et al., 

2006

National Center for

PTSD, 2019

Prochaska et al., 2012

Results

Among this population of individuals seeking services at a range of CBOs in NYC, clinically 

relevant mental health symptoms were quite prevalent, based on established scoring thresholds 

(Table 8.2). As noted earlier, the study eligibility screener included measures of five common 

mental health conditions (depression, generalized anxiety, PTSD, alcohol use, substance use), 

with study eligibility determined by meeting a minimum threshold on one or more, and the 

baseline survey assessed general psychological distress. Together, these six mental health 

conditions are the focal outcomes in this chapter. For baseline descriptive purposes, we stratified 

185 
 

 

 

 

 

the sample according to the commonly used and established symptom thresholds for each 

measure (Table 8.2). 

Table 8.2. Baseline Mental Health Symptoms (Weighted)

Mental Health Symptoms Full Sample C2C Group Comparison Group p-Value

Mean (Range) Mean (Range) Mean (Range)

Depression 8.4 (0–24) 7.9 (0–24) 9.0 (0–24) <.001a

Percent Percent Percent

None—minimal depression 26.6 33.8 18.3 <.001a

Mild depression 35.4 30.4 41.1

Moderate depression 22.7 22.2 23.2

Moderately severe depression 11.4 8.9 14.3

Severe depression 3.9 4.6 3.1

Mean (Range) Mean (Range) Mean (Range)

Generalized anxiety 8.4 (0–21) 8.2 (0–21) 8.5 (0–21) 0.251

Percent Percent Percent

Mild 61.3 62.2 60.3 0.192

Moderate 23.2 21.3 25.3

Severe 15.5 16.5 14.4

Mean (Range) Mean (Range) Mean (Range)

PTSD 28.73 (0–80) 28.05 (0–80) 29.5 (0–78) 0.148

Percent Percent Percent

Percentage above the cut point 47.6 (0–80) 47.3 (0–80) 47.9 (0–78) 0.820

Mean (Range) Mean (Range) Mean (Range)

Alcohol use 4.62 (0–40) 4.78 (0–40) 4.43 (0–34) 0.379

Percent Percent Percent

Below either threshold 79.5 79.2 79.9 0.774

Harmful or hazardous drinking 10.4 10.2 10.6

Alcohol dependence 10.1 10.7 9.4

Mean (Range) Mean (Range) Mean (Range)

Substance use 1.71 (0–10) 1.89 (0–10) 1.51 (0–10) 0.009a

Percent Percent Percent

Below any threshold 38.4 35.6 41.6 0.093

Low 37.3 37.2 37.5

Intermediate 15.9 16.8 14.8

Substantial 6.1 7.7 4.4

Severe 2.2 2.7 1.7

Overall no. of positive screens,

across screening measures

Mean (Range)

1.64 (0–5)

Mean (Range)

1.63 (0–5)

Mean (Range)

1.64 (0–5) 0.899

Percent Percent Percent

One positive screen 20.3 21.7 18.6 0.209

Two positive screens 14.9 16.2 13.4

Three positive screens 18.7 18.9 18.3

Four positive screens 8.3 8.1 8.5

Five positive screens 4.9 4.0 5.9

186 
 

 

 

 

 

 

 

Mental Health Symptoms Full Sample C2C Group Comparison Group p-Value

Psychological distress Mean (Range) Mean (Range) Mean (Range)

8.47 (0–24) 8.66 (0–24) 8.26 (0–24) 0.211

Percentage with severe mental

illness

24.0 25.4 22.5 0.482

Percentage with moderate 

psychological distress

47.0 45.8 48.3

Co-occurring disorders

Percentage with positive mental

health and alcohol/substance use 

screen

23.7 22.5 25.0 0.307

a Statistically significant at the p<.05 level.

At baseline, more than one-third (38 percent) of the overall sample reported depressive 

symptoms in the moderate-to-severe range, and an additional 35.4 percent reported mild 

depressive symptoms. Overall, nearly three-quarters (73.4 percent) of the sample had at least 

mild depressive symptoms at the time of their initial study screening, with the comparison group 

experiencing significantly more depressive symptoms at baseline than the C2C group. For 

generalized anxiety, more than one-third (38.7 percent) reported moderate-to-severe anxiety 

symptoms. Almost half (47.6 percent) of the study participants scored higher than the suggested 

cut score on the measure of PTSD symptoms. For alcohol use, 10.4 percent reported alcohol use 

that indicated alcohol dependence, whereas an additional 10.1 percent reported harmful or 

hazardous drinking behavior. For other substance use, 8.3 percent of study participants reported 

severe or substantial substance use, whereas 15.9 percent reported behavior that indicated an 

intermediate level of substance use. The C2C group had a statistically significant higher mean 

score on the substance abuse measure than the comparison group (1.89 vs. 1.51). Overall, 

20.3 percent of study participants reported symptoms above the threshold for only one condition, 

and nearly one-third (31.9 percent) reported symptoms above the thresholds for three conditions 

or more. For psychological distress, 24 percent of the sample reported symptoms above the 

threshold for severe mental illness and 47 percent reported symptoms above the threshold for 

moderate psychological distress. With regard to co-occurring mental health and alcohol/ 

substance use disorders, about one in four persons screened positive on both a mental health 

and alcohol or substance use measure, and there was no difference by group. 

Primary Research Question: Do C2C Participants Show Greater Improvement in Mental 

Health Symptoms Relative to Comparison Group Participants?

To understand whether C2C improved mental health symptoms, we first examined differences 

within groups over time. The columns in Table 8.3 describe differences within groups between 

the baseline and 12-month assessment by comparing each person’s score at 12 months with his 

or her own score at baseline. From baseline to 12 months, both groups experienced statistically 

significant improvements in scores for all of the mental health outcomes, with both groups 

showing decreases in symptoms over this time period. The magnitude of the change depends on 

187 
 

 

 

 

 

 

 

 

Table 8.3. Within-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Depression (score range 0–24)

Generalized anxiety (score range 0–21)

PTSD (score range 0–80)

Alcohol use (score range 0–40)

Substance use (score range 0–10)

Psychological distress (score range 0–24)

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

457

265

455

265

455

265

449

264

447

260

444

261

–2.22

–2.59

–2.66

–2.36

–10.63

–10.94

–1.66

–1.73

–1.09

–0.815

–3.29

–2.91

0.286

0.330

0.286

0.328

0.885

1.07

0.271

0.350

0.101

0.124

0.267

0.375

<.001b

<.001b

<.001b

<.001b

<.001b

<.001b

<.001b

<.001b

<.001b

<.001b

<.001b

<.001b

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately. b Statistically significant at the p<.05 level.

the scoring range for each measure. For example, an estimate of –2.22 for the C2C group on the 

depression measure indicates that the score decreased by about 2.22 points on the 24-point scale. 

As described below, a 10-point decrease on this depression measure is considered to be a clinically 

significant improvement. The 6-month results were similar with smaller, but still statistically 

significant, improvements in scores across all mental health outcomes (see Appendix C, 

section C8.1). 

We next examined intervention effects over time using an intent-to-treat approach, in which 

all individuals in the C2C group were compared with all those in the comparison group, regardless 

of the actual amount of intervention that the C2C group received. With both groups improving 

over time on all of the mental health outcomes, we did not see any statistically significant 

intervention effects at 12 months (Table 8.4). As illustrated in Table 8.4, all 12-month effect 

sizes were very small (Cohen’s d ranging from 0.04 to −0.03). At 6 months, the intervention 

effect for substance use was statistically significant in the propensity score weighted model but 

not the doubly robust model (see Appendix C, section C8.1), such that C2C participants had a 

stronger decrease than comparison participants. The doubly robust standardized effect size was 

very small (d = −0.08). 

Finally, we examined whether C2C participants had more clinically significant or reliable 

decreases in mental health symptoms over time, relative to the comparison group. For these 

analyses, we used thresholds for change established in the research literature for each measure 

(Table 8.1). Specifically, we examined clinically significant improvement as indicated by a 

10-point or greater decrease in scores, and reliable improvement as indicated by a 5-point or 

greater decrease in scores or going from above the threshold for a positive screen to below the 

188 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 8.4. Between-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted

Modela

Estimate SE p-Value

Doubly Robust Modelb

Estimate SE p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Depression 0.374 0.437 0.392 –0.108 0.362 0.765 –0.019

Generalized anxiety –0.304 0.435 0.485 –0.190 0.391 0.627 –0.033

PTSD 0.303 1.39 0.827 0.651 1.14 0.567 0.036

Alcohol use 0.073 0.443 0.869 –0.037 0.300 0.901 –0.007

Substance use –0.279 0.160 0.082 –0.057 0.107 0.598 –0.027

Distress –0.379 0.460 0.410 –0.154 0.373 0.680 –0.027

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

threshold (note: there is not a reliable change threshold for the PHQ-8, our depression measure, 

because we were unable to find such a threshold in the scientific literature). Across the different 

mental health outcomes, there were no statistically significant differences between groups on 

these measures at 12 months, although we did see a marginally significant trend favoring the 

C2C group for clinically significant improvement in generalized anxiety (Table 8.5). At 6 months, 

Table 8.5. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 12 Months

Group-Level Comparison of

Percent with Change

Percent with

Outcome N Change p-Value

Depression (clinically significant change of 10-point or C2C 457

greater decrease in scores) Comparison 265

Generalized anxiety (clinically significant change of 10- C2C 455

point or greater decrease in scores) Comparison 265

Generalized anxiety (reliable change of 5-point or greater C2C 455

decrease in scores) Comparison 265

PTSD (clinically significant change of 10-point or greater C2C 455

decrease in scores) Comparison 265

PTSD (reliable change of 5-point or greater decrease in C2C 455

scores) Comparison 265

Alcohol use (change to below positive screen threshold) C2C 449

Comparison 264

Substance use (change to below positive screen threshold) C2C 447

Comparison 260

Psychological distress (change to below threshold for C2C 444

serious mental illness) Comparison 261

Psychological distress (change to below threshold for C2C 444

moderate distress) Comparison 261

32.93 0.920

33.32

13.36 0.058

8.46

33.33 0.717

31.95

47.30 0.363

51.04

62.86 0.757

61.62

13.73 0.379

11.23

19.88 0.296

16.35

25.61 0.809

24.74

71.67 0.611

69.78

189 
 

 

 

 

 

we found one statistically significant difference for substance use (see Appendix C, section C8.1). 

Although this result was no longer significant after adjusting for multiple comparisons, the trend 

suggested that the C2C group was more likely than the comparison group to move from above 

the threshold for a positive screen to below the threshold. Overall, at 6 months, 16.4 percent of 

the C2C group moved from above to below the threshold for a positive screen, compared with 

9.6 percent of the comparison group. 

Secondary Research Question: Does Program Effectiveness Vary for the Different 

Target Populations or CBO Service Types?

We sought to understand whether C2C program effectiveness varied for different target 

populations (e.g., adults age 18 or older who are unemployed or underemployed, young adults 

ages 16–24 who are not in school and are not employed, and parents/primary caregivers who are 

expecting or who have children ages 4 and younger) or CBO service types (e.g., job training and 

employment program, youth development program). Note that these groups are not mutually 

exclusive. We examined the mental health outcomes using the same approach as used in the 

main analysis. 

Changes over Time Within Each Group

For each subgroup, we first examined differences within each group over time. In this 

comparison, we considered the within-individual change between baseline and the follow-up 

time points and determined the average change and its direction across participants in each 

group. Across almost all of the subgroups, the C2C and comparison groups indicated statistically 

significant decreases between baseline and 12 months for each of the mental health symptoms 

(see Appendix C, section C8.3). The two exceptions were alcohol use, which did not significantly 

change over time for C2C participants in youth development programs or for comparison group 

participants in the young adult target population. 

C2C Intervention Effects

For each subgroup, we also examined the intervention effect over time using the intent-totreat approach described earlier. Below, we summarize the statistically significant intervention 

effect findings across the subgroups at 12 months (see Appendix C, section C8.3). None of the 

other subgroup findings were statistically significant. 

CBO service type

• Job training and employment program: 

- Depressive symptoms at 12 months had a smaller decrease in the C2C group (in 

the propensity score weighted model before adjusting for multiple comparisons). 

The intervention effect was not statistically significant in the doubly robust 

model, where the standardized effect size was very small (d = 0.04). 

190 
 

 

 

 

 

 

 

 

- Alcohol use at 12 months had a greater decrease in the C2C group (in the doubly 

robust model before adjusting for multiple comparisons). The standardized effect 

size was small (d = −0.17). 

• Youth development program: 

- Depressive symptoms at 12 months had a greater decrease in the C2C group (in 

both the propensity score weighted and doubly robust models before adjusting for 

multiple comparisons). The standardized effect size in the doubly robust model 

was small to medium (d = −0.30). 

- Alcohol use at 12 months had a smaller decrease in the C2C group (in the 

propensity score weighted model before adjusting for multiple comparisons). The 

intervention effect was not statistically significant in the doubly robust model, 

where the standardized effect size was very small (d = 0.08). 

C2C target population

• Young adults ages 16 to 24 who are not in school and are not employed: 

- Psychological distress at 12 months had a greater decrease in the C2C group 

(in the doubly robust model before adjusting for multiple comparisons). The 

standardized effect size was small to medium (d = −0.35) in the doubly robust 

model. 

• Parents/primary caregivers who are expecting or who have children up to age 4: 

- PTSD symptoms at 12 months had a smaller decrease in the C2C group (in the 

doubly robust model before adjusting for multiple comparisons). The standardized 

effect size was small to medium (d = 0.29) in the doubly robust model. 

Overall, we saw a few notable trends in the subgroup analysis with an intervention effect in 

the doubly robust models showing greater improvement for the C2C group in alcohol use (job 

training and employment program), depressive symptoms (youth development program), and 

psychological distress (young adult target population). On the other hand, in the parents/primary 

caregivers’ subgroup, the comparison group improved more on PTSD symptoms than the C2C 

group in the doubly robust model. The two other areas where the comparison group performed 

better were in the propensity score weighted models before adjusting for multiple comparisons. 

At 6 months, only the young adult target population had any intervention effects in the doubly 

robust models (prior to adjusting for multiple comparison), with greater decreases in depression 

(small-to-medium standardized effect size; d = −0.34) and anxiety (small-to-medium standardized 

effect size; d = −0.34) in the C2C group relative to the comparison group. 

Discussion

For this segment of the impact evaluation, we tested hypotheses about whether individuals at 

C2C CBOs showed greater positive improvement across the different mental health outcome 

measures than those at comparison CBOs. Overall, we found that both groups improved across 

all of the mental health outcomes and that there were no significant intervention effects for our 

primary research question at 12 months, and only one trend in the propensity score weighted 

191 
 

 

 

 

 

model at 6 months. However, we found some signal of an intervention effect for some of the 

mental health outcomes for different subgroups, and one signal of an intervention effect that 

favored the comparison group. 

In terms of improvement in mental health symptoms over time, we saw consistent and strong 

improvement from baseline to 12 months across all mental health outcome measures within both 

the C2C and comparison groups (Table 8.6). For depression and anxiety, this pattern of decreasing 

symptoms over time is consistent with the literature that shows symptom patterns that generally 

decline over time in a natural recovery process (Boland et al., 2009; Richards, 2011). 

Table 8.6. Summary of Mental Health Outcome Results in the Overall Sample

Mental Health Outcome

Difference Within Groups

over Time Intervention Effect

Clinically Significant

or Reliable Change

Depression Decrease in both groups

Generalized anxiety Decrease in both groups

PTSD Decrease in both groups

Alcohol use Decrease in both groups

Substance use Decrease in both groups Greater decrease in C2C

group (6 months only,

PSW)

Change more likely in 

C2C group (6 months

only)

Psychological distress Decrease in both groups

NOTE: PSW = propensity score weighted.

We did not see greater improvement in mental health symptoms in the C2C groups over time, 

which suggests that C2C did not have an effect on reducing mental health symptoms. However, 

we did observe one significant intervention effect in the propensity score weighted model at 

6 months. The C2C group showed a trend toward greater decreases in substance use symptoms 

over time relative to the comparison group, although the effect size was very small. Similarly, 

we found that a higher percentage of the C2C group relative to the comparison group moved 

from above the substance use threshold, indicating a positive screen, to below the threshold at 

6 months. Although small, we note that about one in four persons had a mental health problem 

co-occurring with alcohol or substance use, meaning that programs will need to address mental 

health in tandem with substance use to reach a significant proportion of clients with unmet needs. 

Overall, C2C’s screening components were designed to identify mental health needs among 

individuals seeking community-based services in NYC. C2C also provided support and referrals 

to persons with identified mental health needs. Across the board, the prevalence of mental health 

issues in the study sample exceeded the prevalence rates from epidemiological studies of 

vulnerable populations (Table 8.7). For depression, 38 percent of study participants reported 

moderate-to-severe depressive symptoms, which is much higher than the 8-percent estimate often 

cited as context for the ThriveNYC initiative (Belkin et al., 2016; McCray, Buery, and Bassett, 

192 
 

 

 

 

 

 

Table 8.7. Summary of Mental Health Status of Study Participants at Baseline

Study Results

Mental Health Issue

Reference 

Pointa (%)

Full Sample

(%)

C2C Group

(%)

Comparison 

Group (%)

Depression 8–20 38 36 41

Anxiety 18 39 38 40

Trauma/PTSD 3–12 48 47 48

Alcohol use 5–29 21 21 20

Substance use 2–10 25 27 21

Co-occurring disorders 4–12 24 23 25

a The reference points represent rates found in the literature for similar at-risk populations (e.g., depression: 

Belkin et al., 2016 and Kessler et al., 2005; anxiety: Kessler et al., 2005 and Muntaner et al., 1998; PTSD:

Anderson et al., 2016 and Kessler et al., 2005; and alcohol use: SAMHSA, 2019; substance use: SAMHSA,

2019; and co-occurring disorders: Kessler et al., 2005 and SAMHSA, 2019).

2015). Further, 39 percent of study participants had baseline anxiety symptoms in the moderateto-severe range, which is higher than the 18 percent national prevalence rate for anxiety disorder 

(Kessler et al., 2012). Although prevalence rates from epidemiological studies are typically in the 

single digits, almost one-half (48 percent) of study participants had PTSD symptoms at a level 

consistent with a probable PTSD diagnosis. Among study participants, the prevalence of harmful 

drinking or alcohol dependence was comparable to what has been found nationally for alcohol 

use disorders and in NYC for binge drinking. Moderate and severe substance use was also more 

prevalent among study participants when compared with national and local prevalence estimates 

of illicit drug use disorder. Owing to the higher prevalence rates overall, we also found that rates 

of co-occurring disorders (i.e., alcohol/substance use that occurs along with depression, anxiety, 

or PTSD) overall and for each group were also substantially higher than the prevalence rates 

from other studies. 

Our descriptive analyses showed a very high degree of disadvantage among those seeking 

services at CBOs with very low income levels, high unemployment rates, and low education 

levels in both the C2C and comparison groups. This relative disadvantage coupled with 

attitudinal and logistical barriers to care, and relatively low rates of clinical outpatient mental 

health treatment utilization, likely contributed to the high rates of unmet mental health need 

described in Chapter 7. 

Overall, the C2C impact study provides important new information about the prevalence of 

mental health issues among individuals seeking community-based services that can help guide 

ongoing ThriveNYC efforts, as well as the replication and refinement of C2C. The high level of 

unmet need points to the importance of finding ways to identify and support these underserved 

New Yorkers, given the well-documented effects of untreated mental health problems on outcomes 

such as employment, education, housing, and incarceration status (explored in Chapter 9). 

193 
 

 

 

Variation in C2C Effectiveness for Different Subgroups

We examined within-group changes and intervention effects over time separately for the 

three C2C target populations and two of the CBO types. Overall, both the C2C and comparison 

group improved between baseline and 12 months for each of the mental health symptom 

domains across almost all of the subgroups. In terms of intervention effects, we observed mixed 

results, with a number of trends showing greater improvement for the C2C group relative to the 

comparison group and others favoring the comparison group, although none of these findings 

remained statistically significant after adjusting for multiple comparisons. These subgroup 

findings may help explain the overall lack of group differences in these outcomes in the overall 

sample. That is, when two subgroups of CBO clients are responding to C2C in different ways, it 

can seem as if there is no intervention effect overall, masking these important differences in how 

the intervention worked at different types of CBOs (i.e., “treatment heterogeneity”). 

Among job training and employment CBO clients, C2C participants had greater decreases in 

alcohol use compared with their comparison peers at 12 months, including in the doubly robust 

model that further adjusted for important covariates. Although the effect size was still small 

(d = −0.17), we note that it was double the effect size observed for alcohol use in the overall 

analysis. Job training and employment programs within C2C may have been particularly effective 

at targeting alcohol misuse in their clients; or they may be better equipped to reduce alcohol use 

by focusing on goals (i.e., obtain and/or retain employment) that could be incompatible with 

alcohol misuse. This aligns with the findings in Chapter 9, in which C2C clients also had 

substantial improvements in employment outcomes (hours worked and monthly pay) at 12 months 

relative to their comparison peers. Among job training and employment clients, we also found 

that C2C clients had a smaller decrease in depressive symptoms. However, this effect size was 

very small (d = 0.04); combined with the lack of statistical significance after the multiple 

comparisons adjustment, we did not consider this to be evidence of a meaningful effect. 

We found the reverse pattern among youth development CBO clients in the C2C group 

(greater improvement in depression symptoms and less improvement in alcohol use). Consistent 

with our hypotheses, C2C clients had a greater improvement in depressive symptoms relative to 

the comparison group. This effect size was small to medium (d = −0.30) and about 15 times the 

magnitude of the effect calculated for the analyses in the overall sample. This finding also held 

(albeit not after multiple comparisons adjustment) in the doubly robust model, lending further 

weight to the finding. This finding is also consistent with the results in Chapter 7, in which C2C 

clients at youth development CBOs had a threefold greater odds of using outpatient mental 

health services during this same period of follow-up. This greater utilization may have helped 

drive the greater decrease in depression in the C2C group at youth development CBOs. In 

contrast, the C2C group had a smaller decrease in alcohol use than the comparison group at 

youth development CBOs. However, this finding only emerged in the propensity score weighted 

model, and the standardized effect size was very small (d = 0.08). 

194 
 

 

 

Among youth and young adults who are not in school and unemployed, we saw that C2C 

participants had greater reductions in psychological distress relative to comparison group young 

adults, supporting our hypothesis. Two observations underscore the robustness of this finding: 

We found the difference in the doubly robust model before the multiple comparisons adjustment, 

and the effect size was small to medium (d = −0.35), about 13-fold the effect size from the 

main analysis. We also note our finding of Chapter 7 for this subgroup, in which C2C clients had 

a 50-percent reduction in the odds of having used an emergency setting over the year of follow-up. 

This decreased emergency utilization may reflect behavioral health needs being adequately 

met through C2C, leading to fewer emergency setting visits. This finding only emerged for 

psychological distress, which may be related to the lack of specificity for this measure: This 

measure represents a more global indicator of unmet mental health needs with ties to depression, 

anxiety, and overall mood. 

Finally, among parents and caregivers, C2C participants’ PTSD symptoms improved less 

than their comparison group counterparts. This difference was observed in the doubly robust 

model, with a small-to-medium effect size (d = 0.29). This counterintuitive finding may be 

related to differential retention in the study across the groups. 

Limitations

As described more fully in Chapter 7, limitations of the evaluation provide additional 

important context for our findings. First, study participants are not representative of C2C and 

comparison CBO clients at large. In the C2C group, study participants represent only a fraction 

of those served by C2C, and study eligibility was based on meeting a threshold of mental health 

symptoms at baseline. Second, given the voluntary nature of the study, there may be bias among 

those who opted to participate in the baseline assessment and continue with the follow-up 

surveys or barriers to retention in the study that differed by group. Third, all data were selfreported and therefore are subject to social desirability and recall bias in responses. Fourth, we 

were not able to include measures of fidelity or C2C implementation or account for other 

services that participants in both groups may have been receiving from their CBO. Fifth, we 

note the high levels of mental health need at baseline in this sample, along with high rates of 

unemployment, housing instability, and incarceration (explored in Chapter 9), which may mean 

that this sample is not generalizable to the larger NYC population. Finally, there are many 

factors related to the quantity and quality of mental health care in NYC that are outside the 

control of the C2C initiative or this study. For example, many C2C participants with clinically 

significant symptoms were referred for formal mental health services (e.g., medication or 

counseling) (Chapter 7), but we were not able to assess the quality or intensity of such services, 

whether evidence-based treatments were delivered, or whether the client engaged in a full course 

of treatment, all of which are limitations to the interpretation of the findings. 

195 
 

Summary

Overall, we did not see that C2C improved mental health symptoms over the course of 6 or 

12 months, with the exception of a very small potential effect favoring the C2C group in substance 

use. Although C2C may not have had an effect on improving mental health symptoms in the 

overall sample, we did find some signal of an intervention effect for some of the mental health 

outcomes for different subgroups. The comparable improvements found in the comparison group 

might reflect the fact that usual care for mental health issues was changing more broadly in NYC 

due to citywide, concurrent efforts within the ThriveNYC initiative, including efforts such as 

citywide MHFA trainings (not part of C2C). Further, there were changes at some of the 

comparison CBOs in their organizations’ approaches, programming, and services related to 

clients’ mental health needs over the course of the study. It is possible that some of these changes 

may have resulted in clients at the comparison CBOs who became more willing to disclose 

mental health problems, staff who were more aware of mental health problems in their client 

population, and staff who received training on specific skills for addressing mental health issues 

among their clients (e.g., MHFA) over the course of the study. The improvement for both groups 

points to the promise of the supports that are offered as part of usual CBO services, even though 

this made it difficult to observe an intervention effect. It is also possible that these improvements 

for both groups reflect natural recovery or other factors that are unrelated to the mental health 

supports offered. Our findings present valuable information regarding the efficacy of C2C in this 

underserved population and provide a critical snapshot of mental health needs in this sample of 

individuals seeking a range of community-based services and programming in NYC. 

196 
 

 

 

 

 

 

 

 

References

Anderson, R. E., T. J. Geier, and S. P. Cahill. “Epidemiological Associations Between 

Posttraumatic Stress Disorder and Incarceration in the National Survey of American Life,” 

Criminal Behaviour and Mental Health, Vol. 26, No. 2, 2016, pp. 110–123. 

Belkin, G., N. Linos, S. E. Perlman, C. Norman, and M. T. Bassett. “A Roadmap for Better 

Mental Health in New York City,” The Lancet, Vol. 387, No. 10015, 2016, pp. 207–208. 

Berner, M. M., L. Kriston, M. Bentele, and M. Harter, “The Alcohol Use Disorders Identification 

Test for Detecting At-Risk Drinking: A Systematic Review and Meta-Analysis,” Journal of 

Studies on Alcohol and Drugs, Vol. 68, No. 3, 2007, pp. 461–473. 

Bisson, J. I., A. Ehlers, R. Matthews, S. Pilling, D. Richards, and S. Turner, “Psychological 

Treatments for Chronic Post-Traumatic Stress Disorder: Systematic Review and MetaAnalysis,” British Journal of Psychiatry, Vol. 190, 2007, pp. 97–104. 

Blevins, C. A., F. W. Weathers, M. T. Davis, T. K. Witte, and J. L. Domino, “The Posttraumatic 

Stress Disorder Checklist for DSM-5 (PCL-5): Development and Initial Psychometric 

Evaluation,” Journal of Traumatic Stress, Vol. 28, No. 6, 2015, pp. 489–498. 

Boland, R. J., M. B. Keller, I. H. Gotlib, and C. L. Hammen, Course and Outcome of Depression, 

Handbook of Depression, New York: The Guilford Press, Vol. 2, 2009, pp. 23–43. 

Capua, J., E. Tuazon, and D. Paone. Binge Drinking and Associated Health-Related Behaviors 

Among Adults in New York City, 2014. New York City Department of Health and Mental 

Hygiene: Epi Data Brief (77), 2016. 

Center for Behavioral Health Statistics and Quality. (2018). “2017 National Survey on Drug Use 

and Health: Methodological Summary and Definitions.” 

https://www.samhsa.gov/data/ 

City of New York, ThriveNYC: A Mental Health Roadmap for All, New York, NY, 2018. 

https://thrivenyc.cityofnewyork.us/wp-content/uploads/2019/08/Thrive-Roadmap.pdf 

De Luca, S. M., J. R. Blosnich, E. A. Hentschel, E. King, and S. Amen. “Mental Health Care 

Utilization: How Race, Ethnicity and Veteran Status Are Associated with Seeking Help,” 

Community Mental Health Journal, Vol. 52, No. 2, 2016, pp. 174–179. 

Hahm, H. C., B. L. Cook, A. Ault-Brutus, and M. Alegria, “Intersection of Race-Ethnicity and 

Gender in Depression Care: Screening, Access, and Minimally Adequate Treatment,” 

Psychiatric Services, Vol. 66, No. 3, 2015, pp. 258–264. 

Hofmann, S. G., and J. A. Smits, “Cognitive-Behavioral Therapy for Adult Anxiety Disorders: 

A Meta-Analysis of Randomized Placebo-Controlled Trials,” Journal of Clinical Psychiatry, 

Vol. 69, No. 4, 2008, pp. 621–632. 

197 
 

 

 

 

 

 

 

Kataoka, S. H., L. Zhang, and K. B. Wells, “Unmet Need for Mental Health Care Among 

U.S. Children: Variation by Ethnicity and Insurance Status,” American Journal of 

Psychiatry, Vol. 159, No. 9, 2002, pp. 1548–1555. 

Kessler, R. C., M. Petukhova, N. A. Sampson, A. M. Zaslavsky, and H. U. Wittchen, “TwelveMonth and Lifetime Prevalence and Lifetime Morbid Risk of Anxiety and Mood Disorders 

in the United States.” International Journal of Methods in Psychiatric Research:, Vol. 21, 

No. 3, 2012, pp. 169–184. 

Kessler, R. C., W. T. Chiu, O. Demler, K. R. Merikangas, and E. E. Walters, “Prevalence, 

Severity, and Comorbidity of 12-Month DSM-IV Disorders in the National Comorbidity 

Survey Replication.” Archives of General Psychiatry. Vol. 65, 2005, pp. 617–627. 

Kroenke, K., R. L. Spitzer, J. B. Williams, and B. Löwe. “The Patient Health Questionnaire 

Somatic, Anxiety, and Depressive Symptom Scales: A Systematic Review.” General 

Hospital Psychiatry, Vol. 32, No. 4, 2010, pp. 345–359. 

McCray, C., R. Buery, and M. T. Bassett. ThriveNYC: A Mental Health Roadmap for All. New 

York, NY: The New York City Mayor’s Office, 2015. 

Mericle, A. A., V. M. Ta Park, P. Holck, and A. M. Arria, “Prevalence, Patterns, and Correlates 

of Co-Occurring Substance Use and Mental Disorders in the United States: Variations by 

Race/Ethnicity,” Comprehensive Psychiatry, Vol. 53, No. 6, 2012, pp. 657–665. 

Metrik, J., K. Jackson, S. S. Bassett, M. J. Zvolensky, K., Seal, and B. Borsari, “The Mediating 

Roles of Coping, Sleep, and Anxiety Motives in Cannabis Use and Problems Among 

Returning Veterans with PTSD and MDD,” Psychology of Addictive Behaviors, Vol. 30, 

No. 7, 2016, pp. 743–754. 

Miranda, J., J. Y. Chung, B. L. Green, J. Krupnick, J. Siddique, D. A. Revicki, and T. Belin, 

“Treating Depression in Predominantly Low-Income Young Minority Women: A randomized 

controlled trial,” JAMA, Vol. 290, No. 1, 2003, pp. 57–65. 

Muntaner, C., W. W. Eaton, C. Diala, R. C. Kessler, and P. D. Sorlie. “Social Class, Assets, 

Organizational Control and the Prevalence of Common Groups of Psychiatric Disorders.” 

Social Science & Medicine, Vol. 47, No. 12, 1998, pp. 2043–2053. 

Nolan, M. L., S. Mantha, E. Tuazon, and D. Paone. Unintentional Drug Poisoning (Overdose) 

Deaths in New York City in 2018. New York City Department of Health and Mental 

Hygiene: Epi Data Brief (116), 2019. 

Richards, D. (2011). “Prevalence and Clinical Course of Depression: A Review.” Clinical 

Psychology Review, Vol. 31, No. 7, 2011, pp. 1117–1125. 

198 
 

 

 

Roy-Byrne, P., M. G. Craske, G. Sullivan, R. D. Rose, M. J. Edlund, A. J. Lang, A. Bystritsky, 

S. S. Welch, D. A. Chavira, D. Golinelli, L. Campbell-Sills, C. D. Sherbourne, and M. B. 

Stein, “Delivery of Evidence-Based Treatment for Multiple Anxiety Disorders in Primary 

Care: A Randomized Controlled Trial,” JAMA, Vol. 303, No. 19, 2010, pp. 1921–1928. 

Santiago, C. D., S. Kaltman, and J. Miranda, “Poverty and Mental Health: How Do Low-Income 

Adults and Children Fare in Psychotherapy?” Journal of Clinical Psychology, Vol. 69, No. 2, 

2013, pp. 115–126. 

Shear, K., E. Frank, P. R. Houck, and C. F. Reynolds, 3rd, “Treatment of Complicated Grief: A 

Randomized Controlled Trial,” JAMA, Vol. 293, No. 21, 2005, pp. 2601–2608. 

Spitzer, R. L., K. Kroenke, J. B. Williams, and B. Lowe, “A Brief Measure for Assessing 

Generalized Anxiety Disorder: The GAD-7,” Archives of Internal Medicine, Vol. 166, 

No. 10, 2006, pp. 1092–1097. 

Substance Abuse and Mental Health Services Administration (SAMHSA), 2016–2017 NSDUC 

State Prevalence Estimates, 2018. 

http://www.samhsa.gov/data/report/2016-2017-nsduh-state-prevalence-estimates 

———, Key Substance Use and Mental Health Indicators in the United States: Results from the 

2018 National Survey on Drug Use and Health (HHS Publication No. PEP19-5068, NSDUH 

Series H-54), Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance 

Abuse and Mental Health Services Administration, 2019. 

https://www.samhsa.gov/data/sites/default/files/cbhsq-reports/NSDUHNationalFindings 

Report2018/NSDUHNationalFindingsReport2018.pdf 

———, Office of Applied Studies, Query Results Provided from the 2006-07 National Surveys 

on Drug Use and Health, provided to the New York City Dept of Health and Mental Hygiene 

by SAMHSA, 2008, unpublished data. 

Wang, P. S., M. Lane, M. Olfson, H. A. Pincus, K. B. Wells, and R. C. Kessler. (2005). 

“Twelve-Month Use of Mental Health Services in the United States: Results from the 

National Comorbidity Survey Replication,” Archives of General Psychiatry, Vol. 62, No. 6, 

pp. 629–640. 

doi:10.1001/archpsyc.62.6.629 

Yudko, E., O. Lozhkina, and A. Fouts, “A Comprehensive Review of the Psychometric 

Properties of the Drug Abuse Screening Test,” Journal of Substance Abuse Treatment, 

Vol. 32, No. 2, 2007, pp. 189–198. 

199 
 

 

 

 

9. Impact of C2C on Employment, Education, Housing, and

Incarceration

Daniel Siconolfi, Dana Schultz, Lynsay Ayer, Joshua Snoke, and Elie Ohana 

Key Findings

• From baseline to follow-up, both C2C participants and those in the comparison group 

reported more full- or part-time employment, more weekly work hours, higher monthly 

pay, more stable housing, and more high school or GED completion. 

• Across all of the employment, housing, education, and incarceration outcomes, there 

was one statistically significant intervention effect at 12 months. C2C resulted in larger 

increases in weekly work hours compared with usual CBO services, with a small effect 

size. 

• C2C had no effect on the other outcomes in the overall sample. 

• There was evidence that C2C may have been more effective for certain subgroups, 

showing an effect in the following areas: 

- increased weekly work hours (small-to-medium effect size) and monthly pay 

(small effect size) at job training and employment CBOs 

- increased weekly work hours (very small effect size) for adults 18 or older 

- increased weekly work hours (small effect size) for parents or caregivers. 

200 
 

Introduction

The analyses in this chapter examine whether C2C’s task-shifting approach improved 

outcomes in the domains of employment, housing, education, and incarceration for those 

individuals receiving services from CBOs. For this segment of the impact evaluation, we 

expected that C2C clients would show greater improvements in these non–mental health 

outcomes compared with their comparison group peers at follow-up. For employment outcomes, 

we expected that C2C participants would report more full- or part-time employment, hours 

worked, and monthly pay. For housing outcomes, we expected that C2C participants would 

report more stable housing and would be less likely to experience homelessness. For educational 

outcomes, we expected that C2C participants would be more likely to have completed a high 

school diploma or GED. Finally, for criminal justice involvement outcomes, we expected that 

C2C participants would be less likely to have experienced incarceration. 

In this chapter, we examine changes in non–mental health outcomes over time to understand 

the effect of C2C on participants’ social and emotional functioning. Because of the wide 

variability in types of CBOs participating, and thus in the clients they serve, we examined effects 

among the entire sample of participants as well as effects for different populations and settings 

within the sample. 

Background

Untreated mental health problems and substance misuse can have far-reaching effects. 

Mental health problems (including substance use) can make it more difficult for people to find 

or keep full-time employment, stay in school, obtain and keep stable housing, and avoid the 

criminal justice system (Antonisse and Garfield, 2018; Esch et al., 2014; Fazel, Geddes, and 

Kushel, 2014; Hawthorne et al., 2012; Kim and Ford, 2006). The outcomes examined in this 

chapter are interrelated in terms of their causes and mutually reinforcing nature (Acker et al., 

2019; Commission on the Social Determinants of Health, 2008; Fedock and Sarantakos, 2017; 

Freudenberg and Ruglis, 2007; Kim and Ford, 2006; National Health Care for the Homeless 

Council, 2016; Prins, 2014). For example, the effect of depression on educational attainment 

may have longer-term implications for employment status (McGee and Thompson, 2015). Below, 

we briefly summarize the evidence base related to the associations between mental health issues 

(including substance use) and employment, housing, education, and incarceration outcomes. 

Employment

Mental health issues are common among unemployed persons, with evidence that about one 

in three unemployed adults experience psychological problems such as anxiety, depression, and 

distress (Paul and Moser, 2009). Among young adults (ages 18–25), a population with especially 

high unemployment, about one in four unemployed persons evidence depression (McGee and 

Thompson, 2015). The association is also bidirectional. For example, unmet mental health needs 

201 
can lead to unemployment, and job loss has been associated with onset of mental health needs 

(Hergenrather et al., 2015; Lerner and Henke, 2008; McKee-Ryan et al., 2005; Paul and Moser, 

2009; van der Noordt et al., 2014; Wanberg, 2012). Further, unemployment and the associated 

lack of income and insurance coverage can present a barrier to treatment (Wanberg, 2012). 

Unmet mental health needs in young adulthood can also indirectly affect employment prospects, 

for example, through its effects on social relationships that could be conducive to employment 

(McGee and Thompson, 2015). 

There is some evidence that early intervention to address mental health needs for young 

adults can potentially forestall long-term unemployment and negative effects (Gmitroski et al., 

2018). Evidence is mixed with regard to whether employment can improve mental health and 

health (Antonisse and Garfield, 2018). However, for persons with mental health needs, 

employment can facilitate mental health improvements or substance use recovery (Harrison, 

Krieger, and Johnson, 2020; van der Noordt et al., 2014). It may also be possible to directly 

address mental health problems attributed to unemployment, with some evidence for intervention 

programs aimed at reducing the psychological distress associated with chronic unemployment 

(Paul and Moser, 2009). 

Housing

Associations between homelessness and mental health needs are well documented (DaveyRothwell, German, and Latkin, 2008; Duke and Searby, 2019; Fazel, Geddes, and Kushel, 2014; 

Goering et al., 2002; Kim and Ford, 2006; Substance Abuse and Mental Health Services 

Administration, 2013; Suglia, Duarte, and Sandel, 2011). Nationally, about one in two homeless 

adults report having ever received treatment for mental illness, and one in five report current 

unmet mental health needs (Baggett et al., 2010). Estimates of the prevalence of substance use 

problems among homeless populations range from 10 to 50 percent (Baggett et al., 2010; Kim 

and Ford, 2006). Further, the risk factors associated with homelessness (e.g., child maltreatment, 

intimate partner violence, familial mental health problems, and poverty) have significant and 

overlapping implications for mental health and substance use (Edidin et al., 2012; Goering et al., 

2002; Kim and Ford, 2006; Sullivan, Burnam, and Koegel, 2000; Wang et al., 2019). 

Unmet mental health needs may also affect outcomes among homeless or unstably housed 

persons. Persons with substance use issues are less likely to transition into stable housing and 

more likely to exit supported housing prematurely (Aubry et al., 2016; Gabrielian et al., 2016). 

This population also faces barriers to mental health care. For example, homelessness also affects 

employment (Kim and Ford, 2006), and lack of insurance and an inability to pay for care is a 

barrier to mental health services for this population (Baggett et al., 2010). 

Education

Mental health and substance use issues are associated with lower educational completion, 

such as dropping out of high school or lower likelihood of college enrollment (Breslau et al., 

202 
2008; Esch et al., 2014; Fletcher, 2010; Fergusson and Woodward, 2002; Kessler et al., 1995; 

McGee and Thompson, 2015). Mental health problems explain about one in ten of the educational 

terminations before high school graduation (Breslau et al., 2008). Both clinically significant 

depression and subclinical symptoms have been associated with lower educational attainment 

(Fletcher, 2010). The associations are also bidirectional, meaning there is some evidence that 

lower educational attainment can lead to subsequent mental health problems and substance use 

(Esch et al., 2014; Freudenberg and Ruglis, 2007) 

Incarceration

Mental health and substance use issues are associated with incarceration and recidivism, 

because mental health problems and substance use are more common among criminal justiceinvolved persons (Bronson and Berzofsky, 2017; Fedock and Sarantakos, 2017; Greenberg and 

Rosenheck, 2009; Hawthorne et al., 2012; Prins, 2014; Sugie and Turney, 2017). Although 

prevalence estimates vary substantially, one systematic review found that the prevalence of 

major depression among persons incarcerated in state prisons ranged from about one in ten 

persons to nearly one in three persons (Prins, 2014). A national study of prisoners and jail 

inmates found that depression was the most common prior diagnosis (Bronson and Berzofsky, 

2017). Another study of persons in a public mental health system who had been incarcerated in 

the past year found that 65 percent had a cooccurring substance use disorder and 13 percent had 

a diagnosis of major depression (Hawthorne et al., 2012). Recent analysis of NYC Community 

Health Survey data found that New Yorkers with prior criminal justice system involvement were 

more likely than those without criminal justice involvement to show evidence of depression and 

alcohol misuse (heavy drinking and binge drinking) (Baquero et al., 2020). 

The association between incarceration and mental health problems may also exist among 

those who are connected to mental health services, suggesting continuing needs. For example, 

about 12 percent of patients in a public mental health system were incarcerated in the span of 

1 year, and about one in four were reincarcerated within 12 months (Hawthorne et al., 2012). For 

persons in community reentry, inability to pay for services represents a barrier for mental health 

and substance use services (Begun, Early, and Hodge, 2016). Providing outpatient mental health 

services and case management can reduce reincarceration or use of emergency psychiatric 

services (Hawthorne et al., 2012). Although one recent study found that utilization of mental 

health services was associated with a greater likelihood of reincarceration within 1 year among 

individuals with severe mental illness, this may be related to clients’ increased visibility while 

receiving clinical mental health services (Domino et al., 2019). 

As described in Chapter 3, it was anticipated that C2C could yield benefits in these domains, 

both directly and indirectly. By addressing mental health symptoms or catching problems early, 

C2C may result in better employment, housing, education, and incarceration outcomes. It is also 

possible that CBO staff with C2C skills are better able to serve their clients in these domains, by 

recognizing and understanding their clients’ mental health needs. For example, through C2C a 

203 
 

 

 

job training and employment program staff member may have a new ability to recognize 

symptoms of depression in a client and provide adaptive coping strategies that allow the client to 

maintain engagement in job training programs, leading to better employment outcomes. However, 

as described in Chapter 8, we anticipated that the size of any intervention effects would be 

relatively small for the 6- and 12-month follow-up periods. 

Research Questions and Hypotheses

At the individual level, the evaluation aimed to assess whether C2C could improve outcomes 

in the domains of education, housing, employment, and incarceration. For the primary research 

question of whether C2C improved these non–mental health outcomes relative to comparison 

group participants over time, we tested whether C2C participants: 

• employment 

- were more likely to have part-/full-time employment 

- were employed for more hours per week 

- reported higher monthly earnings. 

• housing 

- were more likely to have stable housing 

- were less likely to have experienced recent homelessness. 

• education 

- were more likely to have completed a high school diploma or GED. 

• incarceration 

- were less likely to have been incarcerated. 

We also examined one secondary research question related to our hypotheses: 

• Does program effectiveness vary for the specific populations targeted by C2C (i.e., adults 

age 18 or older who are unemployed or underemployed, young adults ages 16–24 who 

are not in school and are not employed, and parents/primary caregivers who are expecting 

or who have children up to the age of 4) or CBO service types (e.g., job training and 

employment programs, youth development programs)? 

Method

Information on data collection and data analyses was provided in the methods section of 

Chapter 7. Below, we provide only information on data collection that differs from what was 

presented in Chapter 7. See Appendix C for complete details. 

Data Collection

Measures. The specific outcome measures for these analyses included self-reported 

measures of employment, housing, educational attainment, and incarceration from the client 

survey (Table 9.1). For employment outcomes, we focused on the aggregate binary indicator of 

“employed full-/part-time” versus “unemployed,” which excludes retirees, students, homemakers, 

204 
 

 

 

 

 

 

Table 9.1. Recall Period for Non–Mental Health Outcomes

Recall Period

Outcome

Current, at each follow-up 

time point Past year, pooled follow-up

Employment Full- or part-time employment –

Hours worked per week

Monthly pay before taxes/deductions

Housing Housing status Any homelessness

Education High school/GED completion –

Incarceration – Any incarceration

and the disabled, an indicator of hours worked per week (among those who worked), and an 

indicator of monthly employment pay before taxes and deductions (among those who worked). 

For housing outcomes, we focused on the aggregate binary indicator for current housing, 

indicating “stably housed or staying with someone else” (i.e., in a friend or family member’s 

room, apartment, or house) versus “transitional/temporary housing, unstable housing, or 

homelessness.” 

We also examined recent homelessness. This variable was pooled across the two follow-up 

time points due to differing recall periods in the baseline survey versus follow-up surveys (past 

year at baseline, past 6 months at follow-up), and also because any homelessness over the past 

year was considered to be a meaningful and important outcome. 

For the education outcome, we focused on the aggregate binary variable indicating “completed 

high school diploma, GED, or greater,” versus not having completed high school or a GED. 

For the incarceration outcome, we examined recent incarceration (spending time in a 

correctional facility, such as a jail or prison). Similar to recent homelessness, we pooled the 

follow-up survey data to permit baseline and follow-up comparisons, which otherwise had 

differing recall periods, and also because any incarceration in the past year was considered a 

meaningful and important outcome. 

Results

We started these analyses by looking descriptively at baseline employment status, housing 

status, educational attainment, and incarceration status (Table 9.2). Detailed tables are in 

Appendix C, section C7.2 

For employment status at baseline, we describe the distribution across all employment 

categories as well as the proportion of persons who were employed full-time or part-time versus 

unemployed. Overall, there were no statistically significant differences between the C2C and 

comparison groups at baseline for either measure. The majority of study participants (60.8 percent) 

were unemployed at baseline. About one in four study participants had full-/part-time employment 

at baseline (27.4 percent). Participants reported working 7.4 hours per week, on average (across 

205 
 

 

 

 

 

 

 

 

 

Table 9.2. Baseline Indicators for Employment, Housing, Educational Attainment, and

Incarceration (Weighted)

Full Sample C2C Group

Comparison

Group

Percent Percent Percent

Indicator Endorsed Endorsed Endorsed p-Value

Employed full-time or part-timea 27.4 26.0 29.0 0.265

Mean (SD) Mean (SD) Mean (SD) p-Value

Hours worked per week 7.43 (14.4) 6.99 (14.4) 7.92 (14.5) 0.268

Current monthly pay before taxes/deductions $332 (887) $301 (807) $366 (968) 0.200

Percent Percent Percent

Endorsed Endorsed Endorsed p-Value

Currently stably housed/staying with 

someone elseb 76.6 74.3 79.2 0.053

Past-year homelessness 18.9 20.3 17.3 0.202

Completed high school or GED or greaterc 68.7 70.1 67.2 0.293

Past-year incarceration 12.8 14.7 10.7 0.066

NOTE: SD = standard deviation. a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline. b Versus transitional/temporary or homeless/unstably housed. c Among persons 18 and older at baseline.

full-time, part-time, and unemployed persons) with no difference by group. The average current 

monthly pay among persons who were employed full-time, part-time, or unemployed was 

$331 USD, with no difference by group. 

For housing status at baseline, we looked at the distribution across all housing categories, as 

well as the proportion of persons who were stably housed or staying with someone else (i.e., in 

a friend or family member’s room, apartment, or house) versus those who were transitionally 

housed, homeless, or unstably housed. There were no statistically significant differences by 

group for either measure. Although the majority of study participants were stably housed or 

staying with someone else (45.1 and 36.4 percent, respectively; 76.6 percent for combined 

category) at baseline, about 8 percent were in transitional or temporary housing, and another 

10 percent were homeless or unstably housed. About one in five persons (18.9 percent) reported 

any homelessness in the past year. 

For educational attainment at baseline, we focused on the proportion of persons who had 

completed high school, a GED, or more. About two-thirds of the sample ages 18 and older had 

completed a high school diploma, GED, or greater (69 percent) with no statistically significant 

differences by group. 

For incarceration status at baseline, about one in eight persons had been incarcerated in the 

past year at baseline, with no statistically significant differences by group. 

206 
 

 

 

 

 

 

 

 

 

 

 

 

 

Primary Research Question: Do C2C Participants Show Greater Improvements in 

Non–Mental Health Outcomes?

To assess whether C2C participation improved employment, housing, education, and 

incarceration outcomes, we first examined changes in outcomes within each group over time. In 

this comparison, we considered the within-individual change between baseline and the follow-up 

time points and report the average change and its direction across participants in each group. 

For employment-related outcomes, we found that both groups showed statistically significant 

improvements across all outcomes (Table 9.3). At 12 months, the proportion of the C2C group 

with full- or part-time employment doubled, to 61 percent and the comparison group increased to 

57 percent from 37 percent. Both groups also reported statistically significant increases in hours 

worked per week and current monthly pay (Table 9.4). For hours worked per week, the average 

increase at 12 months was 13.1 hours in the C2C group and 6.7 hours in the comparison group. 

For monthly employment pay, the average increase at 12 months was $840 in the C2C group and 

$570 in the comparison group. Results were similar at the 6-month follow-up, with all employment 

outcomes in each group showing statistically significant improvements from baseline (see 

Appendix C, section C8.1). 

Table 9.3. Within-Group Differences in Full-Time or Part-Time Employment at 12 Months 

(Weighted)

Employment at 12 Months

Outcome

Baseline Percentage

(Retained Sample)

12 Months

(%)

Difference

(%) p-Value

Full-time or part-time

employment (vs.

unemployed)a

C2C

Comparison

29.13

37.42

61.11

57.24

31.98

19.81

<.001b

<.001b

a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 

12 months.

b Statistically significant at the p<.05 level.

Table 9.4. Within-Group Mean Changes in Hours Worked and Employment Income at 12 Months 

(Weighted)

Hours and Pay at 12 Monthsa

Outcome Estimate SE p-Value

Hours worked per week C2C 13.1 1.23 <.001b

Comparison 6.68 1.43 <.001b

Current monthly pay before taxes C2C $840 USD 107 <.001b

and deductions Comparison $570 USD 123 <.001b

NOTE: SE = standard error. a Within-individual mean changes between baseline and 12 months for each group separately. b Statistically significant at the p<.05 level.

207 
 

 

 

 

 

 

 

 

 

 

 

 

 

For current housing, both groups evidenced a statistically significant improvement in the 

proportion stably housed or staying with someone else at 12 months (Table 9.5). In the C2C group, 

the proportion of persons in this stable housing/doubled up category increased by 8.5 percentage 

points. In the comparison group, this proportion increased by 5 percentage points. For recent 

homelessness, we used the pooled 6- and 12-month imputed data to examine homelessness over 

the prior year and found no statistically significant within-group changes over 1 year (Table 9.6). 

Table 9.5. Within-Group Differences in Housing Status at 12 Months (Weighted)

Housing at 12 Months

Outcome

Baseline Percentage

(Retained Sample)

12 Months

(%)

Difference

(%) p-Value

Currently stably housed/staying 

with someone else b C2C

Comparison

80.31

85.33

88.81

90.31

8.50

4.98

<.001a

<.001a

a Statistically significant at the p<.05 level. b Versus transitional/temporary or homeless/unstably housed.

Table 9.6. Within-Group Changes in Homelessness at 12 Months (Weighted)

Homelessness at 12 Months (Imputed)

Outcome

Baseline Percentage 

(Retained Sample) 1 Year (%) Difference (%) p-Value

Homeless, past year C2C

Comparison

16.41

11.77

17.05

9.66

+0.64

–2.11

>0.99

0.609

For educational attainment, we found statistically significant increases in both the C2C group 

(4.7 percentage points) and the comparison group (3.9 percentage points) in the proportion of 

persons who had completed a high school diploma, GED, or more at 12 months (Table 9.7). 

Results were similar at the 6-month follow-up, both groups showing statistically significant 

improvements from baseline (see Appendix C, section C8.1). 

For incarceration status, each group decreased the proportion incarcerated over 1 year, but 

the improvement was not statistically significant for either group (Table 9.8). 

Table 9.7. Within-Group Differences in Educational Attainment at 12 Months (Weighted)

Educational Attainment at 12 Months

Baseline Percentage 

Outcome (Retained Sample)

Completed high school or C2C 76.92

GED or greatera

Comparison 74.09

a Among persons ages 18 and older at baseline. b Statistically significant at the p<.05 level.

12 Months

(%)

81.66

77.95

Difference (%)

4.74

3.87

p-Value

0.001b

0.019b

208 
 

 

 

 

 

Table 9.8. Within-Group Changes in Incarceration at 12 Months (Weighted)

Incarceration at 12 Months (Imputed)

Outcome

Baseline Percentage 

(Retained Sample) 1 Year (%)

Difference 

(%) p-Value

Incarcerated, past year C2C

Comparison

11.32

6.79

4.38

4.44

−6.94

−2.35

0.122

>0.99

We next examined the intervention effect over time using an intent-to-treat approach, in 

which all individuals in the C2C group were compared with all those in the comparison group 

regardless of the amount of intervention (i.e., dosage or exposure) the C2C group received. 

For continuous outcomes, we report standardized effect sizes. For binary outcomes, we report 

OR, which represents the odds of C2C participants indicating a given outcome, relative to the 

comparison group. An OR of 1.0 means that the groups were equally likely to report the outcome, 

and a 95% CI that includes the value of 1.0 means that we cannot rule out that there is no group 

difference. We do not show model results when the sample size is less than ten participants per 

group for the propensity score weighted model and less than 20 per group for the doubly robust 

models (noted by gray shading in the tables). 

For employment-related outcomes, we considered changes in full- or part-time employment 

versus unemployment, hours worked per week, and current monthly pay at 12 months (Table 9.9). 

For full-/part-time employment, there were no statistically significant intervention effects, in 

either the propensity score weighted or doubly robust models. For weekly work hours, there was 

a statistically significant intervention effect in the propensity score weighted model, such that 

the C2C group had a greater increase in weekly work hours than the comparison group. In the 

doubly robust model, where the standardized effect size was small (d = 0.16), there was evidence 

of a statistically significant intervention effect before the multiple comparisons adjustment. For 

monthly employment pay, there were also no statistically significant intervention effects in the 

propensity score weighted or doubly robust models. At the 6-month follow-up, there were no 

statistically significant intervention effects in the propensity score weighted or doubly robust 

models for any of the employment-related outcomes (see Appendix C, section C8.1). 

For housing-related outcomes, we looked at changes in stably housed/staying with someone 

and homelessness. For stably housed/staying with someone else, there were no statistically 

significant intervention effects in either model at 12 months (Table 9.9) or 6 months (see 

Appendix C, section C8.1). We used the pooled 6- and 12-month data to assess homelessness over 

the past year and found no statistically significant intervention effect in either model (Table 9.8). 

For the educational attainment outcome, there were no statistically significant intervention 

effects in either model at 12 months (Table 9.9) or 6 months (see Appendix C, section C8.1). 

For the incarceration outcome, we used the pooled 6- and 12-month data to assess 

incarceration over the past year and found no evidence of an intervention effect in the propensity 

score weighted model (Table 9.9). 

209 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 9.9. Between-Group Differences in Employment, Housing, Education, and Incarceration at 12 Months (Weighted)

Group-Level Comparison of Outcomes Standardized 

Propensity Score Weighted Modela

Estimate SE or

Doubly Robust Modelb

Estimate SE or

Effect Size

for Doubly

Robust

Outcome or OR 95% CI p-Value or OR 95% CI p-Value Model

Employment at 12-month follow-up

Full-/part-time employment (vs.

unemployment)c OR = 1.13 [0.77–1.66] 0.537 OR = 1.13 [0.68–1.88] 0.651 –

Change in hours worked per week 6.39 1.88 0.001d 3.21 1.59 0.044d,e 0.155

Change in current monthly pay before taxes

and deductions

271 164 0.099 172 152 0.260 0.098

Current housing at 12-month follow-up

Stably housed or staying with someone elsef OR = 0.85 [0.47–1.55] 0.599 OR = 1.06 [0.45–.47] 0.899 –

Recent homelessness—over 1 year

Homeless (past year)—imputed OR = 0.52 [0.19–1.44] 0.898 OR = 0.60 [0.21–1.70] 0.834 –

Education at 12-month follow-up

Completed high school or greaterg OR = 1.26 [0.83–1.92] 0.271 OR = 1.10 [0.54–2.26] 0.796 –

Incarceration—over 1 year

Incarcerated (past year)—imputed OR = 1.03 [0.24–4.07] 0.506

NOTES: Change estimates are not shown when the group size is fewer than ten, and comparisons are not shown when the group size is fewer than ten for

either group. Doubly robust model results are not shown when the group size is fewer than 20 for either group. Affected cells are shaded in gray. SE = standard 

error.

a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status,

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 12 months. Does not include persons who

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Statistically significant at the p<.05 level. e Not statistically significant at p<.05 after adjusting for multiple comparisons. f

Versus transitional/temporary or homeless/unstably housed. g Among persons 18 and older at baseline.

210 
 

 

 

 

 

Secondary Research Question: Does Program Effectiveness Vary for the Different 

Target Populations or CBO Service Types?

We assessed whether C2C program effectiveness varied for different target populations (e.g., 

adults age 18 or older who are unemployed or underemployed, young adults ages 16–24 who are 

not in school and are not employed, and parents/primary caregivers who are expecting or who 

have children ages 4 and younger) or CBO service types (e.g., job training and employment 

program, youth development program). Note that these subgroups are not mutually exclusive; for 

example, there is expected overlap between the target population of job training and employment 

CBO program clients (CBO-type subgroup) and the target population of adults 18 and older who 

are underemployed or unemployed (client-type subgroup). We examined the same housing, 

employment, education, and incarceration outcomes described in the main analyses. 

Changes over Time Within Each Group

For each subgroup, we first examined differences within each group over time. In this 

comparison, we considered the within-individual change between baseline and the follow-up 

time points and examined the average change and its direction across participants in each group. 

Below, we summarize the statistically significant within-group differences at 12 months for 

housing, employment, education, and incarceration outcomes across the different subgroups (see 

Appendix C, section C8.3). None of the other subgroup findings were statistically significant. 

CBO service type

• Job training and employment program: 

- Both C2C and comparison participants had increases in full-/part-time 

employment, weekly work hours, and monthly pay. 

- C2C participants also had increases in current stable housing and decreases in 

past-year incarceration. 

• Youth development program: 

- C2C participants had increases in the percentage with a high school diploma 

or GED. 

- Comparison participants had increases in full-/part-time employment, hours 

worked per week, and before the multiple comparisons correction, an increase in 

monthly employment pay. 

C2C target population

• Adults age 18 or older who are unemployed or underemployed: 

- Both C2C and comparison participants had increases in full-/part-time 

employment, weekly work hours, and monthly pay. 

- C2C participants also had increases in current stable housing and a reduction in 

past-year incarceration. 

211 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

• Young adults ages 16 to 24 who are not in school and are not employed: 

- Both C2C and comparison participants had increases in weekly work hours and 

monthly pay. 

- C2C participants also had a reduction in past-year homelessness. 

• Parents/primary caregivers who are expecting or who have children up to age 4: 

- C2C participants had increases in full-time employment, weekly work hours, 

monthly pay, and stable housing. 

- Comparison participants had increases in monthly pay and stable housing. 

C2C Intervention Effects

For each subgroup, we also examined the intervention effect over time using the intent-totreat approach described earlier. Below, we summarize the statistically significant intervention 

effects across the subgroups at 6 and 12 months, which show mixed results, with some favoring 

the C2C group and others favoring the comparison group (see Appendix C, section C8.3). None 

of the other subgroup findings were statistically significant. Cell sizes for past-year homelessness 

and past-year incarceration were typically too small for analysis of intervention effects. 

CBO service type

• Job training and employment program: 

- C2C participants at 12 months had a greater increase in weekly work hours 

(in the propensity score weighted model and the doubly robust model). The 

standardized effect size for the doubly robust model was small to medium 

(d = 0.37). 

- C2C participants at 12 months had a greater increase in monthly pay (in the 

propensity score weighted model and the doubly robust model before the multiple 

comparisons adjustment). The standardized effect size for the doubly robust 

model was small (d = 0.20). 

- C2C participants at 6 months were less likely to have completed a high school 

diploma or GED (OR = 0.24; 95% CI: 0.06–0.96) in the doubly robust model 

before the multiple comparisons adjustment. 

• Youth development program: 

- C2C participants at 6 months had a smaller increase in weekly work hours (in the 

propensity score weighted model before the multiple comparisons adjustment). 

The standardized effect size for the nonsignificant doubly robust model was very 

small (d = −0.06). 

C2C target populations

• Adults age 18 or older who are unemployed or underemployed: 

- C2C participants at 12 months had a greater increase in weekly work hours (in 

the propensity score weighted model). The standardized effect size for the 

nonsignificant doubly robust model was very small (d = 0.08). 

- C2C participants at 12 months were more likely to report past-year homelessness 

(OR = 3.92; 95% CI: 1.89–8.14) in the propensity score weighted model. 

212 
 

 

 

 

 

 

 

 

• Young adults ages 16 to 24 who are not in school and are not employed: 

- C2C participants at 12 months had a greater increase in weekly work hours 

(in the propensity score weighted model before the multiple comparisons 

adjustment). The standardized effect size for the nonsignificant doubly robust 

model was small and in the opposite direction (d = −0.10) of the propensity score 

weighted model coefficient, which makes it difficult to interpret this finding. 

• Parents/primary caregivers who are expecting or who have children up to age 4: 

- C2C participants at 12 months had a greater increase in weekly work hours 

(in the propensity score weighted model before the multiple comparisons 

adjustment). The standardized effect size for the nonsignificant doubly robust 

model was small (d = 0.20). 

- C2C participants at 6 months had a smaller increase in weekly work hours (in the 

propensity score weighted model before the multiple comparisons adjustment). 

The standardized effect size for the nonsignificant doubly robust model was very 

small (d = −0.04). 

Discussion

C2C was designed to improve access to evidence-informed mental health services for at-risk 

populations through a task-shifting approach that integrated certain mental health skills and 

supports into the work of CBOs with the goal of improving both mental health and non–mental 

health outcomes. With our examination of employment, housing, education, and incarceration 

outcomes, we tested whether the C2C group showed greater improvements than the comparison 

group on these outcomes. Overall, we found an intervention effect showing a greater increase in 

weekly work hours among C2C participants but no other intervention effects in the main analysis 

(Table 9.10). We did, however, find several subgroup intervention effects. 

Our findings suggest that C2C had no effect on these outcomes. We note that C2C clients 

who participated in the evaluation represent a small portion of overall C2C clients. As a quasiexperimental evaluation, we sought to include a diverse sample of participants across CBOs and 

across the implementation timeline and also to include a similar non-C2C comparison group in 

recruitment efforts and in analysis (using propensity weighting). However, this impact evaluation 

may not represent the effects of C2C on all CBO clients. 

Overall Group Findings

This discussion summarizes our findings and describes a similar pattern of within-group 

changes across the outcomes. 

Employment

At baseline, the vast majority of study participants were unemployed; only about one in five 

reported full- or part-time employment. On average, employed participants were working only 

213 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 9.10. Summary of Employment, Educational Attainment, Housing, and Incarceration

Outcomes in the Overall Sample

Outcome

Difference Within Groups

over Time Intervention Effect

Employment status

Full-/part-time employment vs.

unemployment

Increased in both groups

Hours worked per week Increased in both groups Greater increase in C2C group vs.

comparison group (12-month PSW

model; 12-month DRM only before 

MCC)

Monthly employment pay Increased in both groups

Housing status

Stably housed or staying with 

someone else vs. homelessness

Increased in both groups

(current)

Homelessness (past year)

Educational attainment

Completed high school diploma or

general educational

Increased in both groups

Incarceration status

NOTES: PSW = propensity score weighted; DRM = doubly robust model; MCC = multiple comparisons correction.

about 7 hours per week, and current monthly employment pay was about $330 per month. The 

rates of self-reported unemployment in this sample were substantially higher than current 

estimates in the general population nationally (e.g., about 4 percent in February 2020) and in 

the NYC metropolitan region (e.g., about 3 percent in December 2019) (U.S. Bureau of Labor 

Statistics, 2020a; U.S. Bureau of Labor Statistics, 2020b), although these rates are calculated 

very differently. 

Overall, both groups improved significantly over time for all three of the employment-related 

indicators (full- or part-time employment vs. unemployment, hours worked per week, and 

monthly employment pay). With this level of improvement for both groups, we did not find a 

significant intervention effect at 6 months for any of the employment indicators in doubly robust 

or propensity score weighted models. Given that two-thirds of participants at baseline were 

recruited from job training and employment programs, some short-term improvement in these 

outcomes for both groups is not surprising. For full- or part-time employment, we saw withingroup improvements of 23 to 31 percentage points at 6 months, meaning that about two in three 

persons in each group had full-time or part-time employment at 6 months. Prevalence of full- or 

part-time employment was similar at 12 months, with the C2C group doubling its employment 

rate from baseline. 

With longer follow-up and supporting our hypotheses, we found evidence of an intervention 

effect related to the number of hours worked per week at 12 months. In the propensity score 

weighted model, C2C participants had a stronger increase in weekly work hours than comparison 

214 
 

 

 

participants, although the effect size was small (d = 0.16) and no longer significant after adjusting 

for multiple comparisons. This finding was replicated in multiple subgroups as well, indicating 

that C2C may improve this employment indicator overall and also for key subpopulations. The 

greater increase among C2C participants in weekly work hours could reflect greater full-time 

(vs. part-time) employment, a greater likelihood of having one or more jobs, and/or greater 

engagement in one or more jobs. It is notable that we did not find a concordant intervention 

effect for monthly employment pay, which may indicate that, although C2C participants are 

working more, mean earnings in the two groups did not differ as substantially. With longer-term 

follow-up beyond 1 year, it is possible that this increased job engagement (hours worked) could 

spur later increases and/or better sustainment of income over the long-term for the C2C group. 

We did not find intervention effects for the binary indicator of full- or part-time employment, 

which could be explained by a true lack of differences in this outcome, or insufficient power for 

this indicator. 

Housing

With regard to housing status at baseline, three-quarters of study participants had stable 

housing or were staying with someone else at baseline. The relatively high proportion staying 

with someone else in our study might reflect adult children living with other family members or 

staying with friends, owing both to affordable housing issues in NYC and the very high rates of 

baseline unemployment in this sample. We also note the very high prevalence of past-year 

homelessness at baseline, despite the fact that most persons were currently stably housed or 

staying with someone else. About 19 percent of the overall sample reported that there was a 

period of time in the past year when they were homeless by definition (without a place to sleep). 

This prevalence is substantially higher than the estimate of past-year homelessness in a nationally 

representative sample of adults (1.5 percent) (Tsai, 2018). 

Both groups showed statistically significant improvements in their housing status over time. 

The proportion of persons who were stably housed or staying with someone else increased in 

both groups, and there was no intervention effect for current housing status. However, when we 

examined past-year homelessness during the follow-up period rather than current housing, there 

was no significant change between baseline and follow-up for either group, nor was there an 

intervention effect. In a sensitivity analysis for past-year homelessness using non-imputed data, 

we found that C2C participants were more likely to experience homelessness during follow-up, a 

difference that may have been driven by better retention of homeless and unstably housed 

participants in the C2C group, attributed to the research team’s consistent, long-term contact 

with the C2C CBOs (relative to the comparison CBOs) through the broader C2C evaluation. 

These findings suggest that C2C is not able to reduce risk for homelessness or to improve 

housing stability, at least in the short term. This is not entirely surprising, because homelessness 

and housing stability are tied to many other factors beyond mental health, such as the current 

shortage of affordable housing in NYC (City of New York, 2017). 

215 
 

 

 

 

 

 

Educational Attainment

For educational attainment, about one-third of persons ages 18 and older had not completed 

high school at baseline. This is about three times higher than estimates of high school noncompletion in the general population, at about 12 percent (U.S. Census Bureau, 2017). When 

examining changes in educational attainment, we again observed both groups improving over 

time with increases in the proportion of persons with a high school diploma, GED, or greater. 

We did not find statistically significant intervention effects at 6 or 12 months, suggesting that the 

C2C intervention did not outperform usual CBO services in its effect on educational attainment. 

However, it is worth noting that the 12-month follow-up period may have been too short to 

detect larger changes in this outcome; GED programs could take years to complete, particularly 

if individuals are also working and attending to family obligations simultaneously. 

Incarceration

Finally, we also saw high rates of past-year incarceration at baseline (about 13 percent). This 

is higher than the prevalence of lifetime criminal justice involvement found in analysis of 2017 

NYC Community Health Survey data (approximately 8 percent), weighted to be representative 

of adult New Yorkers (Baquero et al., 2020). The high prevalence in the C2C evaluation sample 

is likely attributable to the C2C population being served (e.g., CBOs offering reentry programs). 

Although the prevalence of incarceration appeared to decrease over the 1 year of follow-up, 

these changes were not statistically significant. It is also worth noting that incarceration was so 

infrequent in the retained sample that we were only able to implement the propensity score 

weighted model due to the small sample size, even after imputing data. With nonimputed data, 

we were unable to model intervention effects for incarceration. It is important to note that the 

retained sample is likely to underrepresent incarceration in the evaluation sample because it 

would have been difficult or impossible to reach currently incarcerated participants to schedule 

their follow-up survey, and we could not complete surveys with an individual if they were 

currently incarcerated. 

Variation in C2C’s Effectiveness for Different Subgroups

We also examined within-group changes and intervention effects over time separately for the 

three C2C target populations and two of the CBO types. Overall, subgroups reported important 

within-group improvements in employment, housing, and education, often in indicators that were 

clearly linked with their program’s goals or in indicators that defined the subgroup. For example, 

in both groups, job training and employment program attendees reported improvements in 

full- or part-time employment, hours worked per week, and monthly pay. Similarly, the target 

population of young adults who are not in school and are not employed also showed withingroup improvements in employment indicators. 

For clients of job training and employment programs, we found two intervention effects at 

12 months, which were consistent with our hypotheses: C2C participants had greater increases in 

216 
 

 

 

 

 

weekly work hours and monthly pay than comparison participants in the propensity score 

weighted model. With the doubly robust model that incorporated more control variables, 

C2C clients still showed greater increases in weekly work hours, with a small-to-medium 

standardized effect size (d = 0.37). Unlike the intervention effect for weekly work hours, the 

intervention effect for monthly pay was no longer significant after adjusting for multiple 

comparisons, possibly because it was a smaller effect (d = 0.20). Together, these two positive 

indicators suggest that C2C is associated with positive employment-related effects for this 

CBO subgroup, very much in line with these CBOs’ program missions. 

The evaluation survey included three objective indicators related to employment (full- or 

part-time, hours worked, monthly pay). There are other objective indicators (e.g., duration of 

current employment) and subjective indicators (e.g., job satisfaction) that we did not assess in 

this evaluation that could help provide a multifaceted perspective on employment outcomes. It is 

also possible that the intervention effect observed here reflects higher quality “usual” job training 

and employment programs at the C2C CBOs relative to the comparison CBOs. We were not able 

to systematically measure the quality of the overall job training and employment programs in this 

study, but this was a factor we considered in recruiting comparison CBO sites. Every attempt 

was made to recruit comparison CBOs with a reputation for high-quality programming. 

At 6 months, there was evidence that C2C was associated with a lower likelihood of high 

school or GED completion in this subgroup, but this finding was not significant after adjusting 

for multiple comparisons, and there was no intervention effect at 12 months. As with educational 

attainment, the 6- and 12-month time periods may be too short to allow for observable intervention 

effects in this particular outcome. 

Among the target population of unemployed or underemployed adults, we found two 

intervention effects at 12 months. Both of these findings only emerged in the propensity score 

weighted model. First, C2C participants had a greater increase in weekly work hours relative to 

the comparison group, which may reflect substantial overlap between this target population and 

the CBO programming type of job training and employment programs, where we also saw 

positive intervention effects for employment outcomes. Again, this finding is consistent with our 

hypothesis and with findings in the overall sample. We note, however, that the effect size for 

this target population was much smaller than the effect size for the job training and employment 

programs (d = 0.08 vs. d = 0.37). This may not be surprising, given that the job training and 

employment programs directly target employment outcomes in their programming, while the 

target population of unemployed adults encompasses clients from any CBO type in the C2C 

evaluation. Still, this suggests that C2C may be more potent for improving employment 

outcomes in the specific context of employment programming. 

Second, for this target population of unemployed and underemployed adults, we found that 

C2C clients were more likely to have experienced homelessness in the past year than comparison 

clients. The effect size for this finding was fairly large (OR of about 4), but we also note the very 

large confidence interval for this effect size (about 2–8), which indicates some lack of precision 

217 
 

 

 

in this estimate. In addition, we could not implement the doubly robust model to adjust this 

estimate for other covariates that might explain differential patterns of homelessness. Therefore, 

it is possible that this finding represents unobserved differences in the C2C and comparison 

samples of unemployed/underemployed adults at baseline and/or follow-up. It is also worth 

noting that the C2C sample had a large proportion of participants (about 20 percent of the C2C 

group baseline sample) recruited from a CBO that focuses on recently incarcerated individuals 

and those on probation and parole, a population known to be at high risk for unstable housing 

and homelessness (Cortes and Rogers, 2010). 

Other subgroup-related findings were not statistically significant after adjustment and also 

tended to have small effect sizes. For example, we found potential intervention effects for weekly 

work hours in the additional subgroups of young adults who were not in school and not employed, 

parents and primary caregivers, and youth development CBO clients, but none of these were 

statistically significant after multiple comparisons corrections, and the effect sizes were very 

small. The largest effect was for parents and caregivers, and even this was small (d = 0.20). 

We were almost never able to model intervention effects for recent homelessness and recent 

incarceration in the subgroups due to small cell sizes. Several interpretations are worth noting 

here. First, this may be an encouraging finding, in the sense that these two negative outcomes 

were very rare during follow-up. This could suggest that both C2C-participating CBOs and 

comparison CBOs are addressing client needs and preventing two especially negative outcomes. 

A more tempered explanation may involve differences in study retention among persons with 

these two outcomes in both groups. As noted earlier, we could not conduct follow-up surveys 

with persons who were currently incarcerated; it was also difficult to reach and complete surveys 

with persons who were homeless at the time of the follow-up. 

Limitations

The evaluation findings should be considered in light of limitations outlined more fully in 

Chapter 7. 

First, the evaluation sample is not representative of the entire C2C and comparison client 

populations because they represent only a fraction of the clients served by these CBOs, and 

eligibility was contingent on participants experiencing at least mild mental health symptoms at 

baseline. Second, participants also opted-in to eligibility screening and study participation if 

eligible, and retention rates were low, which is relevant because factors such as employment, 

housing, and incarceration can drive differential attrition. Third, all data were self-reported and 

therefore are subject to social desirability and recall bias in responses. Although we attempted to 

obtain administrative data from NYS and NYC agencies to obtain more objective estimates of 

these non–mental health outcomes, bureaucratic and COVID-19-related delays prevented us 

from obtaining those data in time for them to be used in these analyses. It is possible that selfreported data are under- or overestimates; fourth, we were not able to include measures of 

218 
 

fidelity or C2C implementation, or to account for other services that participants in both groups 

may have been receiving from their CBO. Finally, the C2C skills may not be sufficient on their 

own to directly affect the outcomes we examined in this chapter. 

Summary

In summary, we found occasional evidence for C2C intervention effects, particularly related 

to employment outcomes. At 12 months, C2C participants had greater increases in weekly work 

hours, albeit in the less robust model. We also found several subgroup intervention effects. The 

most notable findings emerged for job training and employment participants, where the C2C arm 

had greater increases in weekly work hours and monthly pay; the finding for weekly work hours 

was especially robust and of a small-to-medium effect size. Finally, we note that nearly all of the 

indicators across outcomes showed statistically significant within-group improvements at followup in the main analysis and often in the subgroups. This may reflect positive effects of the 

underlying services provided by the CBOs per their missions and programming, and also the 

many concurrent ThriveNYC initiatives during the implementation of C2C. 

219 
 

 

 

 

References

Acker, J., P. Braveman, E. Arkin, L. Leviton, J. Parsons, and G. Hobor, Mass Incarceration 

Threatens Health Equity in America. Executive Summary, Princeton, NJ: Robert Wood 

Johnson Foundation, 2019. 

https://www.rwjf.org/en/library/research/2019/01/mass-incarceration-threatens-health-equity 

-in-america.html 

Antonisse, L., and R. Garfield, The Relationship Between Work and Health: Findings from a 

Literature Review, San Francisco, CA: Henry J. Kaiser Family Foundation, 2018. 

https://www.kff.org/medicaid/issue-brief/the-relationship-between-work-and-health-findings 

-from-a-literature-review/ 

Aubry, T., A. Duhoux, F. Klodawsky, J. Ecker, and E. Hay, “A Longitudinal Study of Predictors 

of Housing Stability, Housing Quality, and Mental Health Functioning Among Single 

Homeless Individuals Staying in Emergency Shelters, American Journal of Community 

Psychology, Vol. 58, No. 1–2, 2016, pp. 123–135. 

Baggett, T. P., J. J. O’Connell, D. E. Singer, and N. A. Rigotti, “The Unmet Health Care Needs 

of Homeless Adults: A National Study,” American Journal of Public Health, Vol. 100, No. 7, 

2010, 1326–1333. 

Baquero, M., K. Zweig, S. Y. Angell, and S. B. Meropol, “Health Behaviors and Outcomes 

Associated with Personal and Family History of Criminal Justice System Involvement, 

New York City, 2017. American Journal of Public Health, Vol. 110, No. 3, 2020, 

pp. 378–384. 

Begun, A. L., T. J. Early, and A. Hodge, “Mental Health and Substance Abuse Service 

Engagement by Men and Women During Community Reentry Following Incarceration,” 

Administration and Policy in Mental Health, Vol. 43, No. 2, pp. 207–218. 

Breslau, J., M. Lane, N. Sampson, and R. C. Kessler, “Mental Disorders and Subsequent 

Educational Attainment in a U.S. National Sample, Journal of Psychiatric Research, Vol. 42, 

No. 9, 2008, pp. 708–716. 

Bronson, J., and M. Berzofsky, Indicators of Mental Health Problems Reported by Prisoners and 

Jail Inmates, 2011–12 (NCJ 250612), 2017. 

http://www.bjs.gov/index.cfm?ty=pbdetail&iid=5946 

City of New York, Housing New York: A Five-Borough, Ten-Year Plan, New York, NY, 2017. 

As of May 19, 2020: 

https://www1.nyc.gov/site/housing/about/our-plan.page 

220 
 

Commission on the Social Determinants of Health, Closing the Gap in a Generation: Health 

Equity Through Action on the Social Determinants of Health. Final Report of the Commission 

on Social Determinants of Health, Geneva: World Health Organization, 2008. As of 

September 9, 2020: 

http://www.who.int/social_determinants/thecommission/finalreport/en/ 

Cortes, K., and S. Rogers, Reentry Housing Options: The Policymakers’ Guide, New York, NY: 

Council of State Governments Justice Center, 2010. 

Davey-Rothwell, M. A., D. German, and C. A. Latkin, “Residential Transience and Depression: 

Does the Relationship Exist for Men and Women?” Journal of Urban Health, Vol. 85, No. 5, 

2008, pp. 707–716. 

Domino, M. E., A. Gertner, B. Grabert, G. S. Cuddeback, T. Childers, and J. P. Morrissey, “Do 

Timely Mental Health Services Reduce Re-Incarceration Among Prison Releasees with 

Severe Mental Illness?,” Health Services Research, Vol. 54, No. 3, 2019, pp. 592–602. 

Duke, A., and A. Searby, “Mental Ill Health in Homeless Women: A Review,” Issues in Mental 

Health Nursing, Vol. 40, No. 7, 2019, pp. 605–612. 

Edidin, J. P., Z. Ganim, S. J. Hunter, and N. S. Karnik, “The Mental and Physical Health of 

Homeless Youth: A Literature Review,” Child Psychiatry and Human Development, Vol. 43, 

No. 3, 2012, pp. 354–375. 

Esch, P., V. Bocquet, C. Pull, S. Couffignal, T. Lehnert, M. Graas, L. Fond-Harmant, and 

M. Ansseau, “The Downward Spiral of Mental Disorders and Educational Attainment: A 

Systematic Review on Early School Leaving,” BMC Psychiatry, Vol. 14, 2014, p. 237. 

Fazel, S., J. R. Geddes, and M. Kushel, “The Health of Homeless People in High-Income 

Countries: Descriptive Epidemiology, Health Consequences, and Clinical and Policy 

Recommendations,” The Lancet, Vol. 384, No. 9953, 2014, pp. 1529–1540. 

Fedock, G., and S. Sarantakos, “Physical and Mental Health Disparities for Young Women with 

Arrest Histories,” Health & Social Work, Vol. 42, No. 2, 2017, pp. e102–e110. 

Fergusson, D. M., and L. J. Woodward, “Mental Health, Educational, and Social Role Outcomes 

of Adolescents with Depression,” Archives of General Psychiatry, Vol. 59, No. 3, 2002, 

pp. 225–231. 

Fletcher, J. M. “Adolescent Depression and Educational Attainment: Results Using Sibling Fixed 

Effects,” Health Economics, Vol. 19, No. 7, 2010, pp. 855–871. 

Freudenberg, N., and J. Ruglis, “Reframing School Dropout as a Public Health Issue,” Preventing 

Chronic Disorders, Vol. 4, No. 4, 2007, p. A107. 

221 
 

 

 

 

 

 

Gabrielian, S., A. V. Burns, N. Nanda, G. Hellemann, V. Kane, and A. S. Young, “Factors 

Associated with Premature Exits from Supported Housing,” Psychiatric Services, Vol. 67, 

No. 1, 2016, pp. 86–93. 

Gmitroski, T., C. Bradley, L. Heinemann, G. Liu, P. Blanchard, C. Beck, S. Mathias, A. Leon, 

and S. P. Barbic, “Barriers and Facilitators to Employment for Young Adults with Mental 

Illness: A Scoping Review,” BMJ Open, Vol. 8, No. 12, 2018, p. e024487. 

Goering, P., G. Tolomiczenko, T. Sheldon, K. Boydell, and D. Wasylenki, “Characteristics of 

Persons Who Are Homeless for the First Time,” Psychiatric Services, Vol. 53, No. 11, 2002, 

pp. 1472–1474. 

Greenberg, G. A., and R. A. Rosenheck, “Mental Health and Other Risk Factors for Jail 

Incarceration Among Male Veterans,” Psychiatry Quarterly, Vol. 80, No. 1, 2009, pp. 41–53. 

Harrison, J., M. J. Krieger, and H. A. Johnson, “Review of Individual Placement and Support 

Employment Intervention for Persons with Substance Use Disorder,” Substance Use & 

Misuse, Vol. 55, No. 4, 2020, pp. 636–643. 

Hawthorne, W. B., D. P. Folsom, D. H. Sommerfeld, N. M. Lanouette, M. Lewis, G. A. Aarons, 

R. M. Conklin, E. Solorzano, L. A. Lindamer, and D. V. Jeste, “Incarceration Among Adults 

Who Are in the Public Mental Health System: Rates, Risk Factors, and Short-Term Outcomes,” 

Psychiatric Services, Vol. 63, No. 1, 2012, pp. 26–32. 

Hergenrather, K. C., R. J. Zeglin, M. McGuire-Kuletz, and S. D. Rhodes, “Employment as a 

Social Determinant of Health: A Review of Longitudinal Studies Exploring the Relationship 

Between Employment Status and Mental Health,” Rehabilitation Research Policy and 

Education, Vol. 29, No. 3, 2015, pp. 261–290. 

Kim, M. M., and J. D. Ford, “Trauma and Post-Traumatic Stress Among Homeless Men,” 

Journal of Aggression, Maltreatment & Trauma, Vol. 13, No. 2, 2006, pp. 1–22. 

Lerner, D., and R. M. Henke, “What Does Research Tell Us About Depression, Job Performance, 

and Work Productivity?” Journal of Occupational and Environmental Medicine, Vol. 50, 

No. 4, 2008, pp. 401–410. 

McGee, R. E., and N. J. Thompson, “Unemployment and Depression Among Emerging Adults 

in 12 States, Behavioral Risk Factor Surveillance System, 2010,” Preventing Chronic 

Disease, Vol. 12, 2015, p. E38. 

McKee-Ryan, F., Z. Song, C. R. Wanberg, and A. J. Kinicki, “Psychological and Physical WellBeing During Unemployment: A Meta-Analytic Study,” Journal of Applied Psychology, 

Vol. 90, No. 1, 2005, pp. 53–76. 

National Health Care for the Homeless Council, Social Determinants of Health: Predictors of 

Health Among People Without Homes, Nashville, TN, 2016. 

http://councilbackup.flywheelsites.com/wp-content/uploads/2011/09/fact-sheet_2016_social 

-determinants-of-health1.pdf 

222 
 

 

 

Paul, K. I., and K. Moser, “Unemployment Impairs Mental Health: Meta-Analyses,” Journal of 

Vocational Behavior, Vol. 74, No. 3, 2009, pp. 264–282. 

Prins, S. J. (2014). Prevalence of Mental Illnesses in US State Prisons: A Systematic Review. 

Psychiatric Services, Vol. 65, No. 7, pp. 862–872. 

Substance Abuse and Mental Health Services Administration, “Behavioral Health Services for 

People Who Are Homeless,” 2013. 

https://www.ncbi.nlm.nih.gov/books/NBK138725/ 

Sugie, N. F., and K. Turney, “Beyond Incarceration: Criminal Justice Contact and Mental Health, 

American Sociological Review, Vol. 82, No. 4, 2017, pp. 719–743. 

Suglia, S. F., C. S. Duarte, and M. T. Sandel, “Housing Quality, Housing Instability, and 

Maternal Mental Health,” Journal of Urban Health, Vol. 88, No. 6, 2011, pp. 1105–1116. 

Sullivan, G., A. Burnam, and P. Koegel, “Pathways to Homelessness Among the Mentally Ill, 

Social Psychiatry and Psychiatric Epidemiology, Vol. 35, No. 10, 2000, pp. 444–450. 

Tsai, J. “Lifetime and 1-Year Prevalence of Homelessness in the US Population: Results from 

the National Epidemiologic Survey on Alcohol and Related Conditions-III,” Journal of 

Public Health, Vol. 40, No. 1, 2018, pp. 65–74. 

U.S. Bureau of Labor Statistics, Employment Situation Summary. 2020a. 

www.bls.gov/CES 

U.S. Bureau of Labor Statistics, New York-New Jersey—Labor Force Statistics, 2020b. 

U.S. Census Bureau, Educational Attainment in the United States: 2017, 2017. 

https://www.census.gov/data/tables/2017/demo/education-attainment/cps-detailed-tables.html 

Van der Noordt, M., I. H. Jzelenberg, M. Droomers, and K. I. Proper, “Health Effects of 

Employment: A Systematic Review of Prospective Studies,” Occupational and 

Environmental Medicine, Vol. 71, No. 10, Oct, 2014, pp. 730–736. 

Wanberg, C. R., “The Individual Experience of Unemployment,” Annual Review of Psychology, 

Vol. 63, 2012, pp. 369–396. 

Wang, P. S., M. Lane, M. Olfson, H. A. Pincus, K. B. Wells, and R. C. Kessler. (2005). “TwelveMonth Use of Mental Health Services in the United States: Results from the National 

Comorbidity Survey Replication.” Archives of General Psychiatry, Vol. 62, No. 6, 629–640. 

doi:10.1001/archpsyc.62.6.629 

223 
Part IV: Cost

Before implementing any new program, organizations—particularly community-serving 

organizations that are likely to be resource constrained—will need to have a good understanding 

of the expected cost of the program they are about to implement. C2C was a novel intervention, 

so the cost evaluation sought to keep track of what CBOs needed to implement and maintain 

the program. This information is useful for comparison among the various types of CBOs that 

participated, but also to inform future implementations of C2C or programs like it. 

Chapter 10 examines the resources required to implement and maintain the C2C program, 

both in terms of overall cost and cost per client served, from the perspective of CBOs. This 

analysis uses a “micro-costing” approach in which all resources used by CBOs to implement the 

intervention and maintain it are factored into the cost estimates. Because of the heterogeneity 

across CBOs, we counted the resources used by intervention CBOs to implement and maintain 

the program instead of comparing cost data between intervention and comparison CBOs. We 

measured the incremental changes in resources, from what they spent on baseline programming 

in year 2 compared with spending in year 3. 

Related material in Appendix D provides detailed results from the sensitivity analyses, which 

were conducted to test the results of the cost study against alternative methods or parameters. 

224 
 

 

10. Resources Required to Implement and Maintain the C2C

Program

Michele Abbott and Harry H. Liu 

Key Findings

• The number of CBO staff members trained in at least one C2C skill increased by 

49 percent from year 2 to year 3. However, with the exception of coordinators hired 

specifically for the C2C program, staff members spent a relatively small portion of their 

time on the program. 

• Annual number of labor hours spent on C2C increased by 26 percent from year 2 to 

year 3. CBO labor activities shifted from receiving training, program management, and 

technical assistance to providing training and receiving coaching and supervision. 

• Due mostly to the increase in staff labor hours, the average annual C2C program cost per 

CBO rose about 18 percent, from $437,546 in year 2 to $514,142 in year 3. 

• In the same time period, the average number of clients served per CBO increased 

31 percent, from 817 per CBO in year 2 to 1,066 in year 3. 

• Across the whole C2C program, the average cost per C2C client served decreased by 

10 percent from $536 in year 2 to $482 in year 3. However, only 6 of the 15 CBOs had a 

year-on-year decrease in cost per C2C client served. 

• Nearly half (44 percent) of the resources consumed in the C2C program were not part of 

the planned budget, and the funder sponsored only half of the rest of C2C program costs 

(28 percent). 

• Potential C2C adopters can use these results (i.e., labor hours and clients served) in 

conjunction with local wages and organization-specific overhead ratios to estimate a 

budget for implementing the C2C program. 

225 
 

 

 

 

Introduction

Integrating mental health support into community-based settings via task-shifting has the 

potential to benefit clients’ mental health and other outcomes. Though multiple reviews have 

called for evidence, there is a dearth of research on cost, cost savings, or economic outcomes of 

task-shifting interventions for mental health (Hoeft et al., 2018; van Ginneken et al, 2013). Reviews 

of research on collaborative care or other low-intensity interventions on mental health also lack 

clarity on cost effectiveness (Barrett, Byford, and Knapp, 2005; Rodgers et al., 2012), but studies 

have been published on intervention cost per patient. A cost evaluation of five collaborative care 

programs for older adults was conducted in which a primary care provider worked with a case 

manager, a psychiatric consultant, CBOs, and family members or friends to form a collaborative 

care team to improve the effectiveness of treating depression in older adults (Hoeft et al., 2019). 

The authors found that the intervention cost per patient per month ranged from $154 to $544 

(in 2018 U.S. dollars). However, the costs of collaborative care models may not be directly 

translatable to task-shifting models like C2C. For one, collaborative care is typically implemented 

in primary care or other medical settings with existing infrastructure and systems that are integrally 

related to program costs. For example, these settings have existing medical records systems, and 

providers can be reimbursed by insurance for the services they provide. Therefore, we would not 

necessarily expect the costs required to implement C2C to be similar to these other models. 

This is the first study, to our knowledge, to estimate the costs of such an intervention in the 

United States. Specifically, this cost evaluation sought to address one primary research question: 

What are the costs to CBOs associated with the implementation of the C2C program, 

overall and on average per program client? We were unable to calculate the marginal cost of 

serving an additional client, due to data limitations explained in the Methods section below. 

Instead, we examined a secondary question in this cost analysis: How did the average cost of 

serving a client change over time? At the start of the evaluation, we had also planned to 

analyze whether the C2C program generates net cost savings to the government. However, 

because of challenges obtaining administrative data (e.g., state and local expenses for health 

care, housing, unemployment benefits, etc.) in time for this analysis, we were not able to answer 

the original cost savings research question as planned. This question could be addressed in a 

separate supplemental report, should these data become available. 

To answer our primary research question, we estimated the resources used to implement and 

maintain the C2C program, including labor, nonlabor, overhead costs, and direct payments from 

CBOs to vendors and MHPs. The cost analysis results can inform government agencies deciding 

whether there are sufficient resources to invest in C2C and in how many CBOs. CBOs thinking 

about implementing C2C may also find these results helpful because they weigh the program’s 

costs and consider how to financially support implementing it in their own context. 

226 
 

 

 

Methods

We followed the guidelines proposed by Crowley et al. (2018) to estimate the C2C program 

intervention cost and present findings, according to which researchers should report the perspective 

for the analysis, time horizon, intervention and comparison conditions, monetary units, methods 

used for estimating costs, cost categories, sources of data, and research limitations. In this cost 

analysis, we adopted a micro-costing approach to enumerate resources utilized by CBOs in the 

implementation and maintenance of the C2C program. The cost analysis was conducted from a 

CBO’s perspective, which means we did not take into account the resources spent by CBO 

clients (e.g., payment for transportation or childcare) who received C2C skills. We were not 

able to receive sufficient administrative data to conduct a benefit-cost analysis that could be 

conducted to estimate net savings of the C2C program from the government perspective. Should 

those data become available, such an analysis could be undertaken in the future. 

Ideally, a comparison group of CBOs would have been included in the cost analysis. However, 

because of the heterogeneity across CBOs in service delivery structure, quantity of clients served, 

and length of time a client is engaged in services—even within the same type of CBO—cost data 

from the comparison CBOs used in the impact evaluation chapters would not be comparable to 

the cost data from CBOs that implemented the C2C program (intervention CBOs). Although 

CBOs serving the same client populations (e.g., parent and caregiver-serving CBOs) are likely 

pursuing similar client outcomes, the resources they consume to deliver services may vary 

widely. For example, one CBO may interact with a parent client once for a referral or community 

event, whereas another CBO may maintain consistent interactions with the client over the course 

of a school year. In consideration of these factors, we measured the incremental resources used 

by intervention CBOs to implement and maintain the program. That is, we used CBO baseline 

programming, or business as usual, for within-CBO comparisons. 

We conducted one main analysis and three sensitivity analyses. Our main analysis estimated 

program resources consumed to implement the C2C program using survey data and participant 

assessments from years 2 (March 2017–February 2018) and 3 (March 2018–February 2019). 

However, this is still an estimate and sensitivity analyses were conducted to verify if the results 

were robust to alternative estimation methods or parameters. The first sensitivity analysis utilized 

an alternative method for overhead cost estimation based on government-negotiated indirect cost 

rates. The second sensitivity analysis relied solely on resources invoiced from CBOs to the 

Mayor’s Fund for the project years 1 to 3 (March 2016–February 2019), rather than survey 

data, to estimate program costs. Finally, the third sensitivity analysis reintroduced labor costs 

associated with C2C skill delivery. As we explain in further detail in the Program-Level Cost 

Input Analysis section, because the delivery of C2C skills was integrated with regular CBO 

programming, it was difficult to parse the effect of C2C skill delivery on labor resources 

consumed in comparison to preexisting programming. Therefore, C2C delivery costs were 

excluded from the main analysis, so this third sensitivity analysis used potential estimates of 

227 
 

 

C2C skill delivery time to approximate costs. All costs have been adjusted for inflation to 2019 

U.S. dollars using the Consumer Price Index from the U.S. Census Bureau (U.S. Bureau of 

Labor Statistics, 2019). 

Data Sources

The cost evaluation was conducted with data collected from several sources. The cost 

measures and supporting data are described below. 

Program-Level Resource Inputs

To generate the total cost of the C2C program for each CBO, we identified and collected data 

required (shown in Figure 10.1) for the CBO program-level cost inputs: labor input, nonlabor 

input, overhead, and direct payments from CBOs to their MHPs and other vendors or consultants 

for the C2C program if relevant (e.g., computer system vendors). Labor input was generated 

based on labor time, in hours, with salaries and fringe benefits. Nonlabor input was generated 

from the depreciation of fixed assets, defined as assets that can be used for 1 or more years, and 

other variable expenses that are typically used up within a year. Overhead inputs refer to the 

overall site management; vendor and consultant costs are direct payments from CBOs. 

Figure 10.1. Program-Level Cost Data Collection

To capture these data, we designed four data collection forms: the Cost Study Section embedded 

in the annual staff survey (Appendix B), the cost of labor survey, the staff compensation report, and 

the nonlabor expenses report. Labor input came from the two surveys and the staff compensation 

228 
 

 

 

 

 

 

report; nonlabor input, direct payments to the MHP and other C2C vendors, and overhead input 

came from the nonlabor expenses report, CBO financial statements, and CBO invoices submitted 

to the Mayor’s Fund. When applicable, the staff compensation report and nonlabor expenses 

report were prepopulated before sharing with CBOs, which then confirmed the accuracy of the 

prepopulated fields. 

Annual Staff Survey: Cost Study Section

As part of the implementation evaluation, RAND conducted annual staff surveys to collect 

information on factors affecting program implementation from the perspective of participating 

CBO staff members. The survey covered a range of topics; a full description of the survey and its 

measures is included in Appendix B. The Cost Study Section consisted of eight questions asking 

respondents to report their level of engagement in a range of C2C activities in a typical week of 

the past month. It was intended to capture the hours per week staff members spent on various 

C2C activities, the number of service sessions delivered in a typical week, and the average length 

of time it takes to deliver each service session. Activities included C2C skills delivery (one or 

more skills), client referrals, training, coaching and supervision, technical assistance, data 

collection and reporting, and program management. 

Cost of Labor Survey

Because of potential seasonal variation in the number of clients served and, correspondingly, 

the amount of labor input for C2C skills, we conducted a cost of labor survey3⁄4with the same 

survey items in the Cost Study Section of the annual staff survey3⁄4between the second annual 

staff survey in year 3 and the third in year 4. We timed the administration of the cost of labor 

survey so that it did not overlap with the time period covered by the annual staff survey. 

Staff Compensation Report

CBOs completed the staff compensation form to report information for all staff members 

who were eligible for the year 3 annual staff survey, including annual salaries and fringe benefits. 

To protect the privacy of CBO staff, the compensation information was reported by job category 

by assigning each staff member to one of five job categories: administrative staff, frontline staff, 

supervisors, leadership, and organizational support staff. Staff members were assigned to 

appropriate job categories by C2C leads. Administrative staff included administrative assistants, 

receptionists, and security guards. Frontline staff included family workers, case managers, 

educators, and social workers. Supervisors had frontline staff as direct reports. Leadership 

included executive- and director-level staff. Organizational support staff included finance, 

contracts, information technology, and human resources staff. We used this information in 

combination with the reported number of hours spent on C2C-related activities to estimate the 

annual labor cost of implementing the C2C program. 

229 
 

 

 

 

 

 

 

 

 

CBO Financial Statements

CBOs’ publicly available audited financial statements were either found online or requested 

from the CBOs’ financial departments for fiscal years 2017 through 2019. We used these 

statements to confirm resources captured in the nonlabor expenses report (i.e., organization-level 

depreciation and total expenses for CBO program services and administration). We calculated 

CBO allocation rates for depreciation, nonlabor, and overhead expenses as the ratio of total C2C 

labor costs (as determined by the staff surveys and compensation reports) to total CBO salary 

and fringe benefits spent on all CBO program services. 

Nonlabor Expenses Report

The nonlabor expense report was designed to be completed by a CBO financial staff member 

to collect the amount of nonlabor associated resources spent by CBOs in the last fiscal year as 

well as any in-kind contributions to program implementation. The report included five key 

subcategories: (1) organization-wide asset depreciation, (2) new asset purchases specifically for 

the C2C program, (3) variable nonlabor expenses, (4) direct payments to mental health partners 

or consultants, and (5) overhead expenses. Except for some recorded expenses specifically for the 

C2C program, categories 1, 3, and 5 typically required an allocation of organization-wide nonlabor 

expenses to the C2C program, using the allocation ratio of C2C labor to all CBO program services 

labor, described above. To estimate the upfront investment required to implement the C2C program, 

we collected the amount spent on new assets purchased specifically for the C2C program and 

associated annual depreciation expenses, though this ended up being minimal. 

Quarterly CBO Invoices

We also leveraged the quarterly CBO invoices submitted to the Mayor’s Fund to supplement 

the information collected using other instruments. Because of audit requirements, these invoices 

should be accurate in terms of the amount charged to the Mayor’s Fund. Leveraging these 

invoices enabled us to reduce the reporting burden of CBOs by prepopulating relevant items in 

the staff compensation report and nonlabor expenses report (e.g., variable nonlabor expenses and 

direct payments to MHP vendors), in some cases. However, the invoices did not capture all 

data at the level of detail needed to conduct the cost analysis. It is important to note that these 

invoices are based on budgets, which often drive an organization’s spending and billing patterns 

and do not necessarily reflect actual expenses. For example, they do not include labor expenses 

for staff who were not on the budget but who still engaged in C2C activities such as training or 

C2C skills delivery. In addition, the overhead rate charged on invoices has a maximum set at the 

rate based on federally negotiated indirect cost rate agreements, which could be lower than actual 

overhead rates used by some CBOs. 

CBO Quarterly Reports

We used the quarterly reports that CBOs submitted to the Mayor’s Fund between March 2016 

and June 2019 to estimate the number of unique clients served in a project year, which was then 

230 
 

 

 

 

 

 

used to calculate the average cost per client served. CBOs reported the total number of clients 

served per quarter as well as the new clients who received C2C skills, per quarter. In both 

metrics, “client served” means the client received at least one C2C skill and could include 

anything from a screening to a 10-week PE group. To calculate the number of unique clients 

served by the C2C program per year, we summed the total number of clients served in the first 

quarter and the new clients receiving C2C skills in the last three quarters. This is the closest 

approximation to the unique number of clients served. Summing the total number of clients served 

across quarters would have overestimated client numbers because some of the clients would have 

received ongoing C2C skills across quarters, whereas summing only the new clients served per 

quarter would have underestimated total clients because it would not include clients who received 

C2C skills in that year who had also received C2C skills from the CBO in prior years. 

Qualitative Interviews with CBO and MHP Leadership

Key informant interviews and focus groups were conducted as a part of the implementation 

evaluation. Interviews took place in person in 2017 and 2018 and by telephone in 2019. We were 

particularly interested in gathering CBO and MHP leadership perspectives on anything related to 

financial considerations, budget constraints, resource allocation, or sustainability of the C2C 

project activities. Further details of the key informant interviews are discussed in section B.2 

of Appendix B. 

Measures

For the program-level cost input analysis, we used several measures which are as follows: 

• average annual C2C program cost per CBO, reflecting the total amount of resources 

consumed to implement and maintain the C2C program at the CBO level 

• average annual program cost per C2C client served, representing the efficiency of C2C 

• change in the average cost per C2C client served over time. 

The average cost likely depends on the total number of C2C clients served in a year and the 

economies of scale. We defined a C2C client as any CBO client that received at least one C2C 

skill (screening, MHFA, MI, or PE), but these clients could have received more than one skill. 

All measures are presented in aggregate across all CBOs and by CBO type (described below). 

Data Analyses

Survey Weighting

We used surveys to collect program-level labor input data, and survey nonresponse can bias 

cost estimates if the outcomes being measured by the survey are correlated with the probability 

of responding to the survey. We therefore used a weighted analysis to address potential 

nonresponse bias. For the cost of labor survey, we collected the information about job titles of 

the staff eligible for the survey, classified them into five job categories, and created weights 

based on the response rate to the survey for each job category. For the annual staff surveys, 

231 
 

 

 

 

which do not have the job category information for nonrespondents, we used the inverse of the 

overall response rate as the weight. This way, individuals who are underrepresented among the 

respondents (relative to all surveyed staff) are upweighted in the analysis. The final estimates 

reported in this chapter are based on these weighted analyses of survey responses. 

Program-Level Cost Input Analysis

Program-level cost includes labor, nonlabor, direct payments to MHP and training consultants, 

and overhead cost. We derived the total labor cost for each staff member by multiplying the time 

spent on C2C per year (as reported on staff surveys) by annual salaries and fringe benefits. 

Because it was a program requirement to integrate C2C skill provision into existing CBO 

programming, there is likely a large overlap, in terms of staff labor hours between “usual” 

CBO service delivery (e.g., job training or youth development work) and C2C skills delivery. 

Although some C2C skills, such as mental health screening protocols or PE groups, may be 

provided separately from regular CBO services, MHFA and MI skills are designed to be woven 

into regular CBO programming. As a result, C2C could potentially improve the quality and 

effectiveness of regular CBO services, but it is unclear how much these skills increased the total 

amount of time CBO staff spent on regular service delivery. In our staff labor survey, respondents 

were not able to distinguish between time spent on regular services and additional time spent 

on C2C skills delivery. This poses a major challenge to estimating labor hours in our analysis. 

Therefore, we decided to exclude the amount of time spent on C2C skills delivery in the program 

cost in the main analysis, but we did conduct a sensitivity analysis to further address this (see 

Sensitivity Analyses section, below). 

CBO financial data provided the amount of nonlabor costs, such as depreciation expenses of 

fixed assets or office supply expenses, which can be further classified into two categories: C2C 

costs that can be separated from other CBO activities and those that cannot. For the former, the 

costs were derived from CBO invoices submitted to the Mayor’s Fund (e.g., travel expenses or 

office supplies purchased solely for the C2C project). For the latter, including overhead expenses, 

we used an allocation basis––the ratio of C2C labor cost to all CBO program service labor cost, 

as reported on CBO financial statements––to allocate the cost of the items shared between the 

C2C project and other organizational activities. We did not allocate any cost items that were not 

directly relevant to the C2C operation (e.g., expenses for meals provided to CBOs’ clients). By 

adding direct CBO payments to MHPs or vendors to labor and nonlabor portions of program 

costs, we were able to calculate the total amount of resources used for the C2C program. The 

resources associated with activities for data collection, evaluation, and reporting were excluded 

from the analysis, because they will not be part of the daily operation of the C2C program when 

the study ends. 

We generated estimates for annual program cost and cost per client served for each 

participating CBO and we present the average estimates across all CBOs, as well as averages 

by CBO type. The estimates of CBO-level averages were not weighted by CBO size, but the 

232 
average cost per client served, defined as the ratio of the sum of total program costs across all 

CBOs over the total number of clients served by all CBOs, was weighted. Similar approaches 

were used for each CBO type. CBOs were categorized into four different CBO types, consistent 

with the categories used for the implementation and impact evaluations (Table 5.1). 

In addition, we present annual program costs grouped by other factors, such as an internal or 

external funding mechanism for the MHP, CBO size in terms of the total number of CBO staff, 

and C2C funding source. As detailed in Chapter 4, the structure of CBO-MHP partnerships and 

the financial vehicles used varied across CBOs and over time. Most CBOs subcontracted with 

external MHP organizations to cover MHP staff time, but some CBOs hired and paid MHP 

clinicians as internal staff. Consequently, these expenses show up differently in the cost data, 

either through the “MHP and trainer contracts” category or rolled into the “labor” category. 

Economies of scale may have affected the cost as well, because larger CBOs (e.g., in terms of 

the number of total staff) may have had more resources available and may operate differently 

than smaller CBOs. Thus, we present annual program costs by CBO size, using employment as 

our grouping metric. 

Last, we present the funding sources of annual program costs. CBOs proposed a budget in 

their C2C program applications submitted to the Mayor’s Fund. The program required that 

the total C2C budget be at least $200,000 per year. The Mayor’s Fund was responsible for 

reimbursing 50 percent of the budget; CBOs were to find matching funds from nonfederal 

funders, such as philanthropic contributions or grants from private foundations, to cover the 

remainder. CBOs were allowed to carry over unused funds from 1 year to the next. If more 

resources were spent than the planned budget, CBOs had to absorb the additional cost. 

To examine whether CBOs improved their operating efficiency, we presented the difference 

in the average cost per client between years 2 and 3. In theory, the costs for training should 

decline over time, if they are upfront fixed costs, but we did not observe such a pattern, likely 

due to staff turnover and staggered rollout of training modules. We were not able to determine 

what portion of labor costs should be considered “fixed,” i.e., costs that do not change when 

service volume increases. In addition, because only year 2 and year 3 data were available for the 

analysis, we were not able to judge with confidence whether or when the program entered the 

maintenance stage and therefore did not estimate the cost of implementation separately from the 

cost of maintenance. For this reason, we did not calculate the marginal cost, often defined as the 

increase in variable costs––costs that change when service volume increases––due to serving an 

additional client while fixed costs remain unchanged. Instead, we examined how the average cost 

per client served changed over time. 

Sensitivity Analyses

We conducted three sensitivity analyses to address the uncertainty in the cost estimation. The 

first sensitivity analysis used an alternative method to calculate overhead costs. Whereas the 

main analysis used an overhead rate derived from CBOs’ financial statements, this sensitivity 

233 
 

 

analysis instead used the overhead rate charged by the CBO on C2C project invoices. The 

overhead rate had a maximum rate based on government-negotiated indirect cost rate agreements; 

however, the actual overhead charged per invoice was sometimes less than the allowable rate. We 

calculated the invoice rate as the ratio of total overhead charged on project invoices per year to 

total project expenses charged less overhead. This invoice rate was then applied to the same 

direct project costs in the main analysis. 

The second sensitivity analysis measured only the program costs charged by CBOs on 

the C2C project invoices, and therefore, only the costs for which CBOs were reimbursed. As 

noted above, the Mayor’s Fund reimbursed 50 percent of CBO project costs, and CBOs were 

required to find funding for the remainder from nonfederal funders. By relying solely on 

invoices, this sensitivity analysis allowed us to calculate program costs for the first three project 

years (compared with only years 2 and 3 in the main analysis using surveys), although using 

invoices only would lead to an underestimation of program costs. 

In our third sensitivity analysis, we adjusted labor hours to account for a marginal 

increase in the amount of time C2C staff spent delivering regular client services due to C2C 

skills delivery. Although most of the C2C skills (such as MHFA and MI) were integrated into 

client interactions under existing CBO programming, delivery of some skills could increase the 

length of regular client interactions (i.e., additional screenings) or add new client interactions 

(i.e., new PE groups). Because of the challenge of quantifying the marginal increase in time 

spent on regular CBO service delivery due to the utilization of C2C skills, we assumed that 

implementing C2C skills increased staff time spent on regular CBO services by 5 to 25 percent, 

based on our qualitative interviews with CBO staff. The purpose of this sensitivity analysis was 

to examine how labor costs and annual program costs would change if C2C skills increased 

regular CBO service delivery time by 5 or 25 percent. 

Results

As described in the Methods section, we conducted a main analysis to determine the costs 

associated with the C2C program, as well as three sensitivity analyses. We also used interviews 

with key informants to understand more about what we were seeing in the data. Note that one 

CBO was unable to implement the program adequately and was discontinued from the program 

by the funder in year 3. As a result, the cost of labor survey for year 3 and the staff survey for 

year 4 were not available for this CBO. Because our analysis focused on years 2 and 3, and we 

were able to rely on the staff survey in year 3 to estimate labor cost for the CBO that dropped 

out, we present the cost results for all 15 CBOs. In this section, we present the results of the main 

analysis first. Several calculations were necessary to complete before the average annual cost of 

providing C2C at each CBO and the average annual cost per client could be determined. Here we 

break down our analyses by the number of CBO staff trained and the number of clients served 

with C2C skills, and then the number of labor hours spent and the compensation of CBO staff, 

234 
	 	

	 	 	

	 	

60.0	

51.2	

Average Number of	C2C	Staff	Members50.0	

57%

26%

13%

52%

26%

10%

52%

22%

17%

43.1	 41.6	

38.3	 39.3	 40.0	 36.5	

16%

58%

16%

33.0

52%

12%

31%

25%

30.0	

51%

10%

32%

30%

23%

43%

27.9	

22.8

20.0	 18.3	

10.0	

59%

51%

16%

28%

-

21%

13%

56%

Year	2 Year	3 Year	2 Year	3 Year	2 Year	3 Year	2 Year	3 Year	2 Year	3

Parent/Caregiver- Job	Training & Youth	Development Other (N	=	4) All	CBOs	(N	=	15)

Serving (N	=	3) Employment(N = 5) (N	=	3)

Administrative Staff Frontline Staff Leadership Organizational Support Staff Supervisors

 

 

 

 

 

before using these data to calculate C2C program costs. After the main analysis results, we 

describe our findings from the sensitivity analyses and the key informant interviews. 

Main Analysis

Staff Trained and Clients Served in C2C

Figure 10.2 presents the average number of staff involved in the C2C initiative, by CBO 

type, for years 2 and 3. Staff involvement was defined as having received training in at least one 

of the four C2C skills (screening, MI, MHFA, or PE) in the past year. CBO program managers 

provided this list of C2C staff members as those eligible to take the staff survey. Overall, CBOs 

(N = 15) reported an average of 27.9 C2C-involved staff members in year 2 (range: 6–81) and 

41.6 in year 3 (range: 14–108), a 49-percent increase in average staff members. There was some 

variation in the average number of C2C staff between CBO type, but also within CBO types. 

Parent/caregiver-serving CBOs (n = 3) had the highest growth in average staff numbers, increasing 

from 18.3 in year 2 (range: 10–30) to 38.3 in year 3 (range: 14–71) or 109 percent. On average, 

youth development CBOs (n = 3) had a large number of staff members (19–40 percent larger) 

than other types of CBOs but had a relatively small increase from a year 2 average of 39.3 

(range: 13–81) to an average of 51.2 in year 3 (range: 26–100). 

Figure 10.2. Average Number of Staff Members Who Received C2C Training per CBO, by Job

Category, C2C Project Year, and CBO Type

SOURCE: Staff surveys. Percentages are not labeled for components that are less than 10 percent. 

NOTE: Year 2 staff job category percentages were extrapolated from survey responses, whereas year 3 job category 

percentages are based on the full list of staff eligible for which such information was available.

235 
 

 

The number of unique clients served increased substantially over the first 3 years of C2C 

implementation as the program scaled up (refer to Chapter 5 for more detail). The total number 

of unique clients served under the C2C initiative increased by 198 percent between years 1 and 2 

(from 4,106 to 12,249, respectively) and by 31 percent in year 3 (to 15,992) (data not shown). 

Figure 10.3 shows the average number of unique clients served for years 2 and 3, by CBO type 

and project year. Overall, CBOs served an average of 817 clients in year 2 (range: 68–2,737) and 

1,066 in year 3 (range: 164–2,590), a 30.6-percent increase, but the change varied widely across 

CBOs. Four CBOs had roughly no change in number of clients served, seven CBOs had a 

modest increase (mean: 36 percent), and the remaining four CBOs had a substantial increase 

(mean: 241 percent). The change in number of clients served did not correlate strongly with 

either CBO type or size. 

Also, the average number of unique clients served varied substantially across and within 

CBO types. CBOs serving parents/caregivers reached the fewest clients in year 3 (mean: 641; 

range: 164–1,583), but these CBOs increased the number of unique clients served more than any 

other CBO type between years 2 and 3 (147 percent). Youth development CBOs served the 

largest number of clients in year 3 (mean: 1,595; range: 493–2,590), but their number served 

remained fairly consistent with year 2 (7.5-percent increase). Job training and employment 

CBOs served an average of 1,076 clients in year 3 (range: 632–1,629; 25-percent increase from 

year 2), and other CBOs served an average of 976 clients in year 3 (range: 507–1,590; 43-percent 

increase from year 2). 

Figure 10.3. Average Number of Unique Clients Served per CBO, by CBO Type and Project Year

SOURCE: CBO quarterly reports.

236 
 

 

 

Labor Hours and Compensation

Annual staff surveys collected the breakdown of weekly hours staff spent on C2C by activity. 

Although we were unable to calculate the marginal increase in time spent on regular CBO service 

delivery due to C2C, staff reported spending an average of 4.6 hours per week on C2C skills 

delivery in year 2 and 3.9 hours per week in year 3 (data not shown). This time spent on C2C 

skills varied by job type: frontline staff and supervisors reported more hours on C2C skill 

delivery than did leadership or administrative staff. However, because of the integration of C2C 

skills with existing CBO programming, we were unable to tease out the marginal increase in 

service delivery hours specifically related to the application of C2C skills. Thus, all labor hours 

spent on C2C skill delivery have been excluded from the rest of this analysis. (Figure D.4, from 

the third sensitivity analysis, illustrates the increase in program cost per client assuming a 5- to 

25-percent increase in staff time spent on delivering regular CBO services due to the addition of 

implementing C2C skills.) 

Figure 10.4 reflects the average number of hours per C2C activity in a typical week by staff 

who responded to the survey. These hours were not weighted for total CBO staff working on the 

C2C program, because we were unable to collect job category information for all nonrespondents. 

Among those who did respond to the surveys, staff spent an average of 6.4 and 5.8 hours per 

week on C2C activities in years 2 and 3, respectively. A large portion of staff time was spent on 

receiving training, followed by providing referrals to MHPs and project management activities. 

Notably, there was a decrease in the amount of time staff received training in year 3 compared 

with year 2 (from 2.2 to 1.3 hours per week), which appears to have been complemented by 

receipt of coaching in year 3 instead of training (1.2 hours per week). The amount of time spent 

per week on several tasks declined from year 2 to year 3: project management, from 1.2 hours 

per week in year 2 to 0.6 in year 3; data collection, 0.8 to 0.5 hours; and technical assistance, 

0.6 to 0.4 hours. Figure D.5 further extends this analysis with responses from the year 4 survey, 

from which we observed an increase in the amount of time for referrals (17 percent) and providing 

coaching (75 percent) but a decrease in receiving training (−65 percent), receiving coaching 

(−57 percent), providing training (−15 percent), technical assistance (−21 percent), data collection 

(−46 percent), and project management (−48 percent). 

As seen in Figure 10.4, the amount of time spent on various C2C activities differed by staff 

job category. As expected, leadership staff spent the most hours per week on project management 

(3.3 in year 2 and 2.5 in year 3), data collection (2.0 in year 2 and 1.1 in year 3), and technical 

assistance activities (1.2 in year 2 and 0.7 in year 3). In both project years, frontline staff spent 

the most time receiving training (2.7 and 1.6 hours in years 2 and 3, respectively) and providing 

referrals to MHPs (1.4 and 1.5 hours in years 2 and 3, respectively). Supervisors also spent a 

substantial amount of time providing referrals to MHPs in years 2 and 3 (1.4 and 1.2 hours in 

years 2 and 3, respectively) and providing coaching to frontline and other staff (0.9 and 1.3 hours, 

respectively). In year 3, MHP staff transitioned some of the recurring training over to CBO 

supervisors, who spent an average of 1.1 hours per week providing training in year 3. 

237 
 

 

 

 

 

 

Figure 10.4. Average Weekly C2C Labor Hours per Staff Member, by Job Category, C2C Activity,

and Project Year

SOURCE: Staff surveys.

NOTES: Because we were unable to collect job category information for all nonrespondents, results reflect the

average staff who responded to the staff survey and were not weighted for the total number of staff. Percentages are 

not labeled for components that are less than 10 percent.

Figure 10.5 shows the average annual labor hours per CBO, with weighted staff survey 

responses to reflect all staff working on the C2C program and extrapolated to the full year. 

Weekly labor hours were standardized to a 35-hour workweek, and we assumed 50 workweeks 

per year. On average, CBOs spent 26 percent more labor hours in year 3 (mean: 10,311; range: 

2,031–29,377) compared with year 2 (mean: 8,151; range: 704–24,544). This translates to an 

average of 5.9 full-time equivalents in year 3 (range: 1.2–16.8) compared with 4.7 full-time 

equivalents in year 2 (range: 0.4–14.0). (Figure D.6 extends results using the year 4 staff survey, 

which indicates a 12-percent decline in the average number of annual labor hours, to 9,109, for 

that year per CBO.) 

Youth development and other CBOs had a higher average number of annual labor hours in 

years 2 and 3 compared with the parent/caregiver-serving and job training and employment 

CBOs. However, the number of staff labor hours worked at parent/caregiver-serving CBOs 

increased substantially (230 percent), whereas the increase in the number of labor hours at youth 

development and job training and employment CBOs was more moderate (22 and 19 percent, 

respectively). Labor hours at other CBOs decreased slightly (−2 percent). However, these 

averages mask wide variation at the CBO level with no discernible correlation to CBO type: 

238 
Figure 10.5. Average Annual C2C Labor Hours per CBO, by CBO Type and Project Year

SOURCE: Staff surveys.

four CBOs’ labor hours decreased from year 2 to year 3, five CBOs’ labor hours increased 

between 0 and 100 percent, and five CBOs’ labor hours more than doubled (>100 percent). 

Figure 10.6 shows the average annual staff compensation per CBO in year 3, the only year 

we collected compensation data. Across all CBOs, the average compensation (salary plus fringe 

benefits) was $58,449 per CBO (median: $55,267, range: $43,902–$74,498). But this average 

staff compensation masks variation by staff job category. For example, for administrative staff, 

the average compensation was $41,589; frontline staff, $50,417; organizational support staff, 

$57,789; supervisors, $70,719; and leadership, $104,120 (data not shown). We also observed 

variation in staff compensation by job category across different types of CBOs. For example, the 

average annual compensation for frontline staff in parent/caregiver-serving CBOs was $43,554, 

compared with $60,751 in youth development CBOs. Similarly, the average compensation for 

leadership staff was $83,998 in parent/caregiver-serving CBOs and $121,382 in job training and 

employment CBOs. 

Using the weighted staff-level reported hours worked in a typical week and annual 

compensation, we calculated the average annual labor cost per CBO, as illustrated in Figure 10.7. 

Overall, CBOs spent 20 percent more in year 3, on average, than in year 2 ($324,792 in year 3, 

vs. $271,124 in year 2). However, both the average annual labor costs and the percentage 

increase from year 2 to 3 differed substantially by CBO type. For example, parent/caregiverserving CBOs spent 260 percent more in year 3 than in year 2 (largely driven by substantial 

increase in labor at one CBO), whereas youth development CBOs spent 6 percent more in year 3. 

239 
 

 

Figure 10.6. Average Annual Staff Compensation in Year 3, by CBO Type

SOURCE: Compensation report, 2018.

Figure 10.7. Average Annual Labor Cost per CBO, by CBO Type and Project Year

SOURCE: Staff surveys; compensation report, 2018.

240 
 

 

 

Job training and employment and other CBOs spent closer to the overall average, with only a 

27-percent increase and a 12-percent decrease, respectively. 

Similar to the findings for annual labor hours, averages of annual labor cost mask wide 

variation at the CBO level that are unrelated to CBO type: five CBOs’ labor cost decreased from 

year 2 to year 3 (mean: −43 percent), five CBOs’ labor hours increased between 0 and 100 percent 

(mean: 56 percent), and four CBOs’ labor hours more than doubled (mean: 592 percent). 

Program Cost

We compiled all results and report the average annual program cost per CBO, by project year 

and CBO type. Overall, CBOs spent a total of $6,563,194 in year 2 and $7,712,137 in year 3, an 

18-percent increase. As illustrated in Figure 10.8, the C2C project costs each CBO an average 

of $437,546 in year 2 (median: $315,532; range: $89,653–$1,237,302) and $514,142 in year 3 

(median: $433,614; range: $124,108–$1,047,960). Four CBOs’ program cost decreased in year 3 

(mean: −42 percent), seven CBOs had a modest increase in program cost (mean: 39 percent), and 

the remaining four CBOs’ program cost more than doubled (mean: 200-percent increase). 

Figure 10.8. Average Annual C2C Program Cost per CBO, by Cost Component, Project Year, and

CBO Type

SOURCE: Staff surveys; compensation survey 2018; annual nonlabor reports 2017, 2018; financial statements 2016

to 2018; project invoices.

NOTE: Percentages are not labeled for components that are less than 10 percent.

241 
 

 

 

Annual program cost, and the increase in cost from year 2 to 3, varied across and within 

CBO types. Youth development CBOs had the highest average cost in year 2 (mean: $611,055; 

range: $280,332–$1,237,302) and year 3 (mean: $659,109; range: $433,614–$879,629), and 

their average program cost increased by 8 percent, a relatively small amount. Average annual 

costs for other CBOs decreased by 2 percent, from $520,810 in year 2 (median: $517,041; range: 

$275,096–$744,062) to $509,658 in year 3 (median: $386,679; range: $217,315–$1,047,960). 

Conversely, the average annual C2C program costs of job training and employment CBOs 

increased 21 percent, from $428,516 in year 2 (median: $449,722; range: $189,986–$838,165) 

to $517,088 in year 3 (median: $508,857; range: $384,258–$652,201). Last, although the 

parent/ caregiver-serving CBOs had the lowest annual program costs, $168,069 in year 2 (range: 

$89,653–$297,345) and $370,246 in year 3 (range: $124,108–$579,357), they had the largest 

increase in program spending (120 percent) from year 2 to 3. 

Annual program costs were partially driven by the structure of the funding mechanism for 

C2C. As previously described, CBOs were required to submit annual budgets of at least 

$200,000 to the Mayor’s Fund, and funding was split 50-50 between the Mayor’s Fund CBOidentified funders, typically foundation grants. Figure 10.9 shows the breakdown of average 

annual program cost per CBO, by funding source and project year; “sponsor” represents the 

Mayor’s Fund share of funding. Qualitative interviews (refer to the Interview Results section 

below) indicated that, in some cases, the match requirement created an upper bound on the 

budget requested. In other cases, CBOs used resources not accounted for in the C2C budgets to 

pay for additional staff labor costs and organizational-level overhead costs above the federally 

Figure 10.9. Average Annual Program Cost per CBO, by Funding Source and Project Year

SOURCE: Staff surveys; compensation survey 2018; annual nonlabor reports 2017, 2018; financial statements 2016

to 2018; project invoices.

NOTE: Percentages are not labeled for components that are less than 10 percent.

242 
 

 

 

 

 

 

 

 

negotiated indirect cost rate agreement. On average, CBOs covered substantially more costs than 

they had planned in the budget, accounting for 43 percent of total program expenses in year 2 

and 44 percent in year 3. 

As shown in Figure 10.10 below, across all CBOs (N = 15), the C2C initiative’s cost per 

client was $536 (median: $701; range: $120–$3,203) in year 2 and $482 (median: $500; range: 

$137–$2,483) in year 3, a 10-percent decrease. The largest cost driver of the C2C initiative was 

labor (61 percent in year 2 and 62 percent in year 3), followed by overhead (17 percent in year 2 

and 18 percent in year 3), variable costs (12 percent in year 2 and 11 percent in year 3), and 

MHP and trainer contract costs (9 percent in year 2 and 8 percent in year 3). 

The C2C cost per client varied across and within CBO types. Among the three parent/ 

caregiver-serving CBOs, the C2C cost per client was $649 in year 2 (range: $217–$1,759) and 

decreased about 13 percent to $578 in year 3 (range: $366–$2,483). Among the five job training 

and employment CBOs, the C2C cost per client was $499 (median: $470; range: $120–$1,603) 

in year 2 and decreased about 4 percent to $480 (median: $500; range: $236–$805) in year 3. 

Among the three youth development CBOs, the C2C cost per client served was $412 (range: 

$240–$701) in year 2 and remained constant at $413 (range: $255–$1,347) in year 3. Among the 

Figure 10.10. Average C2C Program Cost per Client Served, by Cost Component, Project Year, and 

CBO Type

SOURCE: Staff surveys; compensation survey 2018; annual nonlabor reports 2017, 2018; financial statements 2016

to 2018; project invoices.

NOTE: Percentages are not labeled for components that are less than 10 percent.

243 
 

 

 

 

 

four other CBOs, the C2C cost per client was $765 (median: $1,349; range: $259–$3,203) in 

year 2 and decreased 32 percent to $522 (median: $568; range: $137–$1,231) in year 3. 

Combining the average cost per client (Figure 10.10) with the funding sources (Figure 10.9), 

we find that the sponsor-funded portion of the cost per client was $150 in year 2 and $135 in 

year 3. With the average cost per client that was reimbursed (sponsor-funded portion plus the 

50 percent match) at $300 in year 2 and $270 in year 3, CBOs had to cover, on average, $230 per 

client in year 2 and $212 per client in year 3 (data not shown). 

The type of MHP partnership model did not affect the average number of clients served but 

did affect the total C2C program cost. The four CBOs with internal mental health clinicians had 

a similar average number of unique clients served in year 3 (1,075 vs. 1,063) but a higher total 

program cost ($459,913 vs. $476,721) in comparison to the 11 CBOs with external MHP 

clinicians (data not shown). Figure 10.11 shows that the average cost per client served was 

higher among CBOs with internal mental health clinicians. 

Figure 10.11. Average Program Cost per Client Served, by Cost Component, MHP Partnership

Model, and Project Year

SOURCE: Staff surveys; compensation survey 2018; annual nonlabor reports 2017, 2018; financial statements 2016

to 2018; project invoices.

NOTE: Percentages are not labeled for components that are less than 10 percent.

We also broke out the average program cost per client served by CBO size, as calculated by 

the total number of CBO staff members at the end of year 3 (Figure 10.12). For small CBOs 

(50 or fewer staff members), the average program cost per client was $482 in year 2 (range: 

$240–$1,554) and $359 in year 3 (range: $137–$805). For medium CBOs (between 51 and 

100 staff), the average program cost per client was $617 in year 2 (range: $120–$1,759) and 

244 
 

 

 

 

 

 

Figure 10.12. Average Program Cost per Client Served, by Cost Component, CBO Employment

Size, and CBO Project Year

SOURCE: Staff surveys; compensation survey 2018; annual nonlabor reports 2017, 2018; financial statements 2016

to 2018; project invoices; CBO quarterly reports.

NOTE: Percentages are not labeled for components that are less than 10 percent.

$550 in year 3 (range: $236–$2,483). The average program cost per client among large CBOs 

(more than 100 staff) was $532 in year 2 (range: $217–$1,143) and $463 in year 3 (range: 

$340–$1,231). Within CBO size groups, there was still a considerable amount of variation in 

CBO-level cost per client, indicated by the wide ranges. 

We originally hypothesized we would see a lower cost per client, and thus greater economies 

of scale, in larger CBOs that had greater preexisting resources (i.e., human, financial, physical 

space). Small- and medium-sized CBOs spent about the same amount in terms of total program 

costs, but small CBOs provided C2C skills to a larger number of unique clients than the medium 

CBOs, bringing the cost per client down (data not shown). Large CBOs spent a larger percentage 

of the average cost per client on labor (73 percent in year 3) compared with medium and small 

CBOs (60 and 41 percent in year 3, respectively) (Figure 10.12). Conversely, in year 3, small 

CBOs spent a larger percentage of average cost per client on overhead (24 percent, compared with 

19 percent for medium and 15 percent for large CBOs) and variable costs (21 percent compared 

with 12 percent for medium and 5 percent for large CBOs). We also examined average cost per 

client by the number of staff involved in C2C and found no notable patterns (data not shown). 

Last, we examined the changes in the average cost per client served between year 2 and year 3. 

Because of survey limitations (described in the Methods section), we were unable to estimate the 

portion of labor costs that should be considered fixed, and thus we were unable to calculate the 

245 
marginal cost of an additional client served within the same time period. Overall, across all 

CBOs, the average cost per client decreased by 10 percent, from $536 in year 2 to $482 in year 3. 

However, there was large variation in CBO-level changes in the average cost per client served, 

ranging from a year-on-year decrease of $2,506 to an increase of $724 per client served. Only six 

of 15 CBOs experienced a decline in the average cost per client served (mean: −56 percent). The 

other nine CBOs’ average cost per client increased from year 2 to year 3. Of these nine CBOs, 

four had a small increase in average cost (mean: 7 percent), whereas the other five had a much 

larger increase (mean: 74 percent). 

The variation was due to changes in program cost and/or the number of clients served between 

the 2 years. The $2,506 decrease was associated with a 41-percent decline in total program cost 

and a 171-percent increase in the number of clients served between the 2 years. In contrast, the 

$724 increase was associated with a 37-percent increase in total program cost but a 3-percent 

decrease in the number of clients served. 

As described above, all CBO types had a decline in the average cost per client served 

except for youth development CBOs (Figure 10.10). CBOs with an internal MHP had a slight 

increase from $557 to $574, but those with an external MHP had a decrease from $530 to $448 

(Figure 10.11). Once grouped into different size categories, a consistent decline in average cost 

per client emerged across all CBO categories (Figure 10.12). 

Sensitivity Analyses

As described in the Methods section, we performed three sensitivity analyses to explore aspects 

of the main analysis for which we encountered substantial uncertainty. The first analysis involved 

using an alternative method to calculate overhead costs, the second restricted the cost analysis to 

using only the C2C project invoices to measure C2C costs, and the third approximated a marginal 

labor cost of providing C2C skills while delivering typical CBO services if time spent increased 

by 5 or 25 percent. Data for each of these analyses are shown in Appendix D, as specified below. 

For the first sensitivity analysis, we used the indirect rate charged on CBO invoices to 

calculate the total overhead cost per CBO. Across all CBOs, our estimates of the average total 

program cost decreased by about $38,172 in year 2 (9 percent) and $51,620 in year 3 (10 percent) 

(Figure D.1). The overall C2C cost per client under this sensitivity analysis was $489 in year 2 

(9 percent less than the main analysis cost per client of $536) and $433 in year 3 (10 percent less 

than the main analysis cost of $482). These differences demonstrate that the overhead rates based 

on negotiated indirect cost rate agreements are lower than the true overhead costs based on 

CBOs’ financial statements. 

The second sensitivity analysis measured only the program costs shown on the CBO invoices 

to the Mayor’s Fund, 50 percent of which were reimbursed. We find that the program costs billed 

on invoices are substantially less than those estimated using staff surveys and financial statements 

(Figure D.2). Overall, based on billed costs, CBOs spent an average of $174,741 in year 1, 

$244,870 in year 2, and $286,582 in year 3. Thus, average invoice-based program costs were 

246 
 

 

 

 

 

 

 

 

about 40 percent less than program costs in the main analysis both in year 2 and year 3. Figure D.3 

shows that the cost per client in year 1 ($638) based on invoice-based program costs was over 

twice as high as in years 2 and 3 ($300 and $269, respectively). 

Finally, we adjusted staff labor to estimate a 5- to 25-percent marginal increase in labor hours 

and cost related to CBO service delivery when implementing C2C. Under this assumption, in 

year 2, CBOs spent an average of $438,348 to $465,520 on C2C (compared with $437,546 in the 

main analysis) and $520,339 to $567,216 in year 3 (compared with $514,142 in the main analysis). 

These costs represent a 0.2- to 6.4-percent increase in average program cost in year 2 and a 1.2- to 

10.3-percent increase in year 3, compared with the main analysis. Using these labor-adjusted 

program costs, the average cost per client served was $544 to $577 in year 2 and $493 to $537 in 

year 3 (Figure D.4). Compared with the main analysis finding of an average cost per client of $536 

in year 2 and $482 in year 3, this sensitivity found a 5-percent increase in time spent on CBO 

service delivery would increase the average cost per client by $8 in year 2 and $11 in year 3, and 

a 25-percent increase would raise the average cost per client by $41 in year 2 and $55 in year 3. 

Interview Results

Three cost-related themes emerged from the qualitative interviews with CBO and MHP 

leadership: (1) budget-related constraints, (2) allocation of resources, and (3) financial 

sustainability. 

Budget

The C2C funding match requirement limited the size of C2C budgets for some of the CBOs, 

who described the 50-percent match requirement as “incredibly difficult” and “certainly a lot of 

work.” In at least two cases, CBOs said they met the match requirement with internal funding or 

by leveraging labor support from interns. One CBO leader said, “the funding of C2C is also 

matched by our organization, dollar for dollar . . . with very little wiggle room.” Many CBOs 

mentioned the burden on staff time not included on the budget, but one CBO remarked, “There’s 

only so many hours in the day, so I don’t know if it costs us more or it just was robbing Peter to 

pay Paul in terms of the hours.” However, another CBO noted that, while hosting meetings and 

tours for potential funders was very time intensive, “it also helped us strengthen our language, 

our ability to talk about [C2C] and really explain what it does.” 

In addition, project administration and data reporting seemed to be more resource-intensive 

than expected for many CBOs, both in terms of staff time and database development, and were 

typically not included in CBOs’ annual budgets. In the words of one CBO leader, 

The administrative and reporting part of C2C has been hugely challenging . . . . 

There’s no way we would have been able to do C2C reporting without a full-time 

Salesforce administrator, and it’s certainly not reflected in our budget. 

Smaller CBOs without dedicated IT staff or data analysts tended to have a more difficult time 

responding to the C2C data reporting requirements than their larger counterparts. 

247 
Some CBOs and MHPs reported feeling a budgetary pinch through a combination of the 

structure of mental health payments and a lack of C2C funding to cover client copayments or 

counseling fees. For example, one MHP said an ideal change to the C2C program would be 

“to have some kind of [financial] carve out so client insurance was a nonissue.” Another MHP 

remarked that it would be useful to have “some funding built into the system . . . to be able to 

make those copayments or meet those deductibles.” In addition, because of the fee-for-service 

structure of mental health in NYC, many C2C clients were dropped by counselors after missing 

two or three appointments because MHPs lose money when appointments become no-shows. 

One CBO reported that, 

If we had a therapist who wasn’t linked to insurance, who could be more fluid, I 

think that would work well, but there’s not enough funding in the grant for us to 

do that. 

Other CBOs, however, preemptively included therapy fees for uninsured clients into their C2C 

program budgets and were able to cover these costs. 

Resource Allocation

Both CBO and MHP leadership remarked on how resource-intensive staff labor and trainings 

were. All CBOs noted that most of the C2C funding was allocated to labor costs. Some CBOs 

mentioned that this was beneficial, in that it allowed for the hiring of extra case managers and the 

reduction of caseloads. Others highlighted the difficulty of the funding constraints on providing 

full-time C2C coordination positions or salary constraints on hiring qualified individuals for such 

positions. Training was the second largest area of resource allocation. One CBO lead explained, 

“How often I can offer a workshop is going to be based on how much money I have to get a 

space outside this building.” Training resources also involve staff labor, in terms of the time of 

staff in training because they do not provide CBO client services, as well as those of MHP 

trainers. 

Financial Sustainability

CBO and MHP leadership also responded to questions on their current plans for sustainability 

of the C2C program. All CBOs reported that funding will be a key in deciding sustainability of 

C2C program activities. Interviews were conducted in year 2 and year 3, so there was substantial 

variation in whether CBOs had clear vision of potential funding sources for future work after the 

program ends in year 5. One CBO noted that C2C had allowed them to increase both the quantity 

of social work services provided and the stability of those services, which in turn demonstrated 

capacity and “allowed us to secure other funding sources for social work roles.” Another CBO 

mentioned that, in addition to an internal desire to sustain C2C, there are business motivations 

for doing so: 

We’ve seen increased engagement in participants in our mental health services, 

and that increased the need for more present and available clinical supervision. 

248 
 

 

 

Small CBOs are not optimistic about sustainability, as reported by one CBO: 

C2C has been a significant portion of our annual [organizational] budget. C2C 

without funding is not sustainable...even though it’s kind of a depressing answer. 

CBOs also varied in the level of resources they needed to sustain various C2C skills. MHFA 

training was perceived as “mostly free,” because the city offers the training, and “low-hanging 

fruit for us to be able to connect folks with that are on their way in the door.” Other C2C skills 

have become so ingrained in some CBOs’ way of providing regular services that they can continue 

without much effort. For example, hiring more staff with a background in mental health, continuing 

exit surveys or interviews, maintaining relationships with MHPs to track referrals and hold 

collaborative team meetings, and bringing in social work interns and peers were all noted as 

“relatively low cost.” However, hiring social workers for on-site counseling services and 

continuing MI (which is training intensive) were mentioned as examples of C2C activities that 

would depend on external funding sources for continued maintenance. One CBO suggested a 

novel pathway to generate revenue as a means of sustaining C2C activities: to allow “the CBOs 

that have done really well” or “made innovative changes” to provide paid expertise as “technical 

support” services “to other cities and organizations in a way that we can generate income.” 

Discussion

Program Changes over Time

The resources needed to administer the C2C program evolved over the study period. For 

example, the average number of staff members involved in C2C increased 49 percent from year 

2 to year 3, from 27.9 to 41.6. The average annual C2C labor hours per CBO increased 26 percent 

from year 2 to year 3, from 8,151 to 10,311 hours per year, yet decreased by 12 percent in year 4. 

From year 2 to year 3, program staff allocated less time to receiving training, technical assistance, 

data collection, and project management, but more time to providing training and receiving 

coaching and supervision. The shift from MHP staff to CBO staff in the number of hours spent 

providing training suggests that CBOs increasingly relied on their own staff and operated more 

independently in year 3 than in year 2. The increase in staff time spent on receiving coaching and 

decrease in time spent on receiving training are consistent with a greater focus on on-the-job 

coaching than in-classroom training, indicating CBOs were applying the knowledge gained to 

regular client services in year 3. In year 4, CBOs further decreased time in training, technical 

assistance, data collection, and project management. The average amount of time CBO staff 

spent receiving coaching decreased but providing coaching increased, indicating a task-shifting 

of coaching from MHP staff to CBO supervisors in year 4. 

Over time, CBOs shifted their labor input from leadership staff to nonleadership staff. 

Compared with year 2, in year 3 there were three times more administrative staff, about one and 

a half as many frontline staff, and twice as many organizational support staff in C2C. In contrast, 

249 
 

 

 

 

 

 

 

 

 

the number of supervisory and leadership staff increased by 13 and 16 percent, respectively. 

The average number of hours per week per C2C staff member decreased by 9 percent in year 3, 

primarily due to the 24-percent decline in leadership hours, though the supervisor hours increased 

by about one-third. Note that, except for a C2C coordinator hired by some CBOs to work fulltime on C2C, other staff spent a relatively small portion of their time on the program. 

Though the average weekly number of labor hours per staff member among survey respondents 

decreased from year 2 to year 3, the weighted average annual number of C2C labor hours per 

CBO increased. Largely because of the 26-percent increase in the annual number of staff hours 

spent on C2C, the average annual program cost per CBO increased 18 percent in year 3 compared 

with year 2, from $437,546 to $514,142. At 18 percent, the magnitude of the increase in average 

annual program cost was smaller than the 26-percent average increase in staff hours during the 

2-year period. This is likely due to greater utilization of non-leadership staff, whose annual 

compensation is typically lower (e.g., $50,417 for frontline staff and $70,719 for supervisors vs. 

$104,120 for leadership staff). The sensitivity analysis based on CBO invoices only, as opposed 

to survey-based estimates in the main analysis, also demonstrated a rapid increase in the average 

annual program cost over the first 3 years, resulting in an increase of 40 percent in year 2 and 

17 percent in year 3. 

From an efficiency standpoint, it seems the C2C program as a whole had completed the 

startup phase by year 3 and reached a level, in terms of the number of clients served, beyond 

which we would expect a decrease in the average cost per client served (hereafter we call this 

increasing returns to scale). Further increases in service volume may lead to constant returns to 

scale under which increasing service volume does not change the average cost per client served; 

that is, CBOs may operate at the bottom of their long-run average cost curve in this scenario if 

their service volume increases up to that point. Overall, the number of unique clients served 

increased 31 percent between year 2 and year 3, which is higher than the increase in program 

cost during the same time period (18 percent). This is consistent with the 10-percent decrease in 

the average program cost per C2C client served in year 3 than year 2, from $536 to $482. 

However, we caution against the interpretation that most CBOs, at an organizational level, 

reached a stage where serving more clients necessarily leads to a lower average cost per client, 

i.e., increasing returns to scale. Only six of 15 CBOs experienced a decline in the average cost 

per client served. These six CBOs had either a large decline in total program cost or a large 

increase in the number of clients served. The decline in program cost was primarily attributed to 

a lower labor cost or lower external MHP and trainer contract expenses. It could be that these 

CBOs received extensive training and coaching among their staff in year 2, making these 

activities less necessary in year 3. A large increase in the number of clients served also suggests 

that these CBOs scaled the C2C operation efficiently without incurring a similar increase in 

labor costs. 

In general, as service volume increases, we observe that labor costs increase. Based on the 

CBO cost data for years 2 and 3, whether the average cost per client served decreases depends 

250 
 

 

 

 

 

 

 

 

 

 

 

largely on whether there are major reductions in other nonlabor cost categories. That is, whether 

a program has increasing returns to scale may not depend only on service volume but also on the 

cost structure of the program. If the magnitude of the total cost increase for an additional client 

served is smaller than the increase in the number of clients served, the average cost per client 

may decrease. 

As the C2C program becomes more mature over time, we expect to see further decreases 

in staff time spent on training, technical assistance, data collection, and project management, 

consistent with the survey data from year 4. That is, the C2C program cost is likely to decrease 

more in year 4 and beyond. Assuming that the time spent on technical assistance and data 

collection become unnecessary once the program fully matures, and that time spent on training, 

coaching and program management, and the nonlabor cost continues to decrease along the 

same trend, the total program cost could decrease by 22 percent. This may be an important 

consideration for potential adopters of C2C. 

The composition of program costs remained similar over time. Labor input accounted for about 

60 percent of program cost, consistent with a service program; payments to MHPs or trainers 

were about 8 percent of the total program cost. Overhead accounted for 17 and 18 percent of the 

total cost in years 2 and 3, respectively. However, more than two-thirds of the program cost 

increase from year 2 to year 3 ($514,142 − $437,546 = $76,596) was due to an increase in labor 

input ($52,441), whereas the increase in overhead ($18,074) accounted for about one-quarter of 

the increase. 

Cost Differences Between CBO Subgroups

Because of several factors, youth development CBOs were associated with a higher annual 

labor and program cost than other types of CBOs. First, youth development CBOs trained more 

staff members (19–40 percent more in year 3 than other types of CBOs) and spent 15 to 17 percent 

more labor hours than parent/caregiver-serving and job training and employment CBOs. Second, 

a larger proportion of staff involved in C2C in youth development CBOs were supervisors or 

leadership staff, with higher compensation than that of nonsupervisory or leadership staff. Last, 

even within the same job category, youth development CBOs paid a similar or greater amount than 

other types of CBOs. As a result, youth development CBOs’ average annual staff compensation 

was 10 to 46 percent higher than that of other types of CBOs, and their average annual labor cost 

was 33 to 76 percent higher. 

The average cost per C2C client served among youth development CBOs, however, was the 

lowest, followed by job training and employment CBOs, other CBOs, and parent/caregiver-serving 

CBOs. This is primarily due to a larger number of clients served among youth development 

CBOs than other types of CBOs (48–149 percent more in year 3). 

When examining the trends in the average cost per client served, we observed a decline from 

year 2 to year 3 across all CBO types, with the largest decline of 32 percent among other CBOs. 

251 
 

 

 

 

 

 

 

A majority of the decline in cost per client served can be attributed to the decreases in depreciation, 

MHP and trainer contract expenses, and variable nonlabor costs. 

We expected that larger CBOs would have a lower average cost per client served. This 

hypothesis held true with the data for youth development CBOs compared with other types of 

CBOs, but similar patterns were not observed when examining CBOs by employment size. 

Larger CBOs, in terms of the number of staff employed and the number of C2C staff members, 

were not associated with a lower cost per client served. These results suggest that, when 

examining the economies of scale of C2C, the output (e.g., service volume or the number of 

clients served) matters more than the input (employment size). We acknowledge that a simple 

count of the number of clients served masks the heterogeneity in CBO services. For example, the 

clients of parent/caregiver-serving CBOs are typically enrolled at the beginning of a year and 

receive CBO services for a long period of time, whereas the clients of youth development CBOs 

may receive services for several weeks. Such heterogeneity in CBO services should be accounted 

for in future analyses, if more detailed data become available. 

The cost structure of CBOs with internal clinicians is different from those with external MHP 

clinicians. Eleven CBOs subcontracted with external MHP organizations to cover MHP staff 

time; the four remaining CBOs had preexisting MH clinics in their organizations, enabling them 

to cover MHP labor costs as internal staff. Consequently, these expenses show up differently in 

the cost data, but we did not find notable patterns in terms of cost efficiency worthy of further 

exploration. 

Invoice-Based Cost Estimates and Funding Sources

Program cost estimates from our main analysis for year 2 and year 3 were 79 percent larger 

than those from the sensitivity analysis based on invoices only. Direct payments to MHPs were 

the same in both, but the labor cost from the invoice-based estimate was about two-thirds of those 

from our main estimate, variable cost was about half, and overhead cost was about one-quarter. 

The invoices did not provide information on fixed asset depreciation expenses. Examination of 

the C2C funding sources showed that nearly half of the resources consumed were not part of the 

planned budget, representing costs that participating CBOs had to absorb. 

There are several possible explanations for why invoice-based estimates were different. A 

number of factors affect the amount billed on invoices. The invoices are based on the budgets 

submitted to the Mayor’s Fund in the original applications, and the actual amount of resources 

spent on the program may have differed from what is in the budget. Because CBOs were obligated 

to raise 50 percent of the total budget to match the amount from the sponsor, the amount CBOs 

can bill may be limited by the funds they were able to raise, rather than the (greater) amounts they 

spent. In the first 2 years, many CBOs had not fully ramped up the program and they underspent 

the allocated funds. On the other hand, CBOs might have been incentivized to spend more 

resources on the program if they had enough matched funds available. Because CBOs were 

allowed to carry over remaining funds from 1 year to the next, CBOs had little incentive to 

252 
 

 

overbill toward the end of a project year. Also, based on our communication with CBOs’ 

financial personnel, some have may billed expenses that are not allowed under C2C to other 

projects. As such, it seems the estimates based on invoices only would bias the program cost 

results downward. 

Considering the limitations discussed above, for our main program cost analysis, we used 

both invoice and survey data, in addition to CBO self-reported nonlabor expenses and financial 

statements. The labor input came from the surveys, whereas nonlabor expenses were based on 

multiple data sources: the invoices, self-reported nonlabor expenses, and financial statements. 

Overhead to direct cost ratios were derived from CBO financial statements. Of course, labor cost 

estimates based on surveys are subject to biases due to survey nonresponse and inaccurate recall 

of respondents. Nonetheless, this approach allows us to leverage the advantages of multiple data 

sources to generate the best program cost estimates. 

Our qualitative interviews with CBO and MHP leadership revealed that CBOs felt that 

finding the matched funds was difficult and that budgets were constrained. Perceptions of budget 

constraints may not always be accurate, depending on the type of activities requiring cash outlays. 

For example, some activities incurred direct costs, such as paying for a conference room for C2C 

training or hiring a new person for C2C. Such activities that require cash outlays and approval 

from leadership are likely to be more closely tracked (and perhaps cut), which could give CBO 

staff the perception of budget constraints and that sponsor funds were important to relieve such a 

constraint. On the other hand, since a majority of cost incurred was for staff time, and existing 

staff are often salaried, it is easier for CBO staff to reallocate their time to C2C from other typical 

tasks they would have done in the absence of C2C. This labor reallocation may not be as obvious 

as activities requiring cash outlays. In other words, real resources were spent in both scenarios, 

but the perception of a budget constraint could be very different. Overall, their perceptions were 

consistent with our estimate that CBOs spent on average 79 percent more resources than what 

was in the budget. 

Comparison to Prior Research

As described earlier in this chapter, little research has been conducted previously to examine 

the cost of task-shifting in improving mental health. The closest comparison is a prior evaluation 

of the cost of a collaborative care program. The study by Hoeft et al. (2019) estimated a cost per 

patient per month of $154 to $544 based on five collaborative care programs that aimed to 

improve depression treatment among older adults. These estimates translated to an annual cost 

of $1,848 and $6,528, respectively, much higher than our average estimates ($536 per client in 

year 2 and $482 in year 3). Of course, the studied collaborative care programs were very different 

from C2C. In a collaborative care program, in addition to the care provided by primary care 

providers, extra visits with a professional MHP were made for depression patients, and case 

management services were also provided. The cost estimates included, among others, the 

incremental time from case managers, primary care providers, and professional MHPs. In C2C, 

253 
 

 

CBO staff were trained in the four C2C skills and incorporated such skills in their regular services. 

If needed, clients were referred to a professional MHP. Our cost estimates do not include the time 

spent by professional MHPs, so the estimates are not directly comparable. However, we offer the 

example here as a point of reference, given a lack of better comparisons in the literature. 

Utility of Cost Estimates to Other Settings

How can potential adopters of C2C use these program cost results? We are aware that the 

15 CBOs participating in the C2C program may differ from those in other states or cities; that is, 

the average program cost estimates are not necessarily generalizable to other geographic areas. 

Our results are organized such that adopters of C2C or a similar program could use the data 

presented here to help their decisionmaking, especially with regard to labor costs, which accounted 

for nearly two-thirds of total program costs. For instance, results regarding the number of staff 

involved, their job categories, and the amount of time required to implement the program may be 

of interest to other locales. Potential C2C adopters could apply local wages for each job category 

to these data to estimate labor costs, and they could adjust the nonlabor costs by comparing the 

cost of doing business between NYC and their locality. Finally, they can use their own overhead 

cost ratio to estimate the cost of overall management, as well as other expenses that are not 

allocated to specific projects such as rental expenses or utilities. 

Limitations

There are several limitations to our cost evaluation. For example, C2C is funded as a 5-year 

program, but due to data availability, the cost evaluation focuses on years 2 and 3. We present 

the labor hours based on the year 4 staff survey in Appendix D, but it is largely unknown 

whether, in year 4 or 5, the program matured and achieved increasing returns to scale—or even 

better—constant returns to scale (at which point a CBO would operate at the bottom of its longrun total average cost curve or achieve the lowest average cost per client served). Because of the 

lack of the survey and nonlabor expense data for year 1 and the limitations of invoices as described 

above in the Invoice-based Cost Estimates section, we were not able to reliably estimate the 

program startup cost. 

We did not collect cost data from the comparison CBOs that participated in the impact 

evaluation (see Part III of this report) due to the heterogeneity in services and clients across 

CBOs. That is, the amount of resources consumed by individual comparison CBOs is not 

necessarily comparable to that of intervention CBOs in the absence of C2C. We therefore 

measured the incremental resources used by intervention CBOs to implement and maintain the 

C2C program. This cost evaluation assumes that C2C cost can be separated from the cost of 

regular services, which is the case for the most part. But we did encounter the problem of 

measuring C2C skills delivery cost. Because of the design of the C2C intervention as being 

integrated into usual CBO programming, we had difficulty in identifying incremental time spent 

delivering C2C skills. Further, in using C2C CBOs as their own control, we assumed that the 

254 
 

cost of regular CBO programming (outside C2C) was not affected by C2C. This assumption may 

not hold and is therefore another limitation to be considered when interpreting these findings. 

We relied on the staff surveys to collect the information on labor input, that is, the amount of 

time spent on the program, which has its own limitations. The response rates were 33, 53, 56, 

and 51 percent for the year 2 staff survey, year 3 staff survey, and year 3 cost of labor survey, 

and year 4 staff survey, respectively. Although we applied weights to account for nonresponse in 

the analyses, we had to assume that the respondents were a representative sample of all C2C 

staff. For year 3, the distribution of staff job categories of all C2C staff was nearly identical to 

that of the respondents, suggesting that our assumption is reasonable. In addition, we had to 

extrapolate the labor input in a week based on year 2 staff survey and two different 1-week data 

points based on the year 3 staff survey and cost of labor survey. There may be seasonal variation 

in terms of the labor hours spent on C2C, and as a result, our estimates may bias labor input 

upward or downward, depending on the timing of the surveys. 

In a majority of cases, C2C activities are not delivered separately from regular CBO services, 

and this has created a challenge for the surveys. For example, MI and MHFA are often delivered 

by CBO staff when they are delivering regular CBO services, although screenings and PE may 

be separate. In staff surveys or cost-of-labor surveys, respondents may have difficulty estimating 

how much time they allocated to C2C skills. Given the lack of an accurate way to capture this 

data point, we excluded C2C skills delivery time in the main analysis. We conducted a sensitivity 

analysis to adjust the labor input, assuming that adding C2C skills would increase the overall 

service delivery time by 5 to 25 percent, based on our qualitative interview results. We found 

that incorporating C2C skills in regular CBO service delivery could potentially increase the total 

program cost by 0.2 to 10.3 percent. Because these adjustment ratios are exploratory, our main 

analysis does not incorporate this adjustment. 

We were not able to separate data collection or technical assistance activities for research 

purposes from those for program implementation and reporting purposes. Because a majority of 

data collection was for reporting requirements of the Mayor’s Fund, the amount of time spent on 

completing surveys, interviews, or financial data should be relatively small. For example, the 

staff survey took an average of 25 minutes to complete and the cost-of-labor survey took about 

10 minutes, once per year. The interviews and financial data collection took about an hour each, 

but they only involved a small number of individuals. Similarly, most technical assistance 

provided to CBOs was for program implementation. The inability to accurately separate hours 

for data collection and technical assistance associated with the evaluation may have led to a 

slight overestimation of labor hours. 

Summary

The C2C program changed over time, and such changes were reflected in the resources 

consumed to implement and maintain the program. Compared with year 2, nearly 50 percent 

255 
 

 

 

 

more staff members participated in C2C, and program staff allocated less time to training, 

technical assistance, data collection, and project management, but more time to providing 

training and receiving coaching and supervision. In addition, CBOs shifted their labor input to 

nonleadership staff over time, with the average number of hours contributed by leadership 

decreasing by about one-quarter. 

The estimated number of unique clients served increased nearly one-third from year 2 to 

year 3, and the average annual cost per CBO increased by 18 percent, largely due to increased 

labor input, which accounted for nearly two-thirds of the total program cost. Nearly half of the 

resources consumed in the C2C program were not part of the planned budget, and the participating 

CBOs had to absorb these costs. 

From an efficiency standpoint, the C2C program overall appears to have completed the startup phase, and six CBOs achieved increasing returns to scale. The average cost per client served 

decreased from $536 in year 2 to $482 in year 3. However, this result is confounded by the 

heterogeneity in the changes of program input and service volume across CBOs. Compared with 

other types of CBOs, for example, youth development CBOs in year 3 had the largest number of 

unique clients served per CBO per year, the greatest annual program cost per CBO, but the 

lowest average cost per client served. 

Funders of C2C or similar programs can use our results to estimate the resources required to 

implement the program according to their local cost of doing business. For planning purposes, 

potential adopters may need to prepare resources required to implement and maintain C2C or 

similar programs for 2 to 3 years before the average cost per client served declines. Our results 

show that some CBOs, although small in terms of the number of clients served, observed a 

declining cost per client served at the end of year 3. To control implementation costs, future C2C 

adopters may want to focus on building resources for labor input because a majority of program 

cost is attributed to staff time, though controlling nonlabor and overhead expenses may shorten 

the time needed to achieve increasing returns to scale. 

256 
 

 

 

 

 

References

Barrett, B., S. Byford, and M. Knapp. (2005). “Evidence of Cost-Effective Treatments for 

Depression: A Systematic Review. Journal of Affective Disorders, Vol. 84, No. 1, 1–13. 

Crowley, M. D., A. D. Kenneth, W. S. Barnett, P. Corson, S. Duffy, P. Graham, M. Greenberg, 

R. Haskins, L. Hill, D. E. Jones, L. A. Karoly, M. R. Kuklinski, and R. Plotnick, Standards 

of Evidence for Conducting and Reporting Economic Evaluations in Prevention Science. 

Prevention Science, Vol. 19, No. 3, 2018, pp. 366–390. 

Hoeft, T. J., J. C. Fortney, V. Patel, and J. Unützer, Task-Sharing Approaches to Improve Mental 

Health Care in Rural and Other Low-Resource Settings: A Systematic Review. The Journal 

of Rural Health, Vol. 34, No. 1, 2018, 48–62. 

Hoeft, T. J., H. Wilcox, L. Hinton, and J. Unützer, “Costs of Implementing and Sustaining 

Enhanced Collaborative Care Programs Involving Community Partners,” Implementation 

Science, Vol. 14, No. 1, 2019, p. 37. 

Rodgers, M., M. Asaria, S. Walker, D. McMillan, M. Lucock, M. Harden, S. Palmer, and 

A. Eastwood, A. “The Clinical Effectiveness and Cost-Effectiveness of Low-Intensity 

Psychological Interventions for the Secondary Prevention of Relapse After Depression: 

A Systematic Review,” Health Technology Assessment (Winchester, England), Vol. 16, 

No. 28, 2012, pp. 1–130. 

U.S. Bureau of Labor Statistics, Historical Consumer Price Index for All Urban Consumers: 

U.S. City Average, all Items, by Month. For U.S. City Average, 2019. As of April 3, 2019: 

https://www.bls.gov/cpi/tables/supplemental-files/home.htm 

Van Ginneken, N., P. Tharyan, S. Lewin, G. N. Rao, S. M. Meera, J. Pian, S. Chandrashekhar, 

and V. Patel, “Non-Specialist Health Worker Interventions for the Care of Mental, 

Neurological and Substance-Abuse Disorders in Low-and Middle-Income Countries. 

Cochrane Database of Systematic Reviews, No. 11, 2013, CD009149. 

257 
Part V. Conclusions and Recommendations

We close this report by highlighting the key themes or takeaways from this evaluation, across 

its three components (implementation, impact, and cost). Each individual chapter has already 

detailed strengths and limitations and interpreted specific findings. Therefore, in this section, we 

take a step back to draw higher level conclusions and generate recommendations for future 

policy, practice, and research related to mental health task-shifting and the C2C model. 

258 
 

 

11. Conclusions and Recommendations

Lynsay Ayer, Michael Stephan Dunbar, Dana Schultz, Harry H. Liu, and Monique Martineau 

Access to mental health care in the United States is not evenly distributed. Communities 

affected by poverty and unemployment—problems that disproportionately affect racial/ethnic 

minorities—often experience the greatest barriers to obtaining mental health care when it is 

needed. C2C sought to address this inequality and improve mental health and well-being by 

integrating mental health skills into the usual work of CBOs that are trusted and established 

in their NYC communities. The C2C program specifically focused on serving three target 

populations: (1) youth/young adults ages 16 to 24 who are not in school and are not employed, 

(2) adults age 18 or older who are unemployed or underemployed, and, (3) parents/primary 

caregivers who are expecting or who have children up to the age of 4. To understand the effects 

of the C2C program, we evaluated its implementation, impact on individual client outcomes, 

and costs to CBOs. The results of these evaluations and discussion of their interpretation were 

detailed in the previous chapters. 

Overall, the results of the impact evaluation did not provide evidence of the effectiveness 

of the C2C model of task-shifting. We found no differences between the C2C group and the 

comparison group overall, although we did find some benefits of C2C for certain subpopulations 

and in certain settings. While C2C participants as a whole improved in most of the outcome 

domains investigated in the impact study, the comparison group as a whole improved by a 

similar amount. 

There are several possible explanations for this. It may be that C2C lacks efficacy. It may 

also be that the C2C toolkit wasn’t powerful or intensive enough to result in measurable impacts in 

the domains we examined relative to the comparison group. Moreover, the similar improvements 

in both the C2C and comparison groups may reflect a natural recovery process in both groups 

or—as discussed earlier—may have been influenced by extensive concurrent mental health 

programming from the ThriveNYC or other mental health initiatives or programs to which many 

comparison participants may have been exposed. Finally, as Kazdin (2015) has noted, it is not 

uncommon for treatments of known efficacy relative to no-treatment or waitlist control groups to 

perform at a level similar to “treatment as usual” (as in the comparison group in this study) 

(Kazdin, 2015). 

While the impact evaluation results do not support broad scale-up of the C2C model of taskshifting, our experience points to three conclusions about the task-shifting approach that should 

be considered by practitioners, policymakers, and/or researchers interested in further exploring 

how to design, implement, and evaluate task-shifting interventions. Below, we describe these 

conclusions and then offer a series of recommendations that derive from them. 

259 
 

 

 

 

Conclusions

Reaching Full Implementation of a Complex Task-Shifting Model Requires Significant 

Investment of Time and Resources—but Is Feasible

Our findings from the implementation and cost studies showed that the first 2 years of C2C 

implementation involved a significant investment of time and resources from CBOs to get the 

program up and running—even with technical assistance from NYU McSilver and the C2C 

Collaborative. It took over a year for most CBOs to set up the C2C program. It is typical for new 

mental health programs to require a lengthy start-up period (Torrey et al., 2012; Williams et al, 

2018; Williams et al., 2020). However, because C2C was implemented in nontraditional settings 

without existing infrastructure and staff for mental health programming, this was especially true. 

C2C required substantial upfront effort and attention on developing relationships (e.g., with the 

MHPs), planning activities, hiring staff, and developing and refining workflows. We also 

found that many of the partnerships between CBOs and MHPs came together in fits and starts, 

requiring time and coordination across two different organizations with different staff, missions, 

and cultures. The cultural shifts experienced by CBO and MHP staff took time to develop 

throughout the project, as the program evolved and matured. 

Cost analyses showed that implementation efforts were still incurring substantial costs 

beyond the initial planning phase. From year 2 to year 3, labor input increased nearly 50 percent 

and the number of clients increased 31 percent. Over that same period, CBOs shifted their labor 

input to nonleadership staff, decreasing the average number of hours contributed by leadership 

by about one-quarter, and the average cost per client served decreased by 10 percent. We would 

expect that, over time, C2C program costs would decrease further in the future because CBOs 

require less spending on the start-up administrative and training costs that were required to set up 

the program and because CBOs continue to improve their operation efficiency. 

Prior literature shows that integrating and scaling-up evidence-based mental health practices 

in community settings requires a multiphased approach to building the partnerships, infrastructure, 

and capacity required by organizations to enact and sustain new practices (e.g., Grant, Simmons, 

and Davey et al., 2018; Saldana and Chamberlain, 2012). Specific timelines may vary from 

weeks to years, depending on a range of factors including existing infrastructure, organizational 

readiness, and intervention complexity, among other factors (Hurlburt et al., 2014). Moreover, 

past work suggests that factors affecting implementation of EBPs—and clinicians’ use of new 

EBPs—in community mental health settings evolve over periods of 2 to 5 years following initial 

implementation (Torrey et al., 2012; Williams et al, 2018; Williams et al., 2020). It is reasonable 

to expect that these factors would be even more pronounced in the case of a complex task-shifting 

intervention like C2C that is implemented within nontraditional settings with little or no mental 

health experience. More specifically, for many or most CBO staff, mental health–related tasks 

were new and different from their usual responsibilities. This was likely reflected in the initial 

investment of time and resources and consistent increases in labor input over the first several years 

260 
 

 

of the project. Despite the initial resource-intensive start-up phase, C2C served a large number of 

clients (41,000) over the first 3 years and trained more than 1,700 CBO staff members—exceeding 

the initial goal of serving 40,000 clients. 

Although C2C Was Well Received by Staff and Providers, Barriers Such as Stigma and 

Access Challenges Were Hard to Overcome

Our analysis of implementation data showed that, by year 4, most clients (more than 80 percent) 

referred for formal mental health assessment and treatment kept their first appointment with the 

MHP. This is notably higher than rates of kept appointments reported in studies on other mental 

health interventions, in which less than 60 percent of referred patients may keep their first 

appointment. This finding aligned with other data from the implementation study showing that 

most CBO staff (77 percent) felt better equipped to address their clients’ mental health needs, 

particularly by using their MI skills and leveraging their relationship with the MHP. 

Despite overall very positive impressions of the C2C program, our findings from the 

implementation and impact studies showed that some barriers to care—particularly mental 

health stigma—were difficult to overcome. Qualitative data from the implementation study 

showed that many CBO staff believed stigma continued to interfere with clients’ engagement in 

mental health services outside the CBOs, and that logistical barriers like a lack of transportation, 

insurance, and childcare continued to interfere with clients’ abilities to access needed mental 

health care. CBOs and MHPs worked together to come up with a range of solutions to address 

such barriers, like providing metro cards and expanding clinic hours. However, the impact study 

showed that, although both C2C clients and those receiving usual CBO services (i.e., the 

comparison group) reported fewer barriers at the 12-month follow-up compared with baseline, 

there was no benefit of C2C on barriers to mental health care overall. An exception to this, 

revealed in the subgroup analyses, was a small-sized effect (d = −0.28) of C2C on attitudinal 

barriers for the targeted out-of-school and out-of-work youth and young adult population. 

The overall lack of a difference between the C2C and comparison groups in terms of effects 

on barriers to mental health care may not be surprising because this evaluation occurred amid a 

larger Thrive NYC initiative, which rolled out many different programs all over the city with an 

intention to promote mental health and reduce stigma and other barriers to care. Comparison 

CBO clients and staff were likely to be exposed to some of these other ThriveNYC programs or 

other efforts to expand mental health supports. Further, in qualitative interviews conducted with 

CBOs for the impact evaluation, we learned that C2C CBOs were often offering assistance to 

other CBOs in the city who were interested in implementing C2C or some of its components. In 

addition, some barriers, such as mental health stigma, operate on a more systemic scale and may 

be out of reach for C2C or any single mental health initiative. In the same vein, logistical barriers 

like a lack of affordable childcare and health insurance are widespread issues beyond the scope 

of C2C. In 2019, NYC Mayor DeBlasio announced NYC Care, a large investment in expanding 

health care to cover those who do not qualify or cannot afford health insurance (NYC Care, 

261 
 

 

 

undated). Future evaluations could examine whether implementation of this program impacted 

utilization of mental health care. Also, adaptations of C2C could more intentionally address 

mental health stigma and attitudinal barriers by including evidence-based stigma reduction 

curricula (e.g., Bulanda et al., 2014; Perry et al., 2014; Pinto-Foltz, Logsdon, and Myers, 2011) 

and campaigns such as National Alliance on Mental Illness (NAMI)’s Ending the Silence 

campaign (Wahl et al., 2018). Our finding in the youth and young adult subgroup suggests 

that the attitudinal barriers of younger participants may be more malleable and responsive to 

interventions as well, perhaps due to generational differences in views about mental health and 

treatment seeking (ADAA, 2015). 

C2C Positively Affected Some Populations and Settings but Not Others

Because it is often difficult to detect intervention effects in comparison to a treatment as 

usual group (Kazdin, 2015), we conducted additional analyses of the effect of C2C among 

sample subgroups by CBO type or target population. In these analyses, we identified some 

meaningful intervention effects, as well as a few instances where the comparison group had 

better outcomes than the C2C group. For example, C2C increased utilization of outpatient 

mental health care in the youth development program subgroup and reduced emergency service 

utilization in the parents and caregivers’ subgroup and the youth and young adult subgroup. 

C2C may have also led to improvements in non–mental health outcomes for some participants, 

as demonstrated by the increase in weekly work hours and monthly pay among participants in 

the job training and employment CBOs. 

This pattern of findings—a lack of differences in the overall sample but evidence for 

intervention effects in some subgroups—may seem counterintuitive. However, there are several 

plausible explanations for it. For one, the populations and settings in which C2C was implemented 

were extremely heterogeneous. The finding that the intervention worked better for some 

populations and in some settings than others (“treatment heterogeneity”) is consistent with 

other mental health intervention studies—particularly those like C2C that test program 

effectiveness in real-world settings —which often find that a program works better for certain 

groups (e.g., racial/ ethnic groups, ages, gender, symptom severity and diagnoses) (Kravitz, Duan, 

and Braslow, 2004; Vardahan and Seeger, 2013). Furthermore, the timeline and process for 

getting C2C fully up and running was not uniform across CBOs; some were at full implementation 

much sooner than others. Because of the timing of the impact study, we were not able to wait for 

C2C to be scaled and fully implemented at each CBO before starting data collection, so some 

were only at partial C2C implementation (e.g., doing screenings, MHFA, and PE but not yet MI) 

at the time impact study data collection began. These implementation timeline variations could 

also explain why C2C appeared to have a stronger influence in some settings and for some 

populations than others. 

This pattern of findings should also be considered in the context of the different studies. As 

described earlier, the implementation study found that task-shifting is feasible and CBO staff 

262 
 

perceived it as helpful to their clients. While this appears to run counter to the overall impact 

study findings, it is important to note that the samples in each of the studies differed in 

meaningful ways. For the implementation study, participants in the surveys and interviews were 

mostly staff and a few clients who were selected for the study by the CBOs—the people best 

equipped to speak about the process of implementing C2C. The impact study, on the other hand, 

enrolled clients with at least a low baseline level of mental health symptoms, regardless of the 

extent to which they actually ended up engaging in C2C. This design helped to ensure the impact 

study findings were not disproportionately influenced by outcomes of highly motivated and 

engaged C2C clients, who may not be representative of the overall target population. Therefore, 

findings from these different implementation and impact study samples can help to contextualize 

one another, but they are not directly comparable. 

Recommendations

Our recommendations, based on the conclusions outlined above, are designed to inform 

future design, implementation, and evaluation of task-shifting interventions. 

Recommendation 1: Design the mental health task-shifting model with evidence-based content

and alternative delivery modes (e.g., telehealth) to reduce barriers to mental health care.

Many barriers to mental health care were not responsive to the C2C program, as shown by 

both implementation and impact study data. Some barriers like lack of health insurance or fears 

about seeking help stemming from undocumented immigrant status could be addressed at the 

broader policy level but may be outside the control of any individual task-shifting program. 

However, evidence-based stigma reduction curricula and campaigns can be integrated into a 

task-shifting model to help reduce other barriers to care. Our impact study findings suggest that 

intentional and systematic inclusion of stigma reduction curricula into a task-shifting intervention 

could be particularly helpful since C2C in its current form did not appear to impact stigma 

overall, compared to as usual CBO services. 

Furthermore, remote delivery of mental health skills such as the ones implemented in C2C 

(e.g., online, telephone or videoconferencing) is another way to reduce logistical barriers like 

transportation. Tele-mental health has been shown to be an acceptable way to deliver mental 

health interventions with similar outcomes to in-person modalities (Jenkins-Guarnieri et al., 

2015; Morland et al., 2020). Data collection for this evaluation ended before the COVID-19 

pandemic led to broad, citywide shutdown of many in-person CBO services, but as of this 

writing, many CBOs have already found ways to continue delivering C2C to their clients and to 

receive coaching and supervision remotely. Future evaluations could examine the implementation, 

impact, and cost of task-shifting models delivered via telehealth modalities compared with in 

person. 

263 
 

 

 

 

 

 

 

 

Recommendation 2: Consider replacing or augmenting the four skills that made up the original

C2C model (screening, MHFA, MI, PE) with other evidence-based strategies.

C2C was designed to be flexible to meet the needs of each CBO and its clientele. The four 

core skills employed in the C2C model could, however, be changed to better meet the needs of 

clients. Other evidence-based mental health strategies, including CBT skills, for example, can be 

very effective in reducing mental health symptoms and promoting well-being. For example, a 

task-shifting program in Zimbabwe, called the Friendship Bench, trained grandmothers in 

problem-solving therapy (Chibanda et al., 2015) to help community members struggling with 

depression. This and other evidence-based strategies would still require proper training, coaching, 

and supervision, as described in Part II of this report. However, they may have a stronger effect 

on mental health outcomes for many clients. CBOs considering adapting C2C or designing 

another task-shifting model may wish to reference our mental health task-sharing guide (Stevens 

et al., 2020), which contains suggestions for other evidence-based mental health strategies that 

could be delivered within a mental health task-shifting program. 

Recommendation 3: Examine the role of systemic barriers to implementation and sustainment

of C2C and other nontraditional mental health delivery models.

A thorough examination of systemic barriers to implementation of and client engagement in 

C2C was beyond the scope of this evaluation. However, as noted previously, many logistical 

barriers like lack of insurance to cover the cost of mental health services could be ameliorated by 

systemic changes such as universal coverage for mental health care or Medicaid reimbursement 

for mental health services delivered by CBOs. Any nontraditional model that delivers mental 

health services outside clinical settings will have to address these issues to be successful and 

sustainable. 

Recommendation 4: Invest resources in reducing barriers to care among youth and young 

adults.

Youth and young adult attitudinal barriers to care appeared to be more responsive to the C2C 

program compared with older age groups. Investing resources (e.g., time, money) in task-shifting 

programs such as C2C that can further eliminate barriers to mental health care in youth and 

young adults may have long-term benefits, by helping this younger generation to access needed 

treatment earlier, before mental health problems worsen and become more impairing. 

Recommendation 5: To implement mental health task-shifting in CBOs plan for at least an 

initial 1-year ramp-up period, and prepare for increasing costs over time until the program

gets to scale.

Implementation of task-shifting may require a significant ramp-up period to put in place 

the necessary protocols and processes. The 15 CBOs and MHPs participating in C2C benefited 

from the public-private partnership that funded the program as well as from technical assistance 

and training support from the C2C Collaborative and NYU McSilver. CBOs interested in 

264 
 

 

 

 

 

implementing task-shifting may not have access to this level of technical support or resources. 

The lessons learned and guidance reported here and in the task sharing guide (Stevens et al., 

2020), can guide future implementation and evaluation of similar task-shifting models. This 

guidance about what to consider for this ramp-up period may help CBOs avoid some of the 

initial challenges confronted in this project. While the time and cost required to set up a taskshifting intervention will likely vary depending on many factors like the design and focus of the 

program, characteristics of the CBO (e.g., staff and leadership buy-in, size), and the community 

context (e.g., cost of living, support from state or local governments), the time and cost estimates 

provided in this report can be used as a reference point or rule of thumb. 

Recommendation 6: Consider the population, setting, and outcomes when determining 

whether and how to implement the model.

The lack of impact study findings in the overall sample suggests that, in deciding how to 

design and implement task-shifting, it is important to consider the target population, setting, and 

outcomes. Not surprisingly given the heterogeneity of the sample, our study showed that C2C 

did not help everyone more than usual CBO services; although certain subgroups benefited more 

than others. Mental health task-shifting interventions—including adaptations of C2C—should be 

designed to best fit the particular needs and goals of the CBO and clients. For example, if the 

highest priority of the program is to increase utilization of outpatient mental health services, it 

may be best to focus on youth and young adults (because our findings showed this subgroup 

benefited from C2C in this domain, whereas others did not), or to consider ways to adapt the 

program to enhance its effect on other groups. 

Recommendation 7: Design future evaluations of the effectiveness of task-shifting to account

for individual and site-level heterogeneity, and to examine mechanisms of change.

Randomization of participants to a control condition was not possible within this impact 

study, nor was the use of a within-CBO comparison group. However, these methods could help 

future research to ensure the ability to control for potential confounders that we could not reliably 

measure in this study (e.g., the effectiveness of “usual” CBO services) within the study design 

itself. Although we are confident that our rigorous propensity score weighting techniques 

balanced the comparison and C2C groups on most individual client characteristics, we cannot 

rule out the possibility that our findings were influenced by other, unmeasured variables like 

CBO-specific characteristics. This is a limitation of the quasi-experimental design, which could 

be addressed using other designs in future studies of task-shifting (e.g., randomized controlled 

trials). 

Future studies of task-shifting with more highly controlled designs and targeted outcomes 

should also ensure greater consistency in training, coaching, and supervision across sites, and 

conduct more rigorous fidelity monitoring than we were able to perform. This would also help to 

minimize the influence of variation in training and implementation quality on study findings. 

265 
 

In addition, given the already extensive set of outcomes examined, we were not able to add 

measures of potential mechanisms of change in this study. However, our findings highlight the 

importance of studying mechanisms in future research. C2C differentially affected certain 

settings and populations and, while we offer some possible explanations in this report, we do 

not have the data to test whether they are supported by the evidence. Understanding how and 

why task-shifting interventions work for certain groups and settings (and not others) could help 

make such programs more efficient, for example, by eliminating components that are less 

impactful and augmenting the program with more powerful ones. 

Final Thoughts

To our knowledge, this is the first U.S.-based evaluation of a mental health task-shifting 

intervention on this scale. As we have noted, the study is not perfectly suited to answer every 

question about C2C’s implementation, impact, and cost. The impact study found no evidence for 

C2C’s impact on a range of outcomes in the overall sample, though we did find some benefits of 

C2C for certain subpopulations and in certain settings. CBOs balanced many factors to tailor the 

program to their clientele, but most CBO staff and leaders reported that C2C changed the way 

they approached mental health with clients and among themselves. The cost evaluation found 

that labor costs accounted for a substantial portion of overall costs, and that less than half of 

CBOs were able to achieve economies of scale sufficient to reduce the cost per client by year 3. 

Yet the positive feedback from CBO and MHP staff and leadership and the effects of C2C on 

certain subgroups within the impact evaluation—however small—demonstrate that CBOs are 

capable and well suited to integrate mental health supports into their usual services, with help 

from partnering MHPs. Over the course of the 5-year project, C2C brought mental health 

services to tens of thousands of low-income New Yorkers and continues to do so, adapting to the 

new reality of delivering these services remotely during the COVID-19 pandemic. Together, the 

three evaluations underscore that C2C is feasible and can provide useful information to those 

interested in further refining task-shifting. These findings can help CBOs, MHPs, policymakers, 

funders, researchers, and communities as they consider mental health task-shifting as a new way 

to expand the mental health workforce and reach vulnerable, low-income communities with 

evidence-based care. 

266 
 

 

 

 

 

 

 

 

 

 

 

References

Anxiety Disorders Association of America (ADAA), editor, College-Aged Adults Face Less 

Mental Health Stigma, 2015. As of July 05, 2020: 

https://adaa.org/college-aged-adults-face-less-mental-health-stigma 

Bulanda, J. J., C. Bruhn, T. Byro-Johnson, and J. Zentmyer, “Addressing Mental Health Stigma 

Among Young Adolescents: Evaluation of a Youth-Led Approach,” Health & Social Work, 

Vol. 39, No. 2, 2014, pp. 73–80. 

Chibanda, D., T. Bowers, R. Verhey, S. Rusakaniko, M. Abas, H. A. Weiss, and R. Araya, 

“The Friendship Bench Programme: A Cluster 267 Randomized Controlled Trial of a Brief 

Psychological Intervention for Common Mental Disorders Delivered by Lay Health Workers 

in Zimbabwe,” International Journal of Mental Health Systems, Vol. 9, No. 1, 2015, p. 21. 

Grant, K. L., M. B. Simmons, and C. G. Davey, “Three Nontraditional Approaches to Improving 

the Capacity, Accessibility, and Quality of Mental Health Services: An Overview,” Psychiatric 

Services, Vol. 69, No. 5, 2018, pp. 508–516. 

Hurlburt, M., G. A. Aarons, D. Fettes, C. Willging, L. Gunderson, and M. J. Chaffin, “Interagency 

Collaborative Team Model for Capacity Building to Scale-Up Evidence-Based Practice,” 

Children and Youth Services Review, Vol. 39, 2014, pp. 160–168. 

Jenkins-Guarnieri, M. A., Pruitt, L. D., Luxton, D. D., and K. Johnson, “Patient Perceptions of 

Tele-Mental Health: Systematic Review of Direct Comparisons to in-Person Psychotherapeutic 

Treatments,” Telemedicine and e-Health, Vol. 21, No. 8, 2015, pp. 652–660. 

Kazdin, A. E., Treatment as Usual and Routine Care in Research and Clinical Practice,” Clinical 

Psychology Review, Vol. 42, 2015, pp. 168–178. 

Kravitz, R. L., N. Duan, and J. Braslow, “Evidence-Based Medicine, Heterogeneity of Treatment 

Effects, and the Trouble with Averages,” The Milbank Quarterly, Vol. 82, No. 4, 2004, 

pp. 661–687. 

Morland, L. A., M. A. Mackintosh, L. H. Glassman, S. Y. Wells, S. R. Thorp, S. A. M. Rauch, 

P. B. Cunningham, P. W. Tuerk, K. M. Grubbs, S. Golshan, M. J. Sohn, and R. Acierno, 

“Home-Based Delivery of Variable Length Prolonged Exposure Therapy: A Comparison of 

Clinical Efficacy Between Service Modalities,” Depression and Anxiety, Vol. 37, No. 4, 

2020, pp. 346–355. 

NYC Care, homepage, undated. 

https://www.nyccare.nyc/ 

Perry, Y., K. Petrie, and H. Buckley, L. Cavanagh, D. Clarke, M. Winslade, D. Hadzi-Pavlovic, 

V. Manicavasagar, and H. Christensen, “Effects of a Classroom-Based Educational Resource 

on Adolescent Mental Health Literacy: A Cluster 267 Randomized Controlled Trial,” 

Journal of Adolescent Health, Vol. 37, 2014, pp. 1143–1151. 

267 
 

Pinto-Foltz, M. D., M. C. Logsdon, and J. A. Myers, “Feasibility, Acceptability, and Initial 

Efficacy of a Knowledge-Contact Program to Reduce Mental Illness Stigma and Improve 

Mental Health Literacy in Adolescents,” Social Science & Medicine, Vol. 72, 2011, 

pp. 2011e–2019e. 

Saldana, L., and P. Chamberlain, “Supporting Implementation: The Role of Community 

Development Teams to Build Infrastructure,” American Journal of Community Psychology, 

Vol. 50, No. 3–4, 2012, pp. 334–346. 

Stevens, C., E. Tosatti, L. Ayer, D. Barnes-Proby, G. Belkin, S. Lieff, and M. Martineau. Helpers 

in Plain Sight: A Guide to Implementing Mental Health Task Sharing in Community-Based 

Organizations, 2020. 

https://www.rand.org/pubs/tools/TL317.html 

Torrey, W. C., G. R. Bond, G. J. McHugo, and K. Swain, “Evidence-Based Practice 

Implementation in Community Mental Health Settings: The Relative Importance of Key 

Domains of Implementation Activity,” Administration and Policy in Mental Health and 

Mental Health Services Research, Vol. 39, No. 5, 2012, pp. 353–364. 

Varadhan, R, and J. D. Seeger, “Estimation and Reporting of Heterogeneity of Treatment 

Effects,” in P. Velentgas, N. A. Dreyer, P. Nourjah P, et al., editors. Developing a Protocol 

for Observational Comparative Effectiveness Research: A User’s Guide. Rockville, MD: 

Agency for Healthcare Research and Quality (US), 2013. 

Wahl, O., J. Rothman, T. Brister, and C. Thompson, “Changing Student Attitudes About Mental 

Health Conditions: NAMI Ending the Silence,” Stigma and Health, Vol. 4, No. 2, 2018, 

pp. 188–195. 

Williams, N. J., C. B. Wolk, E. M. Becker-Haimes, and R. S. Beidas, “Testing a Theory of 

Strategic Implementation Leadership, Implementation Climate, and Clinicians’ Use of 

Evidence-Based Practice: A 5-Year Panel Analysis,” Implementation Science, 15, 2020, 

p. 10. 

Williams, N. J., M. G. Ehrhart, G. A. Aarons, S. C. Marcus, and R. S. Beidas, “Linking Molar 

Organizational Climate and Strategic Implementation Climate to Clinicians’ Use of EvidenceBased Psychotherapy Techniques: Cross-Sectional and Lagged Analyses from a 2-Year 

Observational Study,” Implementation Science, Vol. 13, No. 1, 2018, p. 85. 

268 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix A. C2C Program Summaries1 

Dionne Barnes-Proby, Clare Stevens, Michael Stephan Dunbar, Michele Abbott, and Wing Yi Chan 

Arab American Association of New York

Arab American Association of New York C2C Program Snapshot (as of Y4 Q3)

Mental health provider: NYU Langone Family Health Center

CBO type: Other (agency primarily serving immigrant populations)

On-site mental health counseling available at CBO: Full-time

C2C Initiative target group(s): Expectant parents and parents or caregivers of children ages 0 to 4; out-ofschool, out-of-work young adults ages 16 to 24; unemployed or underemployed low-income working-age adults

ages 18 and older receiving employment services

CBO’s target population(s): Arab Americans and Arab immigrants of all ages, primarily low-income women and 

youth

Target geographical area/s: Brooklyn

Target CBO programs for C2C implementation: All programs: Adult Education, Immigration and Casework,

Youth, and Advocacy

Number of CBO sites providing C2C services: One CBO location, one non-CBO location

Total CBO clients served by C2C (Y1–Y4 Q3): 4,268

Number of current staff and supervisors trained in one or more C2C modalities: 25

Number of staff receiving continuous coaching and supervision (Y4 average): 13

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview 

The Arab American Association of New York (AAANY) is a nonprofit organization that 

functions as a grassroots center situated in a storefront. Its mission is to support and “empower 

the Arab Immigrant and Arab American community by providing services to help them adjust to 

their new home” (AAANY, undated) and become active members of society. AAANY “serves 

as a bridge between the Arab community and the greater NYC community, fostering greater 

understanding of Arab culture and immigrant issues; serving as a liaison between schools, 

government and other institutions and residents to address issues of discrimination; and 

providing a variety of culturally sensitive social services” (AAANY, undated). AAANY 

provides Adult Education courses (English for Speakers of Other Languages, Civic and 

Naturalization Test Prep), Advocacy and Civil Engagement (Law Enforcement Accountability 

1 Each program description summarizes the specific way each CBO and MHP implemented the C2C program. The 

descriptions compile information and materials provided by the CBOs. 

269 
 

 

 

Campaign, Civic Engagement and Community Organizing, Immigration Reform Campaign, 

Muslim Schools Holidays Campaign, Arab Women Activists and Leaders, Deferred Action for 

Childhood Arrivals), Social Services (Case Worker Services, Legal and Immigration Services), 

and Youth Development programs (Teen Grant-Making Initiative, Youth Community 

Organizing).2 

AAANY recognized the need to provide culturally competent, trauma-informed mental 

health services to the Arab community in both English and Arabic. To that end, AAANY aimed 

to create a space where clients, particularly low-income underemployed adults and youth 

suffering from substance use, could feel safe exploring ideas of mental wellness. C2C was 

implemented across the entire organization, with the goal of weaving a deep understanding of 

best mental health practices into program services as part of a holistic model of service delivery. 

Staff were trained to provide C2C services at AAANY’s main site and at the satellite Adult 

Education Site. 

C2C Model Implementation

C2C staffing model. AAANY partnered with the NYU Langone Family Health Centers 

(NYU Langone) network, which provides outpatient primary health care and support services. 

The CBO C2C program lead (licensed master social worker [LMSW]) conducted all screening 

and referral to MH services, as well as psychoeducation, MI, MHFA, CBT, and trauma-informed 

care in English and Arabic. Other CBO staff implemented MI and MHFA and made internal 

referrals to the CBO C2C program lead. 

NYU Langone clinical staff provided all training, coaching, and supervision to AAANY staff 

(Table A.1). All NYU Langone staff were licensed in their field or board certified and trained in 

mental health screening and MI. All AAANY direct service staff, including the CBO C2C program 

lead, immigration caseworkers, operations manager, advocacy and civic engagement staff, 

immigration attorney, capacity building manager, HR manager, case managers, adult education 

coordinator and associates, were trained in all four skills by the MHP. To help staff translate 

mental health–related words or concepts into Arabic and its dialects, MHP facilitated MI and 

MHFA trainings with a focus on mindful translation and cultural competency. NYU Langone 

provided quarterly booster training sessions to staff, rotating the focus on one or more of the 

four core skills for each session. The primary form of ongoing training and supervision for 

AAANY staff was group meetings with the NYU Langone program lead approximately quarterly 

in department “check-in” meetings. These interactions typically consisted of discussions about 

specific client stories/scenarios encountered by staff. The NYU Langone program lead was also 

on-site every Friday for one-on-one weekly meetings with staff. The CBO C2C program lead 

also provided one-on-one ad hoc coaching to staff members. 

2 For more information, please refer to AAANY, homepage, undated. 

270 
 

 

 

 

Screening. Screening was not implemented universally. It was offered to all clients who 

attended psychoeducation as part of the Adult Education program and to immigration services 

clients as needed or requested. Staff members could also refer individual clients in any program 

to the C2C Program Lead for a screening. CBO staff were trained in screening but did not 

implement this service until the second quarter of year 4, where Immigration Services staff 

incorporated the two-item Patient Health Questionnaire into their standard screening tool. 

Screenings that occurred in conjunction with psychoeducation workshops were client selfadministered in a group setting. Individual screenings occurring outside psychoeducation 

workshops were conducted in two steps. First, a client who presented to CBO staff with any 

kind of emotional distress, or who asked to see a social worker, received a basic “rated 

stress assessment” screening. If clients scored positively on this screening, the staff used 

psychoeducation to advise them on mental health services available on-site and, as necessary, 

MI to help clients accept a referral to the CBO C2C program lead for a full screen. The CBO 

C2C program lead then screened these referred clients, completed an intake form, and conducted 

a psychosocial interview to determine more about the client’s history, background, and symptoms. 

All clients offered a screening were screened for depression (PHQ-9) and anxiety (GAD-7). 

AAANY also offered substance use (Alcohol Use Disorder Identification Test–Concise [AUDITC], Drug Abuse Screening Test–10 [DAST-10]), PTSD (primary care PTSD screen, [PC-PTSD]), 

tobacco use, and suicide risk (Columbia-Suicide Severity Rating Scale [C-SSRS]) and violence 

risk assessment (multidimensional violence risk assessment ) screeners to a subset of clients. 

AAANY translated screening tools and adapted screening introduction and follow-up processes 

to fit the specific cultural preferences and focused needs of the Arab immigrant and Arab 

American community. 

Mental health first aid. All staff were trained to deliver MHFA to ensure their ability to 

respond to client needs. If a client showed signs of substantial anxiety or distress, program staff 

administered MHFA, implementing ALGEE (a mnemonic device for MHFA’s five-step action 

plan) to assess an individual’s immediate need and track changes over time. 

Motivational interviewing. All staff were trained to deliver MI when they encountered clients 

in need of services who demonstrated resistance to those services or other related changes. As 

noted above, MI was used as needed when offering an internal referral for screening and could 

also be delivered when staff interacted with clients for a scheduling appointment, during classes, 

and if a client walked in seeking any CBO service. 

Psychoeducation. All staff were trained to provide PE. Formal, group-based PE was provided 

through one of three groups: (1) Telling Our Stories: Immigrant Women’s Resilience; (2) Domestic 

Violence Support; or (3) Brooklyn at Young Women’s Group. PE provided information about 

available mental health resources as well as mindfulness and problem-solving skills. PE was 

also delivered via one-on-one conversation between staff and clients, where specific information 

was presented to educate clients about mental health issues identified as important or beneficial 

to the client or the community at large. 

271 
 

 

 

 

 

 

 

Referral pathway (warm handoff). Referral to NYU Langone occurred when clients needed 

services beyond what AAANY was able to provide. Typically, before offering a referral to NYU 

Langone, the CBO C2C program lead provided on-site, short-term treatment (10–12 sessions) to 

clients who scored positive on any of the screening tools. At the tenth session, the client retook 

the psychosocial assessment/universal screening tool. If the client’s score improved, AAANY 

scheduled the final session and then provided a referral list and helped develop an out-oftreatment action plan for the client. If the client’s score worsened or did not improve, AAANY 

offered a referral to NYU Langone, which was received by the NYU Langone Intake Director. 

AAANY referrals were given priority at NYU Langone, and appointments were made no later 

than 5 days after the referral. Clients also had the opportunity to choose from a list of Arabicspeaking psychiatrists within the community. 

Care coordination. Once a client was referred to NYU Langone, the CBO C2C program 

lead followed up to ensure that an appointment was scheduled and kept. In addition, quarterly 

interdisciplinary case reviews were held, involving AAANY and NYU Langone managers who 

participated in or oversaw screening, referral, and care coordination. This meeting included review 

of individual cases and aggregate data to improve screening, referral, and engagement in care. 

Table A.1. AAANY Training, Coaching, and Supervision

Who Number Who Who Receives

Received of Staff Provided Coaching and 

C2C Skill Training Trained Training Coaching and Supervision Plan Supervision

Screening All staff 27 MHP

MHFA All staff 27 MHP

MI All staff 27 MHP

Psychoeducation All staff 27 MHP

NYU Langone program lead All staff

provided coaching and supervision 

during quarterly departmental

meetings. In addition, the CBO C2C 

program lead provided one-on-one 

supervision throughout the grant

cycle.

NYU Langone program lead All staff

provided coaching and supervision 

during quarterly departmental

meetings.

NYU Langone program lead All staff

provided coaching and supervision 

during quarterly departmental

meetings. In addition, the CBO C2C 

program lead provided one-on-one 

supervision throughout the grant

cycle. CBO C2C program lead

received coaching and supervision 

on MI from NYU Langone program

lead during weekly meetings.

NYU Langone program lead All staff

provided coaching and supervision 

to all staff during one-on-one and 

group continuous coaching 

sessions.

272 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Bedford Stuyvesant Restoration Corporation

BSRC C2C Program Snapshot (as of Y4 Q3)

Mental health provider: The Family Center

Program type: Job training and employment

On-site mental health counseling available at CBO: Part-time

C2C Initiative target group(s): Unemployed or underemployed low-income working-age adults ages 18 and 

older receiving employment services

CBO’s target population(s): Young adults, families, seniors

Target geographical area/s: Brooklyn

Target CBO programs for C2C implementation: Economic Solutions Center

Number of CBO sites providing C2C services: Multiple sites

Total CBO clients served by C2C (Y1–Y4 Q3): 2,748

Number of current staff and supervisors trained in one or more C2C modalities: 32

Number of staff receiving continuous coaching and supervision (Y4 average): 23

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview

Bedford Stuyvesant Restoration Corporation (BSRC) is a nonprofit multiservice community 

development corporation headquartered in Brooklyn, New York, that promotes economic 

self-sufficiency, healthy and stable families, and art and culture. Services include workforce 

development, economic and business development, education, health services, and youth 

development. C2C was housed in BSRC’s Economic Solutions Center, which provides one-onone case management, benefits screening and enrollment, on-site food stamps and Medicaid 

enrollment, legal counseling, financial counseling, career development, educational coaching, 

and job placement and retention services to promote self-sufficiency and mobility for lowincome individuals (BSRC, undated). 

BSRC’s C2C services targeted individuals residing in two high-need Census tracts in 

northern Bedford Stuyvesant (as defined by the proportion of low-income households and high 

costs of development relative to median income), and other individuals served by the Economic 

Solutions Center. BSRC’s C2C program sought to address behavioral health issues that are 

barriers to economic stability—loss of hope, depression, anxiety, anger—and help clients learn 

important life skills such as staying calm in the workplace. C2C services were provided in two 

distinct BSRC locations: Restoration Plaza (main site) and Jobs-Plus (serving NYCHA residents 

in northern Bedford Stuyvesant). 

C2C Model Implementation

C2C staffing model. Initially, BSRC partnered with Brooklyn Community Services to serve 

as the C2C MHP and provide training and supervision in mental health skills (Table A.2). In 

year 3, BSRC partnered with a new MHP—The Family Center—to increase their capacity to 

273 
 

 

provide on-site mental health treatment services to BSRC clients. C2C implementation was led 

by the BSRC program lead who worked in close coordination with the on-site LCSW from The 

Family Center to plan and carry out coaching and training curriculum development. The program 

was also supported by a CBO lead care coordinator who is an LMSW responsible for providing 

on-site counseling, as well as assisting with oversight of CBO care coordinators, junior care 

coordinator, and MSW interns. The CBO lead care coordinator received supervision from the 

MHP LCSW. The program also included several counseling interns and registered nurses from 

local universities. Although the majority of staff were trained, a core team applied C2C skills 

with clients: customer service specialists (intakes/screenings), resource coordinators (screenings, 

other skills, initiate and follow-up on referrals), and career coaches, employment specialists, and 

financial counselor (MI, PE, and MHFA). The Family Center’s trainer was located on-site at 

both BSRC locations part-time to provide counseling to clients, training, coaching and supervision, 

and facilitate referrals. Although the MHP provided training in screening, MI, and PE to all 

BSRC staff, a subset received intensive training so they could serve as local experts, ultimately 

providing coaching, conducting peer-to-peer learning sessions, and informing BSRC C2C site 

coordinators of additional training and coaching needs. BSRC staff utilized city-sponsored 

MHFA trainings. The MHP LCSW and BSRC LMSW were primarily responsible for coaching 

and supervision. 

Screening. Screening was conducted individually, on paper during intake at Restoration 

Plaza and orientation at the Jobs Plus site. Screening results were later entered into the electronic 

database. Initially, staff offered the PHQ-4 to all clients and used results to determine which 

additional screening instruments should be offered. Clients who screened positive on the PHQ-4 

were then screened by the CBO lead care coordinator for depression (PHQ-9), anxiety (GAD-7), 

substance use (modified simple screening instrument for substance abuse [MSSI-SA]), and 

overall stress (Perceived Stress Scale and The Ardell Wellness Stress Test). Based on screenings, 

the CBO lead care coordinator determined the level of services needed (e.g., offer to participate 

in PE workshops, general on-site counseling, referral to MHP) for each client. Anxiety and stress 

management screenings were also incorporated into the PE workshops for individuals who had 

not yet completed these screenings. 

Mental health first aid. MHFA was applied as needed during interactions with clients on 

positive screens and in crisis situations. 

Motivational interviewing. Staff used MI to assist clients in considering lifestyle changes and 

to encourage clients to use mental health services provided on- and off-site. For example, MI was 

used by financial counselors and workforce staff during times when clients are in a transitional 

stage such as changing careers or becoming a homeowner, and so on. 

Psychoeducation. With the guidance of MHP staff, BSRC nonclinical staff implemented 

stress management workshops based on the Whole Health Action Management model once a 

week for 8 weeks. Workshops were open to all clients and addressed issues such as employment 

stress and family relationships—the workshops related mental health issues to BSRC’s services 

274 
 

 

 

 

 

 

 

 

 

 

 

 

 

(e.g., debt reduction, employment). BSRC also partnered with external groups such as NAMI 

and the National Council of Negro Women to provide these workshops to multiple populations, 

including young women, men, and seniors. Graduates became champions and lead additional 

workshops (e.g., stress management, senior shelter group, and job readiness group). 

Referral pathway (warm handoff). The CBO lead care coordinator determined whether 

clients needed short-term on-site counseling or referral to counseling off-site. Once the Family 

Center became the MHP, most referrals were made there, although clients were referred to other 

organizations based on need and preference. BSRC and the Family Center developed a “hot 

handoff” process—the CBO lead care coordinator was trained in the Family Center’s intake 

process and referred directly to the director of intake. In addition, the CBO lead care coordinator 

added appointments to the MHP LCSW’s calendar when she was scheduled to be on-site at 

either location. To help facilitate referral follow-through, BSRC covered client transportation 

costs to off-site care if the client was not able to afford the expense. BSRC staff were also 

responsible for following up with clients and the Family Center to verify whether the clients kept 

their appointments. 

Care coordination. Biquarterly interdisciplinary case conference meetings were led by the 

C2C site coordinators and held with BSRC and MHP staff for the purpose of reviewing client 

cases, data around screening results and service utilization, and service outcomes. 

Table A.2. BSRC Training, Coaching, and Supervision

C2C Skill

Who Received

Training

Number of Staff 

Trained

Who 

Provided 

Training

Coaching and 

Supervision 

Plan

Who Receives

Coaching and 

Supervision

Screening Subset 17 The Family

Center staff, 

MHP LCSW,

CBO lead

MHP/CBO Economic Solutions

Center staff applying 

modalities

MHFA All staff, 

members of

the community

16 DOHMH MHP/CBO Economic Solutions

Center staff applying 

modalities, Economic

Solutions Center C2C

site coordinators

MI All staff 21 BSRC lead

care 

coordinator—

LMSW is MI 

MHP/CBO Economic Solutions

Center staff applying 

modalities, ESC C2C

site coordinators

certified

PE Subset 20 CBO lead,

LMSW

MHP/CBO Economic Solutions

Center staff applying 

modalities

275 
 

 

 

 

 

 

 

 

 

 

 

 

Mental health provider: The Jewish Board of Family and Children’s Services

Program type: Other (homeless shelter)

On-site mental health counseling available at CBO: Part-time

C2C Initiative target group(s): Expectant parents and parents or caregivers of children ages 0 to 4

CBO’s target population(s): Homeless families referred by Department of Homeless Services or Human 

Exploitation Rescue Operative Child Rescue Corps

Target geographical area/s: Queens

Target CBO programs for C2C implementation: The Landing, a homeless family shelter

Number of CBO sites providing C2C services: One location

Total CBO clients served by C2C (Y1–Y4 Q3): 819

Number of current staff and supervisors trained in one or more C2C modalities: 44

Number of staff receiving continuous coaching and supervision (Y4 average): 15

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

 

 

 

 

 

 

 

CAMBA

CAMBA C2C Program Snapshot (as of Y4 Q3)

Program Overview

CAMBA is a nonprofit community-based social service agency, operational since 1977. 

CAMBA offers 160 programs in six service domains: Economic Development, Education and 

Youth Development, Family Support, Health, Housing, and Legal Services. The organization 

serves more than 45,000 individuals and families annually. Approximately 80 percent of clients 

are at or below the poverty level, and more than half are immigrants. CAMBA has three temporary 

family homeless shelters: Flagstone Family Center, The Kensington, and The Landing.3 

C2C was implemented at The Landing, a homeless family shelter in East Elmhurst, Queens, 

that serves 600 individuals annually. At The Landing, the staff provide ongoing case management, 

relocation to permanent housing, and crisis intervention (emergency food, clothing, transportation, 

and supportive services) for any family in need. The primary goal is to assist families to become 

stabilized through securing benefits, employment, searching for permanent housing, and securing 

a better quality of life when returning to the community. CAMBA recognized and implemented 

C2C in response to the need of delivering mental health service to homeless pregnant mothers 

and parents of children aged 0 to 4 who had experienced trauma. 

C2C Model Implementation

C2C staffing model. CAMBA partnered with The Jewish Board of Family and Children’s 

Services to implement an evidence-based Early Childhood Mental Health Consultation model 

at The Landing. Early Childhood Mental Health Consultation is an evidence-based, problemsolving and capacity-building model, implemented collaboratively between a professional 

consultant with mental health training and one or more individuals with other areas of expertise. 

3 For more information, please refer to CAMBA, “About Us,” webpage, undated. 

276 
The Jewish Board has successfully implemented an Early Childhood Mental Health Consultation 

model in other settings (schools, childcare centers). Using this model, The Jewish Board placed a 

licensed early childhood mental health partner on-site at The Landing part-time to train and 

support shelter staff in the delivery of C2C skills (Table A.3). Specifically, the mental health 

partner provided training, consultation, capacity building, and referrals/linkage to care. 

CAMBA’s C2C clinical coordinator (CC)—the main administrator of C2C implementation— 

worked closely with the MHP on planning and executing trainings and coaching and supervision. 

Service delivery evolved over time, with the shelter team, primarily case managers (primary 

client worker) and client care coordinators (social workers), taking on more responsibility for 

screening and service delivery and Jewish Board staff providing less direct intervention. Case 

managers and clinical care coordinators (CCCs) were supervised by clinical case supervisors. All 

CAMBA staff were trained in at least one of the C2C skills. The Jewish Board also provided 

training on topics beyond the four core skills including ambiguous loss, complex trauma, 

vicarious trauma, parent/caregiver-serving development, working with young and LGBTQ 

adults, and a Seeking Safety group therapy program for clients. 

Screening. CAMBA’s CC and CCCs screened clients within 30 days of establishing shelter 

eligibility. Clients who refused screening the first time were offered a screening 1 month later. 

The screenings included the following: PHQ9 for depression, DAST-10 and Alcohol Use 

Disorder Identification Test–10 (AUDIT-10) for substance and alcohol use, and GAD-7 for 

anxiety. The PC-PTSD and the Edinburgh Postnatal Depression Scale were also added later on. 

The MHP and CC met weekly to review the quality of screens. Based on how a client scored, the 

CBO CC either referred the client directly to The Jewish Board or requested a case conference 

with the MHP to determine the next steps. CAMBA considered the following to be positive 

scores that triggered a referral to the CBO CC: a score above 10 on the PHQ-9, a score above 10 

on the GAD-7, a score equal to 3 or higher on the DAST-10, and a score of 4 or above for men 

and 3 or above for women on the AUDIT-C. For individuals who scored above 20 on the 

PHQ-9 or answered positively on the question of suicidality, the MHP and CBO CC helped the 

individual’s case manager determine the level of risk, and whether emergency services needed to 

be called. 

Mental health first aid. Case managers applied MHFA during their required weekly or 

biweekly case management sessions with all clients at The Landing. Individuals who were not 

screened still received MHFA. 

Motivational interviewing. Case managers applied MI during their required weekly or 

biweekly case management sessions with all clients at The Landing, as relevant. The CBO CCCs 

and the CBO CC also used MI on an ongoing basis with clients. The site also planned to offer 

more continuous MI training to operations staff, particularly shift supervisors. 

Psychoeducation. Individual PE was delivered by the CBO CC when clients were referred 

postscreening. CAMBA offered “Baby and Me,” an attachment-focused, interactive parent-child 

group facilitated by the MHP and a trained staff member, and Seeking Safety, a treatment 

module promoting effective coping and emotional regulation skills for participants affected by 

277 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

trauma and substance use, facilitated by the CBO CC and a trained staff member. Clients whose 

case managers identified that they could benefit from either of these classes were invited to 

attend. PE topics included attachment and bonding, sleeping and disciplinary challenges, 

developmental milestones, infant brain development, talking to your child about being homeless, 

trauma, and so on. PE was offered to all clients regardless of whether or not they participated in 

C2C screenings. 

Referral pathway (warm handoff). Clients were primarily referred to Jewish Board’s Rego 

Park clinic during open access hours, which was reserved for new intakes. Depending on 

preference, clients could also be referred to another of The Jewish Board’s 16 licensed 

behavioral health clinic locations throughout the five boroughs and Long Island. All referrals 

were conducted by the CBO CC, CBO CCC, or MHP, who ensured timely appointment for the 

parent, helped coordinate transportation and consent to release information, and ensured that all 

parties were made aware of clients’ receipt of services. 

Care coordination. For those individuals who did not follow through with referrals, a case 

conference was scheduled at The Landing with the client, MHP, CBO CC, and case manager, 

who all used MI to encourage the client to follow through with services. The MHP met weekly 

with the program directors and supervisory staff and held case conferences with CAMBA case 

managers to monitor client progress and the need for referral to additional services. 

Table A.3. CAMBA Training, Coaching, and Supervision

Who Number Who Receives

C2C Skill

Received 

Training

of Staff 

Trained

Who Provided 

Training

Coaching and 

Supervision Plan

Coaching and 

Supervision

Screening Subset 6 CBP CC and 

MHP

Provided by MHP and CC—weekly 

group coaching led by MHP

All social 

services staff

MHFA All staff 34 MHP and 

CBO CC

CAMBA trainer

(LCSW)

Weekly group coaching led by MHP

(for social services staff)

Monthly group coaching led by CC 

(for shift supervisors)

All social 

services staff

Shift supervisors

MI Subset 10 MHP and 

CBO CC

CAMBA trainer

(LCSW)

Provided by MHP and CC, as well as

CAMBA trainer (LCSW) and additional

Jewish Board refreshers

Weekly group coaching led by MHP

(for social services staff)

Monthly group coaching led by CBO 

CC (for shift supervisors)

Discussion of case examples,

observation of MI interventions and 

All social 

services staff

Shift supervisors

coding by MHP, CBO CC, and social 

services supervisors

PE Subset 17 MHP and 

CBO CC

Provided by MHP, CBO CC, and 

other CAMBA and Jewish Board staff

All social 

services staff

Weekly group coaching led by MHP

(for social services staff)

Monthly group coaching led by CBO 

CC (for shift supervisors)

Shift supervisors

278 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Center for Employment Opportunities

Center for Employment Opportunities C2C Program Snapshot (as of Y4 Q3)

Mental health provider: Center for Alternative Sentencing and Employment Services

Program type: Job training and employment

On-site mental health counseling available at CBO: Part-time

C2C Initiative target group(s): Out-of-school or out-of-work young adults ages 18 to 4; unemployed or

underemployed low-income working-age adults ages 18 and older receiving employment services

CBO’s target population(s): Formerly incarcerated individuals

Target geographical area: Citywide

Target CBO programs for C2C implementation: All programs

Number of CBO sites providing C2C services: Multiple sites

Total CBO clients served by C2C (Y1–Y4 Q3): 5,644

Number of current staff and supervisors trained in one or more C2C modalities: 71

Number of staff receiving continuous coaching and supervision (Y4 average): 30

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview

The Center for Employment Opportunities (CEO) is a nonprofit organization headquartered 

in NYC with locations in 29 other cities across 11 states. C2C was implemented exclusively 

in CEO’s NYC locations. CEO provides immediate and comprehensive employment services. 

All CEO participants were recently released from incarceration and/or are involved in the 

criminal justice system and unemployed and without any income at the time of enrollment. 

Program participants are most often referred to CEO by a parole or probation officer, who has 

determined through a validated risk and needs assessment that the individual is a good match for 

CEO’s programs and services. CEO offers a range of services that aim to (1) improve client 

stability and self-sufficiency post-incarceration and (2) reduce recidivism. Program services 

include screening for employment barriers; short-term, paid transitional employment; job 

coaching; full-time job placement; and job retention skills training. All participants complete 

“Pathway 2 Employment” orientation class on enrollment. After this class, participants work 

with a job coach to prepare them for job placement; after placement, they are supported by a job 

retention specialist (RS).4 

CEO recognized that many of their clients experience varying levels of depression. Because 

untreated clinical depression may become a chronic condition that disrupts work, family, and 

personal life, CEO elected to participate in the C2C Initiative to increase access to mental health 

services for individuals with depression and ultimately improve their clients’ employment 

outcomes. C2C was situated in CEO’s Supportive Services department, whose focus is to 

4 For more information, please refer to Center for Employment Opportunities, homepage, undated. 

279 
 

 

connect clients with resources that will improve their overall physical and mental wellness. 

CEO’s Supportive Services team partners with external service providers to offer health 

screenings and education on-site and facilitate connections for other wellness services off-site. 

The department also works closely with government agencies and CBOs to offer Supplemental 

Nutrition Assistance Program and Medicaid enrollment along with tax preparation support at 

point of service. C2C fit into CEO’s Supportive Services as a foundational part of mental health 

support. The C2C project markedly increased the department’s capacity to address mental health 

needs among clientele. 

C2C Model Implementation

C2C staffing model. CEO partnered with the Center for Alternative Sentencing and 

Employment Services (CASES) to train their direct service staff to deliver services using C2C 

skills, including the supportive services manager (SSM), participant wellness specialist (PWS), 

job coaches, life skills educators, business account managers, RSs, and transitional worksite 

supervisors (TWS). The SSM oversees the Supportive Services department at CEO, building and 

maintaining connections with external wellness service provider partners and organizing on- and 

off-site referrals to those partners. Life skills educators facilitate an introductory 1-week course 

for all new CEO clients, whereas transitional worksite supervisors directly supervise participants 

in their daily work responsibilities on CEO-managed job sites around NYC. CASES and CEO 

have been partners for many years in NYC. Both are members of the city’s alternative to 

incarceration/reentry coalition and have collaborated on alternatives to incarceration and other 

initiatives prior to C2C. At the start of C2C, CASES embedded one LCSW at CEO’s program 

headquarters 2 days a week to provide clinical services. Over time, additional social worker 

interns have been assigned to CEO. On-site, short-term counseling with the MHP LCSWs was 

available to clients in person or over the phone. C2C services (both those offered by CBO staff 

and those provided by CASES) were made available to all CEO participants. In all, a lead MHP 

LCSW was on-site at CEO’s Manhattan HQ or Bronx office twice weekly, whereas an MHP 

social work intern worked once weekly (location worked was dependent on client need and MHP 

staff availability). MHP social work interns generally worked on the project for 4 to 6 months 

before a new intern would be assigned. All CEO program staff and managers were trained in at 

least one of the C2C modalities. CEO and CASES staff provided coaching and supervision to 

direct service staff trained in the C2C skills (Table A.4). 

Screening. After program intake, every CEO participant is assigned to a Life Skills Education 

class. During the Pathway 2 Employment class and as coordinated with the orientation instructors, 

the PHQ-9, a mental health screening tool for depression and anxiety, is administered by the PWS 

or SSM. The PWS or SSM utilizes a SBIRT designed script to introduce the tool to program 

participants, administer one-on-one or in a group setting, and collect the PHQ-9 for later scoring. 

The screening tools were then tabulated by one of the Supportive Services staff members and 

280 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table A.4. CEO Training, Coaching, and Supervision

Who 

Who Number Receives 

Received of Staff Who Provided Coaching and Coaching and 

C2C Skill Training Trained Training Supervision Plan Supervision

Screening Subset 4 MHP LCSW (initial);

CEO trainers

(ongoing)

MHFA All staff 61 MHP LCSW and 

NYC DOHMH

(initial); certified

CEO trainers

(ongoing)

MI Subset 71 MHP LCSW and 

MI Institute (initial); 

certified CEO

trainers (midproject); Institute for

Individual and

Organizational

Change (ongoing)

PE Subset 11 MHP LCSW (initial 

and ongoing)

Quarterly group coaching 

sessions for 1 hour; weekly,

biweekly, or as-needed 

individual reflective

supervision sessions

Quarterly group coaching 

sessions for 1 hour; weekly,

biweekly, or as-needed 

individual reflective

supervision sessions

Monthly coaching sessions;

weekly, biweekly, or asneeded individual reflective 

supervision sessions.

Quarterly MI recordings

submitted to and reviewed 

by the Institute for Individual

and Organizational Change

Quarterly group coaching 

sessions for 1 hour; weekly,

biweekly, or as-needed 

individual reflective

supervision sessions

PWS

Job coaches,

life skills

instructors, 

PWS

Job coaches,

PWS, SSM, 

life skills

education 

instructors

Job coaches,

PWS, life skills

education 

instructors

recorded in CEO’s custom client database. After review of the scores, the PWS compiled a list of 

participants that scored more than 5, indicating clinical intervention was needed and forwarded 

the names to the MHP LCSW while on-site and notified the MSS. The PWS then completed a 

warm handoff to the MHP LCSW to ensure that the participants were offered services on the 

days that MHP LCSW was on-site. Those who met with MHP LCSW were offered a 30-minute 

one-on-one meeting in which the PHQ-9 was reviewed, an additional assessment was completed, 

and a care plan developed. All participants were informed of the option for connection to mental 

health services on-site regardless of the outcome of their PHQ-9 score. 

Mental health first aid. MHFA strategies were integrated into daily service delivery to assist 

all direct service staff with responding effectively to participants who had mental health needs 

and to provide participants with important information about dealing with mental health concerns 

(e.g., how to obtain professional help, how to build resilience). Initially, CEO held quarterly 

MHFA trainings for program participants and their families. Starting in 2019, a specialized 

MHFA trainer with CASES offered monthly trainings for staff and participants alike, both 

on-site at CEO and at a CASES clinic. CEO extended the offering of MHFA to the community 

and served as a host location for MHFA by opening these trainings to the public. 

281 
 

 

 

Motivational interviewing. MI was also integrated into daily service delivery. Prior to C2C, 

CEO staff were already using MI with participants to help them develop their own reasons to 

engage in the program and to overcome barriers to employment. MI was used to build trusting 

relationships between participants and all CEO staff. For example, CEO’s job coaches used MI 

to help participants explore and resolve ambivalence around seeking and obtaining mental 

health services. In addition, the MHP LCSW used MI techniques during the one-on-one 

sessions with CEO participants who screened positive to discuss screening results, administer 

supplemental assessments, establish a care plan, provide psychosocial and support needs 

associated with assessment findings, and discuss options for ongoing clinical treatment, if 

warranted. 

Psychoeducation. PE was introduced to participants during Pathway 2 Employment classes 

through the curriculum Staying Well While You Work. Instructors were trained in the delivery 

of the PE curriculum and associated coping skills. Participants practiced a coping skill while in a 

group setting. PE was also integrated into daily service delivery by CEO’s job coaching staff. 

They delivered PE as needed during weekly one-on-one meetings with participants as part of the 

job start readiness process. This PE provision included a review of various evidenced-based 

coping skills that would apply to the issues that surfaced during the session, including the coping 

skills that were already practiced during orientation class. CEO worked with the MHP LCSW to 

identify common situations experienced by participants that would be ideal topics to address 

through PE and focused on trauma-based responses to the stressful life-based situations faced by 

program participants while reentering community and engaging in employment (e.g., identifying 

previous patterns of stress response and associated behaviors, discussing common barriers faced 

by those returning back to community post-incarceration, and identifying alternate ways to cope 

with stress). 

Referral pathways (warm handoff). The MHP LCSW was responsible for coordinating all 

referrals from CEO for mental health services, which included referrals to CASES Harlem-based 

Nathaniel Clinic and other CASES treatment providers. CASES also made referrals to five 

external providers: New York Psychotherapy and Counseling, Safe Horizon, Brightpoint Health, 

Institute of Family Health, and Hudson Guild. Because CEO participants lived in locations 

throughout the city, the MHP LCSW attempted to connect participants to clinics or MHPs in 

participants’ communities. CEO staff and the MHP LCSW prioritized establishing relationships 

with providers in the Bronx and Brooklyn, where a large proportion of CEO participants resided. 

Participants were also provided transportation support (e.g., MetroCards) to commute to 

treatment providers. 

While on-site, the MHP LCSW received referrals from CEO staff or identified participants 

who needed to be referred based on the screening results. When the MHP LSCW was off-site, 

CEO SSM met with any participants to provide emotional support and referral to MHP or 

external MHPs as indicated. When referred internally, clients met with the MHP LCSW for an 

282 
 

initial consultation and further mental health assessment. Clients could elect to continue to meet 

with the on-site MHP LCSW for up to 3 months for short-term therapy but could also be referred 

to CASES Nathaniel Clinic or other external treatment providers for long-term care. The CEO 

SSM would continue to triage any referrals for consultation made by staff in addition to the 

formal C2C referral process. CEO SSM attended monthly case conferences with CEO teams for 

discussion and follow-up related to participant mental health–related issues and would recommend 

and reinforce with the staff the path for reengagement with MHP LCSW or assist with other 

mental health resources as needed. 

Care coordination. During C2C implementation, leadership at CEO and CASES held monthly 

planning meetings and/or implementation telephone conferences. During each meeting, the 

project partners prepared for upcoming staff trainings, reviewed project performance indicators 

(number of screenings complete, number of referrals made, etc.), and discussed feedback from 

CASES and CEO managers on the status of the quality assurance monitoring and coaching of 

direct service staff following trainings. In addition, CBO and MHP staff convened joint care 

coordination meetings twice a quarter to discuss client engagement, appointment followthrough, retention in care, as well as client well-being (screening results, treatment adherence 

or other health-promoting behaviors, the effects of MHP and CBO services on symptoms or 

other indicators of well-being). In addition, the MHP LCSW participated in case conference 

meetings with treatment providers serving participants to coordinate care. 

283 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Hetrick-Martin Institute

HMI C2C Program Snapshot (as of Y4 Q3)

Mental health provider: HMI counseling department, on-site

Program type: Youth development

On-site mental health counseling available at CBO: Full-time

C2C initiative target group(s): Out-of-school, out-of-work young adults ages 16 to 24 and unemployed or

underemployed low-income working-age adults ages 18 and older receiving employment services

CBO’s target population(s): Low-income youth of color and LGBTQ youth, many who face homelessness and 

lack of family support

Target geographical area/s: Citywide

Target CBO programs for C2C implementation: All

Number of CBO sites providing C2C services: One site

Total CBO clients served by C2C (Y1–Y4 Q3): 4,033

Number of current staff and supervisors trained in one or more C2C modalities: 22

Number of staff receiving continuous coaching and supervision (Y4 average): 25

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview

The Hetrick-Martin Institute (HMI) is a nonprofit that provides free year-round programs and 

services to LGBTQ youth and allies aged 13 to 24. HMI offers a continuum of services designed 

to provide youth with the skills necessary for a productive transition into healthy adulthood. HMI 

functions as the Department of Health–designated Center of Expertise for Sexual Health and 

Gender Identity. HMI provides services to two groups of LGBTQ youth: (1) students at Harvey 

Milk High School and (2) individuals from the community. HMI partners with the Harvey Milk 

School, a NYC Department of Education school, to provide students with in-school college 

counseling and access to HMI programs and services.5 

HMI provides the following services: health and wellness, job readiness, arts and culture, 

academic enrichment, counseling and case management, on-site pantry, homeless outreach 

services, and referrals to additional services. HMI implemented C2C across multiple program 

areas to address critical issues affecting their client population, including depression, substance 

use and risky sexual behavior, verbal harassment, physical abuse, eating disorders, and selfharming behaviors. 

C2C Model Implementation

C2C staffing model. During the first year of C2C implementation, HMI attempted to partner 

with Mt. Sinai Hospital to provide mental health services. However, extensive administrative 

challenges prevented the partnership to succeed. Starting in year 2 of implementation, HMI’s 

youth development department partnered with HMI’s in-house counseling services department to 

5 For more information, please refer to Hetrick-Martin Institute, homepage, undated. 

284 
 

 

serve as the C2C MHP (Table A.5). The partnership aimed to bridge the gap between counselors 

and youth development workers within the organization. Mental health counselors trained youth 

development staff on C2C modalities, and youth development staff shared their expertise on 

outreach and client engagement. HMI staff had full access to HMI’s MHP clinical team: A 

counselor was on-call every day of operation and the director of clinical and counseling services 

was on-call via agency cell phone 24/7. Staff also had access to HMI’s psychiatrist, who was 

on-site 1 day per week for psychiatric evaluation and medication management services, and 

available throughout the week to consult with staff. 

Screening. All potential HMI clients are required to complete an intake process. As a part of 

C2C, mental health screenings, administered by an MHP licensed counselor or an MHP counseling 

intern, were included in the intake process. MHP counseling staff used Beck Anxiety and 

Depression Inventories, DAST-10 for substance use, and AUDIT for alcohol abuse, then added 

the PHQ-9, CRAFFT (Car, Relax, Alone, Forget, Friends, Trouble), of Multi-Attitude Suicide 

Tendency Scale, and Columbia suicidality scales. If a client declined screening at intake, a youth 

development worker sometimes screened the client at a later time. Even if a client did not screen 

positive, they were still eligible to utilize counseling services. Clients were rescreened at the 

termination of a group/internship cycle. 

Mental health first aid. MHFA was delivered primarily during crisis situations and as-needed 

to facilitate de-escalation and to engage clients in referral services. HMI extended MHFA training 

to peer educators, who are HMI clients employed by HMI to do outreach within and outside the 

organization. 

Motivational interviewing. MI was implemented by intake, counseling, and youth programs 

staff as part of individual check-ins and ongoing counseling and case management. MI was used 

to aid youth in thinking through difficult situations where they may be experiencing a great deal 

of ambivalence. In addition to using MI during individual counseling sessions, HMI programmatic 

staff also used MI during individual conversations and interactions with clients participating 

in activities such as college prep, job readiness, portfolio making, and health and wellness 

internships. 

Psychoeducation. MHP counseling staff implemented workshops on a range of topics, 

including signs and symptoms of mental illness, accessing services/benefits, health education for 

LGBTQ youth/young adult community, harm reduction/risk reduction, stress management, 

relationship issues, and trauma-informed care. In year 2, MHP counseling center also developed 

a curriculum-based, evidence-informed PE program that used art therapy skills to help HMI 

clients reduce stress. In year 3, MHP expanded on this program to create a self-care toolkit, an 

evidence-informed PE resource that can be used in group or individual sessions and includes 

strategies to help participants strengthen resilience and leadership skills. In November 2019, 

HMI also developed and piloted a PE support group that is curriculum driven and activity based, 

focused on coping with grief and loss. 

285 
 

 

 

 

 

 

 

 

 

 

MHP referral process (warm handoff). If a client screened positive, a referral to an on-site 

MHP counselor was made. MHP counselors were available 5 days a week and clients could drop 

in to see a counselor. If an MHP counselor was available and/or the client was in need of services 

immediately, the client could start counseling the same day. In addition, HMI employed several 

strategies to support client engagement with the MHP counseling center including providing 

opportunities for HMI clients to get to know counseling center clinicians, providing information 

during referral to demystify the counseling process, and quickly addressing scheduling conflicts 

to ensure clients were connected to care in a timely manner. During the course of C2C, MHP 

changed its counseling model from long-term care to short-term counseling, so clients were 

limited to 12 sessions and were referred to group counseling—if needed—as they transitioned 

out of care. 

Care coordination. MHP counselors and youth development staff participated in biweekly 

interdisciplinary care coordination meetings in which youth development staff set the agenda. 

These meetings served as a forum for discussing staff and client successes, assigning action 

items, and discussing adaptations to C2C programming to best meet the needs of HMI clients. 

Although hampered at times by staffing transitions and scheduling conflicts, implementation of 

routine supervision improved over time. HMI implemented additional group and individual 

supervision sessions to meet its quarterly coaching and supervision targets. 

Table A.5. HMI Training, Coaching, and Supervision

C2C Skill

Who 

Received 

Training

Number

of Staff 

Trained

Who Provided 

Training

Coaching and 

Supervision 

Plan

Who Received Coaching and 

Supervision

Screening Subset 12 MHP MHP program

lead and CC

Supervising counselor; CC; 

counseling staff; graduate 

interns

MHFA Subset 14 MHP; DOHMH MHP lead and 

CC

Education specialists from

health and wellness, services for

homeless youth, and job 

readiness tracks; teaching artists

from arts and culture 

programming; counseling 

graduate interns

MI Subset 13 MHP; also hired 

outside trainer

Three staff 

participating in 

supervision 

track through

MI Institute

Counselor; program coordinator;

coordination/education 

specialists from health and 

wellness program

PE Subset 17 MHP Program

Lead

MHP lead and 

CC

Education specialists from

health and wellness, services for

homeless youth, and job 

readiness tracks; teaching artists

from arts and culture 

programming; counseling 

graduate interns

286 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Hudson Guild

Hudson Guild C2C Program Snapshot (as of Y4 Q3)

Mental health provider: Paula B. Balser Counseling Center

Program type: Child and parent services

On-site mental health counseling available at CBO: None (note: MHP clinic location adjacent to main CBO

site)

C2C Initiative target group(s): Expectant parents and parents or caregivers of children ages 0 to 4

CBO’s target population(s): Low-income families, most of whom live in public and supportive housing in

Chelsea, Hell’s Kitchen, and Lincoln Square neighborhoods

Target geographical area/s: Manhattan

Target CBO programs for C2C implementation: Early Childhood Education program

Number of CBO sites providing C2C services: Five sites

Total CBO clients served by C2C (Y1–Y4 Q3): 590

Number of current staff and supervisors trained in one or more C2C modalities: 56

Number of staff receiving continuous coaching and supervision (Y4 average): 7

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview

Hudson Guild is a nonprofit, multiservice community agency serving low-income families 

who live, work, or go to school in Chelsea, Hell’s Kitchen, and Lincoln Square with programs 

located throughout these Manhattan neighborhoods.6 Programs include early childhood education, 

youth development and education, adult services, art, and community building. Through its 

Paula B. Balser Counseling Center and two school-based clinics, Hudson Guild also offers 

mental health services. 

Hudson Guild recognized that needs such as anxiety, posttraumatic stress, depression, 

substance use, domestic violence, and parent/child conflict may affect the families served by 

their programs and implemented C2C to address these needs. C2C was implemented in the Early 

Childhood Education (ECE) program, which offers Head Start, Pre-K for all, and EarlyLearn NYC 

Child Care. Through C2C, Hudson Guild aimed to decrease the number of crisis interventions 

among ECE families, increase awareness of and access to mental health services, destigmatize 

mental illness, and help to improve the overall well-being of families. 

C2C Model Implementation

C2C staffing model. Hudson Guild partnered with the Paula B. Balser Counseling Center, 

who served as the MHP. The Balser Counseling Center provides the following services: outreach, 

initial assessment (including health screening), psychiatric assessment, crisis intervention, 

psychotropic medication treatment, psychotherapy services, family/collateral psychotherapy, 

group psychotherapy, and complex care management. 

6 For more information, please refer to Hudson Guild, homepage, undated. 

287 
 

 

 

 

 

 

 

 

The Balser Center hired a C2C coordinator (LCSW) to train ECE staff members. Hudson 

Guild family workers were the primary staff members delivering C2C skills. Family workers 

used C2C skills to develop individualized client plans and to facilitate clients’ problem-solving 

by educating and counseling clients where appropriate. The MHP C2C coordinator (LCSW) 

facilitated implementation by serving as a bridge between Hudson Guild ECE programs and the 

Balser Center, which despite close proximity and being housed under the same organization, 

rarely interacted before C2C. 

All family services workers and a family services supervisor received training on PE and 

screening from the MHP C2C coordinator and MHFA training from the NYC DOHMH 

(Table A.6). Both the MHP C2C coordinator and an outside contractor provided MI training to 

family workers. In addition, booster trainings were offered regularly in each C2C skill. Coaching 

and supervision occurred weekly, in a group setting, for 1 hour, and staff completed supervision 

checklists and self-assessments (forms created in-house that assess knowledge, skills, and 

actions) biannually. Coaching and supervision also included experiential learning and direct 

practice observation. CBO supervisors (the director of Early Childhood Education and assistant 

director of Early Childhood Education) monitored the quality of coaching and supervision 

through weekly meetings with MHP C2C coordinator and observations of staff. 

Screening. Initial screenings for depression (PHQ-9), anxiety (GAD-7), substance use 

(MSSI-SA), and PTSD (PCL-5) were included as a regular part of the intake process for clients. 

Family workers aimed to conduct screening with clients within 2 months of enrolling their child 

in the ECE program. Clients could also receive a screening at any time through self-referral or 

Table A.6. Hudson Guild Training, Coaching, and Supervision

Who Number Who Receives

C2C Skill

Received 

Training

of Staff 

Trained

Who Provided 

Training

Coaching and 

Supervision Plan

Coaching and 

Supervisions

Screening Subset 8 MHP C2C coordinator MHP C2C coordinator

provides coaching and 

supervision weekly, in 

a group setting, for

1 hour, as a standalone activity (C2C

supervision group)

Family workers, 

family services 

supervisor

MHFA All ECE staff 56 MHP C2C coordinator Same as above Same as above

Four staff sent to NYC

DOHMH training

MI Subset 10 MHP C2C coordinator Same as above Same as above

All ECE staff 

received 

(initial); outside 

contractor (ongoing)

introductory

MI training

PE Subset 8 MHP C2C coordinator Same as above Same as above

288 
 

through a referral from Hudson Guild staff (e.g., reception staff) who noticed changes in clients’ 

mood or demeanor. 

Mental health first aid. Family workers, teachers, and reception staff were trained in MHFA 

to facilitate recognition and detection of mental health symptoms and provide appropriate 

assistance and resources. Direct service staff integrated MHFA during interactions with clients 

who exhibited signs of mental health symptoms or crises. 

Motivational interviewing. MI was used in conjunction with screening. For clients who 

screened positive, family workers engaged clients using MI techniques to discuss concerns and 

habits the clients wanted to change, evoking clients to change behavior and developing practical 

steps to implement change. 

Psychoeducation. All clients had the opportunity to attend PE workshops. Family workers 

and the MHP C2C coordinator cofacilitated monthly workshops on a range of topics including 

self-care, positive parenting strategies, and anger management. Individual PE was also provided 

by direct service staff, as needed, in conjunction with screenings. 

Referral pathways (warm handoff). If a client received a positive screening or requested a 

referral, a family worker referred her/him to the Balser Center. If the client accepted the referral, 

the family worker completed the required paperwork and collected insurance information from 

the client. MHP referral intakes took place over the phone, but clients could also walk in to the 

Balser Center, which offered open access hours for intake completion. The Balser Center’s 

intake clinician received all referrals and verified insurance status. All Medicaid eligible clients 

were provided with metro cards to cover transportation costs to and from the clinic. 

MHP clinicians notified family workers when a client made the first intake appointment so 

family workers could remind the client to attend. MHP clinicians also alerted family workers 

when a client was not attending sessions so family workers could help with outreach and 

encourage the client to make a new appointment and/or troubleshoot barriers to appointment 

completion. 

Care coordination. Once an MHP therapist was assigned to a client, family workers and the 

C2C coordinator were in ongoing communication with the MHP therapist and hosted quarterly 

case conference meetings in which they discussed care coordination and caregiver engagement 

issues. 

289 
 

 

 

 

 

 

 

 

 

 

 

 

 

Northern Manhattan Improvement Corporation

NMIC C2C Program Snapshot (as of Y4 Q3)

Mental health provider: The Dean Hope Center for Educational and Psychological Services at Teacher’s

College, Columbia University

Program type: Job training and employment

On-site mental health counseling available at CBO: Full-time

C2C Initiative target group(s): Out-of-school, out-of-work young adults 16 to 24

CBO’s target population(s): Disconnected youth; un-/underemployed adults

Target geographical area/s: Washington Heights/Inwood/South Bronx

Target CBO programs for C2C implementation: Education & Career Services department; Legal, Organizing,

and Advocacy department; domestic violence program

Number of CBO sites providing C2C services: One site

Total CBO clients served by C2C (Y1–Y4 Q3): 3,292

Number of current staff and supervisors trained in one or more C2C modalities: 41

Number of staff receiving continuous coaching and supervision (Y4 average): 30

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview

Northern Manhattan Improvement Corporation (NMIC) is a community-based settlement 

house founded in 1979. It has evolved into a leading multiservice agency with a staff of more 

than 150 persons, serving all of NYC. Their mission is “to serve as a catalyst for positive change 

in the lives of the people” in the community.7 Their legal, organizing, and advocacy services 

include immigration legal services, housing court representation/eviction prevention, and 

counseling for immigrant communities. Their education and career services programs provide 

the community with the additional tools necessary to build secure and prosperous futures. They 

also offer weatherization services to improve the housing stock in Upper Manhattan. 

NMIC implemented C2C to build a sustainable bilingual Spanish-English mental health case 

management service program to address challenges they noted in their client population including 

low self-esteem, low self-efficacy, strained family and social relationships, and mental health 

issues (e.g., depression, anxiety, stress, reactivity/impulse control, and substance and/or alcohol 

abuse). C2C was implemented in the Education & Career Services and then subsequently into 

the Legal, Organizing, and Advocacy (LOA) department, as well as their domestic violence 

program. 

C2C Model Implementation

C2C staffing model. NMIC partnered with the Dean Hope Center for Educational and 

Psychological Services at Teacher’s College (TC), Columbia University. NMIC hired a C2C 

7 NMIC, “NMIC Overview,” webpage, undated. 

290 
 

 

 

 

 

 

counseling coordinator to provide substantive support to NMIC staff and TC graduate student 

counselors, while overseeing program operations and handling a direct service caseload. Five 

bilingual master-level counseling graduate students from TC were on-site 9:00 a.m. to 5:00 p.m. 

Monday through Friday to provide services to clients when needed. Licensed clinical staff at 

TC trained NMIC staff. Beyond the four core C2C skills, the NMIC staff were trained on HIPAA 

issues, cultural competency, crisis management, and de-escalation. Because NMIC serves many 

Spanish-speaking clients, staff received separate training on how to deliver the four skills in 

Spanish (Table A.7). 

Screening. Clients were offered screening by their assigned case managers who were already 

building trusting relationships and were trained to use strategic, strength-based messaging while 

providing responsive and respectful services. Although all clients of the NMIC Education and 

Career Services Department were deemed eligible for screening, NMIC focused on offering 

screening to clients engaged in cohort-based programs on-site to maximize the potential for 

follow-up. NMIC’s LOA clients were also offered screening and referral if their paralegal 

assessed a mental health need. In addition, staff in the domestic violence program (i.e., 

counselor/advocates and LMSW) offer their clients screening and referrals to the C2C program. 

Case managers conducted screenings for clients in on-site, cohort-based programs including 

Youth Workforce Adult Education and Adult Workforce/On-site career training. Case managers 

screened clients for depression (PHQ-9), anxiety (GAD-7), and substance use/abuse (CRAFFT 

for clients younger than 21 and CAGE-AID for clients 21+). Screening occurred after the clients 

completed programmatic intake and orientation. Sometimes, clients completed self-administered 

screens in a group setting if the cohort was in an off-site location or a once-a-week class, which 

typically made one-on-one screening difficult for administration. 

The determination of which clients to screen in the LOA department was made on a 

case-by-case basis by the Director of LOA Services and the CBO C2C counseling coordinator. 

Screenings for clients in the legal department were conducted by the NMIC paralegal. 

In the NMIC model, client needs were triaged into one of three tiers based on screening 

scores. And then the four core C2C skills were delivered accordingly: 

Tier I: basic services. If clients screened positive for mild anxiety or minimal depression, 

NMIC case managers provided C2C skills directly to clients. Delivery of MHFA, MI, and PE 

was incorporated into the regular meetings NMIC case managers had with clients to track 

progress in their relevant program. If a client wanted to receive counseling or if case managers 

thought a client could benefit from seeing a counselor, case managers referred clients to Tier II 

or III services without a positive screening. LOA clients did not receive Tier I services because 

those who were identified as needing mental health services were referred to Tier II or III based 

on their screening results. 

Tier II: on-site graduate student counseling. Tier II was indicated for a screening result on 

the PHQ-9 of 10 or above, GAD7 of 10 or above, and positive response of 3 and above on the 

CRAFFT/CAGE-AID. The case managers and MHP C2C counseling intern coordinated for 

291 
 

on-site warm handoff referrals. This in-house counseling service was intended to be short term 

(6–8 sessions). 

Tier III: client in need of higher level of services. If a client’s screening results indicated 

severe need or if it was clear that the client’s mental health needs were not being met by NMIC 

staff or the on-site TC students, the client was referred to the CBO C2C counseling coordinator. 

The CBO C2C counseling coordinator assessed the client’s need for a higher level of care and 

offered a referral to the appropriate referral agency. Higher level of need meant suicide or 

homicide ideation or psychosis. Often the need for a psychiatric evaluation or medication 

management would indicate a need for a referral to an agency with a psychiatrist. Clients were 

assessed by utilizing a risk assessment instrument, C-SSRS instrument, and the rescreening 

results. Referrals were completed in a collaborative process that considered the client’s 

individualized needs. The MHP was included in individual supervision and case 

conceptualization and review. 

Mental health first aid. Depending on the client’s screening results, MHFA was delivered 

by NMIC case managers (Tier I), TC graduate students (Tier II), or the CBO C2C counseling 

coordinator (Tier III). 

Motivational interviewing. MI was completely integrated into service delivery for all clients. 

All NMIC staff utilize these skills in implementation of PE sessions, individual case management 

sessions, and counseling sessions. MI also served as the framework for how staff members 

interact with each other at NMIC. 

Psychoeducation. In addition to any individual PE provided after screening, TC graduate 

students conducted PE sessions in all Education and Career Services classrooms. The purpose of 

these PE sessions was twofold: (1) to reduce stigma and (2) to introduce C2C services to clients. 

These PE sessions supported the increase of utilization of C2C counseling services. 

Referral pathway (warm handoff). A warm handoff was completed internally at NMIC by 

the case manager who completed the C2C Wellness Survey (screening instrument) and the 

assigned MHP C2C counseling intern. The assigned intern reached out to the client to schedule 

an appointment for Tier II. If the client was referred to Tier III, the intern scheduled an 

appointment with an external provider during the session with the client. An additional session 

with the client was scheduled to check the status of the first meeting with the external provider. 

The warm handoff helped to increase the utilization of counseling services by supporting the 

client to engage in counseling services by utilization of the role of the case manager. In addition, 

the warm handoff process was explained to the client to ensure complete understanding before 

the client transitioned to another tier. 

Care coordination. The CBO C2C counseling coordinator coordinated monthly meetings 

between NMIC and TC to review screenings, referrals, and care coordination data. The group 

reviewed the number of screeners completed, data indicating engagement and utilization of C2C 

services, next steps toward improvement of C2C services provided, and how to provide more 

support/trainings to case managers utilizing C2C skills. 

292 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table A.7. NMIC Training, Coaching, and Supervision

Number of Who Receives

Who Received Staff Who Provided Coaching and Coaching and 

C2C Skill Training Trained Training Supervision Plan Supervision

Screening Subset 32 CBO C2C 

counseling 

coordinator and 

MHP consultant

CBO C2C 

counseling 

coordinator

started in year 2

MHP consultant

started in year 3

MHFA Subset 30 DOHMH

MI Subset 35 NMIC has three 

staff trained as

trainers (each 

completed MI

Institute and

trained as 

trainers)

PE Subset 20 CBO C2C 

counseling 

coordinator and 

MHP consultant

Provided by CBO

C2C counseling 

coordinator and MHP

consultant.

Weekly to staff

members who 

delivered C2C skills.

Monthly “advanced 

coaching” required for

all Education &

Career Services staff, 

optional for legal

services staff.

Ad hoc availability of

CBO C2C counseling 

coordinator.

MHP clinician

monitors the quality of

coaching and 

supervision.

Direct coaching and 

feedback and

reflective supervision 

at least twice per

quarter.

Twice per year, staff 

will assess their own 

knowledge, skills,

beliefs, and practice.

Same as above

Same as above

Same as above

All members in

Education &

Career Services

who are delivering

a C2C skill

Same as above

Same as above

Same as above

293 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Red Hook Initiative

Red Hook Initiative C2C Program Snapshot (as of Y4 Q3)

Mental health provider: NYU Langone Health (formerly NYU Lutheran Family Health Centers)

Program type: Youth development

On-site mental health counseling available at CBO: Full-time

C2C Initiative target group(s): Out-of-school, out-of-work young adults ages 16 to 24

CBO’s target population(s): Young people; Brooklyn native; public housing assistance

Target geographical area/s: Red Hook

Target CBO programs for C2C implementation: All programs

Number of CBO sites providing C2C services: Multiple sites

Total CBO clients served by C2C (Y1–Y4 Q3): 710

Number of current staff and supervisors trained in one or more C2C modalities: 39

Number of staff receiving continuous coaching and supervision (Y4 average): 7

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview

Red Hook Initiative (RHI) is a nonprofit organization focused on youth development and 

community-building. RHI’s approach is structured around a youth empowerment pipeline, which 

provides young people intensive academic support, job readiness and leadership training, parttime entry-level employment, college application assistance, college retention support, and a 

range of services that promote resiliency and prepare youth for independent, self-sufficient 

adulthood. The organization primarily serves young people, from middle school through age 24, 

who live in Red Hook, Brooklyn, in the NYC Housing Authority Red Hook Houses— 

Brooklyn’s largest public housing development. RHI has three youth development programs 

(Middle School Program, High School Youth Leader Program, and Young Adult Program) and a 

fourth program for residents 25 and older (Community Building) focused on building a healthy 

and sustainable Red Hook community.8 

RHI implemented C2C to provide access to mental health care that will ultimately improve 

the organization’s ability to support youth toward high school graduation, acceptance into a college 

or workforce training program, graduation from college, and securing a job with opportunities 

for advancement. Particular mental health issues of concern for the population include depression, 

anxiety, suicidal ideation, social isolation and hopelessness, substance use, and trauma. 

C2C Model Implementation

C2C staffing model. RHI’s C2C coordinator was in charge of training and evaluation, and RHI 

also utilized staff with a master’s degree in social work to provide mental health counseling to their 

clients (Table A.8). The following RHI staff were trained in C2C skills: director of professional 

development, senior director of programs, young adult program manager, high school program 

8 For more information, please refer to Red Hook Initiative, homepage, undated. 

294 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table A.8. RHI Training, Coaching, and Supervision

C2C Skill

Who 

Received 

Training

Number of

Staff Trained

Who Provided 

Training

Coaching and 

Supervision Plan

Who Receives 

Coaching and 

Supervision

Screening Subset Social work

staff (6), 

education 

support staff 

(1), case 

manager (1)

Initial trainer:

MHP clinical 

staff

Ongoing 

trainers: RHI

director of 

professional

development

Provided by MHP clinical staff, 

young adult program manager,

high school program manager,

young adult social worker

Social workers receive weekly

individual coaching

Education staff and case 

manager receive weekly

supervision

Social work staff, 

education and 

case management

staff

Social workers receive monthly

group clinical supervision

Reflective supervision included 

in individual and group 

supervision sessions

Compliance completed by NYU

Langone clinical staff annually

Self-assessments completed 

twice annually

MHFA All staff 28 FT

23 PT

Initial trainer: 

MHP clinical 

staff

Ongoing 

trainers:

ThriveNYC

Learning 

Center

Provided by MHP clinical staff, 

RHI senior director of programs,

director of community building,

young adult program manager,

high school program manager,

director of professional

development

Social workers receive weekly

individual coaching

Clinical staff receive weekly

supervision

Social workers receive monthly

group clinical supervision

Young adult social 

worker, C2C social 

work contractor,

high school social 

worker, education 

coordinator,

employment

coordinator, college 

RS, case manager,

high school group 

leaders, community

organizers

Reflective supervision included 

in individual and group 

supervision sessions

Self-assessments completed 

twice annually

MI Subset 21 total—

15 FT

6 PT

Initial trainer: 

MHP clinical 

staff and MI

Institute

Ongoing 

trainer: RHI 

director of 

professional

development/

MI network of 

trainers

Social workers receive monthly

group clinical supervision

Reflective supervision included 

in individual and group 

supervision sessions

Compliance completed by MHP

clinical staff annually

Self-assessments completed 

twice annually

MI refresher and series of six 

MI practice labs are held 

annually

Young adult social 

worker, C2C social 

work contractor,

high school social 

worker, education 

coordinator,

employment

coordinator, college 

RS, case manager,

high school group 

leaders, community

organizers

295 
 

 

 

 

 

 

C2C Skill

Who 

Received 

Training

Number of

Staff Trained

Who Provided 

Training

Coaching and 

Supervision Plan

Who Receives 

Coaching and 

Supervision

PE Subset Social work

staff (6)

MHP clinical 

staff

Provided by MHP clinical staff, 

RHI young adult program

manager, high school program

manager, reproductive health 

educator

Social workers receive weekly

individual coaching

Young adult social 

worker, high 

school social 

worker, C2C social 

work contractor,

reproductive 

health educators

One-on-one case management

weekly supervision

Social workers receive monthly

group clinical supervision

Reflective supervision included 

in individual and group 

supervision sessions

Self-assessments completed 

twice annually

NOTES: FT = full-time; PT = part-time.

manager, college retention specialist, reproductive health manager, employment coordinator, 

programs associate, youth jobs developer, academic advisers, education coordinator, program 

associate, reception staff, technology instructor, media and design associate, referral specialist, 

tech program assistant, and social workers. RHI partnered with NYU Langone Health, formerly 

NYU Lutheran Family Health Centers, an Article 31 clinic, to provide mental health services, 

training, and supervision. NYU Langone clinical staff (senior psychologist) was on-site at least 

once a week and responsible for clinical supervision and training. In addition to training in the 

four skills, training in therapeutic crisis intervention was delivered to staff. Therapeutic crisis 

intervention is an evidence-informed model of crisis prevention and de-escalation used 

internationally in youth-based organizations. It utilizes best practices in social work, youth 

development, and trauma-informed practice perspectives (including the sanctuary model of 

trauma-treatment). 

Screening. Clients were screened by RHI social workers, case managers, and college RSs on 

a rolling basis and during youth leader and young adult enrollment/orientation periods. Clients 

completed a counselor administered in-person Unified Behavioral Health Screening Tool, 

assessing for depression (PHQ-9), trauma (race-based traumatic stress scale and University of 

California, Los Angeles (UCLA) trauma screen, PC-PTSD), anxiety (GAD7, Young Beck 

Anxiety Inventory), and drug (DAST-10) and (AUDIT-C) alcohol use. The CBO social workers 

received in-depth training in screening tool utilization, whereas nonclinical CBO staff learned a 

“screening-lite” version aligned with their roles and level of experience with mental health. 

Enrolled clients had four entry points to C2C services: (1) Clients could mention to a CBO 

social worker directly if services were of interest due to stressors, (2) a program facilitator or 

staff member could notice signs and symptoms of mental health issues and refer the client to the 

CBO social worker at RHI, (3) during client “check-ins” with a CBO social worker or case 

manager, which happens twice annually for high school students and at the point of entry for 

296 
 

 

young adults, and (4) during a peer or CBO social worker–led PE workshop. At each entry point, 

CBO social workers offered screening to the young person. If he/she is not interested in receiving 

services, the CBO social worker followed up within 2 weeks. If the client still refused, the CBO 

social worker let him/her know that there was an open-door policy and encouraged the client to 

return whenever needed. 

After screening, clients were triaged for C2C services according to the following three tiers: 

Tier 1: Individuals received services provided by RHI trained staff using the three skills (MI, 

MHFA, and PE). These services include academic support and advocacy, reproductive health 

services, and case management support. 

Tier 2: Individuals in need of counseling services were provided on-site clinical counseling 

by RHI clinical social work staff, supervised by NYU Langone. CBO clinical social work staff 

provided trauma-informed therapy, psychodynamic therapy, and short-term solution-focused 

therapy to meet client needs. 

Tier 3: Individuals who required more intensive mental health services, including psychiatric 

assessment and crisis services, were referred to NYU Langone Sunset Terrace and NYU Langone 

Psychiatric Hospital. The NYU Langone intake coordinator facilitated completion of a screening 

assessment, full biopsychosocial assessment, and a psychiatric evaluation. NYU Langone 

clinical staff provided different treatment approaches (beyond C2C skills) such as cognitive 

behavioral approaches, psychodynamic techniques, and so on. 

Mental health first aid. Implementation of MHFA was diffuse throughout the organization. 

Staff used MHFA when providing intake and case management, educational counseling, 

reproductive health counseling, discussing screening results with clients, and when referring 

clients to RHI social workers or NYU Langone. 

Motivational interviewing. Implementation of MI was diffuse throughout the organization. 

RHI staff used MI skills to assess and improve their relational work with clients. This focused on 

highlighting client strengths, promoting self-determination, and helping clients make progress 

toward their self-identified goals. Staff also used MI skills to address participant behavior during 

programs, identify new strategies to proactively address triggers, and progress toward change 

behavior. CBO social work staff used MI when discussing screening results, and the CBO social 

workers utilized the skills to encourage clients to attend the first scheduled counseling session 

with on-site RHI clinical social workers and with NYU Langone. 

Psychoeducation. Initially, PE occurred in three ways: (1) peer counselor–led workshops for 

youth/young adult clients such as mental health awareness panels and community health fair 

workshops, (2) individually with a CBO social worker when a youth/young adult client screened 

positive, and through (3) mental health awareness “campaigns” of information posted throughout 

the organization. In April 2018, RHI adopted the Seeking Safety curriculum, an evidence-based 

approach for PE. The curriculum was adapted to be age appropriate for high school participants 

and to address maladaptive coping exhibited by those with positive screenings for drug and 

alcohol use. The adapted curriculum included PE on the symptoms of PTSD, mindfulness 

grounding exercises, and identifying triggers. As a result, participants reported having more 

297 
 

 

 

tools for grounding and centering when experiencing anxiety and stress and requested additional 

workshops. 

Referral pathway (warm handoff). Immediately following a positive screen or if a need was 

indicated by a client, the RHI social worker provided (1) an appointment for in-house CBO 

social workers to provide clinical counseling if the symptoms were mild to moderate, (2) a 

referral to NYU Langone if the symptoms were severe or if psychiatric services were needed, 

and (3) an accompaniment to crisis services if a client was at risk for self-harm or experiencing 

psychiatric crisis. If the staff member conducting screening was not a social worker, the staff 

member referred the client to the CBO social worker for referral support. If a client declined, 

the RHI staff member did biweekly check-ins to assess needs and reoffer support. If an off-site 

referral was needed, the intake coordinator at Lutheran scheduled the client for a clinician 

session or psychiatric evaluation appointment, ideally within 5 days of the referral. Insurance 

was verified at RHI. If the client did not have insurance, CBO social work staff worked to secure 

insurance for him/her. If that was not possible, the RHI social worker worked with NYU Langone 

to obtain a sliding-scale quote and then develop a plan with the client to make the payments. RHI 

provided emergency assistance to clients unable to pay to ensure continuity of service. The RHI 

social worker escorted the client to the first session and provided metro cards and refreshments. 

The CBO social worker referring then became the lead contact, liaising with NYU Langone and 

checking with the intake coordinator about attendance. If clients missed any appointments, the 

referring CBO social worker explored the reasons and provided additional support. In some 

cases, clients were referred to local MHPs that were more geographically accessible. 

The CBO social worker tracked information on what happened with the referral. RHI observed 

very low completion rates for referrals to NYU Langone and determined that this was due to 

several barriers, such as lack of transportation, lack of insurance, unfamiliarity with the workings 

of the medical clinic setting, and increased stigma on engaging in psychiatric services. In the 

summer of 2018, RHI proposed a shared patient model pilot with NYU Langone Brooklyn that 

allowed clients in need of higher-level clinical care to be referred to NYU Langone for psychiatric 

care only (e.g., psychiatric evaluation), whereas the RHI clinical social worker continued to 

provide individual and group counseling on-site. Though the initial pilot was small, it allowed 

strong relationship building with a client and the psychiatric team during the transition period of 

shared care. In January 2019, NYU Langone and RHI agreed to continue the shared patient 

model long-term. 

Care coordination. CBO and MHP staff met at least once quarterly to review aggregate data 

about screening, referral, and engagement rates and strategize around process improvements. 

These meetings typically included RHI’s C2C coordinator and social work staff, and NYU 

Langone’s senior psychologist, intake coordinator, and senior medical director; however, 

additional clinical and administrative staff at NYU Langone often participated in these meetings. 

In addition, RHI and NYU Langone addressed process improvements and individual-level care 

coordination in interdisciplinary team meetings that include NYU Langone, RHI social workers, 

and RHI nonclinical staff. 

298 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Safe Horizon

Safe Horizon C2C Program Snapshot (as of Y4 Q3)

Mental health provider: Safe Horizon Counseling Center

Program type: Other (domestic violence organization)

On-site mental health counseling available at CBO: Part-time

C2C Initiative target group(s): Expectant parents and parents or caregivers of children ages 0 to 4

CBO’s target population(s): Unemployed/underemployed women with young children

Target geographical area/s: Citywide

Target CBO programs for C2C implementation: Domestic Violence Shelter Program

Number of CBO sites providing C2C services: Multiple sites

Total CBO clients served by C2C (Y1–Y4 Q3): 2,059

Number of current staff and supervisors trained in one or more C2C modalities: 113

Number of staff receiving continuous coaching and supervision (Y4 average): 75

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview

Safe Horizon is a nonprofit that operates a domestic violence shelter program to provide 

safety and support for thousands of adults and children each year in all five boroughs. In addition 

to domestic violence shelters, Safe Horizon programs include a 24-hour domestic violence 

hotline, food and clothing assistance, legal and court programs, child advocacy, childcare and 

safety assistance, and financial and housing assistance. 

Many Safe Horizon clients have experienced harms like domestic violence, sexual assault, 

child abuse, stalking, and/or human trafficking. Safe Horizon implemented C2C at its eight 

domestic violence shelters to support residents who might need mental health services as a result 

of surviving these and other extremely difficult life events.9 

C2C Model Implementation

C2C staffing model. Safe Horizon’s domestic violence shelter program partnered with Safe 

Horizon Counseling Center (SHCC), the only NYS licensed mental health clinic that focuses 

solely on trauma-focused treatment for survivors of crime and abuse of all ages. The CBO C2C 

project coordinator managed C2C implementation. SHCC staff facilitated trainings and provided 

coaching and supervision to DVSP supervisors in the delivery of all the core C2C skills except 

MI (Table A.9). The program hired an external consultant to provide MI training and ongoing 

coaching to a subset of staff—directors, social workers, case managers, and housing specialists. 

All clinical services were offered out of the SHCC program locations in Brooklyn or Harlem. 

Screening. All DSVP residents were screened for C2C services using a three-step screening 

process. First, as part of acceptance into the shelter, all new residents met with a social worker or 

9 For more information, please refer to Safe Horizon, homepage, undated. 

299 
 

case manager to conduct a required face-to-face intake. As part of this intake, residents were 

given an initial brief screening to check for the presence of mental health symptoms. Second, 

within the first week of residence in the shelter, all residents met with a CBO social worker and 

completed a clinical assessment that included screening for depression (PHQ-9), substance use 

(MSSI-SA), and PTSD (PCL-C). Initially, the site planned for the final step to occur 4 to 6 weeks 

into a resident’s shelter stay and consist of residents completing the same screening tools again 

to determine if there were any continuing mental health issues. However, this became too 

cumbersome. Although they did not conduct rescreenings, they did continue to monitor residents 

who exhibited symptoms. In addition, they planned to pilot test a rescreening that would involve 

rescreening clients at 4 to 12 weeks after an initial positive screen. Case managers were trained 

in screening residents during regularly scheduled appointments. 

Mental health first aid. Throughout the duration of an individual’s stay in the shelter, DSVP 

staff used MHFA skills on an as-needed basis to address any mental health crises that arose. 

Motivational interviewing. MI techniques such as change talk were incorporated into sessions 

and interactions with clients especially around tough decisions such as whether clients would see 

their abusers again. No decisions were made for the clients, but instead, the pros and cons of 

each situation were weighed to empower clients to make the decision for themselves with the 

support of staff. 

Psychoeducation. The site used Risking Connection®10 as their model to train staff on PE 

at all eight sites. Staff used their knowledge in their one-on-one interactions with residents as 

needed. In particular, PE was used at the time of the mental health screenings, which were found 

to provide an intervention in itself. 

Referral pathway (warm handoff). When screening results indicated any symptoms, social 

workers offered residents a referral to clinical care. For clients who accepted the referral, CBO 

social workers initiated a warm handoff to the intake coordinator at the SHCC. CBO social 

workers sent an email to the SHCC intake coordinator with initial information about the resident 

being referred (including a referral and HIPPA form). Usually within 24 hours, the intake 

coordinator called the CBO social worker to discuss the referral and gather any additional 

helpful information. The SHCC intake coordinator or a SHCC staff member then called the 

resident to complete a 15- to 20-minute phone screening. CBO social workers also followed up 

with residents to make sure they received a call from the intake coordinator and to answer any 

questions or concerns they had about treatment. 

If the counseling center seemed like an appropriate fit for the resident (e.g., was in a location 

safe from the abuser), then the intake coordinator set up an appointment with an available 

clinician. Appointments were scheduled to take place within 2 weeks of the referral acceptance. 

The intake coordinator let the CBO social worker know when an appointment was scheduled. If 

10 Traumatic Stress Institute, “Risking Connection® (RC) Training,” webpage, undated. 

300 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

the resident did not make the appointment, then the CBO social worker contacted the resident to 

help problem-solve any obstacles to appointment attendance. If the counseling center was not an 

appropriate fit for the resident, the intake coordinator provided outside referrals or recommendations 

based on the resident’s needs, safety, and convenience of location. 

Care coordination. The CBO C2C project coordinator organized and led joint quarterly 

meetings between DSVP and SHCC staff to review screening, referral, and care coordination 

data. Safe Horizon’s VP was responsible for monitoring screening, referral, and care coordination 

data with technical assistance from the SHCC clinical social worker. 

Table A.9. Safe Horizon Training, Coaching, and Supervision

Who Receives

Who Received Number of Who Provided Coaching and Coaching and 

C2C Skill Training Staff Trained Training Supervision Plan Supervision

Screening All staff, 98 SHCC staff 

however, only and external

a subset trained consultants

on screening 

administration

MHFA All staff 99 SHCC staff

MI Subset 22 Contractor

through MI

network of 

trainers

PE All staff 98 SHCC staff 

and external

contractors

Safe Horizon staff 

supervisors 

(directors, 

supervising social

workers, supervising 

residential

specialists) provide

with technical

assistance from CBO

clinical social worker

Biweekly individual

supervisions,

including reflective

supervision

Incorporated into

monthly group 

supervisions

Same as above

Two to three on-site

coaching sessions

are provided from MI

consultant (usually

for social workers 

and case managers)

Same as screening 

and MHFA

All staff except 

program assistants

(directors, supervising 

social workers, social 

workers, case 

managers, supervising 

residential specialists,

residential specialists,

childcare aides, and 

housing specialists)

Same as above

Directors, supervising 

social workers, case 

managers, residential

managers housing 

specialists, and some 

supervising residential

specialists

Same as screening 

and MHFA

301 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Sheltering Arms

Sheltering Arms C2C Program Snapshot (as of Y4 Q3)

Mental health provider: Sheltering Arms Jamaica Clinic, Sheltering Arms Bronx Clinic, and Child and Family

Treatment and Support Services

Program type: Child and parent services

On-site mental health counseling available at CBO: Part-time

C2C Initiative target group(s): Expectant parents and parents or caregivers of children ages 0 to 4

CBO’s target population(s): Low-income families, single-parent households

Target geographical area/s: Queens, Bronx, Harlem

Target CBO programs for C2C implementation: Six Early Childhood Education program sites

Number of CBO sites providing C2C services: Multiple sites

Total CBO clients served by C2C (Y1–Y4 Q3): 2,327

Number of current staff and supervisors trained in one or more C2C modalities: 58

Number of staff receiving continuous coaching and supervision (Y4 average): 8

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview

Sheltering Arms is a nonprofit that strengthens the education, well-being, and development 

of high-need children, adults, and families across NYC. The organization provides parent/ 

caregiver-serving programs (Early Head Start, Head Start, and Early Learn); afterschool 

programs; a community school program; programming in foster care/group homes; family 

preservation, juvenile justice placement and aftercare services; fatherhood programs; and 

services for developmentally disabled adults.11 

Sheltering Arms implemented C2C to connect the caregivers of young children in ECE 

programs to mental health services to address depression, alcohol and drug use, immigration 

challenges, postpartum depression, and various types of trauma. Through provision of mental 

health services, Sheltering Arms aimed to help clients provide a more stable and nurturing 

home for their children, improve relationships between families, enhance children’s academic 

performance, and engage parents in school activities. 

C2C Model Implementation

C2C staffing model. The CBO C2C CC (LMSW) was the primary contact for ECE staff 

across six sites. The CC and the ECE clinical supervisor were responsible for the overall 

planning and implementation, training and coaching staff, and interfacing with MHPs and 

linkage agreements. The MHPs for Sheltering Arms included Sheltering Arms Jamaica Clinic, 

Sheltering Arms Bronx Clinic, and Children and Family Support Services. Their linkage 

agreements included The Door, Catholic Charities, Northside, New York Psychotherapy and 

11 For more information, please refer to Sheltering Arms, “About Us,” webpage, undated. 

https://shelteringarmsny.org/about-us/ 

302 
 

 

 

Counseling Center, Emma Bowen Clinic, Harlem Family Institute, and University Settlement. 

The MHP and CBO staff provided MI/reflective check-ins to directors (C2C supervisory team) 

who then provided MI/reflective check-ins with the family service worker (C2C support team). 

All ECE staff were trained in MHFA to help them recognize and respond to the signs and 

symptoms of mental health disorders and substance use. The C2C support team (family service 

worker) were trained in screening, MI, MHFA, and PE (Table A.10). Sheltering Arm’s Director 

of Mental Health Services, LCSW, a PhD candidate, led development and implementation of 

training efforts and serves as a consultant. 

Screening. All participants at the six targeted ECE sites were eligible for screenings. 

Screenings (called wellness surveys at Sheltering Arms) were conducted for depression (PHQ-9), 

alcohol (AUDIT), and drug use (DAST-10). With the help of the CQI project, Sheltering Arms 

implemented wellness survey events at each site where the family workers and C2C interns 

implemented surveys. In addition, they provided self-care activities, including satchel making, 

yoga, and hand massages. They were planning to provide two large wellness survey events and 

four smaller survey events at each site throughout the year. They also provided wellness surveys 

during tabling events. The CBO C2C CC and the ECE mental health clinical supervisor also 

planned to speak to caregivers about C2C and mental health support during caregiver orientations. 

Whether a client indicated moderate, mild, or no symptoms, they were invited to attend PE 

workshops called Trauma Smart or Nurturing Parenting. All clients were asked if they would 

like a referral to an MHP. If a client indicated severe symptoms, staff administering the screening 

used MHFA’s ALGEE skills to assess whether the client was in crisis and then staff contacted 

their immediate supervisor; if the client was not in crisis, they were offered a referral to an MHP 

and encouraged to attend a workshop (PE). 

Mental health first aid. All ECE staff used MHFA in their day-to-day work with caregivers 

to identify risk factors and warning signs for mental health disorders and substance use disorders. 

These staff helped caregivers who appeared to be experiencing mental health challenges 

connect with the support team. In addition, staff helped each other connect to the services they 

learned about during MHFA. 

Motivational interviewing. All staff conducting screenings used MI in their work to 

encourage the caregivers to agree to screenings and, as needed, to accept referrals to MHPs. 

Clients who screened positive received follow-up from staff using MI skills. Staff began by 

asking an open-ended question. For example, if a caregiver scored positive on the PHQ-9, they 

asked a question about the checked responses. “Can you tell me more about these feelings that 

you checked?” 

Psychoeducation. The CBO C2C CC, ECE mental health clinical supervisor, C2C intern, 

fatherhood specialist, health service coordinator, one family worker, and one director provided the 

Trauma Smart curriculum. This curriculum helps inform caregivers about trauma-informed skills 

and resources for themselves and their children. All caregivers were invited to the group PE 

sessions. All caregivers regardless of whether they scored positive or not were invited to the 

303 
 

 

 

 

 

 

 

Trauma Smart workshops. The workshops were very interactive and allowed caregivers to do 

activities with their children to help them feel calm and regulated. 

Another PE curriculum the program implemented was the Nurturing Parenting curriculum. It 

is a family-centered and trauma-informed initiative that teaches nurturing parenting skills. There 

are different programs depending on the needs of the families. The most relevant program for 

ECE families was community-based education in Nurturing Parenting. This program included ten 

independent lessons, including Understanding Feelings, Alternatives to Spanking, Communication 

with Respect, and Building Self-Worth in Children. Family service workers and C2C interns 

led Nurturing Parenting workshops with families at four of the C2C sites. Trauma Smart and 

Nurturing Parenting were very similar and were interwoven together. 

Referral pathway (warm handoff). Caregivers were screened by family workers and C2C 

interns at the wellness survey events. After the surveys were completed, the CBO clinical 

supervisor and the CBO CC discussed referral sources in the neighborhood. Clients’ names and 

numbers were written down. Caregivers were called within a week of the wellness event by a 

C2C support team member, the CBO CC, or the CBO mental health clinical supervisor, about a 

referral. Caregivers were provided the option of being accompanied to the clinic by the CBO CC 

and the C2C intern. The CBO mental health clinical supervisor provided the first appointment 

and then informed the caregiver about where they can continue services with an external provider. 

Clients from the Malcolm X site were typically referred to the Sheltering Arms Jamaica 

Clinic. The family service worker could make a referral to the Jamaica Clinic or refer the 

caregiver to the CBO clinical supervisor or the CBO CC to make the referral. Clients at the 

Paul’s House, Betances, Mother Hale, and Morningside sites were primarily referred to the 

Sheltering Arms Bronx Clinic. MHP LMSW saw families at any of the six C2C sites for a first 

appointment. A psychiatrist only saw families who needed wellness support at Paul’s House. In 

addition, families could be referred from any of the six sites to Child and Family Treatment and 

Support Services, which provides in-home therapy for the child and family. There were many 

MHPs to meet the various needs of the families served. 

The C2C CC was in continual contact with the various MHPs, the C2C support and 

supervisory Team, and the caregivers to coordinate referrals and support caregiver engagement 

in ongoing MH services. If a caregiver asked for specific help calling the MHP or going to an 

intake appointment at the provider, the CBO CC or C2C intern was available to accompany the 

client. If a client refused a referral after indicating moderate or severe symptoms on a screening, 

the C2C support team used MHFA and MI to encourage referral acceptance. If a client did not 

attend a scheduled mental health appointment, the CBO CC called the client to explore obstacles 

and barriers in attending their session and helps the client find solutions to these barriers. 

Care coordination. Each site had access to a data management system that contained 

information about client attendance at MH services and the MHP’s name. C2C support team 

members or the CBO CC conversed with the MHP or the caregiver within a week after the first 

appointment. The CBO CC followed up with the MHP or the caregiver once every 3 months for 

304 
 

 

 

 

 

 

 

 

 

up to a year. There were quarterly care coordination meetings between Sheltering Arms’ CBO 

and MHP staff to review screening, referral, and care coordination data. 

Table A.10. Sheltering Arms Training, Coaching, Supervision, and Continuous Quality

Improvement

Who Number Who Receives

C2C Skill

Received 

Training

of Staff 

Trained

Who Provided 

Training

Coaching and 

Supervision Plan

Coaching and 

Supervision

Screening Subset 8 C2C CC provides initial

and ongoing 

training/coaching

Conducted by

C2C CC

Quarterly 45- to

60-minute coaching

sessions as 

Family workers

needed

MHFA All staff 57 MHP MH director, C2C

CC ongoing booster

sessions led by CC

As needed All staff at six ECE 

sites

MI Subset 7 MI Institute, MHP

provide initial 2-day

training and ongoing

MI coaching

45- to 60-minute 

quarterly

MI/reflective

check-in

Family workers and 

directors

PE Subset 11 C2C CC, trauma smart

trainers initial and 

ongoing

45- to 60-minute

quarterly coaching 

as needed

Family workers, 

C2C intern, director

of Malcolm X, 

health service

coordinator, 

fatherhood 

specialist

305 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

STRIVE International

STRIVE C2C Program Snapshot (as of Y4 Q3)

Mental health provider: Union Settlement, on-site part-time; Silberman School of Social Work at Hunter College

Program type: Job training and employment

On-site mental health counseling available at CBO: Part-time

C2C Initiative target group(s): Unemployed or underemployed low-income working-age adults ages 18 and 

older

CBO’s target population(s): Youth, the formerly incarcerated, public assistance recipients, noncustodial parents,

the homeless, the long-term unemployed, the working poor and recovering from substance use disorder

Target geographical area/s: All NYC

Target CBO programs for C2C implementation: All programs

Number of CBO sites providing C2C services: One site

Total CBO clients served by C2C (Y1–Y4 Q3): 2,158

Number of current staff and supervisors trained in one or more C2C modalities: 32

Number of staff receiving continuous coaching and supervision (Y4 average): 22

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview

STRIVE provides services to New Yorkers from all boroughs with the mission of helping 

individuals acquire the skills and attitudes they need to overcome challenging circumstances, 

find sustained employment, and become valuable contributors to their families, employers, and 

communities. STRIVE services include job training, parenting classes, and youth programs. 

STRIVE’s service delivery model combines intense period of training in attitude, selfpresentation, job skills, and job search techniques with rapid employment placement and longterm follow-up.12 

STRIVE implemented C2C to provide clients with the skills necessary to cope with mental 

health challenges and thereby meet goals related to job training and employment. STRIVE’s 

clients included youth, the formerly incarcerated, public assistance recipients, noncustodial 

parents, the homeless, the long-term unemployed, the working poor and those recovering from 

substance use, all groups at elevated risk for experiencing mental health challenges. C2C was 

implemented across the organization in the following STRIVE programs: Core Attitudinal and 

Job Readiness; vocational skills training; Strong Fathers, Strong Families; FOCUS job readiness 

(for formerly incarcerated individuals); and STRIVE Future Leaders (for justice-involved youth). 

C2C Model Implementation

C2C staffing model. Implementation of C2C skills was integrated into STRIVE programming 

and carried out primarily by case managers and the C2C program lead. The Silberman School of 

12 For more information, please refer to STRIVE, “STRIVE New York,” webpage, undated. 

306 
 

 

Social Work (SSSW) provided initial training and ongoing consultation. Union Settlement (US), 

a licensed mental health clinic located in close proximity to STRIVE, received referrals and 

the MHP lead (LMSW) from US was on-site at the CBO 2 days a week to observe C2C 

implementation, provide brief trainings, and deliver C2C skills to clients. MHP clinical service 

interns aided in service delivery to clients by taking on short-term counseling under MHP lead 

supervision, following up with clients who scored positive on a wellness survey to help identify 

necessary services, conducting PE sessions throughout classes, modifying PE and service delivery 

to tailor directly to client needs, and assisting with resource identification and referrals as needed. 

The MHP provided group coaching sessions to CBO staff and supervisors, incorporating 

reflective supervision and direct observation of staff implementing C2C skills (Table A.11). 

Each coaching session also incorporated an experiential learning component, planned practice 

exercises, and role-plays. STRIVE also adapted a virtual gaming system to enhance knowledge 

of C2C skills and promote skill building. Staff also completed self-assessment process surveys to 

staff to solicit feedback about learning needs related to the core C2C skills. In addition to training 

in the four C2C skills, STRIVE also offered supplementary wellness trainings (e.g., mindfulness 

meditation) to staff. 

Screening. Screening processes were customized to fit the program in which they were 

implemented. For the core program, STRIVE clients undergo an extensive intake and interview 

process before entering. They complete enrollment forms, fulfill education eligibility requirements, 

and meet with a case manager to complete an entrance interview. The entrance interviews primed 

clients for the MH screenings. Once a client was accepted into the core program, or any of the 

other participating programs, the case manager administered C2C screening in a group setting 

within a week of the program beginning. STRIVE clients were screened for depression, anxiety, 

and stress (DASS 21); PTSD (PC-PTSD); and substance use (AUDIT, DAST-10). If clients 

indicated moderate-to-severe mental health symptoms on any of the screening instruments, they 

were referred to the C2C program lead or MHP lead to discuss the possibility of referral to US 

and/or additional mental health services. Clients who declined screening were encouraged to 

discuss their decision with their case manager and reconsider at a later date. 

Mental health first aid. Case managers and trainers used MHFA as needed when mental 

health symptoms interfered with a client’s ability to engage in STRIVE activities. 

Motivational interviewing. Case managers used MI as needed with clients in one-on-one 

meetings, and trainers implemented MI throughout core workshops. MI was used throughout 

conversations with clients in both group and individual settings. Particularly when clients 

demonstrated resistance to change or ambivalence, staff utilized principles of MI during these 

conversations to identify true willingness to change and help clients self-identify that desire for 

change. MI was also used through harm-reduction conversations, de-escalation, and career track 

identification. 

Psychoeducation. All clients in the CORE workshops received weekly group PE sessions 

developed by the clinical service interns and CBO staff. Sessions focused on mental health 

307 
 

versus mental illness; stress management during job search; strategies to combat anxiety before, 

during, and after a job interview; and self-care and job performance. STRIVE adapted this 

curriculum for clients in the Future Leaders program after noticing particularly high scores on the 

PTSD screening among this group. PE sessions for this group included information and exercises 

to support participants in developing coping skills to manage trauma needs. The curriculum also 

incorporated elements of dialectical behavioral therapy, including mindfulness skills. 

The curriculum and workshops were tailored to specific client populations to make the 

content more appealing and relevant. For example, the Future Leaders curriculum integrated 

“credible messengers” (e.g., well-known celebrities) to destigmatize and normalize mental health 

problems. Clients also received an “orientation to clinical services” module, so they were better 

prepared to receive clinical services at US or elsewhere. In addition, case managers provided 

individual PE on a range of mental health topics as needed with clients in weekly one-on-one 

meetings. Also, the underlying goal of all PE sessions was employability and job readiness. 

Referral pathway (warm handoff). If clients indicated moderate or severe symptoms on any 

screening, their case managers referred them to the on-site MHP MSW interns. Interns met with 

participants to find out more about clinical needs, make a referral to US if needed/desired, and 

provide additional information about clinical services to ease the transition with the referral 

process. The interns worked closely with the MHP lead, who communicated directly with a 

US mental health services intake worker to obtain a date for an intake appointment. When 

possible, US’s intake was conducted on-site at STRIVE. If on-site intake was not possible, a 

STRIVE staff member offered to escort clients to the US clinic for intake. The MHP lead let the 

client and the client’s case manager know the date and time of the intake appointment as well as 

the kind of documentation the client needed to bring to the first appointment. If a client did not 

attend his or her intake appointment, the MHP lead was notified and followed up with the client 

to understand any barriers to attendance. At the time of intake, if the client agreed, a release of 

information was signed, so STRIVE and the clinic could exchange information verifying the 

client’s attendance at the clinic. Clients who screened positive and declined referrals to US or 

any other provider suggested by STRIVE were given a list of care providers. Assigned clinical 

service interns followed up with these clients to discern whether or not they wanted these 

services. 

Care coordination. STRIVE-US care coordination meetings were held at least once per 

quarter and sometimes biweekly. In these meetings, CBO and MHP team members (including 

MSW interns) reviewed individual client needs as well as summary information on screening 

results referral outcomes, STRIVE training and advancement outcomes, and client retention in 

ongoing clinical care. 

308 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table A.11. STRIVE Training, Coaching, and Supervision

Who Number Who Receives

C2C Skill

Received 

Training

of Staff 

Trained

Who Provided 

Training

Coaching and 

Supervision Plan

Coaching and 

Supervision

Screening Subset 8 SSSW and 

STRIVE staff

Provided by MHP lead,

C2C program lead

Clinical service

interns

Stand-alone individual Fatherhood case 

coaching one per quarter

Direct practice observation 

and individual supervision 

session once per quarter

manager

FOCUS career

coach

Staff self-assessment

surveys twice per year

MHFA All staff 17 SSSW staff,

outside 

contractor

Staff self-assessment

surveys twice per year

Case managers

Facilitators

Job developers

Retention 

coaches

MI All staff 31 SSSW,

outside 

contractor,

coaching 

sessions led 

in-house

Provided by MHP lead, C2C

program lead, and CBO

advancement and retention 

coordinator

Stand-alone group coaching 

one per quarter

All CBO client

facing staff

members

Direct practice observation 

and individual supervision 

session once per quarter

Case conference supervision 

once per program cycle 

(varies based on program)

Staff self-assessment

surveys twice per year

PE Subset 11 STRIVE staff Provided by MHP lead, C2C

program lead

Stand-alone group coaching 

one per quarter

All CBO client

facing staff

members

Direct practice observation 

and individual supervision 

session once per quarter

Case conference supervision 

once per program cycle 

(varies based on program)

Staff self-assessment

surveys twice per year

309 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Committee for Hispanic Children and Families

CHCF C2C Program Snapshot (as of Y3 Q4)

Mental health provider: Urban Health Plan, Inc., Comunilife

Program type: Child and parent services

On-site mental health counseling available at CBO: None

C2C initiative target group(s): Expectant parents and parents or caregivers of children ages 0 to 4; unemployed 

or underemployed low-income working-age adults; out-of-school, out-of-work adults 16 to 24

CBO’s target population(s): Low-income Latino families living in the Bronx

Target geographical area/s: Bronx

Target CBO programs for C2C implementation: Early Care and Education Institute services, youth 

development programs

Number of CBO sites providing C2C services: Multiple sites

Total CBO clients served by C2C (Y1–Y3): 370

Number of current staff and supervisors trained in one or more C2C core skills (as of Y3 Q4): 21

Number of staff receiving continuous coaching and supervision (Y3 average): 12

NOTE: Discontinued participation in C2C at the end of implementation year 3.

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2018.

Program Overview

Since 1982, the mission of the Committee for Hispanic Children and Families (CHCF) has 

been to expand opportunities for children and families through education and advocacy, and to 

strengthen the voice of the Latino community in NYC. CHCF is a nonprofit that facilitates youth 

development and afterschool programs in partnership with K-12 public schools. CHCF’s Early 

Care and Education Institute (ECEI) provides courses, workshops, and one-on-one support for 

parent/caregiver-serving educators in underserved communities, particularly those who run 

home-based childcare businesses. CHCF’s advocacy work aimed at improving education and the 

well-being of Latino children. CHCF employs bilingual, bicultural staff with knowledge of and 

experience with the communities it serves.13 

CHCF recognized that families seeking their services experience depression, anxiety and 

substance use, child abuse and neglect, and intimate partner violence. Although CHCF’s original 

application for C2C proposed to infuse C2C into ECEI programming, the C2C Collaborative did 

not approve the design based on programmatic concerns related to the organizational structure, 

staffing, and service methods within CHCF’s ECEI program; as such, the C2C Collaborative 

recommended that CHCF focuses only on the parents in its youth programming for their C2C 

program. CHCF aimed to bolster caregiver abilities to address stressors and understand and 

manage their mental health issues. CHCF integrated C2C into youth development afterschool 

programming, at three schools in the Bronx: The Community School of Technology (P.S. 59), 

13 For more information, please refer to CHCF, “About Us,” webpage, undated. 

310 
Captain Manuel Rivera Jr. School (P.S./M.S. 279), and Bronx High School of Business. For the 

youth development component, C2C was primarily integrated into workshops for parents whose 

children attended youth development afterschool programming. 

C2C Model Implementation

C2C staffing model. CHCF initially partnered with Comunilife to serve as the C2C MHP. 

Comunilife is a multiservice nonprofit providing housing and culturally sensitive, bilingual 

(English/Spanish) mental health support services to individuals throughout NYC. In Year 2, 

Comunilife transitioned management of its outpatient mental health clinic to Urban Health Plan, 

Inc. (UHP). Following this transition, UHP became CHCF’s MHP. UHP is a federally qualified 

community health center licensed as a diagnostic and treatment center under Article 28 of the 

New York State Public Health Law and Article 31 of the New York State Office of Mental 

Health. UHP provides comprehensive and affordable primary and specialty health care services 

to families in the Bronx, Harlem, and Queens. At CHCF, coordination of C2C was initially led 

by the CBO manager of parent engagement, who met with program staff, reviewed monthly 

progress reports, solicited feedback from program participants, and maintained regular 

communications with the MHP. Later in implementation, this coordination role was carried out 

by the CBO chief program officer. Comunilife provided some initial training to CHCF staff; 

after the MHP change, UHP staff provided training, coaching, consultation, and supervision. 

Screening. Over the course of C2C implementation, C2C worked to implement a systematic 

screening workflow for parents of their participating youth but found it difficult given the 

structure of their model. CHCF’s final approach to screening involved offering screening to all 

parents who participated in PE workshops. CHCF staff were also trained to offer screenings to 

families during individual conversations and interactions that occurred through their regular job 

duties. Individuals were screened for depression (PHQ-9) and anxiety (GAD-7). 

Mental health first aid. CHCF implemented MHFA to develop awareness and enable staff 

members to show support for those experiencing mental health challenges. MHFA was delivered 

during crisis situations, in conjunction with required protocols for use within schools. 

Motivational interviewing. All full-time youth development staff were trained in MI. Staff 

used MI in their interactions with students, parents, and fellow colleagues at the schools to 

encourage behavior change, as appropriate. MI was also used when sharing screening results 

with parents to encourage them to explore options and decide on follow-up actions. 

Psychoeducation. CHCF implemented the Parenting Journey curriculum as their PE 

programming. Two trained CHCF staff facilitated the Parenting Journey workshops with parents 

at each of CHCF’s three C2C implementation sites. Parenting Journey uses a combination of 

hands-on activities and guided discussions to help caregivers develop knowledge and skills that 

will support them as parents and role models. CHCF also hosted occasional wellness workshops 

that focused on additional mental health issues, with topics and content jointly developed by 

CHCF and the UHP. 

311 
 

 

 

 

Referral pathway (warm handoff). CHCF’s protocol was to make referrals to individuals who 

indicated moderate or severe symptoms during screening. If the individual accepted the referral, 

CHCF explained the next steps in the referral process and submitted a referral form to UHP. 

Within 48 hours of receiving the referral, UHP reached out to the parent/caregiver to schedule a 

first appointment. For individuals who initially declined a referral offer or who failed to attend 

their first scheduled appointment with UHP, CHCF followed up by phone or in person to check in 

and continue to offer resources for support, including another connection to UHP. 

Care coordination. CHCF and UHP staff met on a weekly basis to discuss referrals and 

address any scheduling challenges. In addition, CHCF shared monthly data reports (including 

screenings, referrals, etc.) with UHP and held calls to discuss trends in the reports and any action 

that might need to be taken to improve services. Case counseling was incorporated into biquarterly 

coaching and supervision meetings between CHCF and UHP staff. 

Discontinuation of C2C: CHCF experienced a number of challenges related to staffing 

vacancies and transitions, changes in organizational leadership and C2C leadership, and other 

challenges that led to persistent difficulties in meeting programmatic implementation 

requirements. One particular challenge was that CHCF project leads, without mental health 

backgrounds, were uncertain of how to coordinate with and what to ask of the MHP, which 

delayed some of the initial implementation planning (e.g., CHCF project leads did not realize 

the scope of assistance they should be asking/receiving from their MHP until they received 

additional implementation guidance during year 2 of C2C implementation). After continued 

program issues in its first 3 years, CHCF’s performance did not meet the program performance 

criteria for continuation funding beyond Year 3. They never reached full implementation of all 

required components, and the funder ended their participation in C2C after year 3. 

312 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Door

The Door C2C Program Snapshot (as of Y4 Q3)

Mental health provider: University Settlement

Program type: Youth development

On-site mental health counseling available at CBO: Full-time

C2C Initiative target group(s): Expectant parents and parents or caregivers of children ages 0 to 4; out-ofschool, out-of-work young adults ages 16 to 24; and/or unemployed or underemployed low-income working-age 

adults ages 18 and older receiving employment services

CBO’s target population(s): At-risk youth, out-of-school, out-of-work young adults ages 16 to 24

Target geographical area/s: Citywide

Target CBO programs for C2C implementation: Runaway and Homeless Program, Center Space Program,

Supportive Housing Program and Career and Education Programs

Number of CBO sites providing C2C services: Multiple sites

Total CBO clients served by C2C (Y1–Y4 Q3): 8,067

Number of current staff and supervisors trained in one or more C2C modalities: 95

Number of staff receiving continuous coaching and supervision (Y4 average): 60

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview

The Door—A Center of Alternatives is a nonprofit multiservice youth development agency. 

The Door provides a range of free integrated services to young people (ages 12–21) in eight 

program areas: career and education; adolescent health center; legal services center; runaway 

and homeless youth; supportive housing; arts; food and nutrition; and a mental health services 

department supported by the MHP, University Settlement. The Door operates in four locations: 

lower Manhattan; the Bronx; and two supportive housing sites for young people with a mental 

health diagnosis and a history of homelessness. The agency serves nearly 10,000 young 

people annually, the majority of whom come from low-income families and are racial/ethnic 

minorities.14 

C2C was implemented at The Door’s Manhattan location, Bronx Center, and two supportive 

housing sites across four programs (Center Space, Career and Education, Runaway and Homeless 

Youth, and Supportive Housing) to empower clients by providing them with mental health– 

focused services in a nurturing, diverse space. Clients come to The Door with a wide range of 

stressors and mental health conditions—many have histories of trauma and/or family disruptions 

or experience other issues that affect their ability to accomplish their goals. 

14 For more information, please refer to The Door, “About the Door,” webpage, undated. 

313 
 

C2C Model Implementation

C2C staffing model. The Door’s supervisor of training and capacity building and the MHP’s 

director of mental health and wellness served as the CBO and MHP program leads, respectively. 

Together, they developed C2C implementation and training plans, identified and tested solutions 

to implementation challenges, and served as the bridge between programmatic and mental health 

services (Table A.12). Program staff (including support staff) in the Runaway and Homeless 

Youth Program, Career and Education Department, Center Space and Supportive Housing were 

trained and coached in at least one of the four core C2C skills. The MHP’s director of mental 

health and wellness was on-site at The Door between 2.5 and 3 days a week during the initial 

start-up phase of the project (first 4–6 months) and was available on an ad hoc basis, if needed. 

MHP training staff (seven staff members) were periodically on-site at The Door to deliver trainings. 

Screening. Although all staff were trained in administering and debriefing screens, each 

program had the flexibility to determine who completed screens with clients. In many programs, 

direct service staff were responsible for administering screens. In some cases, designated staff 

took on the responsibility of administering screens and debriefing clients. Similarly, programs 

determined when in the program cycle clients were screened. In some programs, screening was 

part of the initial intake process. Other programs waited until they had established a stronger 

relationship with clients. 

Clients were screened for depression (PHQ-9), alcohol and substance use (CRAFFT), and 

adverse childhood experiences/trauma (Adverse Childhood Experiences survey). The Adverse 

Childhood Experiences survey was only completed once with a client and the PHQ-9 and 

CRAFFT were completed quarterly (every 3 months) after the initial screen. 

Mental health first aid. MHFA was integrated into the overall practices of the agency, with 

emphasis on the target C2C programs. The Door staff used MHFA assessment and skills with all 

clients to recognize mental health issues, proactively address behaviors and symptoms, and link 

clients more quickly to mental health services. 

Motivational interviewing. Core principles and practices of MI were integrated into many 

aspects of work at The Door, with emphasis on the target C2C programs. Staff used MI with 

clients who screened positive to address issues that affected their ability to progress forward 

with goals. The Door integrated the use of MI tools, such as MI session logs, decisional balance 

worksheets, change plan worksheets, and SMART goal-setting worksheets. 

Psychoeducation. Much of the PE provided to clients occurred in individual sessions with a 

client and a counselor or case worker as a part of sharing screening results and as a way to 

normalize and validate symptoms. Staff at The Door, including a mix of clinical and engagement 

personnel, provided specific PE group sessions geared toward addressing common needs of 

clients. For a time, sessions were based on the Seeking Safety curriculum; The Door piloted the 

Project Alert curriculum in Year 2 of implementation, then returned to Seeking Safety. PE was 

also incorporated into regular group programming such as art and recreation groups, during 

which staff discussed strategies for anger management, destigmatizing mental health, and stress 

314 
 

 

 

 

 

 

management techniques, among other topics. Staff also held PE sessions based on the Seeking 

Safety curriculum. 

In addition to being trained in the four C2C skills, one cohort of staff members was trained in 

SPARCs, a 16-week evidence-based group model for adolescents who have experienced complex 

or chronic trauma. This training was provided to a variety of staff but has implemented mostly 

with the staff who facilitate The Door’s anger management group. 

Referral pathway (warm handoff). On completing screenings, CBO staff used C2C skills to 

score screening results, assess for safety, engage in meaningful dialogue about the results of the 

screenings, and refer clients to the appropriate level of intervention: either to C2C integrated 

services within the organization (e.g., PE, counseling) or services at University Settlement. 

Clients who indicated more severe symptoms on any of the screeners were offered a mental 

health consult with an on-site counselor, usually the clinical director of The Door’s counseling 

program. These clients were escorted by the staff member who conducted the screening directly 

to the counselor who completed a more comprehensive assessment. The counselor then discussed 

options, including a referral to University Settlement’s Article 31 Satellite Clinic at The Door or 

other mental health services in the community. 

University Settlement prioritized clients from The Door for intake and appointment scheduling. 

Staff at The Door call a representative at US with the client to help them arrange an appointment. 

If possible, an in-person introduction happened (for a year they had a social work intern who 

traveled between the two sites). 

All referrals (both to University Settlement and off-site providers) were tracked for completion. 

Staff members who made and assisted with referrals were responsible for outreach to providers 

to determine if clients completed the referral process. The Door also hired an on-site care 

navigator for the C2C program to provide additional capacity for outreach and follow-up, 

helping troubleshoot challenges with clients who missed appointments. 

Care coordination. The CBO and MHP program co-leads were jointly responsible for 

coordinating quarterly meetings with CBO and MHP staff members to review screening, 

referral, and care coordination data. 

Table A.12. The Door Training, Coaching, and Supervision

Who Number of Who Receives

C2C Skill

Received 

Training

Staff 

Trained

Who Provided 

Training

Coaching and Supervision 

Plan

Coaching and 

Supervision

Screening Subset 65 MHP/CBO Provided by CBO All staff

There are multiple sessions

offered throughout the month 

in which teams are mixed 

across programs. Coaching 

combines screening, MHFA,

and psychoeducation.

315 
 

 

 

 

 

 

 

Who Number of Who Receives

C2C Skill

Received 

Training

Staff 

Trained

Who Provided 

Training

Coaching and Supervision 

Plan

Coaching and 

Supervision

MHFA All staff 46 MHP/CBO Provided by CBO All staff

There are multiple sessions

offered throughout the month 

in which teams are mixed 

across programs. Coaching 

combines screening, MHFA,

and psychoeducation.

MI Subset 58 MHP/CBO and 

MI Institute

Provided by CBO

Select leadership staff attend 

a monthly ongoing practice 

group at the Institute.

Staff who

attended the 

2-day MI 

workshop or

staff members

Direct service staff and

supervisors attend monthly

coaching groups for 6 months

after initial workshop and 

then ad hoc for refreshers 

with equivalent

MI knowledge 

are invited to 

coaching

after that.

PE Subset 5 MHP/CBO and 

outside 

contractor

Provided by CBO

Integrated into monthly 

coaching sessions

Open to all

staff

316 
 

 

 

 

 

 

 

 

 

 

Mental health provider: Brookdale University Hospital and Medical Health Center; Institute for Family Health

Program type: Job training and employment

On-site mental health counseling available at CBO: Part-time

C2C Initiative target group(s): Unemployed or underemployed low-income working-age adults ages 18 and 

older receiving employment services

CBO’s target population(s): Unemployed, low-income adults 18+

Target geographical area/s: Brooklyn, Bronx

Target CBO programs for C2C implementation: All

Number of CBO sites providing C2C services: Multiple sites

Total CBO clients served by C2C (Y1–Y4 Q3): 1,495

Number of current staff and supervisors trained in one or more C2C modalities: 32

Number of staff receiving continuous coaching and supervision (Y4 average): 27

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

 

 

 

 

 

 

 

The HOPE Program

HOPE C2C Program Snapshot (as of Y4 Q3)

Program Overview

The HOPE Program is a workforce development nonprofit agency that aims to empower 

New Yorkers living in poverty to achieve economic self-sufficiency through employment and 

career advancement. Each year, HOPE serves approximately 300 low-income clients facing 

significant barriers to employment, including homelessness (52 percent), histories of criminal 

justice involvement and substance use (47 percent), low educational attainment (27 percent), and 

poor work history. More than 90 percent of clients are black and/or Hispanic, 30 percent live in 

shelters or with friends or family, and 27 percent reside in substance use treatment facilities at 

the time of their enrollment in HOPE. HOPE’s service population also varies by gender and age. 

Fifty-one percent of their green job trainees are women, whereas 49 percent identify as men. 

They also serve a wide age range, with 37 percent of their clients aged 18 to 29, 22 percent aged 

30 to 39, 22 percent aged 40 to 49, and 19 percent aged 50 and older. 

HOPE’s core programming aims to connect unemployed individuals living in poverty with 

employment, long-term job retention, and successful entry into career pathways. HOPE has a 

comprehensive set of work-readiness services, with infrastructure for managing client 

engagement and retention and a referral protocol to connect clients to supplementary services, 

including childcare and mental health counseling. Their programs include a suite of life and job 

skills modules, including Work Wellness courses that emphasize stress management and 

interpersonal/conflict resolution skills training.15 HOPE implemented C2C at both their Brooklyn 

and Bronx locations to build on their existing Work Wellness courses in hopes of addressing 

substance use, mood and anxiety disorders, sleeplessness, and other mental health issues faced 

by their clients aiming to seek employment. 

15 For more information, please refer to The HOPE Program, homepage, undated. 

317 
 

 

 

 

 

C2C Model Implementation

C2C staffing model. HOPE partnered with Brookdale University Hospital and Medical 

Health Center in Brooklyn and the Institute for Family Health in the Bronx to implement C2C. 

C2C skills were integrated into existing services, including intake, classroom training, clients’ 

one-on-one meetings with instructors and employment specialists, and career counseling for 

HOPE program graduates. The director of Work Wellness and the C2C coordinator provided fulltime management of the program (Table A.13). Trained MHPs from the MHP were on-site at 

HOPE programs part-time—the mental health educator who was a licensed mental health 

clinician was on-site 14 hours per week, and the MHP lead (LCSW) was on-site 3 to 5 hours per 

week to provide supervision and oversight. In Brooklyn, the mental health educator was on-site 

12 hours a week and the MHP lead was there 4 hours a week. This was a change due to need that 

the mental health educator at Brookdale was to fill in some staffing gaps. Prior to this, he was 

on-site 2.5 days a week. In the Bronx, the mental health educator was on-site every Wednesday 

from 9 a.m. to 5 p.m. 

Screening. The program did a lot of experimenting with the screening process. They initially 

planned to administer the MSSI (substance use) to all clients, but ultimately opted not to use this 

screening with most clients because many of them were already in treatment for substance use. 

Instead, they used MSSI optionally if they suspected someone who was not in treatment may 

have substance use. For the last couple of years, they offered the PHQ-9, GAD-7, and PC-PTSD 

as their primary screening tools. They worked through a trauma-informed lens because they 

believed that their population had experienced a lot of trauma and that this was probably the cause 

of mental health symptoms. For that reason, they incorporated a trauma screen from the beginning. 

They usually did screens during the third week of class, and this was after two to three work 

wellness classes. At the Brooklyn site, they conducted one-on-one screenings, but at Bronx, 

screenings were conducted as a group because the clients were in transitional employment and 

not as readily available. 

Mental health first aid. MHFA was delivered by direct service staff as needed. CBO staff 

delivered MHFA in crisis situations, such as explosive anger directed toward staff and/or other 

clients, verbal/physical conflicts between clients, inconsolable crying, and addiction relapse. 

All staff were trained in MHFA to ensure that anyone was equipped to respond. Once MHFA 

was used, the next step was to follow up with the client to discuss a referral to the MHP and 

implement the referral if the client agreed. 

Motivational interviewing. MI was primarily provided by the CBO C2C coordinator but was 

delivered by direct service staff on an as-needed basis. MI skills could be used in any staff-client 

interactions, especially one-on-one encounters with clients (e.g., clients who report difficulties 

with job placement and discussions about referring clients to counseling). MI was conducted in 

the privacy of the staff member’s office to address behavioral/performance concerns in the 

318 
 

 

 

 

 

 

 

 

classroom, at the internship site, and in employment situations; and also used in group settings, 

such as the classroom, to immediately resolve problems. 

Psychoeducation. PE was delivered for 1 hour per week by the CBO C2C coordinator 

with assistance from the graduate counseling interns as part of the ongoing Work Wellness 

programming. HOPE added content on mental health (e.g., recognizing symptoms of depression, 

anxiety disorders; treatment options for mental health issues) to the Work Wellness module in 

both its HOPEworks and FOODworks programs. Over time, HOPE discontinued implementation 

in the FOODworks program. In addition, they transitioned from a 10-week class to a 5-week 

class model, with four 2-hour Work Wellness sessions included in programming. PE was also 

delivered individually to clients during screening and when a referral was recommended to a 

client. Last, brochures on specific mental health topics were distributed and also made available 

in HOPE’s existing resource shelf located in the Work Wellness area. 

Referral pathway (warm handoff). If the screening process revealed that clients were 

experiencing significant mental health symptoms, the CBO C2C coordinator or other staff 

referred them to the MHP. Clients were offered written materials about the mental health 

condition for which they showed symptoms. Once the client agreed to the referral, the CBO staff 

member called one of the two designated intake coordinators at the MHP. Once an appointment 

was scheduled, the CBO staff member provided the client with a map and additional information 

about the appointment. Clients who declined a referral were invited to meet with the CBO C2C 

coordinator and graduate interns for follow-up and to speak with the MHP clinician on-site. 

Clients who had negative screens but requested to see a clinician were referred to the MHP, or 

other providers as needed (e.g., for substance use treatment). The CBO worked with the MHP to 

find appropriate external providers for these clients. 

Care coordination. The MHP lead and CBO C2C coordinator comanaged and tracked the 

referral process using Epic and Salesforce. In Brooklyn, they met every Friday in person at 

HOPE to review current referrals. At the Bronx location, they met or talked regularly, but not 

necessarily on a weekly basis. 

Table A.13. HOPE Training, Coaching, and Supervision

Who Number

Received of Staff Who Provided Coaching and Who Receives Coaching

C2C Skill Training Trained Training Supervision Plan and Supervision

Screening Subset 11 MHP and C2C

coordinators

Provided primarily by

MH educator, with staff 

receiving 30 minutes

of individual coaching/

supervision every

other week.

MH educator provided a 

brief group coaching and 

supervision four times per

month as part of the

Brooklyn: director of 

Brooklyn programs,

clinical intake coordinator,

employment specialists,

director of recruitment,

work readiness

instructors, technology

instructors, business 

development manager,

retention associate,

319 
 

 

 

Who Number

Received of Staff Who Provided Coaching and Who Receives Coaching

C2C Skill Training Trained Training Supervision Plan and Supervision

MHFA All staff 32 ThriveNYC

MI Subset 5 MHP and an 

external trainer

PE Subset 30 C2C 

coordinator;

mental health 

educators

CBO’s existing weekly 

case conference.

C2C coordinator provided 

coaching and supervision 

during weekly team

meetings.

Coaching is offered in all

four modalities based on 

individualized staff needs.

Same as above

Same as above

advancement manager,

and graduate interns

Bronx: director of Bronx

programs, retention 

associate, employment

specialists, administrative 

coordinator and 

admissions coordinator,

director of digital literacy, 

digital literacy instructor, 

case manager, and 

horticultural

superintendent

Same as above

Same as above

Same as above

320 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Voces Latinas

Voces Latinas C2C Program Snapshot (as of Y4 Q3)

Mental health provider: Catholic Charities Brooklyn and Queens

Program type: Agency primarily serving immigrant Latinx populations

On-site mental health counseling available at CBO: Part-time

C2C Initiative target group(s): Expectant parents and parents or caregivers of children ages 0 to 4; unemployed 

or underemployed low-income working-age adults ages 18 and older

CBO’s target population(s): Low-income, Latinx immigrant natives of Queens, 18 to 40 years

Target geographical area/s: Queens

Target CBO programs for C2C implementation: Case management services in Domestic Violence and HIV

Prevention programs: the Promotoras Program and HIV Testing and Linkage to Care Program

Number of CBO sites providing C2C services: One site

Total CBO clients served by C2C (Y1–Y4 Q3): 2,780

Number of current staff and supervisors trained in one or more C2C modalities: 16

Number of staff receiving continuous coaching and supervision (Y4 average): 5

SOURCE: C2C model summaries provided to RAND by CBO leaders in July 2019 and data from quarterly CBO

reports provided to RAND staff, March 2016 to December 2019.

Program Overview

Voces Latinas (Voces) “aims to reduce the rate of HIV transmission and violence among 

immigrant Latinas by empowering, educating, and providing leadership and advocacy training to 

enable them to make healthier decisions for themselves and their families” (Voces Latinas, 

webpage, undated). Through collaborative relationships, Voces tries to connect immigrant 

Latinx with culturally and linguistically sensitive services. Voces programs include domestic 

violence/intimate partner violence services, case management, HIV/sexually transmitted 

infection screening, C2C community mobilization, linkage to pep/prep, linkage to health 

insurance services, navigation services, mental health counseling, and skills-building classes.16 

Voces implemented C2C to address depression, anxiety, trauma, and suicidality in a culturally 

competent manner using bilingual/bicultural staff that fully understand clients and their immigration 

experience. Voces clients are new immigrants; they are often survivors of violence and/or HIV 

positive. Many clients also participate in sex work to earn a living. C2C was embedded in all 

programs and services. It is currently part of the intake/assessment for all programs. 

C2C Model Implementation

C2C staffing model. Voces is a culturally specific organization with approximately 

15 employees. All staff were trained in at least one of the C2Cs, but a core team of approximately 

five staff delivered most of the C2C skills. The director of programs provided oversight to C2C 

implementation and helped coordinate trainings and care coordination meetings. 

16 For more information, please refer to Voces Latinas, “About Us,” webpage, undated. 

321 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Voces partnered with Catholic Charities Brooklyn and Queens (CC), which offers more than 

160 programs and services for children, youth, adults, seniors and those struggling with mental 

illness. The MHP therapist was on-site at Voces 1 day each week to provide clinical services to 

clients and meet with program staff regarding cases. The MHP clinical supervisor trained staff 

and provided supervision and coaching (Table A.14). Voces was in the process of reviewing 

proposals for new MHP candidates in summer and fall 2019. 

Table A.14. Voces Latinas Training, Coaching, and Supervision

Who Who Receives

C2C Skill

Received 

Training

Number of

Staff Trained

Who Provided 

Training

Coaching and 

Supervision Plan

Coaching and 

Supervision

Screening Subset 24 CC clinical 

supervisor and 

therapist (initial);

Elmcor substance 

specialist (as part of

ongoing training and 

refreshers); NYS 

Office of Addiction 

Services and 

Supports downstate 

office (ongoing)

Provided by CC

clinical supervisor

and therapist

Weekly group 

supervision,

including reflective

supervision

CC available for

individual consult by

request, including 

direct observation

Women’s 

services program

manager,

women’s services 

coordinator,

prevention 

coordinators,

case manager,

assistant director, 

patient 

navigators,

outreach 

Quarterly individual

coaching

specialists

MHFA Subset 20 MHFA USA (initial);

CC clinical 

Same as above Same as above

supervisor and 

therapist (ongoing)

MI Subset 17 MI Institute (initial);

NYC Department of

Health Training 

HIV/AIDS Technical

Assistance

Same as above Same as above

(ongoing)

PE Subset 13 CC clinical Same as above Same as above

supervisor and 

therapist (initial)

Screening. Voces’ program assistant was the first contact clients had when they came to the 

organization requesting services. The program assistant administered a basic mental health 

assessment to all clients as part of the program intake process, then guided clients to the 

appropriate programs. Program staff (case manager or coordinator) administered screenings for 

depression (PHQ-9) and trauma (PCL-C); some clients also received screenings for alcohol and 

drug use (AUDIT-10 and DAST-10). Clients who indicated moderate-to-severe symptoms on 

any of the screenings were referred to an MHP. 

322 
 

Mental health first aid. All Voces staff, including reception, case managers, and Promotoras, 

utilized MHFA to identify signs and symptoms of mental illness and intervene in crises, as 

needed. 

Motivational interviewing. MI was conducted by case managers and other staff in contact with 

clients after clients completed their initial program intakes, which now included the depression, 

trauma, and substance use screenings. MI was provided to clients who expressed hesitation or 

unwillingness to use mental health services/counseling sessions, to domestic violence survivors 

ambivalent about leaving their abuser, and to HIV testing clients expressing high-risk behaviors. 

Psychoeducation. PE was provided to clients individually and/or in groups. As part of 

individual PE, program staff provided clients information that empowered them to understand 

and manage their mental health symptoms. PE topics included symptoms and signs of trauma, 

depression, isolation, anxiety, substance use, and mental health services. The MHP Voces 

Latinas coordinator of women’s services implemented PE groups designed to address PTSD and 

substance use using the Seeking Safety curriculum. This curriculum was implemented with the 

domestic violence group. 

Referral pathway (warm handoff). If a client’s screening result indicated moderate-to-severe 

depression or trauma symptoms, Voces staff provided a warm handoff to Catholic Charities either 

on-site to MHP therapist or off-site to the Catholic Charities clinic. Also, at any point, if program 

staff felt a client was in need of more intense clinical services, they consulted with MHP staff to 

get the client to the MHP clinic for a psychiatric evaluation. MHP therapist provides care on-site 

at Voces to a limited number of Voces clients. Program staff were available to accompany clients 

to off-site appointments. Clients who did not accept referrals or other mental health services were 

given resources and offered a follow-up on their request. 

Care coordination. CC provided monthly follow-up reports to Voces staff on clients who 

were referred for mental health services. Voces and CC staff consulted during weekly case 

conferences to develop service plans for clients. Quarterly interdisciplinary meetings were held 

with Voces and CC staff to review screening results, use of C2C skills, mental health plans, 

additional social services, and other topics. 

323 
References

The Arab American Association of New York (AAANY), “Our History and Mission,” webpage, 

undated. 

https://www.arabamericanny.org/our-mission 

———, homepage, undated. 

 https://www.arabamericanny.org/home-2 

Bedford Stuyvesant Restoration Corporation, “About Us,” webpage, undated. 

https://restorationplaza.org/about-us/ 

CAMBA, “About Us,” webpage, undated. 

https://camba.org/about-us/ 

Center for Employment Opportunities, homepage, undated. 

 https://ceoworks.org/ 

CHCF, “About Us,” webpage, undated. 

https://www.chcfinc.org/about-chcf/ 

The Door, “About the Door,” webpage, undated. 

https://door.org/about-door/ 

Hetrick-Martin Institute, homepage, undated. 

https://hmi.org/ 

The HOPE Program, homepage, undated. 

https://www.thehopeprogram.org/ 

Hudson Guild, homepage, undated. 

https://hudsonguild.org/ 

NMIC, “NMIC Overview,” webpage, undated. 

https://www.nmic.org/about/overview/ 

Red Hook Initiative, homepage, undated. 

https://rhicenter.org/ 

Safe Horizon, homepage, undated. 

https://www.safehorizon.org/ 

Sheltering Arms, “About Us,” webpage, undated. 

https://shelteringarmsny.org/about-us/ 

STRIVE, “STRIVE New York,” webpage, undated. 

https://strive.org/nyc 

324 
Traumatic Stress Institute, “Risking Connection® (RC) Training,” webpage, undated. 

https://traumaticstressinstitute.org/services/risking-connection-training/about-rc-training/ 

Voces Latinas, “About Us,” webpage, undated. 

http://voceslatinas.org/about-us/ 

325 
 

Appendix B. Implementation Evaluation Methods

Michael Stephan Dunbar, Wing Yi Chan, Michele Abbott, and Vivian L. Towe 

B.1. Overview of Implementation Evaluation Design and Methods

The goals of the C2C implementation evaluation were to examine how C2C is implemented 

within and across CBOs; whether CBO staff exhibit improved knowledge, behaviors, and 

attitudes about mental health issues and services; how C2C implementation changes CBO client 

access to mental health services; and the facilitators of and barriers to C2C implementation. 

The implementation evaluation used a mixed-method approach (i.e., collecting qualitative and 

quantitative data) and prospective design (i.e., follows the same group over time and includes 

multiple assessment waves over time). Specific analytic approaches are detailed in Chapters 4 

to 6. The primary data sources for the implementation evaluation findings described in this report 

are (1) annual key informant interviews/focus groups with CBO and MHP leadership and staff, 

as well as with CBO clients who have participated in C2C; (2) annual staff surveys; and (3) CBO 

quarterly reports. 

B.2. Key Informant Interviews and Focus Groups

We conducted annual interviews with CBOs and their MHP partners and held separate key 

informant interviews/focus groups with leadership, staff, and clients. We developed separate 

interview guides for CBO leadership, MHP leadership, CBO staff, and CBO clients to collect 

qualitative information on a range of topics, including, but not limited to 

• implementation challenges and areas for improvement 

• implementation facilitators and successes 

• experiences with training on C2C supports 

• delivery of C2C supports to clients 

• collaboration between CBOs and their MHPs 

• client engagement in C2C and referrals to MHPs 

• client perspectives on C2C experience/quality/satisfaction. 

Interviews in 2017 and 2018 were conducted in person with 15 CBOs and their affiliated 

MHPs. In 2019, we conducted interviews with leaders from 14 CBOs/MHPs over the phone. 

Table B.1 provides information on the type of informant interviewed by year. CBO leaders 

included executive directors of organizations and C2C program directors, whereas MHP leaders 

were clinical directors and counselors. CBO frontline staff are defined as staff trained in and 

providing C2C skills to CBO clients. CBO clients are people who received C2C skills. In 2019, 

326 
 

 

Table B.1. Number of Key Informants Interviewed by Year and Type

Type of Informant

Year CBO Leaders MHP Leaders CBO Frontline Staff CBO Clients

2017 35 29 80 38

2018 36 26 61 35

2019 24 16 0 0

Total 95 71 141 73

phone interviews were held mainly to assess any changes in the implementation model since the 

previous round of data collection and to obtain specific information on program sustainment and 

QI practices. 

Leadership and frontline staff interviewees included organizational leaders and program 

managers, psychologists, social workers, client intake specialists, case managers, job counselors, 

and life skills instructors, among others. These individuals were involved in roles such as 

coordination and oversight of C2C supports within the CBO, supervision and training of CBO 

staff, and direct delivery of C2C supports to clients. 

Interviews were mainly conducted in the form of focus groups, with multiple interviewees 

attending per session. This format was chosen to reduce burden for participating interviewees, 

facilitate scheduling during 1- or 2-day site visits, and limit disruption to regular programming at 

CBOs and MHPs. Some interviews with CBO leaders and clients were conducted with a single 

interviewee because of availability and/or interviewee preferences. After obtaining consent from 

each interviewee, the interviews were audio recorded. If an interviewee consented to participate in 

the interview but declined to be audio recorded, the interview was not audio recorded. Interviews 

were conducted by two research staff: One led the interview, whereas the second captured as 

much of the conversation as possible in written notes in real time; audio recordings were used to 

confirm accuracy and completeness of real-time notes. 

Different interview protocols were developed for CBO leaders, MHP leaders, CBO staff, and 

CBO clients. Below are examples of questions included in each protocol: 

• CBO leaders: 

- How has C2C changed how your organization approaches client mental health or 

substance use issues? 

- Could you talk about ways in which the program has changed over the course of 

the past year? 

- Can you tell me a little bit about how the training process has gone over the past 

year? 

- How do you deliver each of the C2C modalities at your organization? 

- How well do you think the referral process is working at this stage? 

- What challenges have you encountered in working with the MHP? 

- What has worked well with respect to engaging clients in C2C? 

327 
 

 

 

• MHP leaders: 

- What has been difficult about implementing the C2C program, if anything? 

- What is your (MHP) role in supporting CBO staff in delivering each of the C2C 

modalities? 

- How are supervisors monitoring fidelity? 

- Can you tell me more about the collaboration between your organization and the 

CBO? 

- What things have been difficult in terms of keeping clients engaged once they 

have been referred to services? 

- Have you received any feedback from C2C clients who have been referred to your 

clinic on their experiences with the C2C program? 

• CBO staff: 

- How has the C2C program changed the way that your organization serves clients? 

- What has your experience been with supervisors overseeing your delivery of C2C 

modalities to clients? 

- What aspects of training/coaching/supervision have been most helpful for you? 

- When and how often do you deliver C2C modalities to clients? 

- What has been challenging about implementing the C2C modalities in your work 

with clients? 

- How do clients typically respond when offered the C2C program? 

- What else could be done to improve the MHP referral process at your organization 

to help ensure that clients in need are able to access mental health services? 

• CBO clients: 

- How you been offered any (mental health and wellness OR C2C services)? What 

specific types of services have you been offered? 

- Could you tell me a little bit more about some of your reasons for accepting or not 

accepting some of the services that you were offered? 

- Did CBO or MHP program staff help with your concerns or talk to you about any 

issues that might have initially prevented you from participating? 

- What did you think of the C2C services you received from the CBO and/or MHP? 

- What did you think of the staff who provided the C2C services? 

- If you knew someone who needed similar services, would you refer him or her to 

this program? 

B.3. Annual Staff Survey

RAND conducted the three waves of annual staff surveys in the summers (May–September) 

of 2017 to 2019 to gain a broader view on program implementation from the perspective of CBO 

program staff who were trained in at least one of the four C2C skills and provided services to 

clients. The purpose of the survey was to collect quantitative data on 

• the educational and occupational backgrounds of CBO staff participating in C2C 

• experiences with C2C training and supervision 

• attitudes toward mental health, including mental health stigma 

• organizational climate and access to resources 

328 
 

 

 

 

• helping behaviors toward clients with mental health issues 

• level of confidence in ability to deliver C2C supports. 

CBOs provided researchers with contact information for staff members who had completed 

training in at least one of the C2C supports. These individuals were invited by email to complete 

an online survey. In 2017, a total of 140 CBO staff members responded to the survey, for a 

response rate of 33 percent. A total of 252 staff members completed surveys in wave 2 (response 

rate 53 percent) and a total of 320 staff members completed surveys in wave 3 (response rate 

51 percent). CBO staff who had ever received training in any of the four core mental health skills 

and who were still actively working at the CBO with a valid email address were eligible to 

participate. Staff surveys covered topics such as staff experiences with C2C training and service 

delivery, confidence in one’s ability to administer C2C skills, knowledge about mental health 

issues, organizational climate, perceptions of the C2C program within the organization (e.g., 

organizational support for the C2C mission; communication), and staff use of specific resources 

and strategies during client interactions. 

Demographics

These questions included age, gender identity, race/ethnicity, language, education/degree, 

mental health, and substance use. 

Background

These questions included CBO/organization, occupation/job role, experience in the industry 

(less than 5 years, 5 to 10 years, more than 10 years), and job details (full-time paid employee, 

part-time paid employee, unpaid employee, intern). 

Training and Use of Skills

These items were developed by the evaluation team and assessed the following: 

• training in modalities (behavioral health screening, MHFA, MI, PE) 

• training duration (How many total hours of training for each of the C2C modalities?) 

• manual or protocol (receipt of a written training manual or protocol) 

• booster sessions/coaching (receipt of booster sessions, supervision or coaching in C2C 

modalities after initial training; type of coaching/supervision) 

• satisfaction with training (satisfied with training in C2C modalities; satisfied with 

supervision; training in C2C modalities has improved my ability to help my clients with 

mental health problems; I could use additional training, coaching, and/or supervision in 

C2C mental health modalities) 

• use of skills (With what proportion of clients have you used any of the C2C modalities?) 

• delivery of C2C modalities (How often do you provide screening, MHFA, MI, PE?) 

• confidence to administer C2C modalities (How confident are you in your ability to . . . 

administer behavioral health screening to clients? Provide PE to clients? 

• Provide MHFA to clients? Provide MI to clients?). 

329 
 

 

Time Allocated to C2C

These six items assessed the amount of time spent delivering C2C mental health supports to 

clients, the number of separate occasions (e.g., discrete service delivery sessions) in which staff 

delivered C2C supports to clients, and the amount of time spent with clients during a typical C2C 

service delivery session. 

Time Allocated to C2C

These items were developed by the evaluation team to assess time/effort devoted to C2C 

activities and included the following: 

• During the past year, how many hours per week did you typically work at this job? 

• Out of the past 12 months, how many months did you work on the C2C program? 

• In a typical week in the past month, how many hours did you spend working on C2C 

program activities? 

• In a typical week in the past month, how did you allocate your time spent on C2C 

activities? 

Delivering C2C modalities to clients: 

• screening 

• MI 

• MHFA 

• PE 

• referring C2C clients to behavioral health providers 

• participation in technical assistance from RAND Corporation/NYU McSilver, including 

email, phone, webinar, or in-person interactions with RAND Corporation or NYU 

McSilver staff 

• receiving or giving training on C2C modalities and referrals for C2C clients 

• data collection and reporting to Mayor’s Fund or C2C evaluator (RAND Corporation/ 

NYU McSilver) 

• supervising CBO staff on C2C program, coordinating with the MHP partner, and 

ensuring that the C2C program operates as expected 

• clinical supervision of C2C modality delivery 

• other activity. 

Occasions per Modality Service Delivery

If you delivered behavioral health services or made referrals in a typical week in the past 

month, on how many separate occasions did you deliver one or more of the following 

behavioral health services? 

• behavioral health screening 

• MI 

• MHFA 

• PE 

• mental health and/or substance use counseling 

330 
• coping skills or stress management training 

• case management for mental health/substance use treatment 

• referring clients to behavioral health providers 

• other. 

Time per Occasion of Modality Service Delivery

If you delivered behavioral health services or made referrals in a typical week in the past 

month, how much time (in minutes) did you spend on delivering one or more of the following 

behavioral health services on a typical occasion? 

• screening 

• MHFA 

• MI 

• PE 

• mental health and/or substance use counseling 

• coping skills or stress management training 

• case management for mental health/substance use treatment 

• referring clients to behavioral health providers 

• other. 

Stigma

Twelve stigma survey questions were adapted from the devaluation-discrimination measures 

from Link et al.’s (1989) modified labeling theory. These items assess the extent to which 

respondents believe that most people will devalue or discriminate against a person with a history 

of psychiatric treatment. Items were answered with a 6-point Likert scale from 1 (strongly 

disagree) to 6 (strongly agree). Items were summed (with six items reverse-scored [see below]) 

and averaged to generate a total score with scores ranging from 1 to 6. Higher scores indicated 

greater perceived community acceptance/lower stigma surrounding individuals with mental 

health conditions. 

1. Most people would willingly accept someone with a history of behavioral health 

problems as a close friend. 

2. Most people believe that someone with a history of behavioral health problems is just as 

intelligent as the average person. 

3. Most people believe that someone with a history of behavioral health problems is just as 

trustworthy as the average citizen. 

4. Most people would accept someone with a history of behavioral health problems as a 

teacher of young children in a public school. 

5. Most people feel that entering a treatment facility of behavioral health problems is a sign 

of personal failure. (Reverse scored) 

6. Most people would not hire someone with a history of behavioral health problems to take 

care of their children, even if he or she has been in recovery for some time. (Reverse 

scored) 

331 
7. Most people think less of someone with a history of behavioral health problems. (Reverse 

scored) 

8. Most employers will hire someone with a history of behavioral health problems if he or 

she is qualified for the job. 

9. Most employers will pass over the application of someone with a history of behavioral 

health problems in favor of another applicant. (Reverse scored) 

10. Most people in my community would treat someone with a history of behavioral health 

problems just as they would treat anyone. 

11. Most young people would be reluctant to date someone with a history of behavioral 

health problems. (Reverse scored) 

12. Once they know a person has a history of behavioral health problems, most people will 

take his or her opinions less seriously. (Reverse scored) 

Mental Health Knowledge and Attitudes

Twelve items were adapted from the Mental Health Knowledge Schedule (Evans-Lacko et 

al., 2010). This scale assesses stigma-related mental health knowledge among the general public. 

Items were answered with a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree). 

1. Most people with behavioral health problems want to have paid employment. 

2. If a friend had a behavioral health problem, I know what advice to give them to get 

professional help. 

3. Medication can be an effective treatment for people with behavioral health problems. 

4. Psychotherapy (e.g., talking therapy or counseling) can be an effective treatment for 

people with behavioral health problems. 

5. People with severe behavioral health problems can fully recover. 

6. Most people with behavioral health problems go to a health care professional to get help. 

7. Depression is a mental illness. 

8. Stress is a mental illness. 

9. Schizophrenia is a mental illness. 

10. Bipolar disorder (manic depression) is a mental illness. 

11. Drug addiction is a mental illness. 

12. Grief is a mental illness. 

Organizational Climate and Support

Twenty-five survey items assessed organizational climate and support in three domains: 

access to resources, efficacy, and organizational climate. The access to resources and efficacy 

items were adapted from the Survey of Knowledge, Attitudes, and Gatekeeper Behaviors for 

Suicide Prevention in Schools (Wyman et al., 2008; Tompkins and Witt, 2009). Organizational 

climate questions were adapted from Lehman, Greener, and Simpson (2002). Access to resources 

and efficacy items were answered with a 6-point Likert scale from 1 (strongly disagree) to 

6 (strongly agree); organizational climate items were answered with a 5-point Likert scale from 

1 (strongly disagree) to 5 (strongly agree). 

332 
 

 

 

Access to Resources

1. I have easy access to the educational or resource materials I need to learn about 

behavioral health. 

2. My organization has access to an adequate number of resources or people to whom I 

could refer clients experiencing behavioral health difficulties. 

3. I can identify the places or people where I should refer clients experiencing behavioral 

health difficulties. 

Efficacy

1. My organization encourages me to ask clients about behavioral health difficulties. 

2. I feel comfortable discussing behavioral health issues with clients. 

3. I am aware of the warning signs of behavioral health problems. 

4. I can recognize clients experiencing behavioral health difficulties by the way they 

behave. 

5. My training to assist clients who are experiencing behavioral health issues is insufficient. 

6. I do not have the necessary skills to discuss behavioral health issues with clients. 

7. I know most clients well enough to question them about behavioral health issues. 

Organizational Climate

1. Ideas and suggestions from staff get fair consideration by program management. 

2. The formal and informal communication channels here work very well. 

3. Program staff are always kept well informed. 

4. More open discussions about program issues are needed here. 

5. Staff members always feel free to ask questions and express concerns in this program. 

6. Some staff get confused about the main goals for this program. 

7. Program staff understand how this program fits as part of the treatment system in your 

community. 

8. Your duties are clearly related to the goals of this program. 

9. This program operates with clear goals and objectives. 

10. Management here has a clear plan for this program. 

11. You are under too many pressures to do your job effectively. 

12. Staff members often show signs of stress and strain. 

13. The heavy workload here reduces program effectiveness. 

14. Staff frustration is common here. 

15. Frequent staff turnover is a problem for this program. 

Staff Behaviors and Perceptions

Additional items assessed staff intervention behaviors, important factors in referral 

decisions, and perceptions of clients’ comfort with discussing mental health–related issues. The 

items assessing staff intervention behaviors were adapted from a survey of knowledge, attitudes, 

and gatekeeper behaviors for suicide prevention in schools. This survey was used to evaluate a 

gatekeeper training for university resident advisers aiming to improve detection and referral 

of at-risk students (Tompkins and Witt, 2009; Shaffer et al., 1991). Items were answered with a 

5-point Likert scale from 1 (never) to 5 (always), with 0 = not applicable. For important factors 

333 
 

 

in referral decisions, items were answered with a 6-point Likert scale from 1 (not at all important) 

to 6 (very much important) and were developed by the RAND evaluation team. Items assessing 

staff perceptions of clients’ comfort in discussing behavioral health-related concerns (also 

developed by the RAND evaluation team) were answered with a 5-point Likert scale from 

1 (strongly disagree) to 5 (strongly agree). Items assessing perceived client barriers to accepting 

referrals were adapted from existing survey items used in National Comorbidity Survey 

Replication study (Mojtabai et al., 2011). 

Intervention Behaviors

When you suspect a client is experiencing behavioral health issues, how often do you 

1. ask the client about behavioral health issues 

2. convince the client to seek help 

3. escort a client to a counselor or other resource 

4. get advice from a coworker 

5. keep it a secret 

6. leave him/her alone until he/she feels better 

7. refer him/her to a behavioral health care provider (e.g., a psychologist) 

8. provide the client with information (e.g., how to get help for behavioral health issues; 

education about behavioral health) 

9. spend time listening to the client 

10. talk to a supervisor about the client 

11. other. 

Important Factors in Referral Decisions

Please rate how important each of the following factors are to you in deciding where to refer 

a client for behavioral health services: 

1. Confidentiality 

2. Convenience 

3. Length of time expected to get an appointment 

4. Policies of your organization 

5. Quality of care 

6. Other. 

Client Comfort

Please rate how much you agree or disagree with the following statements: 

1. My clients do or would feel comfortable talking about their behavioral health issues 

with me. 

2. My clients do or would feel comfortable accepting a referral to a behavioral health 

provider inside my organization. 

3. My clients do or would feel comfortable accepting a referral to a behavioral health 

provider outside my organization (i.e., an external referral). 

334 
 

 

 

 

 

 

 

 

 

 

Perceived Client Referral Barriers

Which of these statements explains why your clients do not accept behavioral health 

referrals? (Check all that apply) 

• cannot afford the cost 

• concerned that getting behavioral health services might cause friends, family, or 

community to have a negative opinion of them 

• concerned that getting behavioral health services might have a negative effect on their job 

• health insurance does not cover any behavioral health services 

• health insurance does not pay enough for behavioral health services 

• do not feel comfortable leaving the CBO to get behavioral health services 

• concerned that the information they give the counselor might not be kept confidential 

• concerned that they might be committed to a psychiatric hospital or might have to take 

medicine 

• too difficult to schedule an appointment for behavioral health services 

• too difficult to get time off work 

• too difficult to arrange transportation 

• too difficult to arrange for childcare 

• too difficult to find a behavioral health provider who is culturally/linguistically 

competent 

• other (specify): _________________. 

B.4. CBO Quarterly Report Data

CBOs submitted quarterly report data to the C2C Collaborative once per quarter between 

March 2016 and September 2019. Reports included aggregate data on a range of implementation 

metrics, including staff training, supervision and coaching activities, delivery of C2C supports to 

clients, and client referrals (Table B.2). Reports were reviewed for completeness and accuracy by 

the C2C Collaborative as well as the RAND evaluation team. CBOs worked closely with the 

C2C Collaborative to resolve any errors in data entry. 

Table B.2. Quarterly Progress Report Data Elements

Evaluation Area Data Element

Staff training • Number of people currently employed by CBO

• Number of CBO staff

• Number of staff trained in at least one modality

• Number of staff trained in each modality

• Number of staff trained in all four modalities

• Number of CBO supervisors

• Number of supervisors trained

• Number of supervisors trained in at least one modality

• Number of supervisors trained in each modality

• Number of supervisors trained in all four modalities

• Total (staff and supervisors) trained for the first time

335 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Evaluation Area Data Element

• Number (percenta) of current staff and supervisors trained in at least one modality

• Number (percenta) of current staff and supervisors trained in each modality

• Number of training hours by CBO staff

• Number of training hours by MHP

Staff coaching • Number of staff who received continuous coaching and supervision

• Number of supervisors who received continuous coaching and supervision

• Total (staff and supervisors) who received continuous coaching and supervision

• Number of coaching and supervision hours provided by MHP and modality

• Number of coaching and supervision hours provided by CBO and modality

Client receipt of • Number of CBO program participants

services • Number of new C2C participants

• Number (percenta

) of program participants who received a C2C modality

• Number of program participants who received screening

• Number of program participants who screened positive

• Number of program participants who received PE

• Number of program participants who were referred to the MHP partner

• Number of program participants who accepted (declined) a referral to the MHP partner

• Number (percenta

) of program participants who completed a referral at the MHP partner

• Number of program participants who were referred to an external mental or behavioral

health provider

• Number of program participants who accepted (declined) a referral to an external

mental or behavioral health provider

• Number (percenta

) of program participants who completed a referral at the external

provider

• Number of participants who screened positive and were referred to the MHP or an 

external provider

Client • Gender

demographics • Age

• Race/ethnicity

• Education status

• Employment status

a Data elements were calculated based on CBO-reported data.

336 
 

 

References

Evans-Lacko, S., K. Little, H. Meltzer, D. Rose, D. Rhydderch, C. Henderson, and G. Thornicroft, 

“Development and Psychometric Properties of the Mental Health Knowledge Schedule,” 

The Canadian Journal of Psychiatry, Vol. 55, No. 7, 2010, 440–448. 

Lehman, W. E., J. M. Greener, and D. D. Simpson, “Assessing Organizational Readiness for 

Change,” Journal of Substance Abuse Treatment, Vol. 22, No. 4, 2002, 197–209. 

Link, B. G., F. T. Cullen, A. Struening, P. E. Shrout, and B. P. Dohrenwend, “A Modified 

Labelling Theory Approach to Mental Disorders: An Empirical Assessment,” American 

Sociological Review, Vol. 54, No. 3, 1989, pp. 400–423. 

Mojtabai, R., M. Olfson, N. A. Sampson, R. Jin, B. Druss, P. S. Wang, K. B. Wells, H. A. Pincus, 

and R. C. Kessler, “Barriers to Mental Health Treatment: Results from the National Comorbidity 

Survey Replication,” Psychological Medicine, Vol. 41, No. 8, 2011, pp. 1751–1761. 

Shaffer, D., A. N. N. Garland, V. Vieland, M. Underwood, M., and C. Busner, “The Impact of 

Curriculum-Based Suicide Prevention Programs for Teenagers,” Journal of the American 

Academy of Child & Adolescent Psychiatry, Vol. 30, No. 4, 1991, pp. 588–596. 

Tompkins, T. L., and J. Witt, “The Short-Term Effectiveness of a Suicide Prevention Gatekeeper 

Training Program in a College Setting with Residence Life Advisers,” The Journal of 

Primary Prevention, Vol. 30, No. 2, 2009, pp. 131–149. 

Wyman, P. A., C. H. Brown, J. Inman, W. Cross, K. Schmeelk-Cone, J. Guo, and J. B. Pena, 

“Randomized Trial of a Gatekeeper Program for Suicide Prevention: 1-Year Impact on 

Secondary School Staff,” Journal of Consulting and Clinical Psychology, Vol. 76, No. 1, 

2008, p. 104. 

337 
 

 

 

 

 

Appendix C. Impact Evaluation Methods

Dana Schultz, Daniel Siconolfi, Lynsay Ayer, Joshua Snoke, Elie Ohana, and Emily Hoch 

The impact evaluation tested the extent to which C2C improves participants’ access to 

mental health care and well-being on a variety of dimensions by comparing C2C clients with 

clients who received similar services, but not C2C services, from similar local CBOs. The impact 

evaluation used a quasi-experimental design, with longitudinal assessments administered at 

baseline, 6 months, and 12 months via a web-based data collection system and NYS and NYC 

agency administrative data. CBO program records were also utilized for participants in the C2C 

intervention arm. In this appendix, we describe the overall impact evaluation design (Section C1), 

client survey data collection (Section C2), CBO program data collection (Section C3), NYS 

and NYC administrative data collection (Section C4), development of our analytic database 

(Section C5), and our analytic approach (Section C6). We then provide details on study enrollment, 

retention (Section C7), and sample characteristics and results from subgroup analysis not included 

in the main body of the report (Section C8). 

C1. Overview of Impact Evaluation Design

The quasi-experimental design allowed for comparisons between the intervention group 

(C2C group) and a similar (comparison group) population without randomization of CBO clients 

to intervention or control conditions. With this design, we were able to examine the effect of 

C2C on participants at two levels: (1) pooled effect of C2C across all intervention participants 

compared with all comparison participants and (2) the effect of C2C on specific subgroups of 

participants, including the three C2C target groups (e.g., underemployed or unemployed adults, 

unemployed/out-of-school youth, caregivers of children ages 0–4) and on participants receiving 

different types of CBO services (e.g., job training and employment, youth development). We did 

not estimate intervention effects at each CBO individually, because the expected (and actual) 

sample sizes at many CBOs would provide insufficient statistical power to detect the expected 

small treatment effect sizes. 

C2C was expected to serve more than 8,000 individuals per year. However, not all clients 

served were included in the impact evaluation. For each group (C2C and comparison), the target 

number of individuals who needed to be screened for study eligibility was approximately 3,125 

over the study enrollment period. In most C2C CBOs, screening was the first C2C skill offered 

and delivered to clients. We assumed a 50-percent eligibility rate (i.e., have at least mild symptoms 

on one or more of the five mental health screeners [depression, anxiety, PTSD, alcohol use, drug 

use]), based on preliminary data from the C2C CBOs, the underserved nature of the C2C client 

338 
 

populations (e.g., homeless, domestic violence victims, recently incarcerated, etc.), and 

epidemiological data showing that approximately 44 percent of the general U.S. population 

ages 18 to 29 (a lower-risk population and higher threshold than the one used here for study 

eligibility) met full diagnostic criteria for a mood, anxiety, or substance disorder in the past 

12 months (see National Comorbidity Survey, undated). Thus, the number of individuals 

expected to be study eligible was 1,563. In both groups, we assumed that there would be some 

attrition between the screening and the completion of the baseline assessment. Assuming that 

80 percent of those who screened positive would consent to the study and complete the baseline 

assessment, target study enrollment (i.e., completion of the baseline assessment) for each group 

was about 1,250. Although evaluation best practices originally led us to target an 80-percent 

retention rate for the follow-up surveys, our sample of individuals seeking services at the 

participating CBOs had very low incomes, high unemployment, unstable housing, and recent 

incarceration. Other studies of underserved populations have had a diverse range of retention 

rates (Altena, Brilleslijper-Kater, and Wolf, 2010; Davis, Broome, and Cox, 2002; Leonard et al., 

2003; Teague et al., 2018) suggesting more realistic benchmarks below 80 percent. Nonetheless 

with an 80-percent retention rate, we anticipated needing a sample size of 1,000 in each group at 

each follow-up to detect a small effect size (d ≈ 0.20). 

C1.1. C2C and Comparison Group CBOs

Thirteen of the 15 C2C CBOs participated in the impact evaluation (see Appendix A for a 

description of each C2C CBO). The impact evaluation conducted analyses of groups of CBOs 

(i.e., it did not examine findings at the individual CBO level), under the assumption that the 

interventions across C2C CBOs were similar to each other. Given this analytic assumption, it 

was not possible to include two organizations in the impact component of the evaluation. One 

CBO had contractual problems with its MHP that led to an extended delay to its implementation; 

the other launched implementation with an intervention model that was substantially different 

from the other CBOs in C2C. 

We successfully recruited ten comparison group CBOs to participate in the impact evaluation 

(Table C.1). Comparison group participants were recruited from CBOs in NYC that served 

similar populations (e.g., geographic location, race/ethnicity, primary language, age) and 

provided similar services (e.g., job training and employment, youth development, homeless 

shelter, domestic violence shelter, immigrant-serving organization, parent/caregiver-serving 

organization) but were not implementing C2C or providing other mental health services. 

Initially, we approached potential comparison group CBOs suggested by C2C CBOs and the 

Mayor’s Fund via email to explain the study and request a meeting. During the initial meeting, 

we provided a memo that described the study and detailed the roles and responsibilities of the 

comparison CBO and RAND related to developing data collection plans, collecting data, 

tracking study participants, and monitoring study recruitment and retention at each comparison 

group CBO. Once the CBO agreed to participate, we initiated a subcontract to compensate the 

339 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

CBO for the time, effort, and resources spent participating in the study. We also conducted 

several follow-up meetings to further develop plans for data collection at each comparison 

group CBO. Prior to beginning data collection, RAND presented a brief informational session 

to staff at each comparison CBO. The presentation sought to raise staff awareness of the 

opportunity for clients to participate in the study. Booster informational sessions were provided 

on request or if a CBO indicated significant staff turnover since its original session. RAND data 

collectors conducted all study eligibility screenings and baseline surveys at the comparison CBOs 

(see Data Collection Procedures section below). 

Table C.1. Comparison Group CBOs

CBO Location Type of Services

Goodwill Industries One site in Bronx Job training and employment program that offers job training 

and employment support to residents of neighboring housing 

developments

Eastside Settlement

House

One site in Bronx Job training and employment program that offers employment

activities, financial services, and educational services

St. Nick’s Alliance One site in Brooklyn Job training and employment program that partners with local

employers for job coaching and training

Cypress Hills One site in Brooklyn Youth job training and internship program

Eckerd Connects Two sites (Bronx, 

Queens)

Youth college or career preparation program that offers

education and job training

Mosholu Montefiore 

Community Center

Two sites in Bronx Parent/caregiver-serving organization that offers free and lowcost services to children younger than 5 and a youth 

development program that offers job training

Sanctuary for Families One site in Bronx and 

one in Manhattan

Shelter program that offers transitional housing as well as case 

management and housing assistance, individual and group 

counseling, children’s services, recreational activities, aftercare

support, and on-site services

Violence Intervention 

Program

One site in Queens Counseling, advocacy, legal, and housing support program for

victims of domestic violence

Children’s Aid Society One site in Bronx and 

three in Manhattan

Parent/caregiver-serving organization that offers caregiver

engagement

CAMBA King’s Inn One site in Queens Temporary family homeless shelter that offers crisis 

intervention, case management, and transition to permanent

housing

For both groups, CBOs were categorized by their primary offering or organizational type: job 

training and employment, youth development, parent/caregiver-serving organization, homeless 

shelters, domestic violence organizations, and agencies specializing in services for specific 

populations. The number of CBOs in each category is detailed in Table C.2. Many of the CBOs, 

both C2C and comparison, had multiple locations involved in the study. Across the 13 C2C 

CBOs, clients were enrolled at 24 locations. At the ten comparison CBOs, data were collected at 

17 locations. 

340 
 

 

 

 

 

 

 

 

Table C.2. CBO Information

CBO Service Type C2C CBOs Comparison CBOs

Job training and employment 5 3a

Youth development 3 3a

Parent/caregiver-serving organizations 2 1a

Homeless shelter 1 1

Domestic violence organization 1 1

Agency serving a specific population 1 1

a One comparison CBO recruited study participants from both a job training and 

employment program and a youth development program; one comparison CBO

recruited study participants from both a job training and employment program and 

a parent/caregiver-serving organization.

C1.2. Power Analysis

For the impact evaluation, our target retained sample size was 2,000 participants, with equal 

numbers coming from the treated and comparison groups (i.e., 1,000 per group). The study was 

designed to yield good power to detect effect sizes that are conventionally considered small 

(Cohen’s d = 0.2). This more conservative approach would give us a higher level of confidence 

about whether C2C had at least a small effect on participants. Further, we would expect only 

small to very small effect sizes with a community-based, non-intensive intervention such as 

C2C. To estimate the average treatment effect on participants, we applied propensity score 

weights to individuals in the comparison group. This weighting process decreased the “effective 

sample size” somewhat, which can make the comparison sample statistically behave as though it 

were smaller in size than its nominal value. 

Table C.3 shows the minimum detectable effect for a range of sample sizes, assuming 

80-percent power and a 95-percent level of significance. We assumed that the effective sample 

size for the comparison sample would be half as large as the nominal sample size, which 

Table C.3. Minimum Detectable Effect Size with 80-Percent Power, Accounting for Propensity 

Score Adjustments

Number Treated Minimum Detectable Effect Size

50 0.70

100 0.49

200 0.35

300 0.29

400 0.25

500 0.22

600 0.20

800 0.19

900 0.18

1,000 0.16

341 
 

experience suggested would be typical for propensity score applications of this type. Hence, our 

power calculations assumed that the analytic sample size for the comparison group would be half 

as large as the nominal value. The power analysis indicated that with a sample size of 1,000 per 

group, we would have the power to detect a small effect size of 0.16, which would be common 

for nonintensive mental health interventions. 

C2. Client Survey Data Collection

Study enrollment for the impact evaluation began in June 2017 and continued through 

March 2019. The primary inclusion criterion for the impact evaluation was meeting a minimum 

threshold on one or more of the eligibility screening measures for the common mental health 

conditions detailed below. We set eligibility thresholds lower than clinical thresholds (“cut 

points”) typically used to inform further screening diagnosis to be inclusive of persons with 

subclinical levels of mental health symptoms in addition to those with clinically significant 

symptoms. All eligibility screeners were conducted in person. 

The baseline assessment was conducted after the eligibility screener and included measures 

of functioning, trauma history, barriers to accessing mental health care, and mental health service 

utilization. The follow-up assessment repeated the administration of the screening and baseline 

assessment measures in addition to asking questions about any services or supports received 

from CBO staff related to mood, thinking, or behavior. The baseline and follow-up surveys 

were conducted in person or via telephone in Spanish or English, with data collection continuing 

through February 2020. 

The following subsections provide more details on client survey measures, study eligibility, 

data collection procedures, and enrollment and data collection timelines. 

C2.1. Client Survey Measures

To assess individual-level outcomes, we identified a standardized set of measures to capture 

background and contextual characteristics, and a broad array of outcomes domains, including 

depression, anxiety, PTSD, alcohol use, and substance use as well as functioning, barriers to 

mental health care, mental health service utilization, services or supports received from the CBO, 

and exposure to C2C (for C2C group). In selecting measures, our goal was to identify measures 

meeting the following criteria: 

• To accommodate budgetary and staffing constraints, lay interviewers, not just highly 

trained clinicians, could administer the measure. 

• To maximize credibility with CBOs and the ultimate audience for the evaluation results, 

the measure had been widely accepted and widely used in the field. 

• The measure was brief so that the burden on participants could be minimized. 

• Measures that had demonstrated sensitivity to change in prior intervention studies were 

prioritized because of the importance of being able to detect changes in the current 

evaluation. 

342 
 

 

 

 

• The measure had good reliability and validity in similar samples. 

• The measure was available in Spanish, or translation was feasible. 

• For screeners, we selected measures that assessed common types of mental health 

problems and prioritized measures that were already being used by C2C CBOs to screen 

clients to minimize client and staff burden. 

The measure selection process involved identifying potential measures in each domain and 

then reviewing the measures to determine whether they met the criteria outlined above. Once we 

selected measures for each domain, we provided the Mayor’s Fund and the C2C CBOs with a 

memo that described the source and content of each recommended measure. After reviewing 

stakeholder feedback, we finalized the list of measures. 

Screening Measures

The study eligibility screening survey, the 6-month survey, and the 12-month survey 

included the following mental health symptom measures. 

PHQ-8. The PHQ-8 is an eight-item self-report measure of current depression that asks 

respondents to indicate how often they have been bothered by each symptom of depression in the 

previous 2 weeks. The measure can be used in clinical samples and in the general population. 

Items are coded 0 to 3, and scores are summed to generate a total score, ranging from 0 to 24, 

with higher scores indicating higher depressive symptoms. Scores greater than or equal to 10 

indicate moderate depression. Prior research using the PHQ-8 provides evidence of good internal 

consistency, with Cronbach’s alphas ranging from 0.86 to 0.89 and good sensitivity and 

specificity for identifying cases with diagnoses of major depressive disorder (Kroenke et al., 

2010). In the present study, the Cronbach’s alpha was 0.78 at baseline, 0.83 at 6 months, and 

0.83 at 12 months. 

GAD-7. The GAD-7 is a seven-item self-report measure of current anxiety that asks 

respondents to indicate how often they have been bothered by each symptom of anxiety in 

the previous 2 weeks. Scores range from 0 to 21 with scores of 0 to 4 considered minimal, 

5 to 10 mild, 10 to 14 moderate, and 15 to 21 severe. The measure has been used in a wide 

range of populations and can be used in clinical samples and in the general population to screen 

and monitor (over repeated assessments) for symptoms of anxiety. The GAD-7 has been 

demonstrated to have strong internal consistency (α = 0.89) and good sensitivity and specificity 

for identifying cases with diagnosis of GAD (Löwe et al., 2008; Spitzer et al., 2006). In the 

present study, the Cronbach’s alpha was 0.84 at baseline, 0.88 at 6 months, and 0.88 at 12 months. 

PCL-5. The PCL-5 is a 20-item self-report measure of PTSD symptoms designed for use in 

primary care populations and has been used in a wide range of demographic groups among 

individuals ages 18 and older. Scores range from 0 to 80. Cut scores for predicting PTSD 

diagnostic status have not yet been established for the PCL-5 and range from 30 to 60 depending 

on the population, setting, and assessment goal (Blevins et al., 2015). A cut point of 31 to 33 is 

suggestive of PTSD diagnosis (National Center for PTSD, 2019). A provisional Diagnostic and 

343 
 

 

 

 

 

 

 

 

Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) PTSD diagnosis may be obtained 

by considering items rated moderately or higher as symptoms endorsed and following the DSM-5 

diagnostic rule (at least one B, one C, two D, and two E symptoms present) (Blevins et al., 2015). 

Prior research shows that this measure demonstrates excellent internal consistency (α = 0.94), 

correlates highly with other scales of PTSD symptoms (r > 0.84), and has good sensitivity and 

specificity for identifying cases with PTSD (Blevins et al., 2015). In the present study, the 

Cronbach’s alpha was 0.93 at baseline, 0.94 at 6 months, and 0.95 at 12 months. 

AUDIT-10. The AUDIT-10 is a ten-item self-report scale used to identify risky or harmful 

alcohol consumption as well as alcohol use disorders, and to assess the amount and frequency of 

alcohol intake, alcohol dependence, and problems related to alcohol consumption. Scores range 

from 0 to 40. The cutoff point for potentially hazardous alcohol intake is 8, although the cutoff 

score may differ depending on sex (i.e., men and women have different cutoff points). In prior 

research, the AUDIT-10 has been shown to have good sensitivity and specificity for identifying 

cases with alcohol use disorders (Babor et al., 1992; Berner et al., 2007). In the present study, the 

Cronbach’s alpha was 0.89 at baseline, 0.86 at 6 months, and 0.86 at 12 months. 

DAST-10. The DAST-10 is a self-report measure of problematic substance use that is 

utilized for clinical screening and research. The ten-item instrument has been used in a wide 

variety of populations, including adolescents and adults in a range of demographic groups. 

Scores range from 0 to 10. Scores greater than or equal to 3 indicate moderate substance use. 

The DAST-10 has strong internal consistency (α = 0.74–0.92) and good test-retest reliability 

(r = 0.71), and correlates very highly with the longer DAST-20 instrument (r = 0.98). The scale 

also demonstrates good sensitivity and specificity for identifying cases with substance use/misuse 

disorders (Yudko, Lozhkina, and Fouts, 2007). In the present study, the Cronbach’s alpha was 

0.83 at baseline, 0.86 at 6 months, and 0.85 at 12 months. 

Baseline and Follow-Up Assessment Measures1 

Kessler 6. The Kessler-6 is a widely used measure of general psychological distress. The 

scale has six items and has been validated in primary care clinics, community mental health 

centers, and social welfare offices. Scores range from 0 to 24. Prior research shows that this 

measure demonstrates good test-retest reliability and convergent validity (Kessler et al., 2003). 

In the present study, the Cronbach’s alpha was 0.85 at baseline, 0.86 at 6 months, and 0.87 at 

12 months. 

BACEv2. The 30-item BACEv2 is used as a measure of a range of different barriers to 

accessing mental health care in community populations and can be used to assess change in 

1 The client surveys (baseline, 6 months, and 12 months) that originally included additional items or measures (i.e., 

World Health Organization Disability Assessment Schedule, stressful events questions from the Life Experiences 

Survey, and the Attitudes Toward Seeking Professional Psychological Help Scale-Short Form) were removed in 

December of 2018 to streamline the surveys and further reduce participant burden. 

344 
 

 

 

 

 

barriers to care after the introduction of community interventions to increase care-seeking and 

service use. Prior research provides evidence of good test-retest reliability, and good convergent 

and construct validity (Clement et al., 2012). The BACEv2 has three subscales relevant to C2C. 

These include logistical barriers (e.g., cost), attitudinal barriers that are not related to stigma 

(e.g., concern about medication side-effects), and stigma-related barriers (e.g., concern about 

how others will react). In addition, the stigma barriers scale includes a single item assessing 

internalized stigma (feeling embarrassed or ashamed), which we also examined as an outcome 

given the salience of internalized stigma and shame as a barrier to mental health care. 

We examined the strength of barrier endorsement using subscale averages because this 

approach allows for a wider continuum of potential change. The BACE can also be scored using 

dichotomized responses (e.g., summing the number of barriers endorsed “This has stopped, 

delayed or discouraged me A LOT”); however, this approach discards potentially meaningful 

heterogeneity captured by the continuum of the 4-point Likert-type response scale. Computing 

subscale averages allows for a much greater continuum of change. In the present study, the 

Cronbach’s alpha for logistical barriers was 0.79 at baseline, 0.81 at 6 months, and 0.82 at 

12 months; for attitudinal barriers, it was 0.76 at baseline, 0.86 at 6 months, and 0.86 at 12 months; 

and for stigma-related barriers, it was 0.91 at baseline, 0.93 at 6 months, and 0.93 at 12 months. 

Mental health service utilization questions from the CPIC evaluation. The utilization 

questions were previously utilized in an evaluation of community engagement versus technical 

assistance approaches to implement depression collaborative care in underserved communities 

in Los Angeles (Chung et al., 2014). Eight survey items assessing mental health service 

utilization and one item assessing emergency care utilization were administered to C2C 

participants. At baseline, participants indicated whether they had ever stayed overnight in a 

hospital for emotional, mental health, alcohol, or drug problems (with recall truncated to the past 

6 months at the 6- and 12-month follow-ups). Participants also reported binary utilization of five 

additional indicators: whether they had attended any self-help or family support groups for 

people with emotional or mental health problems (excluding Alcoholics Anonymous-, Cocaine 

Anonymous-, or Narcotics Anonymous-type programs); had gone to any substance abuse 

agencies that have programs for people with drug or alcohol use problems or attend any self-help 

meetings such as AA, CA, or NA; had called a hotline for problems with emotions or nerves, 

mental, alcohol, or drug problems; had gone to any religious or spiritual places such as a church, 

mosque, temple, or synagogue for emotional, mental health, alcohol, or drug problems; and had 

gone to any parks and recreation or community centers for emotional, mental health, alcohol, or 

drug problems. 

In addition, participants reported continuous indicators of utilization including the number of 

nights they had stayed overnight in a residential treatment program for alcohol or drug problems; 

the number of times they had gone to a hospital emergency room or an urgent care facility for 

any health reason; and the number of times they had gone to any MHP, including psychiatrists, 

psychologists, social workers, psychiatric nurses, or counselors. In addition to treating these as 

345 
 

 

 

continuous indicators, we also created binary variables indicating any utilization during the recall 

period for these three originally continuous items. 

For analysis of mental health care utilization, we combined data in two ways. First, sources 

of care were collapsed into four types of settings: clinical outpatient settings (clinical outpatient 

services); nonclinical settings and sources (self-help or family support groups, substance use 

agencies, or 12-step-type programs, calling a hotline, spiritual/religious places for mental health 

services, or parks/recreation centers for mental health services); inpatient settings (residential 

drug/alcohol treatment, or staying overnight in a hospital for mental health or substance use 

needs); and emergency settings (hospital, emergency room, or urgent care facility for any 

reason). Second, these utilization data from 6- and 12-month follow-up surveys were pooled into 

1-year indicators of any use or frequency of use. 

For the pooled 1-year utilization outcomes in the main analysis comparing the overall C2C 

group to the overall comparison group, imputation was performed for each individual for all 

missing data at either time point. Fully conditional specification, using the R package mice, was 

used to create ten imputations across all variables. The imputed outcome variables were then 

collapsed to create pooled outcomes for the year. We did not impute data for the subgroup 

differences (e.g., among job training and employment program clients) due to smaller sample 

sizes for these secondary analyses. 

Employment outcomes. The client survey assessed three employment-related indicators that 

were examined as outcomes of C2C: full-/part-time employment versus unemployment, typical 

hours worked per week, and monthly employment income before deductions and taxes. 

First, employment status was assessed using a single item (“Are you currently employed fulltime, part-time, unemployed, retired and not working, a student, a homemaker, or are you 

disabled or too ill to work?”). Response choices included these categories as well as an “Other” 

category and a free-text response field. Responses that could be recoded into the existing 

categories were recoded as such; other responses that were uninterpretable remained coded as 

“other” and were not utilized in analyses (n = 9, 4, 7, at baseline, 6 months, and 12 months, 

respectively). For analysis of change in the outcome of full-/part-time employment, only persons 

with an employment status of full-time, part-time, or unemployed at baseline and the follow-up 

time point were included in analysis of this outcome, because changes into or out of the other 

categories (retired and not working, student, homemaker, disabled) could not be definitively 

interpreted as positive or negative change without additional context. For the intervention 

outcome derived from this measure, we focused on the aggregate binary indicator of employed 

full-/part-time versus unemployed. 

Second, participants who indicated full-time or part-time employment were asked to report 

the number of hours in a typical workweek (“At your current job [or at all current jobs combined], 

how many hours do you typically work during a week, from Sunday to Saturday?”). For our 

analysis of hours worked per week as an employment indicator, unemployed persons were coded 

as zero hours because they did not answer this question due to survey routing logic. Third, 

346 
 

 

 

 

monthly pay before taxes and deductions was derived from several questions. First, persons who 

were not currently full-/part-time employed at the survey time point were coded as zero because 

they did not answer this question due to survey routing logic. For participants with full- or parttime employment, their standardized monthly pay was derived from two additional questions. 

Participants reported the frequency of their pay (“Now think about the pay you receive from your 

current job [or all current jobs combined], what is the easiest way for you to report total pay 

BEFORE taxes and other deductions?”: Hourly, Weekly, Biweekly; Monthly; Annually; Other). 

Participants also reported their typical pay, at the interval selected in the previous question 

(“At your current job [or at all current jobs combined], how much do you earn [pay frequency] 

BEFORE taxes and other deductions?”). This amount was multiplied or divided by the frequency 

of pay to generate an estimate of monthly employment pay for each participant, which was 

examined as a C2C outcome. In exploratory analyses, we identified several respondents with 

unusually high incomes that were likely data entry errors (e.g., hourly pay rates resulting in 

monthly incomes in excess of $15,000/month). We coded these values as missing for analysis 

(n = 8, 6, and 2 at baseline, 6 months, and 12 months). 

Housing outcomes. From the client survey, we derived two indicators of housing status: 

current housing and recent homelessness. First, current housing was assessed with a single item 

(“Where do you currently live? [or probe if needed: what type of housing is the place you usually 

live and sleep?]”). Response categories were grouped for analysis into four categories: stably 

housed (apartment or house that you own; room, apartment, or house that you rent; permanent 

supportive housing for formerly homeless persons; group home or other supervised residential 

care facility); staying with someone else (in a friend’s or family member’s room, apartment, or 

house); transitional/temporary housing (transitional housing for homeless persons, psychiatric 

hospital or other psychiatric facility, substance abuse treatment facility or other detox facility; 

hospital (nonpsychiatric); jail, prison, or juvenile detention facility; half-way or three-quarterway home for persons with criminal offenses; foster care home or foster care group home); and 

homeless/unstably housed (emergency shelter, including hotel or motel voucher paid for by a 

social service or charitable organization; hotel or motel paid for without emergency shelter 

voucher; place not meant for human habitation [street, car, park, etc.]). Additional response 

choices included Other, Don’t know, and Refusal. All responses to Other at baseline and 6 months 

were able to be recoded into existing categories. We retained the stand-alone staying with someone 

else category, rather than collapsing it into the transitional/temporary category because of the 

very large portion of persons in this group at baseline and because the question/answer choice 

wording for this category could not differentiate between truly unstable/transitional housing 

(e.g., couch surfing) versus long-term de facto residency with family members who owned the 

unit or were responsible for rent. For the intervention outcome derived from this measure, we 

focused on the aggregate binary indicator of stably housed or staying with someone else versus 

transitional/temporary housing, unstable housing, or homelessness. 

347 
 

 

 

 

 

 

Second, recent homelessness was assessed with a single item (“During the last [#] months or 

since [date], did you have a period of time when you did not have a place to sleep at night?”). 

The baseline survey specified a recall of the past 12 months, whereas the 6- and 12-month 

surveys each specified the past 6 months. To permit comparisons given these discrepant recall 

periods, we pooled the 6- and 12-month responses, such that an affirmative response in either the 

6- or 12-month surveys was coded as any homelessness in the past year. For the pooled 1-year 

outcome in the main analysis comparing the overall C2C group to the overall comparison group, 

imputation was performed for each individual for all missing data at either time point (see 

description of Mental Health Utilization measures). 

Education outcomes. From the survey, we derived two outcomes related to education: 

completion of high school or a GED and having begun or completed additional education beyond 

their baseline level. At each survey time point, participants were asked, “What is the highest 

grade in school or year of college that you have completed? (Less than high school, completed 

high school or GED, some college, completed college, some graduate or professional school, 

completed graduate or professional school).” For the intervention outcome derived from this 

measure, we focused on the aggregate binary variable indicating completed high school diploma, 

GED, or greater, versus not having completed high school or a GED. 

Incarceration outcomes. The survey assessed a single outcome related to criminal justice 

involvement: recent incarceration. Participants were asked, “During the last 12 months or since 

[date], did you spend time in a correctional facility, such as a jail or prison?” At 6- and 12-month 

follow-up, the recall period was 6 months. To permit comparisons given these discrepant recall 

periods, we pooled the 6- and 12-month responses, such that an affirmative response in either 

the 6- or 12-month surveys was coded as incarceration in the past year. For the pooled 1-year 

outcome in the main analysis comparing the overall C2C group to the overall comparison group, 

imputation was performed for each individual for all missing data at either time point (see 

description of Mental Health Utilization measures). 

Services or supports received from the CBO. On the 6- and 12-month follow-up surveys, 

respondents were asked about different services or supports they might have received from the 

CBO, including vocational training or job readiness classes, adult education or GED classes, ESL 

classes, afterschool programs, academic support or GED classes (e.g., tutoring), postsecondary 

support services (e.g., college counseling and prep), housing assistance and/or income support, 

parenting programs, legal services, support groups, immigration service, health and wellness 

education and services, community organizing, community organizing, HIV/acquired 

immunodeficiency syndrome services, and case management. These variables were included 

for descriptive purposes only, to better characterize the sample. 

Exposure to C2C skills. On the 6- and 12-month follow-up surveys, participants in both 

groups were asked a few questions to assess their level of exposure to the different C2C skills, 

including screening (“Did staff ask you a series of questions about any signs and symptoms 

you may have been experiencing related to your mood, thinking, or behavior?”), MI (“Did staff 

348 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

talk with you about ways to change any specific aspects of your mood or behavior (for example, 

feeling less anxious or down, quitting smoking or drinking)?”), PE (“Did you attend a session 

or receive information from staff about any problems you may have with mood, thinking or 

behavior, and what solutions might help you overcome them?”), and referrals for mental health 

services (“Did staff refer you to a social worker, mental health/substance use counselor, or other 

provider to get help with your mood, thinking, or behavior?). These items were designed to 

reflect the core of each C2C skill, using layman’s terminology rather than the formal skill names. 

For the items reflecting the C2C skills of screening, PE, and MI, participants who reported 

receiving a given skill were asked an additional question regarding the number of times that they 

had received that skill in the past 6 months (one time, two to three times, four or more times). 

Participants who reported that they had received a referral were asked a follow-up question 

regarding the number of times they met with a social worker, mental health/substance use 

counselor, or other provider (zero times, one time, two to five times, six to ten times, 11 or 

more times). 

C2.2. Study Eligibility

The primary inclusion criterion for the impact evaluation was a minimum symptom threshold 

for one or more of the following mental health conditions: depression, anxiety, PTSD, alcohol 

use, or substance use. The study eligibility criteria are shown in Table C.4, along with published 

scoring criteria for each measure. As shown in the table, the evaluation eligibility criteria utilized 

a relatively low symptom threshold. Other eligibility criteria included the ability to provide 

informed consent, aged 16 or older, and oral fluency in English or Spanish. 

Table C.4. Impact Evaluation Eligibility Criteria on Screening Measures

Screening 

Measure

Impact Evaluation

Eligibility Published Scoring Criteria Reference

PHQ-8 Total score≥5 0–4 = none to minimal depression

5–9 = mild depression

10–14 = moderate depression

15–19 = moderately severe depression

20–24 = severe depression

Kroenke, Spitzer, and

Williams, 2001

GAD-7 Total score≥5 5 = mild anxiety

10 = moderate anxiety

15 = severe anxiety

Löwe et al., 2008;

Spitzer et al., 2006

PCL-5 Total score≥28 ≥33 = provisional PTSD diagnosis Blevins et al., 2015

AUDIT-10 Total score≥8 ≥8 = harmful or hazardous drinking

≥13 (female, transgender woman, genderqueer), ≥15 (male, transgender man) = 

alcohol dependence

Babor et al., 1992;

Berner et al., 2007

DAST-10 Total score≥1 0 = none

1–2 = low

Yudko et al., 2007

3–5 = intermediate

6–8 = substantial

9–10 = severe

349 
 

 

 

 

CBO Staff

IF ELIGIBLE FOR STUDY

Recruit client for screening to

determine study eligibility

Complete web-based enrollment

form

Obtain informed consent

Enter C2C screening results

Administer any needed study screenings

 forme 

 LE FO 

Handoff client to study

Conduct C2C screening

 b b 

Study

Screening

RAND Staff

Obtain informed consent

Complete web-based baseline 

assessment

Assessment

 

C2.3. Data Collection Procedures

For the C2C CBOs, RAND data collection staff worked closely with program staff to 

integrate the study data collection procedures into their client workflow. Most CBOs followed 

the process depicted in Figure C.1. Within their client populations, C2C CBOs made decisions 

about whom to target based on the clients they served and programs they offered. Some CBOs 

offered C2C to those within specific programs, whereas others opted to deliver C2C skills 

organization wide to all clients because staff and programs were intertwined, so serving only a 

subset was not feasible. 

The target group was new clients or existing clients beginning new programs at the CBOs. 

Screening protocols varied by site, based on their C2C implementation plans. This is detailed in 

the Implementation chapters (see Chapter 3 and Appendix A). However, clients generally 

enrolled in the C2C study within 2 weeks of beginning services or programming at the CBO. 

Figure C.1. Data Collection Process at C2C CBOs

350 
 

 

 

 

 

 

 

First, CBO staff administered their CBO-specific C2C screenings, explained the opportunity 

to participate in the C2C impact evaluation, and provided them with recruitment materials. 

Clients who were interested in participating in the study were referred by CBO staff to RAND 

data collection staff, who obtained informed consent to administer the eligibility screening. If the 

CBO conducted their C2C screening using one or more of the five impact evaluation eligibility 

screening measures, CBO staff provided the RAND data collection staff with the score(s) from 

their C2C screenings to be used for study eligibility determination so that clients did not need to 

repeat those measures for study purposes. Data collection staff entered results from the C2C 

screening instruments into the web-based eligibility screener and administered any of the 

nonoverlapping eligibility screeners. If a client’s scores on any of the screening measures 

deemed them eligible to participate in the study, RAND data collection staff advised the client 

that he or she was eligible to participate in the C2C impact evaluation. If the client was interested 

in participating in the study, RAND data collection staff obtained informed consent for the 

baseline assessment and administered the web-based baseline survey to the client. These 

procedures varied across organizations depending on their preferred workflow (e.g., CBO staff 

administered the study screening and then referred eligible clients to RAND data collection 

staff). Because of the variability across sites in terms of the screening measures administered, 

and because the CBO-administered screenings were part of their larger C2C implementation, 

C2C clients did not receive an incentive for the study eligibility screener. C2C study participants 

received a $20 gift card for completing the baseline assessment. 

The recruitment flow at comparison sites differed from the C2C approach because 

comparison sites were not conducting C2C screenings. Therefore, comparison site participants 

were initially recruited with support from CBO staff and completed all screenings with RAND 

data collectors because there was no overlap in already-administered screening measures. For the 

comparison CBOs, CBO staff supported the early stages of study recruitment (e.g., through 

recruitment events, referring clients to the study during program intake, posting or distributing 

recruitment flyers), and RAND data collection staff administered the study screening and 

baseline assessment for those eligible for the study (Figure C.2). Comparison group participants 

received a $10 gift card for completing the eligibility screening and a $20 gift card for the baseline 

assessment. For both comparison and C2C participants, incentive gift cards at the 6- and 12-month 

follow-ups were valued at $40 and $60, respectively.2 

After eligibility screening, the data collection procedures were the same for those in the 

comparison and C2C groups. The baseline assessment included the collection of contact 

information from study participants so that evaluators could maintain contact and locate them 

for their 6-month follow-up assessment. 

2 Originally, the participant incentive for the 6 and 12-month follow-up assessment was $20. We increased to $40 at 

6 months and $60 at 12 months in March of 2018 to provide further incentive for study participation. 

351 
 

 

 

Figure C.2. Data Collection Process at Comparison CBOs

Recruit client for screening to

determine study eligibility

Obtain informed consent

Complete web-based enrollment

form and administer study

eligibility screening

RAND Staff

 b-bas 

Obtain informed consent

IF ELIGIBLE FOR STUDY

Complete web-based baseline 

assessment

 LE FO 

Handoff client to study

CBO Staff

Assessment

Study

Screening

For both groups, RAND data collection staff implemented a range of strategies to recruit study 

participants, including providing food, beverages, and C2C promotional materials (e.g., branded 

stress balls, pens) at recruitment events, tailoring recruitment materials for each CBO, offering 

phone-based baseline interviews, offering off-site baseline interviews (e.g., in participant’s home, 

at the public library). RAND data collection staff also worked directly with the CBOs on study 

recruitment with strategies such as 

• collaborating with CBOs on a tailored recruitment and screening process to generate staff 

buy-in 

• providing CBOs and staff with additional plain-language “FAQ” documents to support 

their conversations with clients about the study opportunity 

352 
 

• working with CBOs to make Spanish- and English-speaking data collectors available at 

optimal times for recruitment that were also convenient for CBO staff, including evenings 

and weekends 

• conducting group screenings when possible 

• hiring data collectors who were already staff/interns at CBOs and thus trusted by and 

familiar with the organization and its processes 

• collecting ongoing feedback from CBO staff and sometimes clients about how to improve 

recruitment 

• providing monthly updates to CBOs regarding enrollment and updated monthly pace 

needed to reach target 

• meeting with individual CBOs on their request or when RAND identified potential 

improvements to the strategy at that CBO. 

RAND’s interim contact activities included sending postcards and advance letters, emailing, 

texting, and calling study participants at specified intervals to remind them about the study and 

alert them that someone would be reaching out about their next follow-up assessment. Between 

the baseline and 6-month surveys, and between the 6- and 12-month surveys, clients also 

received a self-administered brief survey via email asking for any updated contact information 

and/or received a phone call from study staff to request any updated contact information. For 

the first 16 months of the study, RAND data collection staff also attempted to reach alternate 

contact persons provided by the participant (e.g., a trusted friend or family member identified 

by the participant) if unable to reach the participant via phone or email. However, RAND data 

collectors ceased alternate contact activities (including soliciting this information from 

participants) in November 2018 because these alternate contacts were often difficult to reach, 

or when the contact was successful, most alternates refused to provide new contact information 

for the participant. 

All participants who completed a baseline survey were eligible for the 6- and 12-month 

follow-up survey. For these follow-up surveys, RAND data collection staff attempted to schedule 

and complete each follow-up assessment within a 2.5-month window. The window began 

2 weeks before the exact 6- or 12-month target date for the follow-up assessment and ended 

2 months after that date. Partway through data collection, an internal scheduling team was 

formed with specialized training and shifts for making scheduling calls. As noted earlier, the 

baseline assessment included the collection of comprehensive tracking information, including 

phone numbers (home, work, mobile), mailing address, and email addresses. The impact 

evaluation team maintained a toll-free study phone number and study email address to allow 

incoming communication from participants. To schedule each follow-up assessment, multiple 

methods were used (e.g., text, email, phone call) to locate the participant and schedule the survey 

in person or over the phone. RAND data collection staff offered to conduct interviews via 

353 
 

 

 

 

 

 

 

 

 

phone3 or at alternate locations (e.g., in homes, public library, etc.), and offered to schedule 

6-month follow-up appointments at the time of the baseline interview. Once scheduled, appointment 

reminders (via email, text message, and/or phone calls) were conducted 1 and 3 days prior to the 

scheduled appointment. To support retention, participants also received a mailing shortly before 

their 6- or 12-month follow-up window opening. This mailing reminded participants of the 

C2C and RAND brand and provided RAND contact information for persons who wanted to 

proactively start their follow-up survey. 

In February 2019, the impact evaluation team transitioned all follow-up survey data 

collection to RAND’s Survey Research Group (SRG). Accordingly, all 6- and 12-month followup surveys moved to phone-based interviewing and data collection. Participants then received 

their incentives via email or postal mail after completing the phone-based follow-up survey. 

After the transition to RAND’s SRG, two non-SRG RAND employees remained on C2C 

evaluation staff to continue other participant retention-related activities. These included the 

interim contact activities described earlier, in addition to monitoring incoming messages from 

study participants via email and voicemail. This ensured that RAND’s SRG received the most 

up-to-date contact information for each participant on a monthly basis. 

On a monthly basis, SRG was provided a datafile containing the contact information for 

participants whose 6- or 12-month follow-up windows were opening in the following month. A 

team of trained interviewers in the RAND SRG conducted follow-up phone calls that began on 

or after the date of a given participant’s follow-up window. RAND’s SRG maintained one to two 

calling shifts 5 days of the week, typically between the hours of 11:00 a.m. and 8:00 p.m. Eastern 

Time on weekdays and 10:00 a.m. to 5:00 p.m. on weekends with 4-hour shifts on alternating 

shifts on different days. Participant status was tracked using participant management and 

scheduling software that permitted prioritization of specific cases based on “close date” for followup window or number of prior attempts (e.g., flagging persons who had not completed their 6-

month survey and the closing date of their follow-up window was approaching as highest 

priority) and scheduled callbacks for participants who requested a specific date or time for their 

phone survey. For participants with disconnected phone numbers, SRG also attempted public 

records research using Lexis Nexis and U.S. Postal Service National Change of Address 

databases. If new contact information was found, SRG attempted to reach the participant using 

that phone number. 

The impact evaluation team and RAND data collection staff also worked closely with each 

CBO on strategies to support follow-up data collection. These efforts included monthly updates 

to CBOs with details on retention progress and monthly paces to reach target, ongoing discussions 

with CBO staff about how to improve retention, and monthly contact with CBO staff about 

specific study participants whose windows were closing. 

3 We began offering telephone-based follow-up interviews in March 2018 to reduce burden on study participants. 

354 
 

 

 

C2.4 Enrollment and Data Collection Timelines

The overall study enrollment period spanned 22 months from June 2017 through March 2019 

with CBOs enrolling participants for an average of 16 months for both groups (Figure C.3). After 

carefully considering study progress at each CBO, we decided to stop study enrollment 3 months 

early at several CBOs (all of the parent/caregiver-serving CBOs and homeless shelters) to 

reallocate data collection capacity and resources for the last few months of study enrollment. 

During the transition, RAND data collection staff continued to meet with individuals whom we 

had already engaged (e.g., those who had already expressed interest in completing the eligibility 

screening), completed any already-scheduled baseline appointments, and maintained any 

already-scheduled recruitment events. It is important to note that all of the study participants 

from these CBOs remained in the study and we continued to conduct follow-up surveys with 

them. 

Data collection extended through February 2020 so that we could attempt to complete 

12-month surveys with all participants enrolled through March 2019. We ended 12-month data 

collection 4 weeks earlier than originally planned to enable sufficient time for analyses prior to 

the end of the project in fall 2020. However, all participant follow-up windows were open for at 

least 4 weeks, which allowed us sufficient time for multiple contacts with each participant to 

attempt to complete the survey. Finally, we suspended data collection at all sites for about 2 weeks 

in February 2019 because of contractual issues with the Mayor’s Fund that were unrelated to the 

study protocol. 

355 
 

 

 

 

 

 

 

 

 

Figure C.3. Study Enrollment Timeline by CBO

C2C Jun-17 Jul-17 Aug-17 Sep-17 Oct-17 Nov-17 Dec-17 Jan-18 Feb-18 Mar-18 Apr-18 May-18 Jun-18 Jul-18 Aug-18 Sep-18 Oct-18 Nov-18 Dec-18 Jan-19 Feb-19 Mar-19

HOPE

BSRC

NMIC

STRIVE

CEO

The Door

RHI

HMI

The Landing

Hudson Guild 

Sheltering Arms

Safe Horizon

Voces

Comparison

Goodwill

Eastside

St. Nick's Alliance

Cypress Hills 

Eckerd Connects

MMCC*

King's Inn 

Children's Aid Society

Sanctuary for Families

VIP

* Study enrollment at early childhood program ended in January 2019

NOTES: MMCC = Mosholu Montefiore Community Center; VIP = Violence Intervention Program.

356 
 

 

 

C3. CBO Program Data Collection

The impact evaluation data collection originally included CBO program data requested from 

CBOs on program participants. These individual-level data on receipt of C2C skills were intended 

to be used to address the secondary research question about whether program effectiveness 

varied within the C2C group depending on the degree of exposure to C2C skills. However, 

quantity and quality issues with these CBO program data made them unusable for the intended 

analysis. 

C4. Administrative Data Collection from New York City and State Agencies

The impact evaluation data collection also originally included administrative data from NYC 

and NYS agencies on specific data elements (e.g., Medicaid claims, emergency department 

visits, unemployment benefits, homeless shelter stays) for C2C program and comparison group 

participants who consented to participate in this aspect of the study. Initially, we planned to 

request administrative data from agencies and organizations such as the New York State 

Department of Labor, the New York City Health and Hospitals Corporation, and the New York 

State Office of Medicaid. However, bureaucratic issues slowed down the process of receiving 

these data, and then additional challenges related to COVID meant that these agencies did not 

have the capacity to provide these data for our impact evaluation. 

C5. Development of an Analytic Database

C5.1 Data Processing and Cleaning

The analytic dataset for the impact evaluation was composed of client survey data (screening, 

baseline, 6 months, and 12 months). These data were merged into a final analytic dataset using a 

C2C study ID number linked to each participant. The dataset preparation is shown as a schematic 

in Figure C.4. 

Client survey data. Baseline client survey data consisted of three sequential datasets created 

by the study enrollment workflow within the web-based data collection system. First, individuals 

who consented to the study screening were assigned a unique study ID number using an electronic 

Add Participant form. This form also captured preload values that were piped into subsequent 

survey questions or routing logic across the baseline and follow-up time points (e.g., CBO name 

piped into questions about services the participant had received at their home CBO). Second, the 

screening survey included basic demographics and the mental health screening measures used for 

eligibility criteria. Third, clients who screened eligible and consented to participate in the study 

then completed the baseline survey. 

As shown in Figure C.4, the baseline dataset was composed of data from these three datasets, 

merged on a unique study ID number for each participant. Based on the level of complexity and 

relevance to survey programming, field-based data entry errors were either corrected shortly 

357 
after the error (e.g., choosing an incorrect CBO assignment for a participant) or logged for later 

filtering at the point when baseline data were compiled (e.g., known duplicates). In addition, at 

the conclusion of study enrollment, the impact evaluation team undertook checks for duplicate 

participants who had either accidentally or intentionally enrolled in the study more than once. 

These duplicate checks included dates of birth, and also exact and “fuzzy” (≥ 90 percent) 

matching of names. For cases identified as duplicates, the first observation was retained. After 

this cleaning process, the compiled baseline dataset served as the gold standard record for 

matching clients’ 6- and 12-month client surveys to their baseline. 

The 6- and 12-month datasets were self-contained (i.e., they were not composed of multiple 

incremental datasets), and therefore each time point was only a single export of data. In cases 

where a duplicate participant had already completed a follow-up survey before they were 

detected as a duplicate, only the follow-up survey that matched the gold standard baseline case 

was retained for follow-up analysis. 

358 
 

 

 

 

-

-

-

C2C Impact Study dataset preparation - Flow

Merged baseline

dataset w/ specified 

variables

Baseline data 

manager

Screener data 

manager

Add participant

data manager

Black = Raw survey data

Orange = Intermediary data, not for final use

Purple = Retained datasets; deidentified

Data corrections 

(field errors) 

working dataset

6 mo data manager

12 mo data manager

Baseline exported 

dataset w/specified 

variables

6 mo exported 

dataset w/ specified 

variables

12 mo exported 

dataset w/ specified 

variables

De ID baseline 

dataset

De ID 6 mo dataset

De ID 12 mo dataset

Derived variables calculated here.

Analytic dataset

Programming and

diagnostic baseline

dataset

Programming and

diagnostic 6 mo

dataset

Programming and

diagnostic 12 mo

dataset

Merges all analytic

variables and 

timepoints by study

ID#

Figure C.4. Impact Study Analytic Database Development Process

359 
C6. Analytic Approach

Overall, the impact evaluation examined the effectiveness of the intervention across all sites. 

The main analyses answered the primary research questions: 

• Do C2C program participants have increased mental health care access to and utilization 

relative to comparison group members? (Chapter 7) 

• Do C2C program participants have improved mental health symptoms relative to 

comparison group members? (Chapter 8) 

• Do C2C program participants have improved educational attainment, housing, 

employment, and criminal justice system involvement relative to comparison group 

members? (Chapter 9) 

Because of the impact evaluation’s quasi-experimental design, we used propensity score 

weighting to account for potential confounders (e.g., age, race/ethnicity, income, baseline mental 

health symptoms). We modeled the probability that each individual received C2C (vs. services as 

usual) as a function of pre-C2C characteristics to estimate propensity scores. We used inverse 

propensity score weights to control for potential group differences. Our primary analysis considers 

within- and between-group changes over time. Within-group differences were estimated separately 

at 6 and 12 months for both C2C and non-C2C participants. Between-group differences, that is, 

the average effect of C2C, were then estimated using a simple weighted regression model that 

incorporated propensity score weights and, as a single covariate, an indicator of whether the 

individual received C2C services. We also included the potential confounders as covariates in 

another propensity score weighted outcome model, in what is termed a doubly robust model 

(Lunceford and Davidian, 2004). Outcome variables were individual-level differences between 

baseline and follow-up measures, for both the C2C group and comparison group. Analyzing 

these differences cancels out the effect of time-invariant factors and is a type of difference-indifferences analysis. 

In addition to the primary C2C effects described above, subgroup analyses by target population 

and select CBO services types were also performed as a secondary research question in each 

chapter. For these analyses, propensity scores were re-estimated for each subgroup, checking 

balance as described above for the full sample. Although these C2C effect estimates are less 

precise than the overall estimates, they still provide information on subpopulations for which 

the intervention is more or less effective. 

C6.1. Propensity Score Weighting

We used propensity score weighting to improve comparability between individuals in the 

treatment and comparison groups. In particular, we estimated propensity scores using a 

generalized boosted model to estimate the average treatment effect on treated individuals 

(McCaffrey, Ridgeway, and Morral, 2004). The generalized boosted model is a flexible, 

nonparametric machine learning algorithm that has been found to substantially outperform 

360 
 

 

 

 

 

 

more standard approaches like logistic regression in propensity score applications (Lee, Lessler, 

and Stuart, 2010). After applying the propensity score weights, the distributions of potentially 

confounding variables should be well balanced between the treatment and control samples. 

Separate sets of weights were generated for baseline, 6-month, and 12-month analyses and for 

each subgroup used. A separate set of weights was also generated for the non–mental health 

outcomes, because these outcomes were used as control covariates in the other analyses. As 

shown in the tables below, all propensity score weights included demographic variables (age, 

gender, race/ethnicity, income level, education level, employment status, housing status, 

incarceration status, target population, and CBO service type) and outcome variables (depression, 

anxiety, PTSD, alcohol use, substance use, psychological distress, logistical barriers, attitudinal 

barriers, and stigma barriers). 

Assessing balance of potential confounders in this way helped evaluate the internal validity 

of the results. The propensity score weighting was used for the analytic models for primary 

outcomes of interest: We estimated treatment effects via propensity score weighted regressions. 

Any baseline covariates that were not brought into sufficient balance via propensity score weights 

were still included as covariates in the outcomes model (Lunceford and Davidian, 2004), which 

minimizes any potential bias in these doubly robust models. The propensity score weighted models 

also contained fixed effects for the CBO service type. This allowed us to account for clustering 

at the CBO level to the extent that distributions of unobserved confounders are the same between 

the treatment and comparison group (Li, Zaslavsky, and Landrum, 2013). In some cases, several 

CBOs of one type were compared with a single CBO of the corresponding type for comparison. 

The CBO service types were job training and employment program, youth development program, 

homeless shelter, parent/caregiver-serving organization, domestic violence agency, and Latino/ 

Hispanic agency. Because we modeled the change in outcomes over time as our outcome of 

interest, the propensity score modeled for 6 and 12 months also include baseline outcome 

values to control for potential imbalance among participants in their baseline outcome values. 

As with all propensity score methods, the weighting process can only account for observed 

variables. Hence, there is the potential for unobserved confounders (i.e., pretreatment variables 

that are imbalanced between the treatment and comparison groups and that are associated with 

outcomes of interest) to bias treatment effect estimates and to compromise internal validity. We 

addressed this concern in two ways. First, we recorded differences between pre- and posttreatment 

outcomes, so that time-invariant confounders were removed from the analysis (a type of 

difference-in-differences analysis). Second, we attempted to identify and measure as many 

potential confounders as possible to include in the weighting process. The primary assumptions 

of the propensity score analysis were that there are no unobserved confounders beyond those that 

are accounted for by the group-level fixed effects and that each individual has some chance of 

having been in the treatment versus comparison condition (i.e., no propensity scores are 0 or 1). 

Propensity score analyses also depend on the “stable unit treatment value assumption,” which 

says that the outcome for one individual does not depend on the treatment status of any other 

361 
 

 

 

 

individual. Our flexible propensity score model does not make parametric assumptions about the 

functional form of the propensity score model, but our outcomes model assumes that means or 

log OR is linear in the covariates included in the outcomes model. 

We used the “ps” function in the “twang” software package in the R language to estimate the 

propensity scores (Ridgeway et al., 2015). This nonparametric approach only produces estimated 

propensity scores and does not produce interpretable parameter estimates. In the propensity score 

weighting analyses, we included potential confounders measured at baseline that were expected 

to be related to the key outcomes (Brookhart et al., 2006). This included baseline demographic 

characteristic, including age (Kessler et al., 2010), gender (Breslau et al., 1997), race/ethnicity 

(Wells et al., 2013), income level (Bassuk et al., 1998; Goodman and Huang, 2002), education 

level, housing status, incarceration status, target population (adults age 18 or older who are 

unemployed or underemployed, young adults ages 16–24 who are not in school and are not 

employed, and parents/primary caregivers who are expecting or who have children up to the 

age of 4), and CBO service type, and baseline levels of the key outcomes (i.e., mental health 

symptoms, barriers). 

We tested for successful balance using Kolmogorov-Smirnov statistics and compared 

standardized mean differences between the two samples, using a scale in which standardized 

mean differences below 0.2 are considered indicators of good balance. Overall, the weight 

generated for the primary analyses showed good levels of covariate balance, minimizing any 

potential bias in the results. For categorical variables, the standardized mean differences were 

evaluated for each value (e.g., the difference in percentage of females, males, and other) between 

groups were evaluated separately. With the baseline propensity score weights, all control variables 

had standardized mean differences below 0.2 (see Tables C.5 and C.6). With the 6-month 

propensity score weights, all but five control values (females, males, baseline unemployment 

status, baseline transitional housing status, and baseline PTSD status) had standardized 

differences of greater than 0.2, and the largest was 0.266 (see Tables C.7 and C.8). This 

represents less than 5 percent of all of the control variables. 

With the 12-month propensity score weights, all but seven control values (females, males, 

baseline unemployment status, baseline transitional housing status, baseline PTSD status, 

12-month recent homelessness status, and baseline incarceration statuses) had standardized 

differences of greater than 0.2, and the largest was 0.256 (see Tables C.9 and C.10). This 

represents 6 percent of all of the control variables. 

For both 6 and 12 months, the K-S test statistics produced p-values which were reasonably 

balanced. Assuming a uniform distribution of p-values under the null, we would expect roughly 

two of the p-values to fall below 0.05. For the 6-month statistics, six were below 0.05 and for 

12 months four p-values were below 0.05. This shows that although we were not able to achieve 

perfect balance, it was reasonably close. Because of the computational limitations of weighting, 

it is not always possible to achieve perfect balance. The implications of this are a small potential 

for bias from unbalanced covariates in the non-doubly robust models. Propensity score weights 

362 
 

were also generated for each of the subgroup analyses. Because of the much smaller sample sizes 

of the subgroups, there was a degradation of the overall balance. This means there was slightly 

more potential for bias in the subgroup analyses due to unbalanced covariates. 

For our outcomes model, we used the “svyglm” function in the “survey” package in R 

(Lumley, 2014). The outcome model incorporated the propensity score weights. The included 

covariates were the treatment indicator, fixed effects for CBO service type, and all pretreatment 

covariates to ensure a doubly robust model as described above. Only the parameter estimate 

related to the treatment indicator is of direct interest. It can be interpreted as either a difference in 

means (for continuous outcomes) or as an OR (for binary outcomes converted from log odds). 

For within-group changes, we tabulate the outcomes using the “svyby” function of the “survey” 

package, which again incorporates the propensity score weights. 

363 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.5. Balance Table for Propensity Score Weighting of Study Participants at Baseline, Demographic Characteristics

Unweighted Sample Propensity Score Weighted Sample

Participant Demographic 

Characteristic

Overall

(n = 1,838)

C2C

(n = 1,232)

Comparison 

(n = 606) p-Value Overall C2C Comparison p-Value

Age (mean, SD) 30.1, 12.5 29.8, 12.1 30.7, 13.2 0.204 30.1, 12.5 29.8, 12.1 30.7, 13.2 0.204

Gender (%, n) – – – <.001 – – – 0.425

Male 44, 809 49.4, 609 33, 200 – 44.2% 45.7% 42.4% –

Female 54.2, 996 48.4, 596 66, 400 – 54.2% 52.3% 56.3% –

Transgender, genderqueer,

or other

1.6, 30 2, 25 0.8, 5 – 1.5% 1.9% 1.2% –

Race/ethnicity (%, n) – – – 0.281 – – – 0.419

Hispanic 42.3, 777 41.4, 510 44.1, 267 – 43.2% 42.2% 44.4% –

Black 47.2, 867 47.3, 583 46.9, 284 – 47.1% 47.1% 47.2% –

White 3.1, 57 3.5, 43 2.3, 14 – 2.7% 3.1% 2.2% –

Other 6.9, 127 7.4, 91 5.9, 36 – 6.4% 7.2% 5.6% –

Income level (%, n) – – – 0.186 – – – 0.993

Less than $5,000 42.9, 789 44.5, 548 39.8, 241 – 43.4% 43.6% 43% –

$5,000–10,000 11.3, 207 11.6, 143 10.6, 64 – 10.9% 11.1% 10.7% –

$10,001–20,000 8.7, 159 8.9, 110 8.1, 49 – 8.3% 8.7% 7.9% –

$20,001–30,000 4.6, 84 4.2, 52 5.3, 32 – 5% 4.8% 5.2% –

$30,001–40,000 3.1, 57 3.1, 38 3.1, 19 – 3.3% 3.1% 3.5% –

More than $40,000 2.1, 39 1.8, 22 2.8, 17 – 1.8% 1.8% 1.9% –

Education level (%, n) – – – 0.051 – – – 0.901

Less than high school 30.6, 562 32.1, 395 27.6, 167 – 31.6% 30.9% 32.3% –

Completed high school

diploma or GED

33.7, 620 34.3, 422 32.7, 198 – 33.3% 34% 32.4% –

Some college 22, 405 20.3, 250 25.6, 155 – 21.4% 20.8% 22% –

Completed college 6.8, 125 7.1, 88 6.1, 37 – 6.4% 7.1% 5.6% –

Some graduate or

professional school

0.6, 11 0.6, 8 – 0.8% 0.6% 0.9% –

Completed graduate or

professional school

1.5, 28 1.5, 19 1.5, 9 – 1.8% 1.6% 1.9% –

Employment status (%, n) – – – <.001 – – – 0.595

Unemployed 57.7, 1,060 63.6, 783 45.7, 277 – 57.8% 59.1% 56.3% –

Student 11.2, 205 11.3, 139 10.9, 66 – 11% 11.4% 10.5% –

Employed part-time 14.5, 266 12.7, 157 18, 109 – 14.6% 14% 15.3% –

364 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Unweighted Sample Propensity Score Weighted Sample

Participant Demographic 

Characteristic

Overall

(n = 1,838)

C2C

(n = 1,232)

Comparison 

(n = 606) p-Value Overall C2C Comparison p-Value

Employed full-time 7, 129 5.7, 70 9.7, 59 – 7.2% 6.8% 7.7% –

Retired and not working 1.1, 21 0.5, 6 2.5, 15 – 0.9% 0.7% 1.2% –

Homemaker 1, 18 0.6, 7 1.8, 11 – 1% 0.9% 1.1% –

Disabled or too ill to work 2.1, 39 0.8, 10 4.8, 29 – 2% 1.5% 2.6% –

Other 0.5, 10 0.6, 8 – 0.5% 0.6% 0.3% –

Housing status (%, n) – – – <.001 – – – 0.111

Stably housed 41.1, 755 39.2, 483 44.9, 272 – 42.3% 40.9% 43.9% –

Staying with someone else 33.9, 623 32.8, 404 36.1, 219 – 34.2% 33.3% 35.3% –

Transitional/temporary 9, 166 11.6, 143 3.8, 23 – 7.7% 9.7% 5.4% –

Homeless/unstably housed 10.2, 187 11.3, 139 7.9, 48 – 9.7% 10.1% 9.2% –

Past-year homelessness 18.4, 338 20.9, 257 13.4, 81 <.001 18% 19.4% 16.4% 0.381

Incarcerations status – – – <.001 – – – 0.132

No 82.1, 1,509 79.7, 982 87, 527 – 83.1% 81.4% 85% –

Yes 13.3, 245 16.5, 203 6.9, 42 – 12.2% 14.1% 10.2% –

Target population – – – – – – – –

Young adults ages 16 to 24 

who are not in school and 

24.5, 451 25.2, 310 23.3, 141 0.636 23.6% 24.1% 22.9% 0.592

not employed

Adults age 18 or older who 

are unemployed (or

underemployed)

65.5, 1,204 69, 850 58.4, 354 <.001 65.6% 65.7% 65.3% 0.937

Caregivers and parents who 

are expecting or who have 

children up to the age of 4

14.3, 262 13.1, 162 16.5, 100 0.033 15.4% 13.8% 17.1% 0.226

CBO service type – – – <.001 – – – 0.253

Job training and employment 64, 1,177 70, 862 52, 315 – 65.6% 66% 65.2% –

program

Youth development program 22.8, 419 17.2, 212 34.2, 207 – 21.9% 21% 22.9% –

Homeless shelter 1.1, 21 3, 18 – 1% 0.5% 1.5% –

Parent/caregiver-serving 

organizations

3.5, 65 2.9, 36 4.8, 29 – 3.6% 3.7% 3.6% –

Domestic violence agency 5.1, 93 5.5, 68 4.1, 25 – 4.9% 5.2% 4.6% –

Latino/Hispanic agency 3.4, 63 4.1, 51 2, 12 – 2.9% 3.6% 2.2% –

NOTES: Cells shaded in gray indicate a sample size less than 5. SD = standard deviation.

365 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.6. Balance Table for Propensity Score Weighting of Study Participants at Baseline, Mental Health Symptoms

Outcomes Unweighted Sample Propensity Score Weighted Sample

Mental health symptoms

Overall (n = 

1,838)

C2C (n = 

1,232)

Comparison 

(n = 606) p-Value Overall C2C Comparison p-Value

Depression (%, n) – – – <.001 – – – <.001

None—minimal 30.2, 555 35.2, 434 20, 121 – 26.1% 33.4% 17.9% –

Mild 32.5, 597 29.1, 359 39.3, 238 – 34.8% 30.1% 40.1% –

Moderate 22.1, 406 20.9, 258 24.4, 148 – 22.3% 21.9% 22.6% –

Moderately severe 9.6, 176 8.6, 106 11.6, 70 – 11.2% 8.7% 14% –

Severe 4.2, 78 4.8, 59 3.1, 19 – 3.9% 4.6% 3.1% –

Anxiety (%, n) – – – 0.183 – – – 0.072

Mild 61.9, 1,138 62.8, 774 60.1, 364 – 60.7% 61.9% 59.4% –

Moderate 21.6, 397 20.7, 255 23.4, 142 – 22.9% 21.2% 24.9% –

Severe 15.7, 288 15.9, 196 15.2, 92 – 15.4% 16.4% 14.2% –

PTSD (%, n) – – – 0.516 – – – 0.518

Positive screen 47.7, 876 47.2, 581 48.7, 295 – 47.6% 47.2% 47.9% –

No positive screen 52.2, 960 52.7, 649 51.3, 311 – 52.3% 52.6% 52.1% –

Alcohol use (%, n) – – – 0.098 – – – 0.885

None 78.4, 1,441 76.9, 947 81.5, 494 – 78.4% 77.9% 78.9% –

Harmful/hazardous drinking 9.7, 178 10, 123 9.1, 55 – 10.2% 10% 10.5% –

Alcohol dependence 10.3, 190 11.4, 140 8.3, 50 – 9.9% 10.5% 9.3% –

Other – –

Substance use (%, n) – – – <.001 – – – 0.069

None 36.8, 677 33.3, 410 44.1, 267 – 37.7% 35.2% 40.6% –

Low 36.3, 667 37.1, 457 34.7, 210 – 36.6% 36.7% 36.6% –

Intermediate 16.1, 296 17, 210 14.2, 86 – 15.6% 16.6% 14.5% –

Substantial 6.7, 123 8, 99 4, 24 – 6% 7.6% 4.2% –

Severe 2.4, 44 3.2, 39 0.8, 5 – 2.2% 2.6% 1.6% –

Psychological distress – – – 0.29 – – – 0.650

Percentage with none/low 27.3%, 502 27.8%, 343 26.2%, 159 – 27.5% 27.3% 27.7% –

Percentage with moderate psychological

distress

44.1%, 810 44%, 542 44.2%, 268 – 44.6% 43.4% 46% –

Percentage with severe mental illness 23.4%, 430 23.6%, 291 22.9%, 139 – 22.8% 24% 21.4% –

Access – – – – – – – –

Logistical (mean, SD) 0.7, 0.6 0.7, 0.6 0.7, 0.7 0.316 0.7, 0.6 0.7, 0.6 0.7, 0.7 0.316

Attitudinal (mean, SD) 1, 0.6 1, 0.6 1, 0.6 0.0892 1, 0.6 1, 0.6 1, 0.6 0.089

Stigma (mean, SD) 0.8, 0.8 0.8, 0.8 0.8, 0.8 0.074 0.8, 0.8 0.8, 0.8 0.8, 0.8 0.074

NOTE: SD = standard deviation.

366 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.7. Balance Table for Propensity Score Weighting of Retained Study Participants at 6 Months, Baseline Demographic

Characteristics

Unweighted Sample Propensity Score Weighted Sample

Participant Demographic 

Characteristic

Overall 

(n = 688)

C2C 

(n = 443)

Comparison 

(n = 245) p-Value Overall C2C Comparison p-Value

Age (mean, SD) 31.4, 13.0 30.9, 12.5 32.4, 13.8 0.152 31.3, 12.8 31, 12.7 31.7, 13 0.507

Gender (%, n) – – – <.001 – – – 0.013

Male 36.9, 254 45.4, 201 21.6, 53 – 34.9% 40.7% 27.9% –

Female 60.8, 418 51.9, 230 – 62.6% 56.7% 69.6% –

Transgender, genderqueer, or

other

2.3, 16 2.7, 12 – 2.5% 2.5% 2.5% –

Race/ethnicity (%, n) – – – 0.481 – – – 0.502

Hispanic 42.3, 291 43.1, 191 40.8, 100 – 42.4% 43.8% 40.7% –

Black 46.1, 317 44.5, 197 49, 120 – 46.2% 44.1% 48.7% –

White 2.9, 20 3.2, 14 2.4, 6 – 2.7% 2.9% 2.4% –

Other 8, 55 8.8, 39 6.5, 16 – 8% 8.8% 7% –

Income level (%, n) – – – 0.486 – – – 0.826

Less than $5,000 39.2, 270 40.2, 178 37.6, 92 – 40.1% 39.7% 40.6% –

$5,000–10,000 12.8, 88 13.3, 59 11.8, 29 – 12.7% 13.6% 11.6% –

$10,001–20,000 11.2, 77 11.7, 52 10.2, 25 – 11.1% 11.8% 10.4% –

$20,001–30,000 4.1, 28 3.4, 15 5.3, 13 – 4% 3.7% 4.4% –

$30,001–40,000 3.8, 26 4.1, 18 3.3, 8 – 3.5% 3.9% 3% –

More than $40,000 3.1, 21 2.3, 10 4.5, 11 – 2.8% 2.1% 3.8% –

Education level (%, n) – – – 0.428 – – – 0.771

Less than high school 28.3, 195 28.7, 127 27.8, 68 – 29.2% 28.5% 30% –

Completed high school diploma or

GED

34, 234 36.1, 160 30.2, 74 – 32.4% 34.4% 29.9% –

Some college 24.7, 170 22.3, 99 29, 71 – 24.7% 23.2% 26.6% –

Completed college 9.9, 68 10.2, 45 9.4, 23 – 10.2% 10.8% 9.4% –

Some graduate or professional

school

0.7, 5 – 0.8% 0.8% 0.8% –

Completed graduate or

professional school

2.2, 15 1.8, 8 2.9, 7 – 2.7% 2.1% 3.3% –

Employment status (%, n) – – – <.001 – – – 0.662

Unemployed 56.5, 389 63.7, 282 43.7, 107 – 55% 60% 48.9% –

Student 12.8, 88 11.5, 51 15.1, 37 – 13% 12.1% 14.1% –

76.7, 188

367 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

7.9, 35

Unweighted Sample Propensity Score Weighted Sample

Participant Demographic 

Characteristic

Overall 

(n = 688)

C2C 

(n = 443)

Comparison 

(n = 245) p-Value Overall C2C Comparison p-Value

Employed part-time 15.8, 109 13.8, 61 19.6, 48 – 17.1% 15.1% 19.5% –

Employed full-time 9, 62 11, 27 – 9.1% 8.9% 9.3% –

Retired and not working 1.5, 10 3.3, 8 – 1.3% 0.6% 2.2% –

Homemaker 1.3, 9 2, 5 – 1.3% 1.1% 1.6% –

Disabled or too ill to work 2.5, 17 1.1, 5 4.9, 12 – 2.6% 1.5% 4% –

Other – 0.6% 0.8% 0.5% –

Housing status (%, n) – – – <.001 – – – 0.021

Stably housed 45.6, 314 42.2, 187 51.8, 127 – 47.1% 44.5% 50.3% –

Staying with someone else 35, 241 35.4, 157 34.3, 84 – 35% 34.7% 35.3% –

Transitional/temporary 8.7, 60 11.7, 52 3.3, 8 – 7.4% 10.5% 3.6% –

Homeless/unstably housed 9.7, 67 10.2, 45 9, 22 – 9.5% 9.9% 9.1% –

Past-year homelessness 15.1, 104 18.3, 81 9.4, 23 0.00148 13.6% 16.6% 10% 0.028

Incarcerations status – – – <.001 – – – 0.097

No 89.7%, 617 86.5%, 234 95.5%, 383 – 90.7% 88.4% 93.5% –

Yes 10%, 69 13.3%, 59 4.1%, 10 – 9% 11.4% 6% –

Target population – – – – – – – –

Young adults ages 16 to 24 

who are not in school and not

21.1%, 145 21.4%, 95 20.4%, 50 0.750 19.8% 19.9% 19.7% 0.947

employed

Adults age 18 or older who are 

unemployed (or are 

underemployed)

67.3%, 463 71.8%, 318 59.2%, 145 67.2% 69.6% 64.2% 0.163

Caregivers and parents who are 

expecting or who have children 

23.1%, 159 22.3%, 99 24.5%, 60 0.664 23.6% 23.8% 23.4% 0.322

up to the age of 4

CBO service type – – – – – – 0.365

Job training and employment 61.3%, 422 66.8%, 296 51.4%, 126 – 59.1% 62.6% 54.9% –

program

Youth development program 23%, 158 19%, 84 30.2%, 74 – 24% 21.3% 27.2% –

Homeless shelter 1.2%, 8 2.4%, 6 – 1.3% 0.8% 2% –

Parent/caregiver-serving 

organizations

6.4%, 44 5.6%, 25 7.8%, 19 – 6.8% 7% 6.5% –

Domestic violence agency 4.7%, 32 4.5%, 20 4.9%, 12 – 5.1% 4.7% 5.6% –

Latino/Hispanic agency 3.5%, 24 3.6%, 16 3.3%, 8 – 3.7% 3.7% 3.8% –

NOTES: Cells shaded in gray indicate a sample size less than 5. SD = standard deviation.

368 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.8. Balance Table for Propensity Score Weighting of Retained Study Participants at 6 Months, Baseline Mental Health Symptoms

Unweighted Sample Propensity Score Weighted Sample

Outcomes

Overall

(n = 688)

C2C 

(n = 443)

Comparison 

(n = 245) p-Value Overall C2C Comparison p-Value

Depression (%, n) – – – <.001 – – – <.001

None—minimal 28.3, 195 34.1, 151 18, 44 – 25.2% 30.7% 18.6% –

Mild 34, 234 29.1, 129 42.9, 105 – 37.6% 32.2% 44.2% –

Moderate 21.9, 151 21.7, 96 22.4, 55 – 21% 21.6% 20.2% –

Moderately severe 9.9, 68 9, 40 11.4, 28 – 10.6% 9.5% 11.9% –

Severe 4.7, 32 5.2, 23 3.7, 9 – 4.4% 5.1% 3.7% –

Anxiety (%, n) – – – 0.248 – – – 0.359

Mild 60.5, 416 60.3, 267 60.8, 149 – 60.6% 60.4% 60.9% –

Moderate 24.3, 167 24.6, 109 23.7, 58 – 24.7% 24.7% 24.6% –

Severe 14.2, 98 14.7, 65 13.5, 33 – 13.7% 14.5% 12.8% –

PTSD (%, n) – – – 0.352 – – – 0.427

Positive screen 50.1, 345 51.5, 228 47.8, 117 – 49.9% 51.4% 48.1% –

No positive screen 49.9, 343 48.5, 215 52.2, 128 – 50.1% 48.6% 51.9% –

Alcohol use (%, n) – – – 0.259 – – – 0.629

None 81.5, 561 79.2, 351 85.7, 210 – 82.7% 81% 84.8% –

Harmful/hazardous drinking 9.6, 66 10.8, 48 7.3, 18 – 9.4% 10.2% 8.5% –

Alcohol dependence – 7% 7.6% 6.2% –

Other – 0.2% 0.4% 0% –

Substance use (%, n) – – – <.001 – – – 0.036

None 45.1, 310 39.7, 176 54.7, 134 – 46.7% 44% 50.1% –

Low 31.7, 218 33.2, 147 29, 71 – 32.5% 32.5% 32.4% –

Intermediate 14.7, 101 16.5, 73 – 13.1% 14.8% 11% –

Substantial 5.4, 37 7.2, 32 – 4.2% 5.9% 2.2% –

Severe 2.2, 15 2.9, 13 – 2% 2.3% 1.5% –

Psychological distress – – 0.781 – – – 0.890

% with none/low 26.9%, 185 26%, 115 28.6%, 70 – 27.7% 26.9% 28.8% –

% with moderate psychological distress 46.7%, 321 47.2%, 209 45.7%, 112 – 46.3% 46.9% 45.6% –

% with severe mental illness 25.7%, 177 26%, 115 25.3%, 62 – 25.2% 25.3% 25.1% –

Access – – – – – – – –

Logistical (mean, SD) 0.8, 0.7 0.8, 0.7 0.7, 0.7 0.223 0.7, 0.7 0.8, 0.7 0.7, 0.7 0.532

Attitudinal (mean, SD) 1, 0.6 1, 0.6 0.9, 0.6 0.0402 1, 0.6 1, 0.6 0.9, 0.6 0.240

Stigma (mean, SD) 0.8, 0.8 0.8, 0.8 0.8, 0.8 0.651 0.8, 0.8 0.8, 0.8 0.8, 0.8 0.976

7.8, 54 8.6, 38 6.5, 16

11.4, 28

NOTES: Cells shaded in gray indicate a sample size less than 5. SD = standard deviation.

369 
 

 

 

 

 

 

 

 

 

 

 

 

Table C.9. Balance Table for Propensity Score Weighting of Retained Study Participants at 12 Months, Baseline Demographic 

Characteristics

Unweighted Sample Propensity Score Weighted Sample

Participant Demographic 

Characteristic

Overall

(n = 732)

C2C 

(n = 464)

Comparison 

(n = 268) p-Value Overall C2C Comparison p-Value

Age (mean, SD) 31.6, 13.4 30.9, 12.6 33, 14.6 0.054 31.7, 13.3 31, 12.8 32.6, 13.8 0.138

Gender (%, n) – – – <.001 – – – 0.013

Male 35, 256 43.3, 201 20.5, 55 – 33.5% 38.5% 27.6% –

Female 62.7, 459 53.7, 249 – 64.2% 58.5% 70.9% –

Transgender, gender

queer, or other

2.3, 17 3, 14 – 2.3% 2.9% 1.5% –

Race/ethnicity (%, n) – – – 0.164 – – – 0.392

Hispanic 41.3, 302 39.7, 184 44, 118 – 41.7% 40.7% 43% –

Black 47.5, 348 47, 218 48.5, 130 – 47.6% 46.6% 48.9% –

White 2.3, 17 2.6, 12 1.9, 5 – 2.3% 2.5% 2.1% –

Other 8.5, 62 10.3, 48 5.2, 14 – 8% 10% 5.6% –

Income level (%, n) – – – 0.590 – – – 0.879

Less than $5,000 43.4, 318 44.4, 206 41.8, 112 – 44% 44.3% 43.6% –

$5,000–10,000 12.7, 93 13.6, 63 11.2, 30 – 11.8% 13.1% 10.3% –

$10,001–20,000 10.2, 75 10.1, 47 10.4, 28 – 10.5% 10.5% 10.5% –

$20,001–30,000 4.5, 33 4.3, 20 4.9, 13 – 4.5% 4.3% 4.7% –

$30,001–40,000 3.4, 25 3.9, 18 2.6, 7 – 3.6% 3.7% 3.4% –

More than $40,000 3, 22 2.4, 11 4.1, 11 – 2.9% 2.3% 3.6% –

Education level (%, n) – – – 0.808 – – – 0.869

Less than high school 27, 198 27.4, 127 26.5, 71 – 27.5% 27.4% 27.6% –

Completed high school with 

diploma or GED

33.7, 247 34.7, 161 32.1, 86 – 33.1% 34.1% 31.8% –

Some college 25.3, 185 23.5, 109 28.4, 76 – 24.6% 23% 26.4% –

Completed college 10.7, 78 – 11% 12.1% 9.7% –

Some graduate or

professional school

0.7, 5 – 0.8% 0.7% 0.9% –

Completed graduate or

professional school

2.3, 17 2.2, 10 2.6, 7 – 2.7% 2.3% 3.2% –

Employment status (%, n) – – – <.001 – – – 0.086

Unemployed 56.6, 414 65.5, 304 41, 110 – 55% 60.3% 48.7% –

Student 12.3, 90 10.8, 50 14.9, 40 – 12.7% 11.5% 14.2% –

78.4, 210

11.4, 53 9.3, 25

370 
 

 

 

 

 

 

 

7.5, 35

Unweighted Sample Propensity Score Weighted Sample

Participant Demographic 

Characteristic

Overall

(n = 732)

C2C 

(n = 464)

Comparison 

(n = 268) p-Value Overall C2C Comparison p-Value

Employed part-time 15.3, 112 13.4, 62 18.7, 50 – 15.9% 15.2% 16.8% –

Employed full-time 9, 66 11.6, 31 – 9.4% 8.5% 10.5% –

Retired and not working 2, 15 4.9, 13 – 1.8% 0.7% 3.2% –

Homemaker 1.2, 9 2.2, 6 – 1.3% 1.1% 1.6% –

Disabled or too ill to work 3.1, 23 1.5, 7 6, 16 – 3.4% 2.5% 4.4% –

Other – 0.1% 0% 0.2% –

Housing status (%, n) – – – <.001 – – – 0.054

Stably housed 46.4, 340 43.8, 203 51.1, 137 – 48% 45.5% 51.1% –

Staying with someone else 33.7, 247 33.4, 155 34.3, 92 – 33.5% 33.1% 34% –

Transitional/temporary 7.5, 55 10.1, 47 3, 8 – 6.4% 9% 3.3% –

Homeless/unstably housed 11.2, 82 11.9, 55 10.1, 27 – 11% 11.5% 10.4% –

Past-year homelessness 14.8, 108 16.4, 76 11.9, 32 0.155 14.2% 15.3% 13% 0.358

Incarcerations status – – – <.001 – – – 0.023

No 90, 659 86, 399 97, 260 – 91.4% 88.3% 95.1% –

Yes 9.8, 72 13.8, 64 3, 8 – 8.5% 11.5% 4.9% –

Target population – – – – – – – –

Young adults aged 16 to 24 

who are not in school and 

21.3, 156 22.4, 104 19.4, 52 0.272 20% 21% 18.9% 0.454

not employed

Adults age 18 or older who 

are unemployed (or are 

underemployed)

66.7, 488 72.4, 336 56.7, 152 <.001 66.4% 69.4% 62.8% 0.202

Caregivers and parents who 

are expecting or have 

children up to age 4

23.4, 171 23.3, 108 23.5, 63 0.971 24.5% 25% 23.8% 0.740

CBO service type – – – <.001 – – – 0.195

Job training and employment 21.9, 160 17.2, 80 29.9, 80 – 59.7% 61.5% 57.5% –

program

Youth development program 1.4, 10 3, 8 – 22.5% 19.9% 25.5% –

Homeless shelter 6.1, 45 5, 23 8.2, 22 – 1.3% 0.7% 2.1% –

Parent/caregiver-serving 

organizations

5.3, 39 5.8, 27 4.5, 12 – 7% 7% 7% –

Domestic violence agency 4.1, 30 5.2, 24 2.2, 6 – 5.7% 6% 5.3% –

Latino/Hispanic agency 21.9, 160 17.2, 80 29.9, 80 – 3.8% 4.9% 2.4% –

NOTES: Cells shaded in gray indicate a sample size less than 5. SD = standard deviation.

371 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.10. Balance Table for Propensity Score Weighting of Retained Study Participants at 12 Months, Baseline Outcomes

Unweighted Sample Propensity Score Weighted Sample

Outcomes

Overall

(n = 732)

C2C 

(n = 464)

Comparison 

(n = 268) p-Value Overall C2C Comparison p-Value

Depression (%, n) – – – 0.024 – – – 0.157

None—minimal 27.5, 201 31.7, 147 20.1, 54 – 24.6% 28.6% 19.8% –

Mild 33.9, 248 31.2, 145 38.4, 103 – 35.7% 33.3% 38.6% –

Moderate 23.1, 169 21.3, 99 26.1, 70 – 23.5% 22.4% 24.8% –

Moderately severe 9.7, 71 9.7, 45 9.7, 26 – 11% 9.9% 12.2% –

Severe 4.6, 34 5, 23 4.1, 11 – 4.2% 4.7% 3.6% –

Anxiety (%, n) – – – 0.607 – – – 0.750

Mild 59.8, 438 59.1, 274 61.2, 164 – 58.7% 57.9% 59.6% –

Moderate 25.1, 184 25.4, 118 24.6, 66 – 27.1% 27.1% 27.1% –

Severe 14.3, 105 15.1, 70 13.1, 35 – 13.7% 14.7% 12.6% –

PTSD (%, n) – – – 0.054 – – – 0.205

Positive screen 51, 373 53.7, 249 46.3, 124 – 51.4% 53.8% 48.6% –

No positive screen 49, 359 46.3, 215 53.7, 144 – 48.6% 46.2% 51.4% –

Alcohol use (%, n) – – – 0.219 – – – 0.601

None 80.9, 592 78.9, 366 84.3, 226 – 81.6% 80.7% 82.7% –

Harmful/hazardous drinking 8.7, 64 9.9, 46 6.7, 18 – 8.7% 9.2% 8.2% –

Alcohol dependence 9.4, 69 9.9, 46 8.6, 23 – 8.9% 8.9% 8.9% –

Other – –

Substance use (%, n) – – – <.001 – – – 0.185

None 42.9, 314 37.7, 175 51.9, 139 – 43.9% 39.7% 48.8% –

Low 32.4, 237 34.1, 158 29.5, 79 – 32.2% 33.9% 30.2% –

Intermediate 14.6, 107 15.7, 73 12.7, 34 – 14.4% 15.2% 13.6% –

Substantial 7, 51 8.8, 41 – 6.5% 8.1% 4.6% –

Severe 2.2, 16 2.8, 13 – 1.8% 2.2% 1.2% –

Psychological distress – – – 0.736 – – – 0.867

% with none/low 24.9, 182 23.7, 110 26.9, 72 – 24.5% 23.6% 25.6% –

% with moderate psychological distress 48.8, 357 48.9, 227 48.5, 130 – 49.6% 49.5% 49.7% –

% with severe mental illness 25.4, 186 26.3, 122 23.9, 64 – 25% 25.7% 24.1% –

Access (mean, SD) – – – – – – – –

Logistical (mean, SD) 0.8, 0.7 0.8, 0.6 0.7, 0.7 0.104 0.8, 0.7 0.8, 0.6 0.7, 0.7 0.644

Attitudinal (mean, SD) 1, 0.6 1.1, 0.6 0.9, 0.6 <.001 1, 0.6 1, 0.6 0.9, 0.6 0.063

Stigma (mean, SD) 0.8, 0.8 0.9, 0.8 0.8, 0.8 0.030 0.8, 0.8 0.9, 0.8 0.8, 0.8 0.183

3.7, 10

NOTES: Cells shaded in gray indicate a sample size less than 5. SD = standard deviation.

372 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.11. Balance Table for Propensity Score Weighting of Retained Study Participants at Either 6 or 12 Months, Baseline 

Demographic Characteristics, Unmet Need Sample for Pooled 1-Year Utilization-Related Outcomes

Unweighted Sample Propensity Score Weighted Sample

Participant Demographic 

Characteristic

Overall

(n = 756)

C2C 

(n = 498)

Comparison 

(n = 258) p-Value Overall C2C Comparison p-Value

Age (mean, SD) 31.1, 12.9 30.7, 12.2 31.9, 14.1 0.257 31, 12.7 30.5, 12.3 31.7, 13.2 0.263

Gender (%, n) – – – <.001 – – – 0.023

Male 37.2, 281 44.4, 221 23.3, 60 – 35.8% 41% 29.7% –

Female 60.6, 458 52.6, 262 – 62.1% 56.2% 69% –

Transgender, genderqueer, or other 2.2, 17 3, 15 – 2.1% 2.8% 1.3% –

Race/ethnicity (%, n) – – – 0.580 – – – 0.725

Hispanic 40.9, 309 40.8, 203 41.1, 106 – 42.4% 41.7% 43.3% –

Black 47, 355 46.2, 230 48.4, 125 – 45.4% 45.3% 45.4% –

White 3, 23 3.6, 18 1.9, 5 – 3% 3.7% 2.2% –

Other 8.3, 63 8.8, 44 7.4, 19 – 8.4% 8.8% 8% –

Income level (%, n) – – – 0.537 – – – 0.728

Less than $5,000 43.8, 331 43.6, 217 44.2, 114 – 45.2% 43.8% 46.8% –

$5,000–10,000 13.5, 102 14.1, 70 12.4, 32 – 12.6% 13.9% 11.1% –

$10,001–20,000 10.2, 77 11.4, 57 7.8, 20 – 10.3% 11.4% 9% –

$20,001–30,000 3.3, 25 3, 15 3.9, 10 – 4.1% 3.4% 5% –

$30,001–40,000 3.4, 26 3.4, 17 3.5, 9 – 4% 3.5% 4.6% –

More than $40,000 2.8, 21 2.2, 11 3.9, 10 – 2.3% 2% 2.7% –

Education level (%, n) – – – 0.247 – – – 0.620

Less than high school 27.8, 210 27.1, 135 29.1, 75 – 29.1% 27.1% 31.5% –

Completed high school with diploma 

or GED

34.4, 260 35.1, 175 32.9, 85 – 34.5% 35.8% 32.9% –

Some college 26.3, 199 24.3, 121 30.2, 78 – 25.6% 24.6% 26.9% –

Completed college 9.4, 71 11, 55 6.2, 16 – 8.8% 10.2% 7% –

Some graduate or professional school – 0.5% 0.6% 0.3% –

Completed graduate or professional

school

1.5, 11 1.6, 8 – 1.4% 1.5% 1.3% –

Employment status (%, n) – – – <.001 – – – 0.612

Unemployed 59, 446 65.9, 328 45.7, 118 – 58.1% 60.7% 55% –

Student 11.8, 89 10.4, 52 14.3, 37 – 11.6% 11.3% 11.8% –

76, 196

373 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

6.2, 31 9.7, 25

3.9, 10

17.3, 86

Unweighted Sample Propensity Score Weighted Sample

Participant Demographic 

Characteristic

Overall

(n = 756)

C2C 

(n = 498)

Comparison 

(n = 258) p-Value Overall C2C Comparison p-Value

Employed part-time 14.8, 112 14.5, 72 15.5, 40 – 15.7% 15.2% 16.3% –

Employed full-time 7.4, 56 – 7.4% 7.3% 7.5% –

Retired and not working 1.6, 12 – 1.3% 0.7% 2% –

Homemaker 0.9, 7 – 1.2% 1.2% 1.3% –

Disabled or too ill to work 3.7, 28 1.4, 7 8.1, 21 – 3.8% 2.8% 5.1% –

Other 0.7, 5 – 0.6% 0.8% 0.5% –

Housing status (%, n) – – – <.001 – – – 0.014

Stably housed 45.8, 346 43.6, 217 50, 129 – 48.1% 45.5% 51.2% –

Staying with someone else 34.8, 263 33.7, 168 36.8, 95 – 34.5% 33.9% 35.3% –

Transitional/temporary 7.8, 59 10.8, 54 1.9, 5 – 5.9% 9% 2.2% –

Homeless/unstably housed 11, 83 11.4, 57 10.1, 26 – 10.8% 11.2% 10.2% –

Past-year homelessness 17.6, 133 20.5, 102 12, 31 0.001 16.4% 18.2% 14.3% 0.092

Incarcerations status – – – 0.001 – – – 0.162

No 88.9, 672 85.9, 428 94.6, 244 – 89.9% 87.9% 92.4% –

Yes 10.8, 82 13.9, 69 5, 13 – 9.8% 11.9% 7.2% –

Target population – – – – – – – –

Young adults aged 16 to 24 who are 

not in school and not employed

22.9, 173 22.3, 111 24, 62 0.589 22.3% 21.7% 23% 0.715

Adults age 18 or older who are 

unemployed (or are underemployed)

68, 514 73.5, 366 57.4, 148 <.001 68.7% 69.3% 68% 0.736

Caregivers and parents who are 

expecting or have children up to 

age 4

21.3, 161 22.1, 110 19.8, 51 0.692 21.2% 22.5% 19.7% 0.435

CBO service type – – – <.001 – – – 0.042

Job training and employment program 62.4, 472 67.3, 335 53.1, 137 – 62.9% 63.5% 62.1% –

Youth development program 22.8, 172 33.3, 86 – 22.6% 20.6% 24.9% –

Homeless shelter 1.2, 9 3.5, 9 – 1% 0% 2.3% –

Parent/caregiver-serving organization 4.2, 32 4.2, 21 4.3, 11 – 4.4% 5.2% 3.6% –

Domestic violence agency 5.2, 39 6, 30 3.5, 9 – 5.1% 5.9% 4.1% –

Latino/Hispanic agency 4.2, 32 5.2, 26 2.3, 6 – 4% 4.8% 3% –

NOTES: Cells shaded in gray indicate a sample size less than 5. SD =	standard	deviation.

374 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.12. Balance Table for Propensity Score Weighting of Retained Study Participants at Either 6 or 12 Months, Baseline Outcomes,

Unmet Need Sample for Pooled 1-Year Utilization-Related Outcomes

Unweighted Sample Propensity Score Weighted Sample

Outcomes

Overall

(n = 756)

C2C 

(n = 498)

Comparison 

(n = 258) p-Value Overall C2C Comparison p-Value

Utilization – – – – – – – –

Visits to outpatient MH providera

(mean, SD)

4.1, 10 4.5, 10.8 3.3, 8.2 0.100 4, 9.4 4.3, 10.4 3.6, 8.1 0.358

Nights in residential treatment

program for alcohol or drug 

problemsa (mean, SD)

3.8, 22.2 5.5, 27.1 0.4, 4.2 <.001 2.5, 18 4.5, 24.2 0.2, 3.1 <.001

Nonclinical providers – – – – – – – –

Used any nonclinical settings or

resources (includes self-help or

family support groups, substance

use agencies, or 12-step-type

programs, called a hotline, or

attended religious/spiritual

places, parks and recreation, or

community centers for mental

health needs)a

37.2, 281 41.4, 206 29.1, 75 0.004 36.3% 39.1% 32.9% 0.273

Inpatient settings

Stayed in inpatient setting 

(includes one or more nights in a

residential treatment program

for alcohol or drug problems or

stayed overnight in a hospital for

emotional, mental health,

alcohol, or drug problems)

b

29.9, 226 33.5, 167 22.9, 59 0.010 27.1% 32% 21.3% 0.012

Emergency settings – – – – – – – –

Went to a hospital, emergency

room, urgent care facility, any

health reasona

40.7, 308 40.6, 202 41.1, 106 0.120 41.5% 41.4% 41.6% 0.126

NOTE: SD = standard deviation. a Past 6 months. b One or more nights in residential treatment program for alcohol/drug problem (past 6 months) or overnight stay in a hospital for emotional, mental health, alcohol,

or drug problems (lifetime at baseline, past 6 months at each follow-up).

375 
Subgroup Balance Tables

Available on request. 

C6.2. Avoiding False Discovery with Multiple Comparisons

When conducting large numbers of simultaneous hypothesis tests, as was done in this study, 

it is important to account for the possibility that some results will achieve statistical significance 

simply by chance. Using a traditional 95% CI, for example, RAND expects 1 in 20 comparisons 

to achieve statistical significance as a result of random error or chance alone. Adjustments 

were therefore made to account for false positives (or Type I errors) due to large numbers of 

comparisons having been made. 

We addressed false positives using the false discovery rate method (Benjamini and 

Hochberg, 1995), which allows the analyst to bound the expected fraction of rejected null 

hypotheses that are mistakenly rejected (i.e., “false discoveries”). The rejection decision for each 

hypothesis in the family of tests is a simple function of the rank of the p-value of the test, the 

total number of tests, and the chosen false discovery rate. 

As described above, we specified confirmatory and exploratory hypotheses. Assessments of 

statistical significance were based on applying the false discovery rate procedure separately to all 

of the confirmatory and exploratory outcome tests in this report using a false discovery rate of 

0.05. We also applied the false discovery rate procedure separately for different groups of 

outcomes as well as the different subgroup analyses. Although this slightly raises the chance of 

false discovery, the false discovery rate procedure assumes all tests are independent. This was 

clearly not the case, so if we were to control for all tests simultaneously, we would expect our 

tests to be underpowered. Accordingly, we ran the procedure within analytic subsets, and we 

control the rate of errors within each subset. Thus, the p-values used to determine significance 

differ for each set of analyses, depending on the number of tests conducted. In each case, we give 

information about the p-values required to determine significance. However, information is also 

presented about nonsignificant trends observed between p <0.05 and the false discovery rate 

cutoff to indicate those results that are approaching statistical significance. 

C6.3. Interpretation of Analysis Results

We first conducted descriptive analyses to summarize the sample baseline characteristics 

and the outcome variables. We tested for differences in characteristics between the C2C and 

comparison groups using t-tests and chi-squared tests. We also examined whether those 

participants who were lost to follow-up differed in any systematic way from those who were 

retained, using t-tests and chi-squared tests. 

For the primary research question in each outcome chapter, we conducted the following 

series of analyses for each outcome using propensity score weights. 

Differences within groups at baseline and each follow-up assessment. We examined 

differences within the C2C group and comparison group at baseline and each follow-up assessment 

for all of the outcomes. Although this method cannot show intervention effects, it can show the 

376 
 

 

 

 

magnitude and direction of changes within both C2C and comparison groups and thus is useful 

for describing the type of change observed. We tested for differences within group using paired 

t-tests for continuous outcomes and McNemar’s test for binary outcomes. 

Differences between groups over time. We examined differences between groups over time 

using chi-squared tests, linear regression, and logistic regression models. For the regression 

models, we present the propensity score weighted models when the sample size is more than 

10 per group, outcome pair, and we present the doubly robust models when the sample size is 

more than 20 per group, outcome pair. These models were weighted using the inverse propensity 

score weights and the coefficient corresponding to the treatment indicator was tested for 

significance using t-tests. 

Intervention effects over time. We examined intervention effects over time using an intentto-treat approach in which we compared all individuals in the C2C group with all those in the 

comparison group, regardless of the actual amount of intervention that the C2C group received. 

• Propensity score weighted models. Because any change in outcomes observed can 

potentially be the result of a time trend observed in all individuals in the study, we used a 

difference-in-differences method with the propensity score weights to assess C2C’s 

unadjusted impact (when the sample is at least ten per group, outcome pairing). With 

continuous outcomes, the unadjusted difference in differences is the difference between 

the average change in an individual’s outcome from baseline to follow-up between the C2C 

and comparison groups. For binary outcomes, the unadjusted difference in differences is 

the change in the proportion between baseline and follow-up, contrasted between the 

C2C and comparison groups, which is transformed and interpreted as an odd ratio. 

• Doubly robust models. At the 6- and 12-month follow-ups, we also conducted multiple 

linear regressions on the continuous outcomes and linear probability regressions on the 

binary outcomes to test for the difference in differences via main effects and the 

interaction between intervention status and time after controlling for demographic 

characteristics at baseline and follow-up (age, gender, race/ethnicity, income, 

employment status, education level, housing status, recent homelessness, and 

incarceration status) and subgroup (target population, CBO service type). We also 

included baseline outcome values for the corresponding outcome models to control for 

any potential differences in changes based on individuals’ starting values. We selected 

these characteristics to correct for any potential imbalance in the groups by relevant 

demographic or other characteristics. We present the doubly robust models when the 

sample size is more than 20 per group. For continuous outcomes, we report Cohen’s d as 

the standardized effect size. For binary outcomes, we report the OR as the effect size. 

The standardized effect size was always derived from the doubly robust model. 

For the secondary research question of whether C2C program effectiveness varied for 

different target populations (e.g., adults age 18 or older who are unemployed or underemployed, 

young adults ages 16–24 who are not in school and are not employed, and parents/primary 

caregivers who are expecting or who have children up to the age of 4) or CBO service types 

(e.g., job training and employment program, youth development program), we conducted this 

same series of analyses using propensity score weighting recomputed for each subgroup analysis. 

377 
 

 

 

 

 

 

 

 

C6.4. Power analysis

Power analysis. For the client survey, we retained a total of 689 participants in the study at 

6 months and 732 at 12 months. Given the sample size, there was sufficient power to detect an 

intervention effect of size 0.223 at 6 months and 0.215 at 12 months, which are medium-sized 

effects according to Cohen’s effect size classification (Cohen, 1988). For the pooled utilization 

analysis that combined 6- and 12-month data, there was sufficient power to detect a small 

intervention effect size (0.215). 

C7. Study Enrollment, Retention, and Sample Characteristics

C7.1. Participant Enrollment and Retention

As described earlier, to be eligible, participants must have screened positive on at least one of 

the five screeners using the lower study eligibility thresholds. Overall, 2,600 C2C eligibility 

screenings were completed, 64 percent of which were from the C2C group and 36 percent from 

the comparison group (Table C.13). This total number of screenings fell below the target of 3,125. 

Table C.13. Final Study Enrollment

Total C2C Group Comparison Group

Completed study eligibility screenings 2,600 1,657 943

Eligible based on screenings 2,106 1,359 747

Percentage screened à eligible 81 82 79

Completed baseline surveys 1,838a 1,232 606

Percentage eligible à completed baselines 87 91 81

a Initially, there were 1,860 completed baseline surveys. After data cleaning and processing, the total decreased to

1,838 usable surveys.

Recruitment for participants from the C2C CBOs typically started earlier than comparison 

CBOs, because the latter entered into agreements with us later in the course of the project. 

Overall, 81 percent of individuals screened were eligible for the study, meaning scores met the 

minimum threshold for at least one of the five measures on the eligibility screener (see Table C.13). 

Individuals who screened as eligible were offered the opportunity to participate in the study. 

Subsequent baseline surveys were completed for 87 percent of eligible screenings overall, above 

the target of 80 percent of eligibility screenings. The baseline completion rate was somewhat 

higher at C2C CBOs (91 percent) than comparison group CBOs (81 percent). 

Some of the client-level challenges to completing study screenings and enrolling participants 

in the study included competing demands for attention (e.g., participating in CBO classes), time 

constraints (e.g., parents from parent/caregiver-serving organization sites being approached at 

drop-off or pickup), and fatigue/disinterest in the client survey after in-depth psychosocial intake 

processes at some C2C sites, especially those serving youth. Other structural issues contributed 

378 
 

 

 

 

 

 

to lower-than-anticipated recruitment and enrollment. These included low client turnover at some 

CBOs (e.g., homeless shelters, parent/caregiver-serving organizations), small client populations 

at comparison CBOs, and a smaller pool of C2C participants due to C2C program enrollment 

lower than originally estimated, or delays in approval to recruit certain groups of clients within 

the CBO (e.g., parolees). 

At both C2C and comparison CBOs, other recruitment challenges related to the data collection 

process, including a need for customization and ongoing adjustments to the data collection 

process at each site (e.g., messaging, workflow, staff training), limited private space to conduct 

screenings, variable staff buy-in and support, and high CBO staff turnover that resulted in breaks 

in continuity for data collection activities. Further, at C2C CBOs, there was often only a small 

window of time to conduct the baseline assessment before clients received more in-depth 

C2C services. The data collection workflow was also sometimes seen as extraneous to C2C 

implementation by CBO staff. Finally, there was a 2-week data collection suspension of study 

enrollment because of contractual issues with the Mayor’s Fund. 

Table C.14 shows the enrollment by group, comparing the actual enrollment with the target 

enrollment needed for power, using our initial assumption of an 80-percent retention rate. With 

total enrollment of 1,838, the study enrolled less than the sample size required to detect a small 

intervention effect (74 percent of target). The overall retention rates at 6 and 12 months meant 

that it retained a total of 688 participants in the study at 6 months and 732 at 12 months, which 

was below the number necessary (34 and 37 percent, respectively) to have an 80-percent chance 

of detecting a small intervention effect (d ≈ 0.20). Given the sample size, there was sufficient 

power to detect an intervention effect of size 0.223 at 6 months and 0.215 at 12 months, according 

to Cohen’s effect size classification (Cohen, 1988). 

Table C.14. Required Versus Actual Enrollment for a Small Effect Size

Requirement Total C2C Group

Comparison 

Group

Enrolled sample needed for power 2,500 1,250 1,250

Total enrollment 1,838 1,232 606

Percentage of needed enrollment 74 99 48

Retained sample needed for power 2,000 1,000 1,000

Retained sample, 6 months 688 443 245

Percentage of needed retention, 6 months 34 44 25

Retained sample, 12 months 732 464 268

Percentage of needed retention, 12 months 37 46 27

Retained at one or more time points (pooled 6 and 12 months) 904 584 320

Percentage of needed retention, pooled 6- and 12-month analysis 45 58 32

As noted above, the study enrolled a total of 1,838 participants in the study, with 1,232 in the 

intervention group and 606 in the comparison group. In Table C.15, we present the number and 

379 
 

 

 

 

 

 

 

percentage of all enrollees who were eligible for participation at each data collection time point. 

As shown, 36 percent of the participants in the C2C group and 40 percent of the participants in 

the comparison group were retained for the 6-month assessment. Retention for participants at 

12 months was 38 percent for the C2C group and 44 percent for the comparison group. 

Table C.15. Retention of Participants Eligible to Participate at Each Time Point

Participants 6 Months 12 Months

1 Year (Pooled 

6 and 12 Months)

Intervention (C2C)

Completed 443 464 584

Eligible 1,232 1,232 1,232

Retention rate (%) 36 38 47

Comparison

Completed 245 268 320

Eligible 606 606 606

Retention rate (%) 40 44 53

Total

Completed 688 732 904

Eligible 1,838 1,838 1,838

Retention rate (%) 37 40 49

Retention-related challenges included the 

• inability to reach participants by phone due to disconnected numbers, temporarily 

suspended numbers, changed phone numbers, or full voicemail boxes 

• Inability to reach participants by email due to participant typographical errors in provided 

email addresses 

• participants who did not recognize the C2C evaluation brand or their prior study 

participation because of differences in C2C messaging, branding, and implementation 

across CBOs 

• participants who no longer engaged with CBO at time of follow-up 

• participants dissatisfied with CBO experience who tended to decline or discontinue 

participation in the evaluation. 

Across the three time points, 28 percent of study participants completed all three assessments 

(Table C.16). About one-fifth of the sample completed only the 6-month follow-up (9 percent) or 

only the 12-month follow-up (12 percent). 

We also faced challenges because the assessment was time intensive (45–60 minutes) and 

covered sensitive and possibly distressing topics. Further, study contacts (e.g., scheduling calls, 

reminder emails and texts) competed with a surge in spam messages and “robo” phone calls. 

Recent studies suggest that telephone survey response rates have been slowly declining for 

reasons such as using caller ID to screen calls, growth in cell phone usage, increase in 

solicitations, and other factors (Czajka and Beyler, 2016). 

380 
 

 

 

 

 

 

 

Table C.16. Study Participants by Time Point

Total C2C Group Comparison Group

Time Point n percentage n percentage n percentage

Baseline only 934 51 648 53 286 47

Baseline + 6 months only 172 9 120 10 52 9

Baseline + 12 months only 216 12 141 11 75 12

Baseline + 6 months + 12 months 516 28 323 26 193 32

Baseline + one or more follow-up time

points

904 49 584 47 320 53

Although we based our initial power analysis on a target 80-percent retention rate, study 

participants had very low income with high unemployment rates, unstable housing, and recent 

incarcerations. Other longitudinal studies, including those with disadvantaged populations, have 

had retention rates ranging from about 20 percent to nearly 100 percent (Altena et al., 2010; 

Davis, Broome, and Cox, 2002; Leonard et al., 2003; Teague et al., 2018). 

We examined sample characteristics by study retention at 6 months to compare characteristics 

of the study participants at baseline with the sample retained at each follow-up. At 6 months, 

there were differences between those retained and those lost to follow-up by gender, age, 

educational attainment, and income (Table C.17). Study participants who were not surveyed at 

6 months were more likely to be male (particularly in the comparison group), younger, have 

lower education levels, and have lower income (particularly the C2C group) at baseline. At 

12 months, there were differences between those retained and those lost to follow-up by gender, 

age, educational attainment, employment status, and housing status (Table C.18). Study 

participants who were not surveyed at 12 months were more likely to be male, younger, have 

lower educational attainment, to have been unemployed, and to be less than stably housed at 

baseline. 

381 
 

 

 

 

 

Table C.17. Baseline Sample Characteristics by Study Retention at 6 Months (Unweighted)

Total C2C Group Comparison Group

Retained 

Sample 

Lost to Retained 

Sample 

Lost to Retained 

Sample 

Lost to 

Sample at 

6 Months

Follow-Up 

at 6 Months

Sample at 

6 Months

Follow-Up 

at 6 Months

Sample at 

6 Months

Follow-Up 

at 6 Months

Characteristics (n = 688) (n = 1150) p-Value (n = 443) (n = 789) p-Value (n = 245) (n = 361) p-Value

Gender <.001 0.059 <.001

Male 36.9% 48.4% 45.4% 51.8% 21.6% 40.8%

Female 60.8% 50.4% 51.9% 46.5% 76.7% 58.9%

Transgender, genderqueer, or

other

2.3% 1.2% 2.7% 1.7% 1.6% 0.3%

Age (mean, SD) 31.4 (3.0) 29.3 (12.1) .001 30.9 (12.5) 29.3 (11.9) 0.020 32.4 (13.8) 29.5 (12.8) 0.013

Race/ethnicity 0.516 0.286 0.649

Hispanic 42.6% 42.4% 43.3% 40.6% 41.3% 46.5%

Black 46.4% 48% 44.7% 49.1% 49.6% 45.7%

White 2.9% 3.2% 3.2% 3.7% 2.5% 2.2%

Other 8.1% 6.3% 8.8% 6.6% 6.6% 5.6%

Education level 0.001 0.029 0.015

Less than high school 28.4% 34.5% 28.7%, 36.2% 27.8% 30.6%

Completed high school with 

diploma or GED

34.1% 36.3% 36.2% 35.4% 30.2% 38.3%

Some college 24.7% 22.1% 22.4% 20.4% 29% 25.9%

Completed college 9.9% 5.4% 10.2% 5.8% 9.4% 4.3%

Some graduate or professional

school

0.7% 0.6% 0.7% 0.7% 0.8% 0.3%

Completed or some graduate or

professional school

2.2% 1.2% 1.8% 1.5% 2.9% 0.6%

Employment status 0.169 0.386 0.357

Unemployed, looking for work 56.5% 63.3% 63.7% 68% 43.7%, 52.6%

Student 12.8% 11% 11.5% 11.9% 15.1% 9%

Employed part-time 15.8% 14.8% 13.8% 13% 19.6% 18.9%

382 
 

 

 

Total C2C Group Comparison Group

Characteristics

Retained 

Sample at 

6 Months

(n = 688)

Sample 

Lost to 

Follow-Up 

at 6 Months

(n = 1150) p-Value

Retained 

Sample at 

6 Months

(n = 443)

Sample 

Lost to 

Follow-Up 

at 6 Months

(n = 789) p-Value

Retained 

Sample at 

6 Months

(n = 245)

Sample 

Lost to 

Follow-Up 

at 6 Months

(n = 361) p-Value

Employed full-time

Retired and not working

Homemaker

Disabled or too ill to work

Other

Individual income level

Less than $5,000

$5,000–10,000

$10,001–20,000

$20,001–30,000

$30,001–40,000

More than $40,000

Housing status

Stably housed

Staying with someone else

Transitional/temporary

Homeless/unstably housed

9%

1.5%

1.3%

2.5%

0.6%

52.9%

17.3%

15.1%

5.5%

5.1%

4.1%

46%

35.3%

8.8%

9.8%

6.3%

1%

0.8%

2.1%

0.6%

62.9%

14.4%

9.9%

6.8%

3.8%

2.2%

42%

36.4%

10.1%

11.4%

0.001

0.336

7.9%

0.5%

0.9%

1.1%

0.7%

53.6%

17.8%

15.7%

4.5%

5.4%

3%

42.4%

35.6%

11.8%

10.2%

4.7%

0.5%

0.4%

0.7%

0.7%

63.7%

14.5%

10%

6.4%

3.4%

2.1%

40.7%

33.9%

12.5%

12.9%

0.011

0.526

11%

3.3%

2%

4.9%

0.4%

51.7%

16.3%

14%

7.3%

4.5%

6.2%

52.7%

34.9%

3.3%

9.1%

9.9%

2.2%

1.9%

5.3%

0.3%

61.1%

14.3%

9.8%

7.8%

4.5%

2.5%

45.2%

42.1%

4.7%

8.1%

0.222

0.238

NOTE: SD = standard deviation.

383 
 

 

 

 

 

Table C.18. Baseline Sample Characteristics by Study Retention at 12 Months (Unweighted)

Total C2C Group Comparison Group

Retained 

Sample at 

12 Months 

Sample Lost

to Follow-Up 

at 12 Months 

Retained 

Sample at 

12 Months

Sample Lost

to Follow-Up 

at 12 Months 

Retained 

Sample at 

12 Months

Sample Lost

to Follow-Up 

at 12 Months

(n = 732) (n = 1,106) p-Value (n = 464) (n = 768) p-Value (n = 268) (n = 338) p-Value

Gender <.001 0.001 <.001

Male 35% 50.1% 43.3% 53.3% 20.5% 43%

Female 62.7% 48.7% 53.7% 45.3% 78.4% 56.4%

Transgender, gender

queer, or other

2.3% 1.2% 3% 1.4% 1.1% 0.6%

Age (mean, SD) 31.6 (13.4) 29.1 (11.8) <.001 30.9 (12.6) 29.2 (11.8) 0.011 33.0 (14.6) 28.8 (11.8) 0.001

Race/ethnicity 0.072 0.011 0.779

Hispanic 41.4% 43.2% 39.8% 42.6% 44.2% 44.6%

Black 47.7% 47.2% 47.2% 47.7% 48.7% 46.1%

White 2.3% 3.6% 2.6% 4.1% 1.9% 2.7%

Other 8.5% 5.9% 10.4% 5.6% 5.2% 6.6%

Education level <.001 <.001 0.025

Less than high school 27.1% 35.7% 27.4% 37.3% 26.6% 31.8%

Completed high school 33.8% 36.5% 34.8% 36.3% 32.2% 37.1%

with diploma or GED

Some college 25.3% 21.5% 23.5% 19.6% 28.5% 26.2%

Completed college 10.7% 4.6% 11.4% 4.9% 9.4% 4%

Some graduate or

professional school

0.7% 0.6% 0.6% 0.7% 0.7% 0.3%

Completed or some 

graduate or professional

school

2.3% 1.1% 2.2% 1.3% 2.6% 0.7%

Employment status 0.001 0.064 0.002

Unemployed, looking 

for work

56.7% 63.5% 65.7% 66.8% 41.2% 55.5%

Student 12.3% 11.3% 10.8% 12.4% 15% 8.6%

Employed part-time 15.3% 15.1% 13.4% 13.2% 18.7% 19.6%

Employed full-time 9% 6.2% 7.6% 4.9% 11.6% 9.3%

384 
 

Total C2C Group Comparison Group

Retained 

Sample at 

12 Months 

(n = 732)

Sample Lost

to Follow-Up 

at 12 Months 

(n = 1,106) p-Value

Retained 

Sample at 

12 Months

(n = 464)

Sample Lost

to Follow-Up 

at 12 Months 

(n = 768) p-Value

Retained 

Sample at 

12 Months

(n = 268)

Sample Lost

to Follow-Up 

at 12 Months

(n = 338) p-Value

Retired and not working

Homemaker

2.1%

1.2%

0.6%

0.9%

0.4%

0.6%

0.6%

0.6%

4.9%

2.2%

0.7%

1.7%

Disabled or too ill to work

Other

Individual income level

Less than $5,000

$5,000–10,000

$10,001–20,000

$20,001–30,000

$30,001–40,000

More than $40,000

Housing status

Stably housed

Staying with someone 

else

Transitional/temporary

Homeless/unstably

housed

3.2%

0.1%

56.2%

16.4%

13.3%

5.8%

4.4%

3.9%

47%

34.1%

7.6%

11.3%

1.6%

0.9%

61.2%

14.8%

10.9%

6.6%

4.2%

2.2%

41.2%

37.3%

11%

10.4%

0.224

0.020

1.5%

0%

56.4%

17.3%

12.9%

5.5%

4.9%

3%

44.1%

33.7%

10.2%

12%

0.4%

1.1%

62.4%

14.6%

11.5%

5.8%

3.6%

2%

39.5%

35.1%

13.5%

11.8%

0.483

0.247

6%

0.4%

55.7%

14.9%

13.9%

6.5%

3.5%

5.5%

51.9%

34.8%

3%

10.2%

4.3%

0.3%

58.4%

15.4%

9.5%

8.6%

5.4%

2.7%

45.3%

42.6%

5%

7%

0.366

0.091

NOTE: SD = standard deviation.

385 
 

 

 

 

C7.2. Sample Characteristics

Table C.19 provides a summary of baseline demographics for the sample of study participants 

(totals may add up to more than 100 percent due to rounding). 

Overall, 54 percent identified as female; 44 percent as male; and 2 percent as transgender, 

genderqueer, or another gender identity. The two groups differed significantly at baseline, with 

50 percent of the C2C group identifying as male compared with 66 percent for the comparison 

group. 

The average age of study participants at baseline was 30.1 years of age. About 44 percent of 

the study participants were 18 to 24 at the time of the baseline survey, 15 percent were 25 to 

30 years of age, and 16 percent were 31 to 39 years of age; just more than one-fifth (21 percent) 

of the study participants were 40 or older at the time of the baseline survey. A small percentage 

of study participants (5 percent) were younger than 18 years at baseline. Although the mean age 

was similar for the two groups, the distribution across age categories differed significantly with 

more of the C2C group younger than 18 and more of the comparison group 18 to 24 years of age. 

A substantial minority (43 percent) of the study participants identified as Hispanic/Latino. 

Forty-seven percent of study participants identified as Black/African American non-Hispanic, 

with smaller percentages identifying as White/Caucasian non-Hispanic (3 percent) and American 

Indian or Alaskan Native, Asian or Pacific Islander, or some other race/ethnicity (7 percent). 

The education level of participants at baseline ranged from less than high school completion 

(32 percent) to completed graduate or professional school (2 percent). More than one-third 

(35 percent) had completed high school with a diploma or GED at the time of the baseline 

survey, 23 percent had completed some college, and 7 percent had completed college. 

Seven percent of study participants had full-time employment and 15 percent had part-time 

employment at the time of the baseline survey. However, nearly two-thirds (61 percent) were 

unemployed and looking for work. Small percentages of participants were retired, students, 

homemakers, disabled or too ill to work, or had some other employment status. There were 

significant differences between groups by employment status, with many more C2C participants 

unemployed compared with comparison group participants (66 percent vs. 49 percent). 

The majority of study participants (59 percent) reported an individual income under $5,000 

per year. Another 27 percent reported annual incomes between $5,000 and $20,000. Eleven 

percent reported income between $20,001 and $40,000 per year. 

Many study participants were stably housed at baseline (44 percent), meaning that they 

were owners, renters, or lived in permanent supportive housing, a group home, or supervised 

residential facility. More than one-third (36 percent) were staying with someone else; 10 percent 

were in transitional or temporary housing such as a psychiatric facility, substance abuse treatment 

facility, correctional facility, or halfway house; and another 11 percent were homeless or 

otherwise unstably housed, meaning that they lived in a shelter or had no place to live. Again, 

there were significant differences by group, with many more C2C participants in transitional or 

temporary housing (12 vs. 4 percent) and homeless or otherwise unstably housed (12 vs. 9 percent) 

than comparison group participants. 

386 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.19. Sample Characteristics: Full Unweighted Sample at Baseline

Full Baseline Sample

Overall C2C Comparison 

Participant Demographic Characteristic (n = 1,838) (n = 1,232) (n = 606) p-Value

Age (mean, SD) 30.1, 12.5 29.8, 12.1 30.7, 13.2 0.279

Age (%, n)

Less than 18 4.6, 84 5.8, 71 2.1, 13 0.010

18–24 43.7, 803 42.4, 522 46.4, 281

25–30 15.2, 280 15.3, 189 15, 91

31–39 15.6, 286 15.7, 193 15.3, 93

40 or older 20.9, 385 20.9, 257 21.1, 128

Gender (%, n) <.001

Male 44.1, 809 49.5, 609 33.1, 200

Female 54.3, 996 48.5, 596 66.1, 400

Transgender, genderqueer, or other 1.6, 30 2, 25 0.8, 5

Race/ethnicity (%, n) 0.280

Hispanic 42.5, 777 41.6, 510 44.4, 267

Black 47.4, 867 47.5, 583 47.3, 284

White 3.1, 57 3.5, 43 2.3, 14

Other 6.9, 127 7.4, 91 6, 36

Income level (%, n) 0.440

Less than $5,000 59.1, 789 60, 548 57.1, 241

$5,000–10,000 15.5, 207 15.7, 143 15.2, 64

$10,001–20,000 11.9, 159 12, 110 11.6, 49

$20,001–30,000 6.3, 84 5.7, 52 7.6, 32

$30,001–40,000 4.3, 57 4.2, 38 4.5, 19

More than $40,000 2.9, 39 2.4, 22 4, 17

Education level (%, n) 0.115

Less than high school 32.1, 562 33.4, 395 29.3, 167

Completed high school with diploma or GED 35.4, 620 35.7, 422 34.8, 198

Some college 23.1, 405 21.2, 250 27.2, 155

Completed college 7.1, 125 7.4, 88 6.5, 37

Some graduate or professional school 0.6, 11 0.7, 8 0.5, 3

Completed graduate or professional school 1.6, 28 1.6, 19 1.6, 9

Employment status (%, n) <.001

Unemployed 60.6, 1,060 66.4, 783 48.8, 277

Student 11.7, 205 11.8, 139 11.6, 66

Employed part-time 15.2, 266 13.3, 157 19.2, 109

Employed full-time 7.4, 129 5.9, 70 10.4, 59

Retired and not working 1.2, 21 0.5, 6 2.6, 15

Homemaker 1, 18 0.6, 7 1.9, 11

Disabled or too ill to work 2.2, 39 0.8, 10 5.1, 29

Other 0.6, 10 0.7, 8 0.4, 2

Housing status (%, n) <.001

Stably housed 43.6, 755 41.3, 483 48.4, 272

Staying with someone else 36, 623 34.6, 404 39, 219

Transitional/temporary 9.6, 166 12.2, 143 4.1, 23

Homeless/unstably housed 10.8, 187 11.9, 139 8.5, 48

NOTE: SD = standard deviation.

387 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

For mental health utilization outcomes (Chapter 5), we isolated analyses to the subsample of 

persons with unmet need at baseline. Unmet mental health need at baseline was defined as 

evidencing one or more screening scores above the moderate or intermediate threshold for the 

baseline mental health indicators (depression, generalized anxiety, psychological distress, PTSD, 

alcohol use, and substance use). At baseline, 81.3 percent of participants had unmet mental health 

need (n = 1,494). This subsample is described in Table C.20. The distribution of demographic 

characteristics was very similar to the overall sample’s distribution. Statistically significant 

differences between the C2C group and comparison group were found for the same characteristics 

as those at baseline (age, gender, employment status, and housing status). In addition, there was a 

difference between groups in educational attainment. 

Table C.20. Sample Characteristics: Full Unweighted Subsample with Unmet Mental Health Need

at Baseline

Full Baseline Sample

Participant Demographic Characteristic

Overall

(n = 1,494)

C2C 

(n = 1,017)

Comparison 

(n = 477) p-Value

Age (mean, SD) 30.0, 12.2 30.0, 11.9 30.0, 12.7 0.829

Age (%, n)

Less than 18 4.2, 63 5.5, 56 1.5, 7 0.001

18–24 44.0, 658 41.6, 423 49.3, 235

25–30 15.5, 231 15.4, 157 15.5, 74

31–39 16, 239 16.9, 172 14.0, 67

40 or older 20.3, 303 20.6, 209 19.7, 94

Gender (%, n)

Male 43.6, 651 48.7, 495 32.7, 156 <.001

Female 54.5, 814 48.8, 496 66.7, 318

Transgender, genderqueer, or other 1.9, 28 2.5, 25 0.6, 3

Race/ethnicity (%, n)

Hispanic 42.5, 631 41.8, 423 44, 208 0.299

Black 46.4, 689 46.2, 468 46.7, 221

White 3.6, 53 4.2, 42 2.3, 11

Other 7.5, 112 7.8, 79 7.0, 33

Income level (%, n)

Less than $5,000 59.5, 658 60.3, 458 57.8, 200 0.330

$5,000–10,000 16.1, 178 15.9, 121 16.5, 57

$10,001–20,000 11.9, 132 12.6, 96 10.4, 36

$20,001–30,000 5.3, 59 4.6, 35 6.9, 24

$30,001–40,000 4.4, 49 4.3, 33 4.6, 16

More than $40,000 2.7, 30 2.2, 17 3.8, 13

Education level (%, n)

Less than high school 31.6, 457 32.7, 323 29.4, 134 0.038

Completed high school with diploma or GED 35.4, 511 35.3, 349 35.5, 162

Some college 23.9, 345 21.8, 215 28.5, 130

Completed college 7, 101 7.8, 77 5.3, 24

Some graduate or professional school 0.6, 9 0.7, 7 0.4, 2

Completed graduate or professional school 1.5, 21 1.7, 17 0.9, 4

388 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Full Baseline Sample

Participant Demographic Characteristic

Overall

(n = 1,494)

C2C 

(n = 1,017)

Comparison 

(n = 477) p-Value

Employment status (%, n)

Unemployed 

Student

Employed part-time

Employed full-time

Retired and not working

Homemaker

Disabled or too ill to work

Other

Housing status (%, n)

Stably housed

Staying with someone else

Transitional/temporary

Homeless/unstably housed

61.3, 885

11.6, 167

14.8, 214

6.9, 100

1.2, 17

1.0, 15

2.4, 35

0.7, 10

42.5, 606

36.2, 517

9.7, 139

11.6, 165

66.3, 656

11.7, 116

13.4, 133

5.5, 54

0.6, 6

0.6, 6

1.0, 10

0.8, 8

40.7, 398

34.4, 336

12.5, 122

12.4, 121

50.4, 229

11.2, 51

17.8, 81

10.1, 46

2.4, 11

2.0, 9

5.5, 25

0.4, 2

46.2, 208

40.2, 181

3.8, 17

9.8, 44

<.001

<.001

NOTE: SD = standard deviation.

Table C.21. Baseline Barriers to Seeking Professional Care for a Mental Health Problem—

Individual Barrier Endorsement (Weighted)

Full Sample C2C Group

Comparison

Group p-Value

Endorsement of Individual Barriers Percent Percent Percent

Within Each Domain Endorsed Endorsed Endorsed p-Value

Logistical barriers

Being unsure where to go to get professional care

Problems with transport or traveling to

appointments

Not being able to afford the financial costs involved

Professionals from my own ethnic or cultural group 

not being available

Being too unwell to ask for help

Difficulty taking time off work

Having problems with childcare while I receive 

professional care

Having no one who could help me get professional

care

Attitudinal barriers

Wanting to solve the problem on my own

Fear of being put in hospital against my will

Thinking the problem would get better by itself

Preferring to get alternative forms of care (e.g.,

traditional/religious healing or

alternative/complementary therapies)

Thinking that professional care probably would 

not help

Dislike of talking about my feelings, emotions, or 

thoughts

46.6

39.1

55.7

28.1

36.3

41.1

29.6

38.6

76.7

46.2

63.0

41.6

51.9

66.4

47.5 45.6 0.520

37.5 40.9 0.226

57.1 54.2 0.300

28.5 27.7 0.758

34.6 38.3 0.188

43.1 38.8 0.169

30.2 29.1 0.751

39.3 37.9 0.632

77.2 76.0 0.614

47.1 45.1 0.491

65.7 59.9 0.036

41.8 41.4 0.885

50.7 53.3 0.364

66.9 65.9 0.690

389 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Endorsement of Individual Barriers Comparison

Within Each Domain Full Sample C2C Group Group p-Value

Concerns about the treatments available (e.g.,

medication side effects)

Having had previous bad experiences with 

professional care for mental health

Preferring to get help from family or friends

Thinking I did not have a problem

Stigma barriers

Concern that I might be seen as weak for having a 

mental health problem

Concern that it might harm my chances when 

applying for jobs

Concern about what my family might think, say, do,

or feel

Feeing embarrassed or ashamed (internalized

stigma)

Concern that I might be seen as “crazy”

Concern that people I know might find out

Concern that people might not take me seriously if

they found out I was having professional care

Concern that I might be seen as a bad parent

Not wanting a mental health problem to be on my

medical records

Concern that my children may be taken into care or

that I may lose access or custody without my 

agreement

Concern about what my friends might think, say,

or do

Concern about what people at work might think,

say, or do

59.0

31.7

45.9

52.0

46.2

46.9

49.9

49.3

45.6

37.7

45.6

40.4

47.0

37.8

33.9

33.6

59.6 58.3 0.649

32.7 30.4 0.413

45.9 45.9 0.986

54.0 49.7 0.142

48.0 44.1 0.180

48.7 45.0 0.209

49.8 50.0 0.944

51.2 47.1 0.151

45.4 45.8 0.893

39.1 36.1 0.290

47.5 43.4 0.155

42.3 38.5 0.303

47.7 46.3 0.628

37.7 37.8 0.980

34.4 33.3 0.686

34.5 32.7 0.552

390 
 

 

 

 

 

 

 

 

 

C8. Additional Analysis Results

C8.1. 6-month Analysis Results

C8.1.1. Barriers to Mental Health Care

Table C.22. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 401 −0.293 0.034 <.001

Comparison 226 −0.285 0.050 <.001

Attitudinal barriers C2C 401 −0.385 0.036 <.001

Comparison 226 −0.340 0.044 <.001

Stigma barriers C2C 400 −0.330 0.039 <.001

Comparison 226 −0.353 0.053 <.001

Internalized stigma C2C 382 −0.417 0.070 <.001

Comparison 214 −0.482 0.097 <.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately.

Table C.23. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Group-Level Comparison of Mean Changes

Propensity Score Weighted

Modela Doubly Robust Modelb

Standardized 

Effect Size for

Outcome Estimate SE p-Value Estimate SE

pValue

Doubly 

Robust Model

Logistical barriers −0.008 0.060 0.892 0.012 0.052 0.813 0.018

Attitudinal barriers −0.045 0.057 0.433 0.016 0.054 0.772 0.023

Stigma barriers 0.023 0.065 0.722 0.016 0.062 0.802 0.020

Internalized stigma 0.065 0.120 0.588 0.094 0.090 0.296 0.069

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

391 
 

 

 

 

 

 

 

 

 

C8.1.2. Mental Health Symptoms

Table C.24. Within-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Depression (score range 0–24) C2C

Comparison

421

236

−1.45

−1.81

0.29

0.36

<.001

<.001

Generalized anxiety (score range 0–21) C2C

Comparison

419

236

−1.90

−1.62

0.28

0.34

<.001

<.001

PTSD (score range 0–80) C2C

Comparison

419

236

−7.75

−7.09

0.83

1.13

<.001

<.001

Alcohol use (score range 0–40) C2C

Comparison

414

234

−1.47

−0.94

0.24

0.37

<.001

0.011

Substance use (score range 0–10) C2C

Comparison

409

228

−0.97

−0.42

0.10

0.10

<.001

<.001

Psychological distress (score range 0–24) C2C

Comparison

403

231

−2.63

−2.21

0.27

0.37

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately.

Table C.25. Between-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted

Modela

Estimate SE p-Value

Doubly 

Robust Modelb

Estimate SE p-Value

Standardized 

Effect Size for

Doubly 

Robust Model

Depression 0.355 0.461 0.442 −0.157 0.403 0.697 −0.027

Generalized anxiety −0.286 0.439 0.515 −0.183 0.395 0.643 −0.033

PTSD −0.660 1.41 0.639 −0.478 1.29 0.711 −0.029

Alcohol use −0.532 0.442 0.228 −0.100 0.281 0.722 −0.019

Substance use −0.544 0.134 <.001 −0.147 0.103 0.154 −0.075

Distress −0.420 0.456 0.358 0.033 0.393 0.934 0.006

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

392 
 

 

 

 

 

 

 

 

 

Table C.26. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 6 Months

Group-Level Comparison of

Percentage with Change

Outcome N

Percentage

with Change p-Value

Depression (clinically significant change) C2C 421 27.88 0.582

Comparison 236 30.01

Generalized anxiety (clinically significant C2C 419 8.35 0.461

change) Comparison 236 6.77

Generalized anxiety (reliable change) C2C 419 31.26 0.211

Comparison 236 26.46

PTSD (clinically significant change) C2C 419 41.26 0.392

Comparison 236 37.66

PTSD (reliable change) C2C 419 53.51 0.873

Comparison 236 52.82

Alcohol use (change to below positive screen 

threshold)

C2C

Comparison

414

234

12.61

9.43

0.265

Substance use (change to below positive 

screen threshold)

C2C

Comparison

409

228

16.36

9.55

0.021a

Psychological distress (change to below

threshold for serious mental illness)

C2C

Comparison

403

231

25.78

24.74

0.785

Psychological distress (change to below

threshold for moderate distress)

C2C

Comparison

403

231

69.65

66.21

0.398

a Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

393 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

C8.1.3. Employment, Education, Housing, and Incarceration

Table C.27. Within-Group Differences in FT/PT Employment at 6 Months (Weighted)

Employment at 6 Months

Outcome

Baseline Percentage 

(Retained Sample)

6 Months

(%)

Difference

(%) p-Value

FT or PT employment (vs.

unemployed)a C2C

Comparison

29.92

40.28

61.04

63.29

31.13

23.01

<.001

<.001

NOTES: FT = full-time; PT = part-time. a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline.

Table C.28. Within-Group Mean Changes in Hours Worked and Employment Income at 6 Months 

(Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-value

Hours worked per week C2C

Comparison

298

145

11.29

9.31

1.22

2.04

<.001

<.001

Current monthly pay before 

taxes and deductions

C2C

Comparison

263

121

$650 USD

$688 USD

90

123

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between baseline and 6 months for each group separately.

Table C.29. Within-Group Differences in Housing at 6 Months (Weighted)

Housing at 6 Months

Outcome

Baseline Percentage

(Retained Sample)

6 Months

(%)

Difference

(%) p-Value

Aggregate current housing

Stably housed or staying 

with someone elsea C2C

Comparison

78.61

85.82

85.13

90.31

6.52

4.49

<.001

<.001

a Versus transitional/temporary or homeless/unstably housed.

Table C.30. Within-Group Differences in Educational Attainment at 6 Months (Weighted)

Educational Attainment at 6 Months

Baseline Percentage Difference

Outcome (Retained Sample) 6 Months (%) (%) p-Value

Completed high school or C2C 74.52 78.90 4.38 0.002

GED or greatera

Comparison 73.47 80.37 6.90 <.001

a Among persons ages 18 and older at baseline.

394 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.31. Between-Group Differences in Employment, Housing, Education, and Incarceration at 6 Months (Weighted)

Group-Level Comparison of Differences

Outcome

Propensity Score Weighted Modela

Estimate or SE or 

OR 95% CI p-Value

Estimate or 

OR

Doubly Robust Modelb

SE or 

95% CI p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Employment

FT/PT employment (vs. 

unemployment)c OR = 0.957 [0.65–1.41] 0.536 OR = 1.30 [0.78–2.16] 0.482 –

Change in hours worked per week 1.98 2.38 0.405 1.73 1.69 0.310 0.081

Change in current monthly pay

before taxes and deductions

−37.49 152 0.806 128 127 0.314 0.096

Aggregate current housing

Stably housed or staying with 

someone else (vs. transitional or

homeless)

Education

OR = 0.606 [0.35–1.05] 0.076 OR = 0.900 [0.35–2.34] 0.829 –

Completed high school or greaterd OR = 0.904 [0.59–1.39] 0.649 OR = 0.442 [0.19–1.03] 0.058 –

NOTES: SE = standard error; FT = full-time; PT = part-time. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status,

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and follow-up. Does not include persons who

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Among persons ages 18 and older at baseline.

395 
 

 

 

 

 

 

C8.3. Subgroup Analysis Results

C8.3.1. CBO Type: Job training and Employment Program

Chapter 7 Mental Health Care Access and Utilization

Table C.32. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 263 −0.348 0.042 <.001

Comparison 117 −0.345 0.082 <.001

Attitudinal barriers C2C 263 −0.416 0.046 <.001

Comparison 117 −0.298 0.070 <.001

Stigma barriers C2C 263 −0.345 0.050 <.001

Comparison 117 −0.314 0.080 <.001

Internalized stigma C2C 250 −0.391 0.092 <.001

Comparison 108 −0.421 0.151 0.006

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately.

Table C.33. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 293 −0.487 0.038 <.001

Comparison 134 −0.466 0.072 <.001

Attitudinal barriers C2C 296 −0.584 0.041 <.001

Comparison 134 −0.417 0.074 <.001

Stigma barriers C2C 294 −0.476 0.042 <.001

Comparison 132 −0.438 0.074 <.001

Internalized stigma C2C 280 −0.542 0.075 <.001

Comparison 124 −0.641 0.130 <.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately.

396 
 

 

 

 

 

 

 

 

Table C.34. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Group-Level Comparison of Mean Changes Standardized 

Outcome

Propensity Score Weighted Modela Doubly Robust Modelb

Estimate SE p-Value Estimate SE p-Value

Effect Size for

Doubly Robust

Model

Logistical barriers −0.003 0.092 0.972 0.006 0.068 0.929 0.009

Attitudinal barriers −0.118 −0.118 −0.118 −0.118 −0.118 −0.118 −0.118

Stigma barriers −0.031 0.094 0.739 0.016 0.086 0.856 0.020

Internalized stigma 0.030 0.177 0.866 0.096 0.141 0.496 0.069

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

Table C.35. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Group-Level Comparison of Mean Changes Standardized 

Outcome

Propensity Score Weighted Modela Doubly Robust Modelb

Estimate SE p-Value Estimate SE p-Value

Effect Size for

Doubly Robust

Model

Logistical barriers −0.021 0.082 0.801 0.008 0.046 0.870 0.011

Attitudinal barriers −0.166 0.085 0.051 −0.068 0.060 0.253 −0.096

Stigma barriers −0.039 0.085 0.649 0.018 0.055 0.736 0.025

Internalized stigma 0.099 0.150 0.511 0.138 0.088 0.121 0.109

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

397 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.36. Within-Group Mental Health Care Utilization over 1 Year Among Persons with Need at

Baseline—Binary (Weighted)

Utilization Among Persons with Baseline Unmet Need

Baseline 

Percentage

(Retained 

Binary (Yes or No) Outcome Sample) 1 Year (%) Difference p-Value

Went to any outpatient MHPa,b C2C 35.44 36.31 +0.88 0.833

Comparison 27.04 30.19 +3.15 0.331

Used any nonclinical settings or C2C 38.69 30.76 −7.93 0.006 a,c resources Comparison 32.96 25.89 −7.07 0.035d

Stayed in inpatient settinge C2C 35.90 8.06 −27.84 <.001

Comparison 22.13 5.31 −16.82 <.001

Used emergency settingsa,f C2C 40.19 40.57 +0.38 0.971

Comparison 38.81 44.43 +5.62 0.124

a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. b Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. c Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended

religious/spiritual places, parks and recreation, or community centers for mental health needs. d Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons. e One or more nights in residential treatment program for alcohol/drug problems (past 6 months) or overnight stay in a 

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up). f

Hospital emergency room or an urgent care facility for any health reason.

Table C.37. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Continuous (Weighted)

Within-Group Mean Change over 1 Year

Continuous (No. of Times) Outcome N Estimate SE p-Value

Number of times went to any outpatient MHPa C2C 309 2.33 0.904 0.010

Comparison 131 1.14 0.951 0.233

Number of nights in a residential treatment C2C 311 −3.83 1.55 0.014

program for alcohol or drug problemsa

Comparison 131 −0.394 0.320 0.219

Number of times went to a hospital, emergency C2C 311 −0.054 0.180 0.766

room, urgent care facility for any health reasona

Comparison 131 0.230 0.245 0.349

NOTE: SE = standard error. a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year.

398 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.38. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Binary (Weighted)

Group-Level Comparison of Utilization over 1 Year

Propensity Score Weighted Modela Doubly Robust Modelb

Binary (Yes or No) Outcome OR 95% CI p-Value OR 95% CI p-Value

Went to any outpatient MHPc 1.37 [0.850–2.22] 0.196 1.50 [0.740–3.04] 0.262

Used any nonclinical settings or d resources1.25 [0.769–2.03] 0.370 1.19 [0.549–2.59] 0.659

Stayed in inpatient settinge

Used emergency settingsf 0.853 [0.541–1.34] 0.492 0.830 [0.405–1.70] 0.610

a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. d Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended

religious/spiritual places, parks and recreation, or community centers for mental health needs. e One or more nights in residential treatment program for alcohol/drug problem (past 6 months) or overnight stay in a 

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up). f

Hospital emergency room or an urgent care facility for any health reason.

Table C.39. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Continuous (Weighted)

Group-Level Comparison of Changes in Utilization over 1 Year

Propensity Score Weighted Doubly Robust Standardized 

Modela Modelb Effect Size for

Continuous (No. of Times) Doubly Robust

Outcome Estimate SE p-Value Estimate SE p-Value Model

Number of times went to any 1.19 1.31 0.364 0.152 1.37 0.912 0.011

outpatient MHPa,b

Number of nights in a residential −3.43 1.58 0.030c −0.015 0.903 0.987 −0.0006

treatment program for alcohol or

drug problemsa

Number of times went to a hospital, −0.284 0.304 0.351 −0.308 0.263 0.243 −0.104

emergency room, urgent care facility

for any health reasona

NOTE: SE = standard error. a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. b Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. c Estimate was not statistically significant at p <.05 after adjusting for multiple comparisons.

399 
 

 

 

 

 

 

 

Chapter 8 Mental Health Symptoms

Table C.40. Within-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Depression (score range 0–24) C2C

Comparison

279

122

−0.477

−1.86

0.374

0.590

0.202

0.002

Generalized anxiety (score range 0–21) C2C

Comparison

277

122

−1.67

−1.73

0.349

0.493

<.001

0.001

PTSD (score range 0–80) C2C

Comparison

277

122

−8.14

−8.13

1.11

2.03

<.001

<.001

Alcohol use (score range 0–40) C2C

Comparison

275

121

−2.18

−1.57

0.367

0.752

<.001

0.039b

Substance use (score range 0–10) C2C

Comparison

270

115

−1.19

−0.372

0.137

0.189

<.001

0.051

Psychological distress (score range 0–24) C2C

Comparison

266

117

−2.57

−1.94

0.355

0.556

<.001

0.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately. b Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

Table C.41. Within-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Depression (score range 0–24) C2C

Comparison

305

138

−1.07

−2.45

0.381

0.460

0.005

<.001

Generalized anxiety (score range 0–21) C2C

Comparison

303

138

−2.40

−1.98

0.370

0.453

<.001

<.001

PTSD (score range 0–80) C2C

Comparison

304

138

−11.14

−10.04

1.11

1.63

<.001

<.001

Alcohol use (score range 0–40) C2C

Comparison

302

137

−2.24

−1.90

0.400

0.533

<.001

0.001

Substance use (score range 0–10) C2C

Comparison

300

133

−1.31

−0.927

0.143

0.189

<.001

<.001

Psychological distress (score range 0–24) C2C

Comparison

296

136

−3.12

−1.96

0.340

0.533

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately.

400 
 

 

 

 

 

 

 

 

 

 

 

Table C.42. Between-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted 

Modela

Estimate SE p-Value

Doubly Robust

Modelb

Estimate SE p-Value

Standardized 

Effect Size for

Doubly Robust 

Model

Depression 1.39 0.699 0.048c −0.191 0.604 0.752 −0.032

Generalized anxiety 0.054 0.604 0.929 −0.579 0.558 0.300 −0.101

PTSD −0.010 2.31 0.997 −1.55 2.03 0.444 −0.088

Alcohol use −0.613 0.837 0.464 −0.746 0.394 0.059 −0.123

Substance use −0.817 0.233 0.001 −0.255 0.176 0.149 −0.112

Distress −0.634 0.660 0.337 0.031 0.569 0.957 0.006

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

Table C.43. Between-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score 

Weighted Modela

Estimate SE p-Value

Doubly Robust

Modelb

Estimate SE p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Depression 1.38 0.597 0.021c 0.264 0.534 0.622 0.043

Generalized anxiety −0.421 0.585 0.472 −0.214 0.549 0.698 −0.035

PTSD −1.10 1.97 0.577 −0.749 1.71 0.661 −0.040

Alcohol use −0.344 0.666 0.606 −1.08 0.460 0.019c −0.168

Substance use −0.385 0.237 0.105 −0.233 0.179 0.196 −0.103

Distress −1.15 0.633 0.069 −0.542 0.544 0.320 −0.094

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity,

education level, employment status, housing status, and incarceration status in the model. c Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

401 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.44. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 6 Months

Group-Level Comparison of Percentage

with Change

Percentage with

Outcome N Change p-Value

Depression (clinically significant change of 10-point

or greater decrease in scores)

Generalized anxiety (clinically significant change of

10-point or greater decrease in scores)

Generalized anxiety (reliable change of 5-point or 

greater decrease in scores)

PTSD (clinically significant change of 10-point or

greater decrease in scores)

PTSD (reliable change of 5-point or greater

decrease in scores)

Alcohol use (change to below positive screen 

threshold)

Substance use (change to below positive screen 

threshold)

Psychological distress (change to below threshold 

for serious mental illness)

Psychological distress (change to below threshold 

for moderate distress)

C2C 279

Comparison 122

C2C 277

Comparison 122

C2C 277

Comparison 122

C2C 277

Comparison 122

C2C 277

Comparison 122

C2C 275

Comparison 121

C2C 270

Comparison 115

C2C 266

Comparison 117

C2C 266

Comparison 117

23.41 0.136

31.53

7.99 0.786

7.19

31.97 0.396

27.20

43.07 0.711

40.82

54.45 0.856

55.54

16.71 0.274

11.78

19.78 0.133

12.08

23.73 0.744

25.56

70.88 0.226

64.07

Table C.45. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 12 Months

Group-Level Comparison of

Percentage with Change

Percentage with

Outcome N Change p-Value

Depression (clinically significant change of 10-point 

or greater decrease in scores)

Generalized anxiety (clinically significant change of

10-point or greater decrease in scores)

Generalized anxiety (reliable change of 5-point or

greater decrease in scores)

PTSD (clinically significant change of 10-point or

greater decrease in scores)

PTSD (reliable change of 5-point or greater

decrease in scores)

Alcohol use (change to below positive screen 

threshold)

Substance use (change to below positive screen 

threshold)

Psychological distress (change to below threshold 

for serious mental illness)

Psychological distress (change to below threshold 

for moderate distress)

C2C 305

Comparison 138

C2C 303

Comparison 138

C2C 303

Comparison 138

C2C 304

Comparison 138

C2C 304

Comparison 138

C2C 302

Comparison 137

C2C 300

Comparison 133

C2C 296

Comparison 136

C2C 296

Comparison 136

28.08 0.637

30.49

13.34 0.051

6.70

33.59 0.150

26.09

49.42 0.770

47.76

65.60 0.042a

54.51

18.43 0.149

11.89

22.89 0.373

18.46

23.84 0.640

21.52

69.65 0.442

65.63

a Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

402 
 

 

 

 

 

 

 

 

 

 

 

 

Chapter 9 Employment, Education, Housing, and Incarceration

Table C.46. Within-Group Differences in FT/PT Employment at 6 Months (Weighted)

Employment at 6 Months

Outcome

Baseline Percentage

(Retained Sample) 6 Months (%) Difference p-Value

FT or PT employment (vs.

unemployed)

a C2C

Comparison

19.24

34.28

57.69

66.64

38.45

32.35

<.001

<.001

NOTES: FT = full-time; PT = part-time. a

Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and

follow-up.

Table C.47. Within-Group Mean Changes in Hours Worked and Employment Income at 6 Months 

(Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Hours worked per week C2C

Comparison

227

73

14.83

13.92

1.36

3.62

<.001

<.001

Current monthly pay before 

taxes and deductions

C2C

Comparison

205

59

719

924

94.9

214

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between baseline and 6 months for each group separately.

Table C.48. Within-Group Differences in FT/PT Employment at 12 Months (Weighted)

Employment at 12 Months

Outcome

Baseline

Percentage

(Retained Sample) 12 Months (%) Difference p-Value

FT or PT employment (vs.

unemployed)a C2C

Comparison

19.90

36.58

60.89

59.10

+40.99

+22.53

<.001

<.001

NOTES: FT = full-time; PT = part-time. a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and

follow-up.

403 
 

 

 

 

 

 

 

 

 

 

 

 

Table C.49. Within-Group Mean Changes in Hours Worked and Employment Income at 12 Months

(Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Hours worked per week C2C

Comparison

228

74

16.85

6.94

1.39

2.21

<.001

0.002

Current monthly pay before 

taxes and deductions

C2C

Comparison

208

62

983

540

120

142

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between baseline and 12-month scores for each group separately.

Table C.50. Within-Group Differences in Housing at 6 Months (Weighted)

Housing at 6 Months

Outcome

Baseline Percentage

(Retained Sample)

6 Months

(%) Difference p-Value

Aggregate current housing

Stably housed or staying 

with someone elsea C2C

Comparison

78.43

89.19

86.83

94.72

+8.40

+5.53

<.001

0.003

a Versus transitional/temporary or homeless/unstably housed.

Table C.51. Within-Group Differences in Housing at 12 Months (Weighted)

Housing at 12 Months

Outcome

Baseline Percentage

(Retained Sample)

12 Months

(%) Difference p-Value

Aggregate current housing

Stably housed or staying 

with someone elsea C2C

Comparison

81.52

89.79

88.29

92.94

+6.76

+3.15

0.002

0.184

a Versus transitional/temporary or homeless/unstably housed.

Table C.52. Within-Group Changes in Homelessness over 1 Year (Weighted)

Within-Group Difference over 1 Year

Outcome

Baseline Percentage

(Retained Sample) 1 Year (%) Difference p-Value

Homeless, past year C2C

Comparison

16.81

13.81

16.46

3.07

−0.35

−10.74

0.944

<.001

404 
 

 

 

 

 

 

 

 

 

Table C.53. Within-Group Differences in Educational Attainment at 6 Months (Weighted)

Educational Attainment at 6 Months

Baseline

Outcome

Percentage

(Retained 

Sample) 6 Months (%) Difference p-Value

Completed high school or GED

or greatera C2C

Comparison

77.13

71.03

80.02

76.67

+2.89

+5.64

0.083

0.004

a Among persons ages 18 and older at baseline.

Table C.54. Within-Group Differences in Educational Attainment at 12 Months (Weighted)

Educational Attainment at 12 Months

Baseline

Outcome

Percentage

(Retained 

Sample)

12 Months

(%) Difference p-Value

Completed high school or GED

or greatera C2C

Comparison

77.86

73.03

80.89

72.21

+3.03

−0.82

0.123

0.827

a Among persons ages 18 and older at baseline.

Table C.55. Within-Group Changes in Incarceration over 1 Year (Weighted)

Within-Group Difference over 1 Year

Outcome

Baseline

percentage

(Retained 

Sample)

1 Year

(%) Difference p-Value

Incarcerated, past year C2C

Comparison

16.36

9.62

4.97

3.17

−11.39

−6.45

<.001

<.001

405 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.56. Between-Group Differences in Employment, Housing, Education, and Incarceration at 6 Months (Weighted)

Group-Level Comparison of Differences

Propensity Score Weighted Modela Doubly Robust Modelb Standardized 

Effect Size for 

Outcome Estimate or OR

SE or

95% CI p-Value Estimate or OR

SE or

95% CI p-Value

Doubly Robust

Model

Employment

FT/PT employment (vs. unemployment)c OR = 0.773 [0.447–1.34] 0.357 OR = 1.32 [0.584–2.97] 0.507 —

Change in hours worked per week 0.907 3.87 0.815 0.612 2.78 0.826 0.029

Change in current monthly pay before 

taxes and deductions

−205 234 0.382 −131 162 0.421 −0.098

Aggregate current housing

Stably housed or staying with someone 

else (vs. transitional or homeless)

OR = 0.361 [0.143–0.911] 0.032d OR = 1.52 [0.360–6.42] 0.569 —

Education

Completed high school or greatere OR = 1.20 [0.701–2.05] 0.508 OR = 0.239 [0.059–0.962] 0.045d —

NOTES: SE = standard error; FT = full-time; PT = part-time. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status, 

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 6 months. Does not include persons who

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Estimate was not statistically significant at p <.05 after adjusting for multiple comparisons. e Among persons ages 18 and older at baseline.

406 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.57. Between-Group Differences in Employment, Housing, Education, and Incarceration at 12 Months and 1 Year (Weighted)

Group-Level Comparison of Outcomes

Propensity Score Weighted Modela Doubly Robust Modelb Standardized 

Effect Size for

Outcome

Estimate or

OR

SE or

95% CI p-Value

Estimate

or OR

SE or

95% CI p-Value

Doubly 

Robust Model

Employment at 12-month follow-up

FT/PT employment (vs. unemployment)c OR = 1.11 [0.641–1.93] 0.706 OR = 1.94 [0.870–4.35] 0.106 —

Change in hours worked per week

Change in current monthly pay before taxes 

and deductions

9.91

443

2.61

186

0.004

0.018

7.50

304

2.08

150

<.001

0.044d

0.369

0.196

Current housing at 12-month follow-up

Stably housed or staying with someone elsee — 

Recent homelessness—over 1 year

Homeless — 

Education at 12-month follow-up

Completed high school or greaterf OR = 1.60 [0.945–2.72] 0.081 OR = 1.93 [0.713–5.23] 0.196 —

Incarceration—over 1 year

Incarcerated

NOTES: Change estimates are not shown when the group size is fewer than ten, and comparisons are not shown when the group size is fewer than ten for either

group. Doubly robust model results are not shown when the group size is fewer than 20 for either group. Affected cells are shaded in gray. SE = standard error;

FT = full-time; PT = part-time. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status,

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 12 months. Does not include persons who 

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Not statistically significant at p < .05 after adjusting for multiple comparisons. e Versus transitional/temporary or homeless/unstably housed f

Among persons 18 and older at baseline

407 
 

 

 

 

 

 

C8.3.2. CBO Type: Youth Development Program

Chapter 7 Mental Health Care Access and Utilization

Table C.58. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 81 −0.257 0.082 0.002

Comparison 68 −0.284 0.087 0.002

Attitudinal barriers C2C 81 −0.401 0.074 <.001

Comparison 68 −0.500 0.079 <.001

Stigma barriers C2C 81 −0.285 0.075 <.001

Comparison 68 −0.463 0.092 <.001

Internalized stigma C2C 78 −0.511 0.139 <.001

Comparison 66 −0.549 0.166 0.002

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately.

Table C.59. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 75 −0.211 0.072 0.004

Comparison 78 −0.420 0.075 <.001

Attitudinal barriers C2C 76 −0.521 0.070 <.001

Comparison 78 −0.463 0.076 <.001

Stigma barriers C2C 75 −0.481 0.079 <.001

Comparison 78 −0.485 0.082 <.001

Internalized stigma C2C 74 −0.715 0.139 <.001

Comparison 75 −0.506 0.164 0.003

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately.

408 
 

 

 

 

 

 

 

 

 

 

 

Table C.60. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Outcome

Group-Level Comparison of Mean Changes

Propensity Score Doubly Robust

Weighted Modela Modelb

Estimate SE p-Value Estimate SE p-Value

Standardized 

Effect Size

for Doubly

Robust Model

Logistical barriers 0.027 0.120 0.824 0.048 0.123 0.698 0.068

Attitudinal barriers 0.099 0.108 0.363 0.156 0.120 0.199 0.239

Stigma barriers 0.178 0.119 0.137 0.078 0.132 0.554 0.109

Internalized stigma 0.038 0.217 0.863 0.153 0.200 0.446 0.119

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

Table C.61. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted Modela

Estimate SE p-Value Estimate

Doubly Robust

Modelb

SE p-Value

Standardized 

Effect Size

for Doubly

Robust Model

Logistical barriers 0.209 0.104 0.045c 0.093 0.077 0.231 0.154

Attitudinal barriers −0.058 0.103 0.571 0.012 0.087 0.894 0.019

Stigma barriers <.001 0.114 0.973 −0.034 0.081 0.675 −0.053

Internalized stigma −0.209 0.215 0.332 0.067 0.145 0.644 0.053

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

409 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.62. Within-Group Mental Health Care Utilization over 1 Year Among Persons with Need at

Baseline—Binary (Weighted)

Utilization Among Persons with Baseline Unmet Need

Baseline 

Percentage

(Retained 

Binary (Yes or No) Outcome Sample) 1 Year (%) Difference p-Value

Went to any outpatient MHPa,b C2C 54.42 53.57 −0.85 0.996

Comparison 34.04 26.51 −7.53 0.125

Used any nonclinical settings or C2C 47.45 35.89 −11.56 0.024d

a,c resources Comparison 28.19 23.56 −4.63 0.411

Stayed in inpatient settinge C2C 34.62 5.17

Comparison 27.42 4.44

Used emergency settingsa,f C2C 40.80 38.55 −2.25 0.762

Comparison 43.84 35.35 −8.49 0.106

a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. b Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. c Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended

religious/spiritual places, parks and recreation, or community centers for mental health needs. d Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons. e One or more nights in residential treatment program for alcohol/drug problem (past 6 months) or overnight stay in a 

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up). f

Hospital emergency room or an urgent care facility for any health reason.

Table C.63. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Continuous (Weighted)

Within-Group Mean Change over 1 Year

Continuous (No. of Times) Outcome N Estimate SE p-Value

Number of times went to any outpatient C2C 82 2.42 1.53 0.117

MHPa

Comparison 82 0.196 1.01 0.846

Number of nights in a residential treatment C2C 83 −2.20 1.64 0.185

program for alcohol or drug problemsa

Comparison 82 0.115 0.094 0.225

Number of times went to a hospital, C2C 83 0.084 0.260 0.747

emergency room, urgent care facility for Comparison 82 0.025 0.331 0.941

any health reasona

NOTE: SE = standard error. a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year.

410 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.64. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Binary (Weighted)

Group-Level Comparison of Utilization over 1 Year

Propensity Score Weighted 

Modela Doubly Robust Modelb

Binary (Yes or No) Outcome OR 95% CI p-Value OR 95% CI p-Value

Went to any outpatient MHPc 3.10 [1.54–6.26] 0.002

Used any nonclinical settings or 1.99 [0.966–4.09] 0.064 d resources

Stayed in inpatient settinge

Used emergency settingsf 1.22 [0.627–2.36] 0.563 0.989 [0.236–4.15] 0.988

a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education

level, employment status, housing status, and incarceration status in the model. c Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. d Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended

religious/spiritual places, parks and recreation, or community centers for mental health needs. e One or more nights in residential treatment program for alcohol/drug problem (past 6 months) or overnight stay in a

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up). f

Hospital emergency room or an urgent care facility for any health reason.

Table C.65. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year Among

Persons with Need at Baseline—Continuous (Weighted)

Group-Level Comparison of Changes in Utilization over 1 Year

Propensity Score Weighted Doubly Robust Standardized 

Modela Modelb Effect Size for

Continuous (No. of Times) Doubly Robust

Outcome Estimate SE p-Value Estimate SE p-Value Model

Number of times went to any 2.23 1.83 0.226 −1.84 1.75 0.299 −0.158

outpatient MHPa,b

Number of nights in a residential −2.31 1.65 0.162 −0.153 0.168 0.365 −0.011

treatment program for alcohol or

drug problemsa

Number of times went to a hospital, 0.060 0.421 0.888 −0.261 0.382 0.496 −0.105

emergency room, urgent care facility

for any health reasona

NOTE: SE = standard error. a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. b Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors.

411 
 

 

 

 

 

Chapter 8 Mental Health Symptoms

Table C.66. Within-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Depression (score range 0–24) C2C

Comparison

83

70

−3.14

−1.93

0.514

0.714

<.001

0.009

Generalized anxiety (score range 0–21) C2C

Comparison

83

70

−2.36

−1.96

0.613

0.638

0.003

<.001

PTSD (score range 0–80) C2C

Comparison

83

70

−7.51

−10.29

1.60

1.99

<.001

<.001

Alcohol use (score range 0–40) C2C

Comparison

81

69

−0.214

−1.43

0.396

0.480

0.590

0.004

Substance use (score range 0–10) C2C

Comparison

81

69

−0.808

−0.787

0.170

0.209

<.001

<.001

Psychological distress (score range 0–24) C2C

Comparison

81

70

−3.16

−3.33

0.525

0.694

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately.

Table C.67. Within-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Depression (score range 0–24) C2C

Comparison

78

80

−3.92

−2.19

0.534

0.691

<.001

0.002

Generalized anxiety (score range 0–21) C2C

Comparison

78

80

−2.66

−3.00

0.675

0.568

<.001

<.001

PTSD (score range 0–80) C2C

Comparison

77

80

−9.23

−14.47

1.98

2.09

<.001

<.001

Alcohol use (score range 0–40) C2C

Comparison

75

80

−0.343

−1.93

0.447

0.602

0.445

0.002

Substance use (score range 0–10) C2C

Comparison

76

80

−1.01

−0.769

0.248

0.228

<.001

0.001

Psychological distress (score range 0–24) C2C

Comparison

77

78

−3.78

−4.39

0.700

0.662

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately.

412 
 

 

 

 

 

 

 

 

 

 

 

 

Table C.68. Between-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted 

Modela

Estimate SE p-Value

Doubly Robust

Modelb

Estimate SE p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Depression −1.21 0.880 0.170 0.198 0.802 0.805 0.040

Generalized anxiety −0.393 0.885 0.658 0.571 0.941 0.545 0.106

PTSD 2.78 2.56 0.279 3.81 2.41 0.118 0.262

Alcohol use 1.22 0.623 0.053 0.761 0.617 0.220 0.197

Substance use −0.021 0.270 0.937 0.013 0.262 0.959 0.009

Distress 0.176 0.870 0.840 0.906 1.04 0.385 0.180

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

Table C.69. Between-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Group-Level Comparison of Mean Changes Standardized 

Propensity Score Weighted 

Modela Doubly RobustModelb

Effect Size

for Doubly

Robust

Outcome Estimate SE p-Value Estimate SE p-Value Model

Depression −1.73 0.873 0.049c −1.51 0.759 0.050c −0.298

Generalized anxiety 0.335 0.882 0.705 0.379 0.808 0.640 0.075

PTSD 5.25 2.88 0.070 4.56 2.38 0.058 0.288

Alcohol use 1.59 0.750 0.036c 0.370 0.530 0.487 0.081

Substance use −0.237 0.336 0.482 −0.089 0.239 0.709 −0.046

Distress 0.602 0.963 0.533 0.080 1.02 0.938 0.015

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

413 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.70. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 6 Months

Group-Level Comparison of

Percentage with Change

Percentage with

Outcome N Change p-Value

Depression (clinically significant change of 10-point

or greater decrease in scores)

Generalized anxiety (clinically significant change of

10-point or greater decrease in scores)

Generalized anxiety (reliable change of 5-point or

greater decrease in scores)

PTSD (clinically significant change of 10-point or

greater decrease in scores)

PTSD (reliable change of 5-point or greater

decrease in scores)

Alcohol use (change to below positive screen 

threshold)

Substance use (change to below positive screen 

threshold)

Psychological distress (change to below threshold 

for serious mental illness)

Psychological distress (change to below threshold 

for moderate distress)

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

83 33.61 0.690

70 30.47

83 9.30 0.834

70 8.30

83 34.32 0.632

70 30.56

83 42.42 0.705

70 45.58

83 53.06 0.317

70 61.32

81 8.05 0.776

69 9.34

81 14.57 0.850

69 15.71

81 34.20 0.969

70 34.51

81 73.33 0.886

70 74.40

Table C.71. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 12 Months

Group-Level Comparison of

Percentage with Change

Percentage with

Outcome N Change p-Value

Depression (clinically significant change of 10-point

or greater decrease in scores)

Generalized anxiety (clinically significant change of

10-point or greater decrease in scores)

Generalized anxiety (reliable change of 5-point or

greater decrease in scores)

PTSD (clinically significant change of 10-point or

greater decrease in scores)

PTSD (reliable change of 5-point or greater

decrease in scores)

Alcohol use (change to below positive screen 

threshold)

Substance use (change to below positive screen 

threshold)

Psychological distress (change to below threshold 

for serious mental illness)

Psychological distress (change to below threshold 

for moderate distress)

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

C2C

Comparison

78 36.40 0.856

80 34.96

78 10.26 0.822

80 9.15

78 32.81 0.454

80 38.76

77 42.64 0.028

80 61.12

77 60.55 0.105

80 73.48

75 5.29 0.077

80 14.36

76 19.74 0.985

80 19.61

77 37.32 0.893

78 38.44

77 78.19 0.556

78 81.99

414 
 

 

 

 

 

 

 

 

 

 

 

Chapter 9 Employment, Education, Housing, and Incarceration

Table C.72. Within-Group Differences in FT/PT Employment at 6 Months (Weighted)

Employment at 6 Months

Outcome

Baseline Percentage

(Retained Sample) 6 Months (%) Difference p-Value

FT or PT employment (vs.

unemployed)a C2C

Comparison

61.48

41.91

72.69

73.42

+11.20

+31.52

0.235

<.001

NOTES: FT, full-time; PT = part-time. a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 

follow-up.

Table C.73. Within-Group Mean Changes in Hours Worked and Employment Income at 6 Months 

(Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Hours worked per week

Current monthly pay before 

taxes and deductions

C2C

Comparison

C2C

Comparison

30

46

25

38

2.37

11.53

174

751

3.18

2.97

350

159

0.462

<.001

0.624

<.001

NOTE: SE = standard error. a Within-individual mean changes between baseline and 6 months for each group separately.

Table C.74. Within-Group Differences in FT/PT Employment at 12 Months (Weighted)

Outcome Employment at 12 Months

Baseline Percentage

(Retained Sample)

12 Months

(%) Difference p-Value

FT or PT employment (vs.

unemployed)a C2C

Comparison

69.57

35.41

67.90

60.66

−1.77

+25.25

0.992

0.002

NOTES: FT, full-time; PT = part-time. a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and

follow-up.

415 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.75. Within-Group Mean Changes in Hours Worked and Employment Income at 12 Months

(Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Hours worked per week

Current monthly pay before 

taxes and deductions

C2C

Comparison

C2C

Comparison

30

50

27

42

−0.428

9.74

−30.07

990

4.19

3.11

321

454

0.919

0.003

0.926

0.035b

NOTE: SE = standard error. a Within-individual mean changes between baseline and 12 months for each group separately. b Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

Table C.76. Within-Group Differences in Housing at 6 Months (Weighted)

Housing at 6 Months

Baseline

Outcome

Percentage

(Retained 

Sample)

6 Months

(%) Difference p-Value

Aggregate current housing

Stably housed or staying with 

someone elsea C2C 88.28 89.78

Comparison 98.34 98.34

a Versus transitional/temporary or homeless/unstably housed.

Table C.77. Within-Group Differences in Housing at 12 Months (Weighted)

Housing at 12 Months

Baseline Percentage 12 Months

Outcome (Retained Sample) (%) Difference p-Value

Aggregate current housing

Stably housed or staying with 

someone elsea C2C 89.52 88.39

Comparison 97.50 97.50

a Versus transitional/temporary or homeless/unstably housed.

Table C.78. Within-Group Changes in Homelessness over 1 Year (Weighted)

Within-Group Difference over 1 Year

Baseline Percentage 1 Year

Outcome (Retained Sample) (%) Difference p-Value

Homeless, past year C2C 17.96 14.50 −3.46 0.387

Comparison 13.31 11.78

416 
 

 

 

 

 

 

 

 

 

 

Table C.79. Within-Group Differences in Educational Attainment at 6 Months (Weighted)

Educational Attainment at 6 Months

Baseline Percentage

Outcome (Retained Sample) 6 Months (%) Difference p-Value

Completed high school or C2C 63.31 73.73 +10.42 0.023b

GED or greatera

Comparison 80.27 94.61 +14.34 0.001

a Among persons ages 18 and older at baseline. b Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

Table C.80. Within-Group Differences in Educational Attainment at 12 Months (Weighted)

Educational Attainment at 12 Months

Baseline Percentage

Outcome (Retained Sample) 12 Months (%) Difference p-Value

Completed high school or GED C2C 65.58 83.30 +17.72 <.001

or greatera

Comparison 80.54 88.27

a Among persons ages 18 and older at baseline.

Table C.81. Within-Group Changes in Incarceration over 1 Year (Weighted)

Within-Group Difference over 1 Year

Baseline Percentage 1 Year

Outcome (Retained Sample) (%) Difference p-Value

Incarcerated, past year C2C 0.00 1.19

Comparison 1.15 2.49

417 
 

 

 

 

 

 

 

 

 

 

 

 

Table C.82. Between-Group Differences in Employment, Housing, Education, and Incarceration at 6 Months (Weighted)

Group-Level Comparison of Differences

Outcome

Propensity Score Weighted Modela

Estimate or SE or

OR 95% CI p-Value

Estimate or

OR

Doubly Robust Modelb

SE or

95% CI p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Employment

FT/PT employment (vs.

unemployment)c

Change in hours worked per

week

OR = 0.785

−9.16

[0.347–1.78]

4.35

0.563

0.039d −1.16 3.10 0.709 −0.057

Change in current monthly pay

before taxes and deductions

−577 385 0.139 −28.52 239 0.906 −0.020

Aggregate current housing

–

Stably housed or staying with 

someone else (vs. transitional or

homeless)

Education

Completed high school or

greatere

NOTES: SE = standard error; FT, full-time; PT = part-time. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status,

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 6 months. Does not include persons who

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Estimate was not statistically significant at p < .05 after adjusting for multiple comparisons. e Among persons ages 18 and older at baseline.

418 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.83. Between-Group Differences in Employment, Housing, Education, and Incarceration at 12 Months and 1 Year (Weighted)

Group-Level Comparison of Outcomes

Propensity Score Weighted Modela Doubly Robust Modelb Standardized 

Effect Size for

Outcome

Estimate

or OR

SE or

95% CI p-Value

Estimate

or OR

SE or

95% CI p-Value

Doubly Robust

Model

Employment at 12-month follow-up

FT/PT employment (vs. unemployment)c OR = 1.20 [0.542–2.66] 0.653

Change in hours worked per week −10.17 5.21 0.055 −2.65 5.07 0.604 −0.113

Change in current monthly pay before taxes

and deductions −1,020 556 0.071 −525 426 0.227 −0.274

—

Current housing at 12-month follow-up

Stably housed or staying with someone elsed —

Recent homelessness—over 1 year

Homeless —

Education at 12-month follow-up

Completed high school or greatere —

Incarceration—over 1 year

Incarcerated

NOTES: Change estimates are not shown when the group size is fewer than ten, and comparisons are not shown when the group size is fewer than ten for either

group. Doubly robust model results are not shown when the group size is fewer than 20 for either group. Affected cells are shaded in gray. SE = standard error;

FT, full-time; PT = part-time. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status,

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 12 months. Does not include persons who 

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Versus transitional/temporary or homeless/unstably housed. e Among persons 18 and older at baseline.

419 
 

 

 

 

 

 

 

C8.3.3. Target Population: Adults Age 18 or Older Who Are Unemployed or Underemployed

Chapter 7 Mental Health Care Access and Utilization

Table C.84. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 285 −0.284 0.043 <.001

Comparison 135 −0.327 0.073 <.001

Attitudinal barriers C2C 286 −0.377 0.044 <.001

Comparison 135 −0.334 0.060 <.001

Stigma barriers C2C 286 −0.288 0.050 <.001

Comparison 135 −0.325 0.079 <.001

Internalized stigma C2C 270 −0.377 0.090 <.001

Comparison 125 −0.438 0.139 0.002

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately.

Table C.85. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 320 −0.432 0.037 <.001

Comparison 148 −0.460 0.061 <.001

Attitudinal barriers C2C 323 −0.553 0.042 <.001

Comparison 148 −0.427 0.065 <.001

Stigma barriers C2C 321 −0.493 0.042 <.001

Comparison 147 −0.510 0.067 <.001

Internalized stigma C2C 303 −0.548 0.074 <.001

Comparison 135 −0.688 0.127 <.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately.

420 
 

 

 

 

 

 

 

 

 

 

 

 

Table C.86. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Outcome

Group-Level Comparison of Mean Changes

Propensity Score Weighted Doubly Robust

Modela Modelb

Estimate SE p-Value Estimate SE p-Value

Standardized

Effect Size

for Doubly

Robust Model

Logistical barriers

Attitudinal barriers

0.043

−0.043

0.084

0.075

0.612

0.562

0.035

0.089

0.064

0.067

0.581

0.186

0.051

0.126

Stigma barriers 0.037 0.094 0.693 0.079 0.075 0.295 0.098

Internalized stigma 0.061 0.165 0.710 0.128 0.113 0.259 0.091

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

Table C.87. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Outcome

Group-Level Comparison of Mean Changes

Propensity Score Weighted Doubly Robust

Modela Modelb

Estimate SE p-Value Estimate SE p-Value

Standardized 

Effect Size

for Doubly

Robust Model

Logistical barriers

Attitudinal barriers

0.028

−0.126

0.072

0.077

0.701

0.103

−0.019

−0.106

0.045

0.055

0.665

0.054

−0.029

−0.151

Stigma barriers

Internalized stigma

0.018

0.141

0.079

0.147

0.825

0.339

0.017

0.227

0.049

0.085

0.733

0.008c

0.022

0.174

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

421 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.88. Within-Group Mental Health Care Utilization over 1 Year Among Persons with Need at

Baseline—Binary (Weighted)

Utilization Among Persons with 

Baseline Unmet Need

Baseline

Percentage

(Retained 

Binary (Yes or No) Outcome Sample) 1 Year (%) Difference p-Value

Went to any outpatient MHPa,b C2C 37.90 37.49 −0.41 0.947

Comparison 32.83 37.22 +4.38 0.135

Used any nonclinical settings or C2C 43.37 31.77 −11.60 <.001 a,c resources Comparison 35.63 24.29 −11.35 0.001

Stayed in inpatient settingd C2C 36.93 6.65 −30.28 <.001

Comparison 25.77 3.68 −22.09 <.001

Used emergency settingsa,e C2C 39.53 37.44 −2.10 0.521

Comparison 37.94 41.34 +3.40 0.342

a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. b Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. c Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended

religious/spiritual places, parks and recreation, or community centers for mental health needs. d One or more nights in residential treatment program for alcohol/drug problems (past 6 months) or overnight stay in a 

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up). e Hospital emergency room or an urgent care facility for any health reason. f

Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

Table C.89. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Continuous (Weighted)

Within-Group Mean Change

over 1 Year

Continuous (No. of Times) Outcome N Estimate SE p-Value

Number of times went to any outpatient MHPa C2C 341 2.17 0.788 0.006

Comparison 144 1.93 1.04 0.066

Number of nights in a residential treatment C2C 342 −3.12 1.35 0.021

program for alcohol or drug problemsa

Comparison 144 0.044 0.036 0.218

Number of times went to a hospital, emergency C2C 342 −0.155 0.156 0.319

room, urgent care facility for any health reasona

Comparison 144 0.204 0.218 0.350

NOTE: SE = standard error. a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year.

422 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.90. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Binary (Weighted)

Group-Level Comparison of Utilization over 1 Year

Propensity Score 

Weighted Modela Doubly RobustModelb

Binary (Yes or No) Outcome OR 95% CI p-Value OR 95% CI p-Value

Went to any outpatient MHPc,d

Used any nonclinical settings or c,e resources

1.04

1.43

[0.664–1.63]

[0.892–2.29]

0.860

0.138

0.891

1.06

[0.492–1.62]

[0.571–1.96]

0.706

0.860

Stayed in inpatient settingf

Used emergency settingsc,g 0.857 [0.555–1.33] 0.489 0.817 [0.450–1.48] 0.506

a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. d Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. e Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended

religious/spiritual places, parks and recreation, or community centers for mental health needs. f

One or more nights in residential treatment program for alcohol/drug problems (past 6 months) or overnight stay in a 

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up). g Hospital emergency room or an urgent care facility for any health reason.

Table C.91. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year Among

Persons with Need at Baseline—Continuous (Weighted)

Group-Level Comparison of Changes in Utilization over 1 Year

Propensity Score Weighted Doubly Robust Standardized 

Modela Modelb Effect Size for

Continuous (No. of Times) Doubly Robust

Outcome Estimate SE p-Value Estimate SE p-Value Model

Number of times went to any 0.245 1.31 0.851 0.452 1.23 0.713 0.033

outpatient MHPc,d

Number of nights in a residential −3.17 1.35 0.019e 0.400 0.780 0.609 0.016

treatment program for alcohol or

drug problems

Number of times went to a hospital, −0.359 0.267 0.180 −0.192 0.210 0.361 −0.069

emergency room, urgent care facility

for any health reason

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. d Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. e Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

423 
 

 

 

 

 

 

Chapter 8 Mental Health Symptoms

Table C.92. Within-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Depression (score range 0–24) C2C

Comparison

302

139

−1.37

−2.14

0.36

0.48

<.001

<.001

Generalized anxiety (score range 0–21) C2C

Comparison

300

139

−2.00

−1.98

0.36

0.46

<.001

<.001

PTSD (score range 0–80) C2C

Comparison

300

139

−7.94

−8.16

1.07

1.50

<.001

<.001

Alcohol use (score range 0–40) C2C

Comparison

298

137

−1.85

−1.62

0.33

0.45

<.001

<.001

Substance use (score range 0–10) C2C

Comparison

293

135

−1.15

−0.58

0.13

0.15

<.001

<.001

Psychological distress (score range 0–24) C2C

Comparison

288

136

−2.80

−1.90

0.33

0.47

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately.

Table C.93. Within-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Depression (score range 0–24) C2C

Comparison

330

151

−2.28

−2.74

0.353

0.443

<.001

<.001

Generalized anxiety (score range 0–21) C2C

Comparison

328

151

−2.94

−2.37

0.357

0.458

<.001

<.001

PTSD (score range 0–80) C2C

Comparison

329

151

−11.29

−11.71

1.13

1.43

<.001

<.001

Alcohol use (score range 0–40) C2C

Comparison

327

150

−2.06

−1.95

0.332

0.470

<.001

<.001

Substance use (score range 0–10) C2C

Comparison

325

148

−1.33

−1.10

0.126

0.176

<.001

<.001

Psychological distress (score range 0–24) C2C

Comparison

321

149

−3.49

−2.90

0.338

0.496

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately.

424 
 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.94. Between-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted 

Modela

Estimate SE p-Value

Doubly Robust

Modelb

Estimate SE p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Depression 0.767 0.599 0.201 −0.027 0.498 0.957 −0.005

Generalized anxiety −0.017 0.586 0.977 0.014 0.523 0.979 0.002

PTSD 0.224 1.84 0.903 0.251 1.65 0.880 0.015

Alcohol use −0.230 0.557 0.680 −0.071 0.320 0.825 −0.013

Substance use −0.561 0.195 0.004c −0.143 0.140 0.308 −0.065

Distress −0.898 0.575 0.119 0.231 0.482 0.632 0.043

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

Table C.95. Between-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted 

Modela

Estimate SE p-Value

Doubly Robust

Modelb

Estimate SE p-Value

Standardized 

Effect Size

for Doubly

Robust Model

Depression 0.455 0.566 0.422 −0.364 0.469 0.438 −0.060

Generalized anxiety −0.569 0.581 0.328 −0.356 0.523 0.496 −0.059

PTSD 0.422 1.82 0.817 −0.052 1.45 0.971 −0.003

Alcohol use −0.114 0.575 0.843 −0.405 0.349 0.246 −0.068

Substance use −0.230 0.216 0.288 −0.063 0.126 0.619 −0.028

Distress −0.585 0.600 0.330 −0.131 0.482 0.785 −0.022

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

425 
 

 

 

 

 

 

 

 

 

 

 

Table C.96. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 6 Months

Group-Level Comparison of

Percentage with Change

Percentage with

Outcome N Change p-Value

Depression (clinically significant change of

10-point or greater decrease in scores)

Generalized anxiety (clinically significant change 

of 10-point or greater decrease in scores)

Generalized anxiety (reliable change of 5-point or

greater decrease in scores)

PTSD (clinically significant change of 10-point or

greater decrease in scores)

PTSD (reliable change of 5-point or greater 

decrease in scores)

Alcohol use (change to below positive screen 

threshold)

Substance use (change to below positive screen 

threshold)

Psychological distress (change to below threshold 

for serious mental illness)

Psychological distress (change to below threshold 

for moderate distress)

C2C 302

Comparison 139

C2C 300

Comparison 139

C2C 300

Comparison 139

C2C 300

Comparison 139

C2C 300

Comparison 139

C2C 298

Comparison 137

C2C 293

Comparison 135

C2C 288

Comparison 136

C2C 288

Comparison 136

27.28 0.347

31.93

10.86 0.275

7.51

32.32 0.635

29.93

40.33 0.950

39.99

53.50 0.609

56.31

14.81 0.347

11.12

19.44 0.099

12.20

27.12 0.416

23.15

72.34 0.264

66.63

426 
 

 

 

 

 

 

 

 

 

 

 

 

Table C.97. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 12 Months

Group-Level Comparison of

Percentage with Change

Percentage with

Outcome N change p-Value

Depression (clinically significant change of 10-point

or greater decrease in scores)

Generalized anxiety (clinically significant change of

10-point or greater decrease in scores)

Generalized anxiety (reliable change of 5-point or

greater decrease in scores)

PTSD (clinically significant change of 10-point or

greater decrease in scores)

PTSD (reliable change of – point or greater

decrease in scores)

Alcohol use (change to below positive screen 

threshold)

Substance use (change to below positive screen 

threshold)

Psychological distress (change to below threshold 

for serious mental illness)

Psychological distress (change to below threshold 

for moderate distress)

C2C 330

Comparison 151

C2C 328

Comparison 151

C2C 328

Comparison 151

C2C 329

Comparison 151

C2C 329

Comparison 151

C2C 327

Comparison 150

C2C 325

Comparison 148

C2C 321

Comparison 149

C2C 321

Comparison 149

34.18 0.665

36.41

15.62 0.047a

8.95

36.14 0.425

32.10

48.69 0.365

53.56

63.74 0.936

63.33

15.32 0.619

13.34

22.90 0.551

20.13

26.25 0.377

22.11

72.18 0.929

71.75

a Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

427 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Chapter 9 Employment, Education, Housing, and Incarceration

Table C.98. Within-Group Differences in FT/PT Employment at 6 Months (Weighted)

Employment at 6 Months

Outcome

Baseline Percentage

(Retained Sample)

6 Months

(%) Difference p-Value

FT or PT employment (vs.

unemployed)a C2C

Comparison

18.43

33.16

57.58

59.29

39.15

26.13

<.001

<.001

NOTES: FT = full-time; PT = part-time. a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and

follow-up.

Table C.99. Within-Group Mean Changes in Hours Worked and Employment Income at 6 Months 

(Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Hours worked per week

Current monthly pay before 

taxes and deductions

C2C

Comparison

C2C

Comparison

256

119

230

103

14.24

11.13

833

728

1.27

2.39

93.9

143

<.001

<.001

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between baseline and 6 months for each group separately.

Table C.100. Within-Group Differences in FT/PT Employment at 12 Months (Weighted)

Employment at 12 Months

Outcome

Baseline Percentage

(Retained Sample)

12 Months

(%) Difference p-Value

FT or PT employment

(versus unemployed)a C2C

Comparison

19.20

26.30

57.24

50.00

38.04

23.69

<.001

<.001

NOTES: FT = full-time; PT = part-time. a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and

follow-up.

Table C.101. Within-Group Mean Changes in Hours Worked and Employment Income at 12 Months

(Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Hours worked per week C2C

Comparison

266

119

14.84

8.57

1.32

1.69

<.001

<.001

Current monthly pay before 

taxes and deductions

C2C

Comparison

243

104

986

638

115

154

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between baseline and 12 months for each group separately.

428 
 

 

 

 

 

 

 

 

 

 

Table C.102. Within-Group Differences in Housing at 6 Months (Weighted)

Housing at 6 Months

Outcome

Baseline Percentage

(Retained Sample)

6 Months

(%) Difference p-Value

Aggregate current housing

Stably housed or staying 

with someone elsea C2C

Comparison

74.63

85.44

83.05

88.42

+8.42

+2.98

<.001

0.046b

a Versus transitional/temporary or homeless/unstably housed. b Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

Table C.103. Within-Group Differences in Housing at 12 Months (Weighted)

Housing at 12 Months

Outcome

Baseline Percentage

(Retained Sample)

12 Months

(%) Difference p-Value

Aggregate current housing

Stably housed or staying 

with someone elsea C2C

Comparison

77.91

84.16

87.25

87.74

+9.34

+3.58

<.001

0.095

a Versus transitional/temporary or homeless/unstably housed.

Table C.104. Within-Group Changes in Homelessness over 1 Year (Weighted)

Within-Group Difference over 1 Year

Outcome

Baseline Percentage

(Retained Sample) 1 Year (%) Difference p-Value

Homeless, past year C2C

Comparison

16.17

12.78

13.75

3.92

−2.41

−8.86

0.227

<.001

Table C.105. Within-Group Differences in Educational Attainment at 6 Months (Weighted)

Educational Attainment at 6 Months

Baseline Percentage

Outcome (Retained Sample) 6 Months (%) Difference p-Value

Completed high school or GED C2C 78.50 81.06 +2.57 0.065

or greatera

Comparison 77.00 81.59 +4.59 0.009

a Among persons ages 18 and older at baseline.

429 
 

 

 

 

 

 

Table C.106. Within-Group Differences in Educational Attainment at 12 Months (Weighted)

Educational Attainment at 12 Months

Baseline Percentage 12 Months

Outcome (Retained Sample) (%) Difference p-Value

Completed high school or GED C2C 79.75 83.10 +3.35 0.053

or greater a

Comparison 78.23 79.27 +1.04 0.702

a Among persons ages 18 and older at baseline.

Table C.107. Within-Group Changes in Incarceration over 1 Year (Weighted)

Within-Group Difference over 1 Year

Outcome

Baseline Percentage

(Retained Sample) 1 Year (%) Difference p-Value

Incarcerated, past year C2C

Comparison

13.76

10.35

4.18

1.90

−9.57

−8.44

<.001

<.001

430 
 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.108. Between-Group Differences in Employment, Housing, Education, and Incarceration at 6 Months (Weighted)

Group-Level Comparison of Differences

Outcome

Propensity Score Weighted Modela

Estimate or

OR SE or 95% CI p-Value

Doubly Robust Modelb

Estimate or

OR SE or 95% CI p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Employment

FT/PT employment (vs.

unemployment)c OR = 0.932 [0.586–1.48] 0.766 OR = 1.65 [0.855–3.18] 0.137 —

Change in hours worked per week 3.11 2.70 0.251 1.94 1.88 0.304 0.094

Change in current monthly pay

before taxes and deductions

105 171 0.541 150 148 0.313 0.118

Aggregate current housing

Stably housed or staying with 

someone else (vs. transitional or

homeless)

OR = 0.630 [0.328–1.21] 0.523 OR = 1.38 [0.514–3.70] 0.166 —

Education

Completed high school or greaterd OR = 0.951 [0.533–1.70] 0.865 OR = 0.327 [0.055–1.94] 0.219 —

NOTES: SE = standard error; FT = full-time; PT = part-time. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status,

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 6 months. Does not include persons who 

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Among persons ages 18 and older at baseline.

431 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.109. Between-Group Differences in Employment, Housing, Education, and Incarceration at 12 Months and 1 Year (Weighted)

Group-Level Comparison of Outcomes

Propensity Score Weighted Modela Doubly Robust Modelb Standardized 

Effect Size for

Outcome Estimate or OR

SE or

95% CI p-Value

Estimate or

OR

SE or

95% CI p-Value

Doubly Robust

Model

Employment at 12-month follow-up

FT/PT employment (vs. unemployment)c OR = 1.34 [0.834–2.15] 0.537 OR = 1.01 [0.547–1.85] 0.651 — 

Change in hours worked per week 6.27 2.14 0.004 1.65 1.72 0.338 0.082

Change in current monthly pay before 

taxes and deductions

348 192 0.071 59.90 169 0.723 0.035

Current housing at 12-month follow-up

Stably housed or staying with someone 

elsed OR = 0.956 [0.480–1.90] 0.898

Recent homelessness—over 1 year

Homeless 3.92 [1.89–8.14] <.001

Education at 12-month follow-up

Completed high school or greatere OR = 1.27 [0.726–2.21] 0.406 OR = 1.65 [0.673–4.04] 0.275 — 

Incarceration—over 1 year

— 

— 

Incarcerated

NOTES: Change estimates are not shown when the group size is fewer than ten, and comparisons are not shown when the group size is fewer than ten for either

group. Doubly robust model results are not shown when the group size is fewer than 20 for either group. Affected cells are shaded in gray. SE = standard error;

FT = full-time; PT = part-time. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status,

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 12 months. Does not include persons who 

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Versus transitional/temporary or homeless/unstably housed. e Among persons 18 and older at baseline.

432 
 

 

 

 

 

 

 

 

 

C8.3.4. Target Population: Young Adults Ages 16 to 24 Who Are Not in School and Are 

Not Employed

Chapter 6 Mental Health Care Access and Utilization

Table C.110. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 85 −0.322 0.069 <.001

Comparison 47 −0.210 0.097 0.036b

Attitudinal barriers C2C 86 −0.478 0.077 <.001

Comparison 47 −0.318 0.128 0.017

Stigma barriers C2C 86 −0.383 0.074 <.001

Comparison 47 −0.330 0.113 0.006

Internalized stigma C2C 79 −0.556 0.154 0.001

Comparison 45 −0.953 0.278 0.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately. b Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

Table C.111. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 96 −0.344 0.056 <.001

Comparison 52 −0.375 0.087 <.001

Attitudinal barriers C2C 97 −0.648 0.066 <.001

Comparison 52 −0.424 0.101 <.001

Stigma barriers C2C 97 −0.630 0.073 <.001

Comparison 52 −0.412 0.112 0.001

Internalized stigma C2C 93 −0.830 0.135 <.001

Comparison 50 −0.779 0.261 0.004

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately.

433 
 

 

 

 

 

 

 

 

 

Table C.112. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Outcome

Group-Level Comparison of Mean Changes

Propensity Score Weighted Doubly Robust

Modela Modelb

Estimate SE p-Value Estimate SE p-Value

Standardized 

Effect Size for

Doubly 

Robust Model

Logistical barriers −0.112 0.119 0.347 −0.190 0.142 0.186 −0.301

Attitudinal barriers −0.160 0.149 0.287 −0.124 0.145 0.395 −0.172

Stigma barriers −0.053 0.135 0.696 −0.077 0.156 0.620 −0.109

Internalized stigma 0.397 0.318 0.214 0.210 0.199 0.294 0.148

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

Table C.113. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Outcome Group-Level Comparison of Mean Changes

Propensity Score Weighted Doubly Robust

Modela Modelb

Standardized 

Effect Size for

Doubly 

Robust Model

Estimate SE p-Value Estimate SE p-Value

Logistical barriers

Attitudinal barriers

0.031

−0.224

0.103

0.120

0.767

0.066

0.008

−0.196

0.067

0.089

0.910

0.031c

0.014

−0.284

Stigma barriers

Internalized stigma

−0.218

−0.051

0.134

0.294

0.105

0.863

−0.091

0.082

0.082

0.199

0.269

0.683

−0.137

0.060

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

434 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.114. Within-Group Mental Health Care Utilization over 1 Year Among Persons with Need at

Baseline—Binary (Weighted)

Utilization Among Persons with Baseline Unmet Need

Baseline 

Percentage

(Retained 1 Year

Binary (Yes or No) Outcome Sample) Percentage Difference p-Value

Went to any outpatient MHPa,b C2C 41.78 27.50 −14.28 0.006

Comparison 28.31 20.38 −7.93 0.085

Used any nonclinical settings or C2C 35.89 20.31 −15.58 0.003 a,c resources Comparison 27.08 13.82 −13.26 0.012

Stayed in inpatient settingd C2C 41.47 5.96 −35.51 <.001

Comparison 35.35 7.26 −29.10 <.001

Used emergency settingsa,e C2C 41.84 30.05 −11.79 0.026f

Comparison 56.43 48.09 −8.34 0.169

a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. b Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. c Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended

religious/spiritual places, parks and recreation, or community centers for mental health needs. d One or more nights in residential treatment program for alcohol/drug problems (past 6 months) or overnight stay in 

a hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up). e Hospital emergency room or an urgent care facility for any health reason. f

Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

Table C.115. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Continuous (Weighted)

Within-Group Mean Change over 1 Year

Continuous (No. of Times) Outcome N Estimate SE p-Value

Number of times went to any outpatient MHPa C2C 103 −0.093 0.938 0.921

Comparison 60 −0.592 0.865 0.496

Number of nights in a residential treatment C2C 103 −4.37 2.71 0.110

program for alcohol or drug problemsa

Comparison 60 0.137 0.098 0.168

Number of times went to a hospital, emergency C2C 103 −0.469 0.355 0.189

room, urgent care facility for any health reasona

Comparison 60 0.036 0.225 0.874

NOTE: SE = standard error. a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year.

435 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.116. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Binary (Weighted)

Group-Level Comparison of Utilization over 1 Year

Propensity Score Weighted Modela Doubly Robust Modelb

Binary (Yes or No) Outcome OR 95% CI p-Value OR 95% CI p-Value

Went to any outpatient MHPc,d 1.50 [0.640–3.51] 0.353

Used any nonclinical settings or 1.76 [0.710–4.38] 0.224 c,e resources

Stayed in inpatient settingf

Used emergency settingsc,g 0.469 [0.223–0.987] 0.048h 0.436 [0.077–2.48] 0.351

a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. d Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. e Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended 

religious/spiritual places, parks and recreation, or community centers for mental health needs. f

One or more nights in residential treatment program for alcohol/drug problems (past 6 months) or overnight stay in a 

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up) g Hospital emergency room or an urgent care facility for any health reason. h Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

Table C.117. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Continuous Weighted

Group-Level Comparison of Changes in Utilization over 1 Year

Propensity Score Weighted Standardized 

Modela Doubly Robust Modelb Effect Size for

Continuous (No. of Times) Doubly 

Outcome Estimate SE p-Value Estimate SE p-Value Robust Model

Number of times went to any 0.499 1.28 0.696 1.41 1.33 0.290 0.156

outpatient MHPc

Number of nights in a residential −4.51 2.71 0.099 −0.172 0.214 0.422 −0.008

treatment program for alcohol or

drug problems

Number of times went to a hospital, −0.505 0.420 0.231 −0.013 0.258 0.961 −0.004

emergency room, urgent care 

facility for any health reason

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors.

436 
 

 

 

 

 

 

 

 

Chapter 8 Mental Health Symptoms

Table C.118. Within-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Depression (score range 0–24) C2C 92 −1.17 0.549 0.036b

Comparison 47 −0.33 0.792 0.678

Generalized anxiety (score range 0–21) C2C 90 −2.49 0.546 <.001

Comparison 47 −1.44 0.815 0.085

PTSD (score range 0–80) C2C 90 −9.24 1.81 <.001

Comparison 47 −5.90 1.99 0.005

Alcohol use (score range 0–40) C2C 89 −1.33 0.497 0.009

Comparison 47 −1.72 0.463 0.001

Substance use (score range 0–10) C2C 88 −1.04 0.212 <.001

Comparison 47 −0.465 0.244 0.063

Psychological distress (score range 0–24) C2C 86 −3.19 0.538 <.001

Comparison 46 −2.55 0.628 <.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately. b Estimate was not statistically significant at p <.05 after adjusting for multiple comparisons.

Table C.119. Within-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Depression (score range 0–24) C2C

Comparison

104

52

−2.22

−2.50

0.556

0.821

<.001

0.004

Generalized anxiety (score range 0–21) C2C

Comparison

102

52

−2.97

−2.90

0.512

0.649

<.001

<.001

PTSD (score range 0–80) C2C

Comparison

102

52

−14.50

−11.59

1.74

2.54

<.001

<.001

Alcohol use (score range 0–40) C2C

Comparison

102

52

−1.22

−0.949

0.492

0.597

0.015

0.118

Substance use (score range 0–10) C2C

Comparison

100

52

−1.72

−0.881

0.203

0.239

<.001

0.001

Psychological distress (score range 0–24) C2C

Comparison

100

52

−4.55

−2.91

0.614

0.788

<.001

0.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately.

437 
 

 

 

 

 

 

 

 

 

 

 

Table C.120. Between-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted 

Modela

Estimate SE p-Value

Doubly Robust

Modelb

Estimate SE p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Depression −0.837 0.964 0.386 −1.80 0.701 0.012c −0.340

Generalized anxiety −1.05 0.981 0.286 −1.83 0.675 0.008c −0.336

PTSD −3.33 2.69 0.217 −2.06 2.00 0.306 −0.129

Alcohol use 0.394 0.680 0.563 0.524 0.446 0.244 0.120

Substance use −0.577 0.323 0.076 −0.427 0.238 0.076 −0.230

Distress −0.644 0.827 0.438 0.053 0.738 0.943 0.011

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Estimate was not statistically significant at p <.05 after adjusting for multiple comparisons.

Table C.121. Between-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted 

Modela

Estimate SE p-Value

Doubly Robust Modelb

Estimate SE p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Depression 0.287 0.991 0.773 −0.090 0.616 0.884 −0.016

Generalized anxiety −0.070 0.827 0.933 −0.215 0.776 0.782 −0.044

PTSD −2.91 3.08 0.345 0.012 2.16 0.996 0.001

Alcohol use −0.268 0.773 0.729 −0.119 0.501 0.813 −0.024

Substance use −0.291 0.314 0.355 0.088 0.190 0.643 0.046

Distress −1.64 0.999 0.103 −1.99 0.897 0.029c −0.352

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

438 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.122. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 6 Months

Group-Level Comparison of Percentage

with Change

Outcome N Percentage with Change p-Value

Depression (clinically significant change of

10-point or greater decrease in scores)

C2C

Comparison

92

47

22.80

13.72

0.207

Generalized anxiety (clinically significant change 

of 10-point or greater decrease in scores)

C2C

Comparison

90

47

6.97

7.83

0.857

Generalized anxiety (reliable change of 

5-point or greater decrease in scores)

C2C

Comparison

90

47

36.24

30.47

0.532

PTSD (clinically significant change of

10-point or greater decrease in scores)

C2C

Comparison

90

47

42.02

33.53

0.364

PTSD (reliable change of 5-point or greater

decrease in scores)

C2C

Comparison

90

47

59.09

52.25

0.489

Alcohol use (change to below positive 

screen threshold)

C2C

Comparison

89

47

19.07

15.24

0.649

Substance use (change to below positive

screen threshold)

C2C

Comparison

88

47

24.22

13.95

0.167

Psychological distress (change to below

threshold for serious mental illness)

C2C

Comparison

86

46

29.90

27.17

0.745

Psychological distress (change to below

threshold for moderate distress)

C2C

Comparison

86

46

73.46

78.56

0.577

Table C.123. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 12 Months

Group-Level Comparison of Percentage

with Change

Outcome N Percentage with Change p-Value

Depression (clinically significant change of

10-point or greater decrease in scores)

Generalized anxiety (clinically significant change 

of 10-point or greater decrease in scores)

Generalized anxiety (reliable change of

5-point or greater decrease in scores)

PTSD (clinically significant change of

10-point or greater decrease in scores)

PTSD (reliable change of 5-point or greater

decrease in scores)

Alcohol use (change to below positive screen 

threshold)

Substance use (change to below positive screen 

threshold)

Psychological distress (change to below

threshold for serious mental illness)

Psychological distress (change to below

threshold for moderate distress)

C2C 104

Comparison 52

C2C 102

Comparison 52

C2C 102

Comparison 52

C2C 102

Comparison 52

C2C 102

Comparison 52

C2C 102

Comparison 52

C2C 100

Comparison 52

C2C 100

Comparison 52

C2C 100

Comparison 52

32.70 0.524

38.64

11.17 0.127

4.38

36.49 0.592

31.49

57.54 0.942

56.85

74.15 0.290

65.03

14.46 0.454

9.54

22.70 0.830

24.53

34.17 0.875

35.61

71.53 0.392

79.23

439 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Chapter 9 Employment, Education, Housing, and Incarceration

Table C.124. Within-Group Differences in FT/PT Employment at 6 Months (Weighted)

Employment at 6 Months

Outcome

Baseline Percentage

(Retained Sample)

6 Months

(%) Difference p-Value

FT or PT employment (vs.

unemployed)a C2C

Comparison

N/A, inclusion criteria

N/A, inclusion criteria

52.50

57.44

NOTES: FT = full-time; PT = part-time. a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and

follow-up.

Table C.125. Within-Group Mean Changes in Hours Worked and Employment Income at 6 Months 

(Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Hours worked per week

Current monthly pay before

taxes and deductions

C2C

Comparison

C2C

Comparison

75

42

71

40

16.38

16.95

842

875

2.08

3.32

139

188

<.001

<.001

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between baseline and 6 months for each group separately.

Table C.126. Within-Group Differences in FT/PT Employment at 12 Months (Weighted)

Employment at 12 Months

Baseline Percentage 12 Months

Outcome (Retained Sample) (%) Difference p-Value

FT or PT employment (vs. C2C N/A, inclusion criteria

unemployed)a

Comparison N/A, inclusion criteria

NOTES: FT = full-time; PT = part-time. a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and

follow-up.

440 
 

 

 

 

 

 

 

 

 

 

 

 

Table C.127. Within-Group Mean Changes in Hours Worked and Employment Income at 12 Months

(Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Hours worked per week

Current monthly pay before 

taxes and deductions

C2C

Comparison

C2C

Comparison

77

44

77

41

19.55

12.07

1,213

811

2.33

2.73

189

255

<.001

<.001

<.001

0.003

NOTE: SE = standard error. a Within-individual mean changes between baseline and 12 months for each group separately.

Table C.128. Within-Group Differences in Housing at 6 Months (Weighted)

Housing at 6 Months

Baseline

Outcome

(Retained Sample)

Percentage

6 Months

(%) Difference p-Value

Aggregate current housing

Stably housed or staying 

with someone elsea C2C 79.53 90.43

Comparison 95.35 89.30

a Versus transitional/temporary or homeless/unstably housed.

Table C.129. Within-Group Differences in Housing at 12 Months Weighted

Housing at 12 Months

Baseline Percentage 12 Months 

Outcome (Retained Sample) (%) Difference p-Value

Aggregate current housing

Stably housed or staying 

with someone elsea C2C 84.22 92.02

Comparison 92.49 86.75

a Versus transitional/temporary or homeless/unstably housed.

Table C.130. Within-Group Changes in Homelessness over 1 Year (Weighted)

Within-Group Difference over 1 Year

Baseline Percentage 1 Year

Outcome (Retained Sample) (%) Difference p-Value

Homeless, past year C2C 18.89 12.29 −6.61 0.042a

Comparison 16.74 9.10

a Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

441 
 

 

 

 

 

 

 

 

 

Table C.131. Within-Group Differences in Educational Attainment at 6 Months (Weighted)

Educational Attainment at 6 Months

Baseline Percentage 6 Months 

Outcome (Retained Sample) (%) Difference p-Value

Completed high school or GED C2C 73.30 73.11 −0.19 0.817

or greatera

Comparison 71.21 81.26

a Among persons ages 18 and older at baseline.

Table C.132. Within-Group Differences in Educational Attainment at 12 Months (Weighted)

Educational Attainment at 12 Months

Baseline Percentage 12 Months

Outcome (Retained Sample) (%) Difference p-Value

Completed high school or GED C2C 76.30 78.02 +1.71 0.822

or greater a

Comparison 75.48 76.21

a Among persons ages 18 and older at baseline.

Table C.133. Within-Group Changes in Incarceration over 1 Year (Weighted)

Within-Group Difference over 1 Year

Baseline Percentage 1 Year

Outcome (Retained Sample) (%) Difference p-Value

Incarcerated, past C2C 13.92 3.93

year Comparison 13.43 3.60

442 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.134. Between-Group Differences in Employment, Housing, Education, and Incarceration at 6 Months (Weighted)

Group-Level Comparison of Differences

Outcome

Propensity Score Weighted Modela

SE or

Estimate or OR 95% CI p-Value

Doubly Robust Modelb

SE or

Estimate or OR 95% CI p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Employment

FT/PT employment (vs. unemployment)c 

Change in hours worked per week

Change in current monthly pay before 

taxes and deductions

OR = 0.819

−0.574

−32.99

[0.350–1.91]

3.91

234

0.646

0.884

0.888

−1.12

−154

3.08

171

0.718

0.370

−0.062

–0.137

Aggregate current housing

–

Stably housed or staying with someone 

else (vs. transitional or homeless)

Education

Completed high school or greaterd

NOTES: SE = standard error; FT = full-time; PT = part-time. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status,

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 6 months. Does not include persons who

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Among persons ages 18 and older at baseline.

443 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.135. Between-Group Differences in Employment, Housing, Education, and Incarceration at 12 Months and 1 Year (Weighted)

Group-Level Comparison of Outcomes Standardized 

Propensity Score Weighted Modela

Estimate SE or

Doubly Robust Modelb

Estimate SE or

Effect Size

for Doubly

Robust

Outcome or OR 95% CI p-Value or OR 95% CI p-Value Model

Employment at 12-month follow-up

FT/PT employment (vs. unemployment)c OR = 1.70 [0.740–3.91] 0.213

Change in hours worked per week 7.48 3.59 0.039d −1.96 4.19 0.641 −0.100

Change in current monthly pay before taxes

and deductions

402 317 0.208 −572 462 0.220 −0.324

Current housing at 12-month follow-up

Stably housed or staying with someone elsee

Recent homelessness—over 1 year

Homeless

Education at 12-month follow-up

Completed high school or greaterf

Incarceration—1 year

—

—

—

—

Incarcerated

NOTES: Change estimates are not shown when the group size is fewer than ten, and comparisons are not shown when the group size is fewer than ten for

either group. Doubly robust model results are not shown when the group size is fewer than 20 for either group. Affected cells are shaded in gray. SE = standard 

error; FT = full-time; PT = part-time. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status,

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 12 months. Does not include persons who 

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Not statistically significant at p<.05 after adjusting for multiple comparisons. e Versus transitional/temporary or homeless/unstably housed. f

Among persons 18 and older at baseline.

444 
 

 

 

 

 

 

C8.3.5. Target Population: Parents/Primary Caregivers Who Are Expecting or Who Have 

Children up to Age 4

Chapter 7 Mental Health Care Access and Utilization

Table C.136. Within-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 92 −0.345 0.060 <.001

Comparison 55 −0.409 0.081 <.001

Attitudinal barriers C2C 92 −0.491 0.071 <.001

Comparison 55 −0.484 0.082 <.001

Stigma barriers C2C 91 −0.409 0.077 <.001

Comparison 55 −0.566 0.099 <.001

Internalized stigma C2C 87 −0.690 0.145 <.001

Comparison 51 −0.766 0.194 <.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately.

Table C.137. Within-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Logistical barriers C2C 103 −0.485 0.062 <.001

Comparison 60 −0.397 0.090 <.001

Attitudinal barriers C2C 104 −0.494 0.077 <.001

Comparison 59 −0.474 0.103 <.001

Stigma barriers C2C 103 −0.558 0.064 <.001

Comparison 59 −0.576 0.109 <.001

Internalized stigma C2C 97 −0.643 0.124 <.001

Comparison 56 −0.772 0.191 <.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately.

445 
 

 

 

 

 

 

 

Table C.138. Between-Group Mean Changes in Barriers to Care at 6 Months (Weighted)

Outcome

Group-Level Comparison of Mean Changes

Propensity Score Weighted Doubly Robust

Modela Modelb

Estimate SE p-Value Estimate SE p-Value

Standardized 

Effect Size for

Doubly 

Robust Model

Logistical barriers 0.063 0.100 0.530 0.031 0.083 0.706 0.051

Attitudinal barriers −0.007 0.109 0.947 0.011 0.110 0.920 0.018

Stigma barriers 0.157 0.126 0.212 0.080 0.123 0.518 0.114

Internalized stigma 0.076 0.242 0.754 0.075 0.118 0.524 0.053

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

Table C.139. Between-Group Mean Changes in Barriers to Care at 12 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted 

Modela

Estimate SE p-Value

Doubly Robust

Modelb

Estimate SE p-Value

Standardized 

Effect Size for

Doubly 

Robust Model

Logistical barriers −0.088 0.109 0.420 0.001 0.075 0.994 0.001

Attitudinal barriers −0.020 0.128 0.878 0.058 0.101 0.568 0.079

Stigma barriers 0.018 0.127 0.885 0.059 0.093 0.527 0.083

Internalized stigma 0.129 0.227 0.571 0.118 0.157 0.454 0.092

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

446 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.140. Within-Group Mental Health Care Utilization over 1 Year Among Persons with Need at

Baseline—Binary (Weighted)

Utilization Among Persons with Baseline Unmet Need

Baseline 

Percentage

(Retained

Binary (Yes or No) Outcome Sample) 1 Year (%) Difference p-Value

Went to any outpatient MHPa,b C2C 32.67 34.60 +1.93 0.838

Comparison 29.66 22.72 −6.94 0.377

Used any nonclinical settings or C2C 32.10 24.21 −7.90 0.191 a,c resources Comparison 38.09 25.22 −12.87 0.043d

Stayed in inpatient settinge C2C 28.78 3.47 −25.31 <.001

Comparison 17.50 4.59 −12.91 0.019d

Used emergency settingsa,f C2C 43.53 25.60 −17.92 0.005

Comparison 42.10 50.54 +8.44 0.394

a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. b Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. c Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended

religious/spiritual places, parks and recreation, or community centers for mental health needs. d Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons. e One or more nights in residential treatment program for alcohol/drug problems (past 6 months) or overnight stay in a 

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up). f

Hospital emergency room or an urgent care facility for any health reason.

Table C.141. Within-Group Mean Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Continuous (Weighted)

Within-Group Mean Change over 1 Year

Continuous (No. of Times) Outcome N Estimate SE p-Value

Number of times went to any outpatient MHPa C2C 103 0.672 1.11 0.546

Comparison 49 0.037 1.22 0.976

Number of nights in a residential treatment C2C 104 −1.69 0.774 0.031b

program for alcohol or drug problemsa

Comparison 49 0.090 0.110 0.415

Number of times went to a hospital, emergency C2C 104 −0.970 0.362 0.009

room, urgent care facility for any health reasona

Comparison 49 0.636 0.460 0.174

NOTE: SE = standard error. a Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. b Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

447 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.142. Between-Group Binary Changes in Mental Health Care Utilization over 1 Year Among 

Persons with Need at Baseline—Binary (Weighted)

Group-Level Comparison of Utilization over 1 Year

Propensity Score Weighted Modela Doubly Robust Modelb

Binary (Yes or No) Outcome OR 95% CI p-Value OR 95% CI p-Value

Went to any outpatient MHPc,d 1.91 [0.861–4.24] 0.114

Used any nonclinical settings or resourcesd,e 0.916 [0.367–2.29] 0.851

Stayed in inpatient settingf

Used emergency settingsc,g 0.342 [0.161–0.723] 0.006

a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Past 6 months at baseline; follow-up uses pooled 6- and 12-month data reflecting using over the prior year. d Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors. e Self-help or family support groups, substance use agencies, or 12-step-type programs, called a hotline, or attended

religious/spiritual places, parks and recreation, or community centers for mental health needs. f

One or more nights in residential treatment program for alcohol/drug problems (past 6 months) or overnight stay in a 

hospital for emotional, mental health, alcohol, or drug problems (lifetime at baseline, past 6 months at each follow-up). g Hospital emergency room or an urgent care facility for any health reason.

Table C.143. Between-Group Mean Changes in Mental Health Care Utilization over 1 Year Among

Persons with Need at Baseline—Continuous (Weighted)

Group-Level Comparison of Changes in Utilization over 1 Year

Propensity Score Weighted Doubly Robust Standardized 

Modela Modelb Effect Size

Continuous (No. of Times) for Doubly

Outcome Estimate SE p-Value Estimate SE p-Value Robust Model

Number of times went to any 0.635 1.65 0.701 1.89 1.55 0.229 0.169

outpatient MHPc

Number of nights in a residential −1.78 0.781 0.024c −0.053 0.206 0.798 −0.008

treatment program for alcohol or

drug problems

Number of times went to a −1.61 0.586 0.007 −1.84 0.425 <.001 −0.484

hospital, emergency room, urgent

care facility for any health reason

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. Past 6 months at baseline; follow-up 

uses pooled 6- and 12-month data reflecting using over the prior year. c Including psychiatrists, psychologists, social workers, psychiatric nurses, or counselors.

448 
 

 

 

 

 

Chapter 8 Mental Health Symptoms

Table C.144. Within-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Outcome Individual Mean Change at 6 Monthsa

N Estimate SE p-Value

Depression (score range 0–24) C2C

Comparison

96

57

−2.60

−2.26

0.575

0.687

<.001

0.002

Generalized anxiety (score range 0–21) C2C

Comparison

96

57

−2.64

−2.28

0.488

0.644

<.001

0.001

PTSD (score range 0–80) C2C

Comparison

95

57

−9.57

−7.66

1.64

2.40

<.001

0.002

Alcohol use (score range 0–40) C2C

Comparison

93

56

−2.25

−1.19

0.535

0.622

<.001

0.060

Substance use (score range 0–10) C2C

Comparison

93

55

−0.789

−0.371

0.195

0.137

<.001

0.001

Psychological distress (score range 0–24) C2C

Comparison

91

56

−3.38

−2.87

0.519

0.747

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 6-month scores for each group separately.

Table C.145. Within-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Depression (score range 0–24) C2C

Comparison

107

62

−2.64

−2.69

0.579

0.802

<.001

0.001

Generalized anxiety (score range 0–21) C2C

Comparison

107

62

−3.09

−2.53

0.572

0.798

<.001

0.002

PTSD (score range 0–80) C2C

Comparison

107

62

−11.61

−9.91

1.78

2.23

<.001

<.001

Alcohol use (score range 0–40) C2C

Comparison

105

61

−2.27

−1.83

0.491

0.633

<.001

0.005

Substance use (score range 0–10) C2C

Comparison

105

60

−0.804

−0.499

0.174

0.190

<.001

0.011

Psychological distress (score range 0–24) C2C

Comparison

104

62

−3.49

−3.31

0.519

0.841

<.001

<.001

NOTE: SE = standard error. a Within-individual mean changes between the baseline and 12-month scores for each group separately.

449 
 

 

 

 

 

 

 

 

 

Table C.146. Between-Group Mean Changes in Mental Health Symptoms at 6 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted 

Modela

Estimate SE p-Value

Doubly Robust Modelb

Estimate SE p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Depression −0.340 0.895 0.704 −0.605 0.721 0.404 −0.109

Generalized anxiety −0.365 0.809 0.652 −0.081 0.785 0.918 −0.017

PTSD −1.91 2.91 0.513 −0.362 2.56 0.888 −0.022

Alcohol use −1.06 0.820 0.198 −0.679 0.458 0.142 −0.128

Substance use −0.419 0.238 0.081 −0.062 0.161 0.699 −0.035

Distress −0.506 0.910 0.579 −0.307 0.811 0.706 −0.060

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score 

weighted models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model.

Table C.147. Between-Group Mean Changes in Mental Health Symptoms at 12 Months (Weighted)

Group-Level Comparison of Mean Changes

Outcome

Propensity Score Weighted 

Modela

Estimate SE p-Value

Doubly Robust

Modelb

Estimate SE p-Value

Standardized 

Effect Size for

Doubly Robust

Model

Depression 0.045 0.990 0.964 0.738 0.789 0.352 0.125

Generalized anxiety −0.568 0.982 0.564 0.684 0.805 0.398 0.117

PTSD −1.70 2.85 0.553 5.14 2.36 0.032c 0.293

Alcohol use −0.442 0.802 0.582 0.248 0.315 0.434 0.047

Substance use −0.305 0.258 0.238 0.086 0.194 0.657 0.050

Distress −0.187 0.989 0.850 0.257 0.778 0.741 0.046

NOTE: SE = standard error. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted 

models where the comparison group is weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education 

level, employment status, housing status, and incarceration status in the model. c Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

450 
 

 

 

 

 

 

 

 

 

Table C.148. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 6 Months

Group-Level Comparison of Percentage

with Change

Percentage with

Outcome N Change p-Value

Depression (clinically significant change of 10-point or

greater decrease in scores)

Generalized anxiety (clinically significant change of

10-point or greater decrease in scores)

Generalized anxiety (reliable change of 5-point or

greater decrease in scores)

PTSD (clinically significant change of 10-point or

greater decrease in scores)

PTSD (reliable change of 5-point or greater decrease 

in scores)

Alcohol use (change to below positive screen 

threshold)

Substance use (change to below positive screen 

threshold)

Psychological distress (change to below threshold for

serious mental illness)

Psychological distress (change to below threshold for

moderate distress)

C2C 96 34.08 0.684

Comparison 57 30.76

C2C 96 7.16 0.625

Comparison 57 5.17

C2C 96 30.21 0.927

Comparison 57 29.49

C2C 95 43.29 0.517

Comparison 57 37.77

C2C 95 57.82 0.348

Comparison 57 49.72

C2C 93 15.00 0.267

Comparison 56 8.71

C2C 93 13.07 0.097

Comparison 55 4.16

C2C 91 24.22 0.780

Comparison 56 22.15

C2C 91 68.00 0.780

Comparison 56 65.69

451 
 

 

 

 

 

 

 

 

 

 

 

Table C.149. Clinically Significant/Reliable Change for Mental Health Symptoms and Group-Level

Comparison at 12 Months

Group-Level Comparison of Percentage

with Change

Percentage with

Outcome N Change p-Value

Depression (clinically significant change of 10-point

or greater decrease in scores)

Generalized anxiety (clinically significant change of

10-point or greater decrease in scores)

Generalized anxiety (reliable change of 5-point or

greater decrease in scores)

PTSD (clinically significant change of 10-point or

greater decrease in scores)

PTSD (reliable change of 5-point or greater

decrease in scores)

Alcohol use (change to below positive screen 

threshold)

Substance use (change to below positive screen 

threshold)

Psychological distress (change to below threshold 

for serious mental illness)

Psychological distress (change to below threshold 

for moderate distress)

C2C 107

Comparison 62

C2C 107

Comparison 62

C2C 107

Comparison 62

C2C 107

Comparison 62

C2C 107

Comparison 62

C2C 105

Comparison 61

C2C 105

Comparison 61

C2C 104

Comparison 62

C2C 104

Comparison 62

37.08 0.975

37.33

15.53 0.291

9.71

33.63 0.696

36.71

52.56 0.278

43.50

63.00 0.552

58.16

17.72 0.201

10.08

13.93 0.319

8.09

21.95 0.948

22.40

65.26 0.931

65.94

452 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Chapter 9 Employment, Education, Housing, and Incarceration

Table C.150. Within-Group Differences in FT/PT Employment at 6 Months (Weighted)

Employment at 6 Months

Outcome

Baseline Percentage

(Retained Sample) 6 Months (%) Difference p-Value

FT or PT employment (vs.

unemployed)a C2C

Comparison

27.73

30.07

57.26

48.07

29.53

17.99

<.001

0.010

NOTES: FT = full-time; PT = part-time. a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and

follow-up.

Table C.151. Within-Group Mean Changes in Hours Worked and Employment Income at 6 Months 

(Weighted)

Individual Mean Change at 6 Monthsa

Outcome N Estimate SE p-Value

Hours worked per week

Current monthly pay before 

taxes and deductions

C2C

Comparison

C2C

Comparison

68

37

62

33

10.5

3.33

755

340

2.53

2.48

194

153

<.001

0.188

<.001

0.033b

NOTE: SE = standard error. a Within-individual mean changes between baseline and 6-month scores for each group separately. b Estimate was not statistically significant at p<.05 after adjusting for multiple comparisons.

Table C.152. Within-Group Differences in FT/PT Employment at 12 Months (Weighted)

Employment at 12 Months

Outcome

Baseline Percentage

(Retained Sample)

12 Months 

(%) Difference p-Value

FT or PT employment (vs.

unemployed)a C2C

Comparison

24.81

32.72

56.53

46.04

+31.72

+13.32

<.001

0.077

NOTES: FT = full-time; PT = part-time. a Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and

follow-up.

Table C.153. Within-Group Mean Changes in Hours Worked and Employment Income at 12 Months

(Weighted)

Individual Mean Change at 12 Monthsa

Outcome N Estimate SE p-Value

Hours worked per week C2C

Comparison

83

36

13.00

5.00

2.28

2.66

<.001

0.069

Current monthly pay before 

taxes and deductions

C2C

Comparison

75

34

894

481

197

203

<.001

0.024

NOTE: SE = standard error. a Within-individual mean changes between baseline and 12-month scores for each group separately.

453 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.154. Within-Group Differences in Housing at 6 Months (Weighted)

Housing at 6 Months

Outcome

Baseline Percentage

(Retained Sample)

6 Months 

(%) Difference p-Value

Aggregate current housing

Stably housed or staying 

with someone elsea C2C

Comparison

67.45

73.85

74.91

80.54

+6.69

+7.45

0.067

0.069

a Versus transitional/temporary or homeless/unstably housed.

Table C.155. Within-Group Differences in Housing at 12 Months (Weighted)

Housing at 12 Months

Outcome

Baseline Percentage

(Retained Sample)

12 Months 

(%) Difference p-Value

Aggregate current housing

Stably housed or staying 

with someone elsea C2C

Comparison

72.21

76.39

87.82

87.47

+15.62

+11.08

<.001

0.009

a Versus transitional/temporary or homeless/unstably housed.

Table C.156. Within-Group Changes in Homelessness over 1 Year (Weighted)

Within-Group Difference over 1 Year

Baseline Percentage 1 Year

Outcome (Retained Sample) (%) Difference p-Value

Homeless, past year C2C 15.87 15.84

Comparison 10.52 10.29

Table C.157. Within-Group Differences in Educational Attainment at 6 Months (Weighted)

Educational Attainment at 6 Months

Baseline Percentage 6 Months

Outcome (Retained Sample) (%) Difference p-Value

Completed high school or GED C2C 73.97 73.95 +0.02 0.775

or greatera

Comparison 71.77 74.58 −2.81 0.215

a Among persons ages 18 and older at baseline.

454 
 

 

 

 

 

 

Table C.158. Within-Group Differences in Educational Attainment at 12 Months (Weighted)

Educational Attainment at 12 Months

Outcome

Baseline Percentage

(Retained Sample)

12 Months 

(%) Difference p-Value

Completed high school or

general educational or 

greatera

C2C

Comparison

74.87

72.20

74.98

76.89

+0.11

+4.68

0.800

0.293

a Among persons ages 18 and older at baseline.

Table C.159. Within-Group Changes in Incarceration over 1 Year (Weighted)

Within-Group Difference over 1 Year

Outcome

Baseline Percentage

(Retained Sample) 1 Year (%) Difference p-Value

Incarcerated, past year C2C

Comparison

11.46

5.98

2.60

5.39

455 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.160. Between-Group Differences in Employment, Housing, Education, and Incarceration at 6 Months (Weighted)

Group-Level Comparison of Differences Standardized 

Propensity Score Weighted Modela

SE or

Doubly Robust Modelb

SE or

Effect Size

for Doubly

Robust

Outcome Estimate or OR 95% CI p-Value Estimate or OR 95% CI p-Value Model

Employment

FT/PT employment (vs.

unemployment)c OR = 1.38 [0.640–2.96] 0.414 OR = 0.749 [0.191–2.93] 0.679

Change in hours worked per week 7.14 3.54 0.046d −0.817 3.94 0.837 −0.042

Change in current monthly pay before 

taxes and deductions

415 247 0.096 −0.795 342 0.998 −0.001

Aggregate current housing

Stably housed or staying with someone 

else (vs. transitional or homeless)

OR = 0.919 [0.316–2.67] 0.877

Education

Completed high school or greatere OR = 0.967 [0.428–2.19] 0.936

NOTES: SE = standard error; FT = full-time; PT = part-time. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status,

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 6 months. Does not include persons who 

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Estimate was not statistically significant at p < .05 after adjusting for multiple comparisons. e Among persons ages 18 and older at baseline.

456 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C.161. Between-Group Differences in Employment, Housing, Education, and Incarceration at 12 Months and 1 Year (Weighted)

Outcome

Group-Level Comparison of Outcomes

Propensity Score Weighted Modela Doubly Robust Modelb

Estimate SE or Estimate SE or

or OR 95% CI p-Value or OR 95% CI p-Value

Standardized 

Effect Size

for Doubly

Robust

Model

Employment at 12-month follow-up

FT/PT employment (vs. unemployment)c

Change in hours worked per week

Change in current monthly pay before 

taxes and deductions

OR = 1.30

8.00

413

[0.621–2.74]

3.50

282

0.485

0.024d

0.146

OR = 0.704

3.89

134

[0.227–2.18]

3.46

245

0.546

0.265

0.587

0.199

0.080

Stably housed or staying with someone 

elsee

Current housing at 12-month follow-up

–

Homeless

Recent homelessness—over 1 year

–

Completed high school or greaterf

Education at 12-month follow-up

OR = 0.952 [0.431–2.10] 0.903 –

Incarceration—over 1 year

Incarcerated

NOTES: Change estimates are not shown when the group size is fewer than ten, and comparisons are not shown when the group size is fewer than ten for

either group. Doubly robust model results are not shown when the group size is fewer than 20 for either group. Affected cells are shaded in gray. SE = standard 

error; FT = full-time; PT = part-time. a Group-level comparison of within-individual mean changes from baseline to 12 months in propensity score weighted models where the comparison group is

weighted to “look like” the C2C group through propensity score weights. b Propensity score weighted model with additional (double) control for baseline age, gender, race/ethnicity, education level, employment status, housing status,

and incarceration status in the model. c Among persons who were employed full-time, employed part-time, or unemployed/looking for work at baseline and 12 months. Does not include persons who 

were students, retired and not working, homemakers, disabled or too ill to work, or other. d Not statistically significant at p < .05 after adjusting for multiple comparisons. e Versus transitional/temporary or homeless/unstably housed f

Among persons 18 and older at baseline

457 
 

 

 

 

 

References

Altena, A. M., S. N. Brilleslijper-Kater, and J. L. Wolf, “Effective Interventions for Homeless 

Youth: A Systematic Review,” American Journal of Preventive Medicine, Vol. 38, No. 6, 

2010, pp. 637–645. 

Babor, T. F., J. R. La Fuente, J. Saunders, and M. Grant, AUDIT, the Alcohol Use Disorders 

Identification Test: Guidelines for Use in Primary Health Care. Geneva: Substance Abuse 

Department, World Health Organization, 1992. 

Bassuk, E. L., J. C. Buckner, J. N. Perloff, and S. S. Bassuk, “Prevalence of Mental Health and 

Substance Use Disorders Among Homeless and Low-Income Housed Mothers,” American 

Journal of Psychiatry, Vol. 155, No. 11, 1998, pp. 1561–1564. 

Benjamini, Y., and Y. Hochberg, “Controlling the False Discovery Rate—A Practical and 

Powerful Approach to Multiple Testing,” Journal of the Royal Statistical Society Series 

B-Methodological, Vol. 57, No. 1, 1995, pp. 289–300. 

Berner, M. M., L. Kriston, M. Bentele, and M. Harter, “The Alcohol Use Disorders Identification 

Test for Detecting At-Risk Drinking: A Systematic Review and Meta-Analysis,” Journal of 

Studies on Alcohol and Drugs, Vol. 68, No. 3, 2007, pp. 461–473. 

Blevins, C. A., F. W. Weathers, M. T. Davis, T. K. Witte, and J. L. Domino, “The Posttraumatic 

Stress Disorder Checklist for DSM-5 (PCL-5): Development and Initial Psychometric 

Evaluation,” J Trauma Stress, Vol. 28, No. 6, 2015, pp. 489–498. 

Breslau, N., G. C. Davis, P. Andreski, E. L. Peterson, and L. R. Schultz, “Sex Differences in 

Posttraumatic Stress Disorder,” Archives of General Psychiatry, Vol. 54, No. 11, 1997, 

pp. 1044–1048. 

Brookhart, M. A., S. Schneeweiss, K. J. Rothman, R. J. Glynn, J. Avorn, and T. Sturmer, 

“Variable Selection for Propensity Score Models,” American Journal of Epidemiology, 

Vol. 163, No. 12, 2006, pp. 1149–1156. 

Chung, B., M. Ong, S. L. Ettner, F. Jones, J. Gilmore, M. McCreary, C. Sherbourne, V. Ngo, 

P. Koegel, L. Tang, E. Dixon, J. Miranda, T. R. Belin, and K. B. Wells, “12-Month 

Outcomes of Community Engagement Versus Technical Assistance to Implement 

Depression Collaborative Care: A Partnered, Cluster, Randomized, Comparative 

Effectiveness Trial,” Annals of Internal Medicine, Vol. 161, No. 10, 2014, pp. S23–S34. 

Clement, S., E. Brohan, D. Jeffery, C. Henderson, S. L. Hatch, and G. Thornicroft, “Development 

and Psychometric Properties the Barriers to Access to Care Evaluation Scale (BACE) Related 

to People with Mental Ill Health, BMC Psychiatry, 12, 2012, p. 36. 

Cohen, J. Statistical Power Analysis for the Behavioural Sciences, 2nd ed., Hillsdale, NJ: 

Erlbaum, 1988. 

458 
 

 

 

 

 

 

Czajka, J. L., and A. Beyler. Background Paper Declining Response Rates in Federal Surveys: 

Trends and Implications. Washington, DC: Mathematica Policy Research, 2016. 

Davis, L. L., M. E. Broome, and R. P. Cox, “Maximizing Retention in Community-Based 

Clinical Trials,” Journal of Nursing Scholarship, Vol. 34, No. 1, 2002, pp. 47–53. 

Goodman, E., and B. Huang, “Socioeconomic Status, Depressive Symptoms, and Adolescent 

Substance Use,” Archives of Pediatrics & Adolescent Medicine,, Vol. 156, No. 5, 2002, 

pp. 448–453. 

Kessler, R. C., P. R. Barker, L. J. Colpe, J. F. Epstein, J. C. Gfroerer, E. Hiripi, and A. M. 

Zaslavsky, “Screening for Serious Mental Illness in the General Population,” Archives of 

General Psychiatry, Vol. 60, No. 4, 2003, pp. 184–189. 

Kessler, R. C., H. G. Birnbaum, V. Shahly, E. Bromet, I. Hwang, K. A. McLaughlin, N. Sampson, 

L. H. Andrade, G. de Girolamo, K. Demyttenaere, J. M. Haro, A. N. Karam, S. Kostyuchenko, 

V. Kovess, C. Lara, D. Levinson, H. Matschinger, Y. Nakane, M. O. Browne, J. Ormel, 

J. Posada-Villa, R. Sagar, and D. J. Stein, “Age Differences in the Prevalence and CoMorbidity of DSM-IV Major Depressive Episodes: Results from the WHO World Mental 

Health Survey Initiative,” Depression and Anxiety, Vol. 27, No. 4, 2010, pp. 351–364. 

Kroenke, K., R. L. Spitzer, and J. B. W. Williams, “The PHQ-9: Validity of a Brief Depression 

Severity Measure,” Journal of General Internal Medicine, Vol. 16, No. 9, 2001, pp. 606–613. 

Kroenke, K., R. L. Spitzer, J. B. Williams, and B. Löwe. “The Patient Health Questionnaire 

Somatic, Anxiety, and Depressive Symptom Scales: A Systematic Review.” General 

Hospital Psychiatry, Vol. 32, No. 4, 2010, pp. 345–359. 

Lee, B. K., J. Lessler, and E. A. Stuart, “Improving Propensity Score Weighting Using Machine 

Learning,” Statistics in Medicine, Vol. 29, No. 3, 2010, pp. 337–346. 

Leonard, N. R., P. Lester, M. J. Rotheram-Borus, K. Mattes, M. Gwadz, and B. Ferns, “Successful 

Recruitment and Retention of Participants in Longitudinal Behavioral Research,” AIDS 

Education and Prevention, Vol. 15, No. 3, 2003, pp. 269–281. 

Li, F., A. M. Zaslavsky, and M. B. Landrum, “Propensity Score Weighting with Multilevel 

Data,” Statistics in Medicine, Vol. 32, No. 19, 2013, pp. 3373–3387. 

Löwe, B., O. Decker, S. Muller, E. Brahler, D. Schellberg, W. Herzog, and P. Y. Herzberg, 

“Validation and Standardization of the Generalized Anxiety Disorder Screener (GAD-7) in 

the General Population,” Medical Care, Vol. 46, No. 3, 2008, pp. 266–274. 

Lumley, T., Survey: Analysis of Complex Survey Samples, R Package Version 3.30, 2014. 

Lunceford, J. K., and M. Davidian, “Stratification and Weighting via the Propensity Score in 

Estimation of Causal Treatment Effects: A Comparative Study,” Statistics in Medicine, 

Vol. 23, No. 19, 2004, pp. 2937–2960. 

459 
 

 

McCaffrey, D. F., G. Ridgeway, and A. R. Morral, “Propensity Score Estimation with Boosted 

Regression for Evaluating Causal Effects in Observational Studies,” Psychological Methods, 

Vol. 9, No. 4, 2004, pp. 403–425. 

National Center for PTSD. Using the PTSD Checklist for DSM-5 (PCL-5), 2019. 

https://www.ptsd.va.gov/professional/assessment/adult-sr/ptsd-checklist.asp 

National Comorbidity Survey, homepage, undated. 

http://www.hcp.med.harvard.edu/ncs/ 

Ridgeway, G., D. McCaffrey, A. R. Morral, B. Griffin, and L. Burgette, twang: Toolkit for 

Weighting and Analysis of Nonequivalent Group, R Package Version 1.4-9.5, 2015. 

https://CRAN.R-project.org/package=twang 

Spitzer, R. L., K. Kroenke, J. B. Williams, and B. Lowe, “A Brief Measure for Assessing 

Generalized Anxiety Disorder: The GAD-7,” Archives of Internal Medicine, Vol. 166, 

No. 10, 2006, pp. 1092–1097. 

Teague, S., G. J. Youssef, J. A. Macdonald, E. Sciberras, A. Shatte, M. Fuller-Tyszkiewicz, 

C. Greenwood, J. McIntosh, C. A. Olsson, D. Hutchinson, and Seed Lifecourse Sciences 

Theme, “Retention Strategies in Longitudinal Cohort Studies: A Systematic Review and 

Meta-Analysis,” BMC Medical Research Methodology, Vol. 18, No. 1, 2018, p. 151. 

Wells, T. S., J. L. Horton, C. A. LeardMann, I. G. Jacobson, and E. J. Boyko, “A Comparison of 

the PRIME-MD PHQ-9 and PHQ-8 in a Large Military Prospective Study, the Millennium 

Cohort Study.” Journal of Affective Disorders, Vol. 148, No. 1, 2013, pp. 77–83. 

Yudko, E., O. Lozhkina, and A. Fouts, “A Comprehensive Review of the Psychometric 

Properties of the Drug Abuse Screening Test,” Journal of Substance Abuse Treatment, 

Vol. 32, No. 2, 2007, pp. 189–198. 

460 
 

 

 

 

 

Appendix D. Cost Evaluation and Sensitivity Analyses

Michele Abbott and Harry Liu 

For the first sensitivity analysis, we used the indirect rate charged on C2C project invoices to 

calculate the total overhead cost per CBO. In general, this method decreased overhead costs as 

well as the average total program cost per CBO. Across all CBOs in year 2, the average total 

program cost decreased from $437,546 in the main analysis to $399,374 in the sensitivity 

analysis. The percentage of program costs spent on overhead in year 2 decreased from 17 to 

10 percent. In year 3, the average total program cost decreased from $514,142 in the main 

analysis to $462,522 in the sensitivity analysis. The percentage of program costs spent on 

overhead in year 3 decreased from 18 to 9 percent. 

Findings from the results of the main analysis remain robust to this sensitivity analysis. For 

example, the overall C2C cost per client under this sensitivity specification is $489 in year 2 

(9 percent less compared with a cost per client of $536 in the main analysis) and $433 in year 3 

(10 percent less compared with $482). These reductions demonstrate that the externally set 

indirect rates based on federally negotiated indirect cost rate agreements tend to be lower than 

the true overhead costs incurred by the CBOs. 

Figure D.1. Average Annual Program Cost per CBO Using Invoice-Based Overhead Cost Rates, by 

Cost Component, Project Year, and CBO Type

SOURCE: Staff surveys; compensation survey 2018; annual nonlabor reports 2017 and 2018; financial statements

2016 to 2018; project invoices.

NOTE: Percentages are not labeled for components of 10 percent or less.

461 
 

 

The second sensitivity analysis measured only the program costs charged by CBOs on the 

C2C project invoices, and therefore, the only costs the CBOs were reimbursed for. We find that 

the program costs billed on invoices are substantially less than those estimated using staff 

surveys and financial statements. 

As shown in Figure D.2 below, overall, CBOs spent an average of $174,741 in year 1, 

$244,870 in year 2 (compared with $437,546 in the main analysis), and $286,582 in year 3 

(compared with $514,142 in the main analysis). Thus, on average, billable program costs 

increased by 40 percent from year 1 to year 2 and an additional 17 percent in year 3. Average 

invoice-based program costs were about 40 percent less than program costs in the main analysis 

both in year 2 and year 3. 

The distribution of program costs also differed by CBO type. Parent/caregiver-serving 

CBOs spent the least in year 3 ($216,778) on average and most of the costs were spent on labor 

(76 percent) and variable expenses (14 percent). Conversely, the average youth development 

CBO spent the most in year 3 ($398,883), with 87 percent of program costs spent on labor and 

7 percent on overhead. In year 3, job training and employment and other CBOs spent, on 

average, $271,872 and $273,099, respectively. However, these CBOs tended to spend less on 

labor (63 percent for job training and employment CBOs and 48 percent for other CBOs) and 

Figure D.2. Average Annual Program Cost per CBO Based on Invoices, by Cost Component, 

Project Year, and CBO Type

SOURCE: Project invoices.

NOTE: Percentages are not labeled for components less than 10 percent.

462 
 

 

 

 

 

more on MHP contracts (22 and 23 percent, respectively) and variable costs (6 and 19 percent, 

respectively), compared with the parent/caregiver-serving and youth development CBOs. 

Continuing with the second sensitivity analysis, we estimated an invoice-based cost per client 

served by dividing the total program costs based on invoices for each CBO type by the total 

number of clients served for each CBO type. As illustrated below in Figure D.3, the cost per 

client in year 1 ($638) was over twice as high than in years 2 and 3 ($300 and $269, respectively). 

This is due to the smaller number of clients served during the start-up phase of the C2C Initiative. 

Across all CBOs, the average cost per client was about 40 percent less in the invoice-based 

analysis than in the main analysis (Figure 8.10) in year 2 and year 3. 

Though parent/caregiver-serving CBOs spent, on average, the least amount of total program 

costs, the cost per client served was the largest among CBO types in year 3 ($338; range: 

$164–$1,593). The average cost per client served among the remaining CBO types in year 3 was 

$253 for job training and employment CBOs (range: $131–$534), $250 for youth development 

CBOs (range: $137–$510), and $280 for other CBOs (range: $116–$524). 

For the third sensitivity analysis, we adjusted staff labor to estimate the marginal increase in 

labor hours and cost related to implementing C2C. We assumed a 5- to 25-percent increase in 

Figure D.3. Average Program Cost per Client Served Based on Invoices, by Cost Component,

Project Year, and CBO Type

SOURCE: Project invoices; quarterly reports.

NOTES: Total number of clients served in year 1, not reported in the main text, are as follows: 295 for parent/

caregiver-serving CBOs (average: 98), 2,406 for job training and employment CBOs (average: 481), 1,001 for youth 

development CBOs (average: 334), 404 for other CBOs (average: 101), and 4,106 across all CBOs (average: 274).

Percentages are not labeled for components 10 percent or less.

463 
 

 

 

 

labor hours associated with service delivery based on qualitative interviews with CBO staff and 

leadership. In year 2, CBOs spent an average of $438,348 to $465,520 on C2C (compared with 

$437,546 in the main analysis) and $520,339 to $567,216 in year 3 (compared with $512,449 in 

the main analysis). These costs represent a 0.2- to 6.4-percent increase in average program cost 

in year 2 and a 1.4- to 10.7-percent increase in year 3, compared with the main analysis. 

Figure D.4 shows the average program costs per client across CBOs, by percentage increase 

in time spent on service delivery and project year. Using these labor-adjusted program costs, the 

average cost per client served was $544 to $577 in year 2 and $493 to $537 in year 3 (compared 

with main analysis results of $536 in year 2 and $481 in year 3). In conclusion, assuming that 

implementing C2C included a 5-percent increase in time spent on service delivery increased the 

average cost per client by $8 in year 2 and $12 in year 3. Assuming a 25-percent increase in time 

spent on service delivery increased the average cost per client by $41 in year 2 and $56 in year 3. 

Figure D.4. Average Annual Program Cost per Client Served, Adjusted for Incremental Time Spent

on Delivering CBO Services Due to Utilizing C2C Skills, by Cost Component and Project Year

SOURCE: Staff surveys; compensation survey 2018; annual nonlabor reports 2017, 2018; financial statements 2016

to 2018; project invoices; CBO quarterly reports.

NOTE: Percentages are not labeled for components that are 10 percent .or less

As shown in Figure D.5 below, we observed an increase in the amount of time for referrals 

(17 percent) and providing coaching (75 percent), but a decrease in receiving training (65 percent), 

receiving coaching (57 percent), providing training (15 percent), technical assistance (21 percent), 

data collection (46 percent), and project management (48 percent). However, it is important to 

note that although we had almost all survey respondents estimate the total number of hours 

worked on C2C (Figure D.6), we got a worse response rate for the questions on breakdown of 

464 
Figure D.5. Average Weekly Labor Hours per Staff Member in Year 4, by Job Category and C2C

Activity

 

 

 

 

 

 

SOURCE: The staff surveys in year 4.

NOTES: Because we were unable to collect job category information for all nonrespondents, results reflect the

average staff who responded to the staff survey and were not weighted for the total number of staff. Given a small

number of staff who responded to the labor-hour questions in the year 4 staff survey, the results may not be reliable.

Percentages are not labeled for components that are less than 10 percent.

Figure D.6. Average Annual Labor Hours per CBO in Year 4, by CBO Type

SOURCE: The staff survey in year 4.

465 
 

labor hours by C2C activity, demonstrated in the figure below. Thus, the numbers below 

(especially, e.g., the hours spent on referrals for administrative staff) cannot be considered as 

representative of the full CBO staff working on C2C. 

The staff survey in year 4 shows that the total labor hours for all CBOs decreased by 12 percent, 

compared with year 3, to 9,109 annual labor hours (median: 7,522; range: 1,305–22,549). This 

number has been weighted for survey nonresponse. Annual labor cost in year 4 decreased by 

16 percent compared with year 3 to $265,329 (median: $242,661; range: $38,013–$714,096). 

466