The Fundamentals of Data Collection and 

Evaluation, Version 4.2

Section 1 of the Guide for 

Evaluating Your CCP
The Fundamentals of Data Collection and Evaluation (Version 4.2) Section 1 

2022 Guide for Evaluating Your CCP (Version 4.2) | i

Table of Contents

Introduction—the Basics .............................................................................................................. 1

How is the Guide for Evaluating Your CCP organized? ............................................................. 1 

How should this guide be used? .................................................................................................. 1 

What is the Crisis Counseling Assistance and Training Program? ............................................. 1 

What are the roles of FEMA, SAMHSA, and the SAMHSA Disaster Technical

Assistance Center? ...................................................................................................................... 2 

What are the roles of states and/or service providers with a CCP? ............................................. 2 

Why is it important to collect consistent data on CCP services? ................................................. 2 

Understanding Program Evaluation ............................................................................................3

What is program evaluation? ........................................................................................................ 3 

Why is it important? ..................................................................................................................... 3

How are results used? ................................................................................................................... 3 

How does it work? ........................................................................................................................ 3

What are inputs? ........................................................................................................................... 4 

What are activities?...................................................................................................................... 4 

What are outputs and outcomes, and how are they different? ..................................................... 4 

Evaluation for CCPs ..................................................................................................................... 5

What are the goals? ...................................................................................................................... 5 

What is program reach? ................................................................................................................ 5

What is program quality? ............................................................................................................ 5 

What is program consistency? ...................................................................................................... 5 

Data Management and Analysis .................................................................................................. 6

How are the forms produced? ...................................................................................................... 6 

How are the forms filled in? ........................................................................................................ 6 

Where do completed tools go? .................................................................................................... 6 

How often are they submitted? .................................................................................................... 6 

What do supervisors do?.............................................................................................................. 7 

How are the data entered? ........................................................................................................... 7 

What reports are required? .......................................................................................................... 7 

Who is responsible for this work? ............................................................................................... 7 

How are the data analyzed? .......................................................................................................... 7 

Is this evaluation enough? ............................................................................................................7
The Fundamentals of Data Collection and Evaluation (Version 4.2) Section 1 

2022 Guide for Evaluating Your CCP (Version 4.2) | 1

Introduction—the Basics

How is the Guide for Evaluating Your CCP (Version 4.2) organized?

This guide is organized into four modules.

1. The first section is the Fundamentals of Data Collection and Evaluation (which you

are reading now). The manual is organized as a series of questions that outreach workers,

crisis counselors, supervisors, program planners, and data managers might ask as they

perform ongoing program monitoring. The manual includes an overview of the Crisis

Counseling Assistance and Training Program (CCP) and how evaluation is used in the

program.

2. The second section is the CCP Mobile Application User Manual, which is designed to

aid users in understanding the features of the CCP mobile application.

3. The third section is CCP Data Collection Forms and Instructions, which should be

used by outreach workers and crisis counselors to become familiar with each of the

forms. This section also includes examples of each form that is used by the program.

4. The fourth section is the Online Data Collection and Evaluation System User Manual,

which is designed to assist users of the CCP Online Data Collection and Evaluation

System (ODCES) in understanding the features of the system, including data entry and

reporting functions.

How should this guide be used?

The guide was created to serve two important functions. First, it should be used for training 

direct-service staff (crisis counselors and/or outreach workers) and other relevant program staff 

about CCP evaluation data forms and their appropriate use when out in the field working with 

disaster survivors. In addition, it can be used as an ongoing reference when questions arise 

regarding all facets of CCP data collection and evaluation. 

What is the Crisis Counseling Assistance and Training Program?

The Crisis Counseling Assistance and Training Program (commonly referred to as the Crisis 

Counseling Program or CCP) is funded by the Federal Emergency Management Agency 

(FEMA) through the Robert T. Stafford Disaster Relief and Emergency Assistance Act (Public 

Law 100-707, which amended the Disaster Relief Act of 1974, or Public Law 93-288). U.S. 

states, territories, and federally recognized tribes1 are eligible to apply for a CCP grant after the 

President has made a declaration of disaster for Individual Assistance for the state. The CCP is 

administered through a federal interagency partnership between FEMA and the Substance Abuse 

and Mental Health Services Administration (SAMHSA) Center for Mental Health Services 

(CMHS). The CCP is composed of two grant programs: 

1. Immediate Services Program (ISP), which is 60 days in duration

2. Regular Services Program (RSP), which is up to nine months in duration

CCPs aim to meet short-term mental health and substance use-related needs of affected 

communities through counseling, outreach, public education, training, and referral. In recent

1 After this point, we use the term “state” to refer to a state, territory, or federally recognized tribe.

1.

2.

3.

4.

1.

2.
The Fundamentals of Data Collection and Evaluation (Version 4.2) Section 1 

2022 Guide for Evaluating Your CCP (Version 4.2) | 2

years, there have been approximately 15-20 active CCPs per year, but in some years, there have 

been more (such as after the 2005 Hurricanes Katrina, Wilma, and Rita, and during the COVID19 global pandemic). The CCP has provided brief mental health and substance use-related 

services to millions of disaster survivors since its inception and has become an important model 

for response to a variety of catastrophic events.

What are the roles of FEMA, SAMHSA, and the SAMHSA Disaster Technical 

Assistance Center?

The CCP is a partnership between FEMA and SAMHSA CMHS. CMHS provides states with 

consultation and assistance in implementing the program. As a part of CMHS, the SAMHSA 

Disaster Technical Assistance Center (DTAC) provides technical assistance throughout the 

phases of disaster recovery, including dedicated technical assistance for CCPs. SAMHSA DTAC 

maintains a library of print and electronic resource materials and offers assistance through a tollfree helpline (800-308-3515), a comprehensive website (www.samhsa.gov/dtac), and an email 

account (dtac@iqsolutions.com).

What are the roles of states and/or service providers in a CCP?

CCP services are typically provided to the disaster-affected areas by mental health and substance 

use disorder treatment organizations through contracts with a state’s department of mental 

health. In some cases, crisis counselors are hired directly as state employees. CCP staff members 

usually include a combination of mental health and substance use disorder treatment 

professionals and paraprofessionals trained and supervised to deliver an array of crisis 

counseling services. CCP staff members generally come from the affected communities and are 

sometimes survivors themselves. The CCP-funded state is tasked with ensuring that CCP 

services, reporting requirements, and financial documentation adhere to Section 416 of the 

Stafford Disaster Relief and Emergency Assistance Act. 

Why is it important to collect consistent data on CCP services?

Collecting accurate information about services and service recipients is essential for monitoring 

and evaluating CCPs. In the past, states developed their own procedures and forms. This process 

was time-consuming and often missed finding answers to important questions. SAMHSA CMHS 

recognized that standard reporting methods needed to be implemented in order to make the data 

meaningful and more accurate across disasters as well as across U.S. states.

In 2005, SAMHSA CMHS introduced a data toolkit to standardize program activities, 

definitions, and data collection. These tools were originally evaluated and approved by the 

Office of Management and Budget (OMB) in September 2005 with an expiration date of 

September 2008. A revised set of tools was again submitted to OMB and was approved in 

January 2009 (OMB No. 0930-0270) with an expiration date of January 2012. Another set of 

changes to all seven forms was approved by OMB in the fall of 2012, with an expiration date of 

August 2015. The next iterations of the forms, which had expiration dates of September 2018 

and July 2022, had minor changes to reflect changes to the data collection system and better 

align the demographic and suicidal ideation questions to other data collection instruments. 

Further changes were made, including updating location-of-service and risk categories as well 

as media categories, following the adaptation to virtual services during the COVID-19 

pandemic. The current data collection suite, with an expiration date of August 2025, was

approved and launched in 2022. 
The Fundamentals of Data Collection and Evaluation (Version 4.2) Section 1 

2022 Guide for Evaluating Your CCP (Version 4.2) | 3

Program Evaluation

What is program evaluation?

Program evaluation refers to systematic efforts to collect, analyze, and interpret information 

about the delivery or outcomes of interventions. Program monitoring typically relies on easily 

measurable indicators that can be tracked over time, such as the number of crisis counseling 

encounters or client satisfaction.

Why is it important?

Continuing CCP recognition, acceptance, and support depend, at least in part, on the program’s 

ability to show sponsors and other interested parties that it delivers the services it intends to 

deliver and that survivors benefit from the services provided. Program achievements are 

documented through program evaluation. A useful management tool, evaluation also helps 

program administrators to determine if the project is proceeding according to plan so that they 

can make midcourse corrections when needed.

How are results used?

Ultimately, evaluation is not about gathering data but about using data to draw conclusions. 

Evaluation results are open to interpretation. A program may reach a large number of people but 

only a fraction of the total population at risk. Program results may involve trade-offs. For 

example, crisis counselors who tirelessly throw themselves into their work may realize greater 

consumer satisfaction but experience greater burnout. An innovative program may serve fewer 

clients but get better results for those it does serve. Different stakeholder groups may judge these 

exchanges differently.

Evaluations are useful only if their results are communicated. Program managers should 

regularly share results in staff meetings, quarterly updates, or even graphs posted on the wall. 

This feedback can then facilitate discussion on means to improve services. For example, noting 

that one outreach team dramatically increased its number of counseling encounters may 

encourage workers in the field to share an innovative outreach technique. It also might reveal 

that the team is counting very brief contacts as counseling, which would allow the supervisor to 

give feedback to staff about ways to deepen their discussions with survivors. It is critical that 

results are shared in a climate that is supportive and curious (“What might the data be telling 

us?”), not rigid and punitive. 

How does it work?

Program evaluation or program monitoring is much more likely to be useful and meaningful if it 

is grounded in an understanding of how a program operates: what resources it has, what it does, 

what it produces, and what societal benefits it is trying to achieve. This understanding is often 

termed a “program theory” or “logic model.” A program logic model typically includes inputs, 

activities, outputs, and outcomes, as detailed below. The nature of the inputs, activities, outputs, 

and outcomes, however, may depend upon characteristics of the disaster (such as its type and 

severity) and characteristics of the community (such as how closely together people live, or 

population density, and income and wealth in the area). Figure 1 provides a sample logic model 

of a CCP.
The Fundamentals of Data Collection and Evaluation (Version 4.2) Section 1 

2022 Guide for Evaluating Your CCP (Version 4.2) | 4

What are inputs?

Figure 1. Sample CCP Logic Model

Inputs are the resources available to the program to use in achieving its goals. Some inputs are 

tangible resources: funding, program staff, office space, office supplies and other consumables, 

transportation, etc. Others are less concrete but equally important: the skills and expertise of 

program staff, the relationships between staff and local community leaders, and the delineation 

of responsibilities among the different agencies involved. Lack of one or more of these needed 

contributions can greatly limit an organization’s ability to deliver services.

What are activities?

Activities are the means used to bring about program objectives. Different CCPs aim for 

different mixes of public education and crisis counseling services according to what program 

leaders believe is best for their community. Advanced training, another activity, helps crisis 

counselors do their jobs more effectively, especially when the disaster is especially severe or 

complicated. CCPs vary in their activities in response to diversity in the population and in 

identifying and referring individuals with more severe mental health or substance use needs.

What are outputs and outcomes, and how are they different?

Outputs are the measurable units of product from a program’s activities. Outcomes, on the other 

hand, are the societal benefits. While outputs assess “how much” was done, outcomes focus on 

“how much good” was done. An output might be how many children were served by a particular 

CCP in a given month, and an associated outcome might be an improvement in mental health 

among children in the area the CCP is serving. Outcomes can be considered in the short 

(immediate), intermediate, and long term. Immediate outcomes are those that can be observed 

directly after completing an activity. Intermediate outcomes are those that derive from immediate 

outcomes, such as alleviation of symptoms; reduced substance use; or improved functioning in 

family, community, professional, and other social roles. Long-term outcomes may include 

community cohesion, increased disaster preparedness, or community resilience in dealing with 

subsequent crises.
The Fundamentals of Data Collection and Evaluation (Version 4.2) Section 1 

2022 Guide for Evaluating Your CCP (Version 4.2) | 5

Evaluations often focus on the outputs of the service delivery process, such as the number of 

individuals who received crisis counseling and number of educational presentations made. In 

some cases, evaluations conclude with outputs, which are used as a proxy for outcomes. In other 

cases, outcomes need to be measured directly in order to assess whether services are truly having 

an impact.

Evaluation for CCPs

What are the goals?

Evaluation for a CCP answers questions about three critical areas of performance: (1) program 

reach, (2) program quality, and (3) program consistency. 

What is program reach?

How many people in the community were served by the CCP and what were their 

characteristics? The CCP aims to deliver services to large numbers of residents who are diverse 

in age, ethnicity, and needs. This aspect of the evaluation makes use of data from all data 

collection forms that are routinely completed by counselors. The question is not only about the 

actual numbers of people served but also about how well these numbers align with the 

distribution of the state’s population in the affected areas. 

What is program quality?

Were the services perceived by disaster survivors and providers to be appropriate and beneficial? 

To assess service quality, consumer feedback is essential. This aspect of the evaluation relies 

upon brief anonymous surveys (i.e., Participant Feedback Surveys) that capture service 

recipients’ perceptions of service quality and personal improvements in functioning. (More detail 

about how this is done will follow.) Disaster survivor feedback can also help program managers 

reach a better understanding of factors that influence recipients’ perceptions of service quality.

Because a different perspective on service quality can be obtained from service providers, CCP 

evaluation also includes a survey for crisis counselors and their supervisors (the Service Provider 

Feedback Form). 

What is program consistency?

Many CCPs involve multiple jurisdictions such as counties, parishes, or townships, as well as 

collaborative arrangements with provider organizations. Did these areas or providers vary in 

performance (reach and quality), and can this be explained by differences between them in 

population and experiential characteristics? If CCP evaluation shows that some providers attain 

higher reach (in proportion to population), recipient satisfaction, or provider satisfaction, then 

this will inform project managers that further study or corrective action is needed. Advanced 

analyses can also contribute to knowledge about characteristics of settings (such as low 

population density) and events that make it more challenging to implement the CCP. 
The Fundamentals of Data Collection and Evaluation (Version 4.2) Section 1 

2022 Guide for Evaluating Your CCP (Version 4.2) | 6

Data Management and Analysis

How are the forms produced?

The CCP data forms come in PDF or via the CCP mobile app. The PDF versions of the forms 

can be photocopied and used immediately, as well as throughout the life of the program. It is a 

violation of OMB policies to change the forms in any way. 

All PDF versions of the forms are intended for duplex (two-sided) printing. The three forms that 

are used most often involve collection of information and have instructions on the last page or 

pages of the form (Individual/Family Crisis Counseling Services Encounter Log, Group 

Encounter Log, and Weekly Tally Sheet). The instructions are part of the OMB-approved form 

and should always be printed. When possible, the forms should be printed on both sides of one 

sheet of paper. The Child/Youth Assessment and Referral Tool and the Adult Assessment and 

Referral Tool both have general instructions on the last page of the form. The Child/Youth 

Assessment and Referral Tool has important questions and referral information on pages 2 and 3. 

Some programs may choose to use the CCP mobile app. Crisis counselors are able to fill out all 

five forms on their program-issued mobile device and upload them directly to the ODCES. 

How are the forms filled in?

The PDF versions of the tools have been designed to require little more than numbers or X’s in 

boxes that correspond to the selected answer. The marks should be made firmly and neatly with a 

black pen. Many data validation fields have been added to questions on the mobile app forms. 

Many fields within the mobile app simply require users to select one of the available options.

Where do completed tools go?

When using the PDF versions of the data collection forms, crisis counselors turn in completed 

Individual/Family Crisis Counseling Services Encounter Logs, Group Encounter Logs, Weekly 

Tally Sheets, Adult Assessment and Referral Tools, and Child/Youth Assessment and Referral 

Tools to their supervisors or enter their data directly into the data system. Mobile app users 

simply need to upload completed forms to ODCES at the end of each day. Participants can either 

mail their Participant Feedback Surveys to the state’s evaluation coordinator or complete the 

survey online via a weblink. Crisis counselors either complete the Service Provider Feedback 

Form online or on paper, and then they mail paper forms to their state leadership. 

How often are they submitted?

Programs can decide whether completed Individual/Family Crisis Counseling Services 

Encounter Logs and Group Encounter Logs are to be submitted daily or weekly to the data entry 

staff. In large programs serving many people, it is better to do this daily to avoid a backlog of 

work. Weekly Tally Sheets are submitted by crisis counselors to their supervisors on a weekly 

basis. Forms collected via the CCP mobile app are saved locally on the device and uploaded 

daily or more frequently, depending on program policy.
The Fundamentals of Data Collection and Evaluation (Version 4.2) Section 1 

2022 Guide for Evaluating Your CCP (Version 4.2) | 7

What do supervisors do?

Supervisors check the completeness of submitted forms and note errors. When crisis counselors 

fail to follow the instructions, they should be shown what to do in the future. It is likely that the 

most time-consuming part of the supervisor’s monitoring work is dealing with counselor errors, 

so good counselor training is important. Once forms have been entered or uploaded, team leaders 

are responsible for reviewing and approving those in the pending queue. 

How are the data entered?

The data are entered through the CCP ODCES website at www.ccpdata.org/CCP2Field or the 

CCP mobile app at www.ccpdata.org/ccpmobile. For technical assistance with CCP data forms 

or data entry via the online system, please contact SAMHSA DTAC at 800-308-3515 or 

dtac@iqsolutions.com. 

What reports are required?

Two quarterly progress reports and one final report for the nine-month RSP grant must be 

submitted to FEMA and SAMHSA CMHS project officers and a SAMHSA DTAC technical

assistance specialist. Quarterly reports are due 30 days after the end of the three-month reporting 

period. The final program report is due to the FEMA and CMHS project officers within 120 days 

of the final day of program services. Evaluation data are required in the quarterly reports and the

final program report. Please refer to the Notice of Award letter and the terms and conditions of 

your grant award for guidance on the evaluation reporting requirements. 

Who is responsible for this work?

Programs are responsible for entering the data from Individual/Family Crisis Counseling

Services Encounter Logs, Group Encounter Logs, Weekly Tally Sheets, Participant Feedback

Surveys, and Adult and Child/Youth Assessment and Referral Tools, whether the program uses 

PDF or mobile versions of the forms.

The Service Provider Feedback Form is administered online. SAMHSA DTAC provides the 

CCP program manager with a link to the Service Provider Feedback Form. The program 

manager then distributes the link to the crisis counselors and team leaders during a designated 

time period. Data entered into this online survey by crisis counselors and team leaders are then 

automatically uploaded into ODCES. 

How are the data analyzed?

The CCP ODCES website has reporting functions that correspond to results required on the RSP 

quarterly reports. The CCP ODCES website also allows for downloads of the data files so that 

they may be exported into statistical software, such as SPSS and SAS, for additional analyses as

warranted.

Is this evaluation enough?

This evaluation plan may or may not be enough, depending upon the size and complexity of the 

program. Good evaluators assist program planners and managers in identifying other information

needs specific to their locations that are not part of the evaluation required by the sponsor. For 

example, as the program unfolds, innovative approaches may emerge that warrant special
The Fundamentals of Data Collection and Evaluation (Version 4.2) Section 1

2022 Guide for Evaluating Your CCP (Version 4.2) | 8

evaluation procedures to capture outcomes as well as outputs. There could be occasions where 

the program needs qualitative data on select, focused issues. In other words, the plan described 

here provides basic information on service reach, quality, and consistency, but it does not 

preclude the possibility of states’ adding other components to their own program evaluations.