1

Practitioners

GUIDE

PRINCETON, NJ - ANN ARBOR, MI - CAMBRIDGE, MA - CHICAGO, IL - OAKLAND, CA - SEATTLE, WA 

TUCSON, AZ - WASHINGTON, DC - WOODLAWN, MD

EDI: HIGH WYCOMBE, UK - BUKOBA, TANZANIA - DAR ES SALAAM, TANZANIA

SEPTEMBER 2019

Data are multipurpose and offer many opportunities to organizations. Many organizations that 

help improve people’s lives collect valuable information on participants in their programs and on 

how an intervention is being implemented, but they often use the data only for monitoring the 

intervention and reporting the results to funders. However, organizations can also use data to 

provide a thorough understanding of how their intervention operates, to improve upon it, and to 

show how it affects the participants they serve. This more robust application of data can make an 

intervention more efficient, generate strong evidence for its effectiveness, and create additional 

opportunities for funding.1,2 Several resources in the Further Reading section describe how the 

use of data can strengthen interventions. Maximizing data for these particular uses requires a 

system—a management information system or some other computerized process for collecting, 

analyzing, and reporting data—to standardize the data items and the methods of collecting them.

This guide highlights issues for organizations to consider when using, collecting, and managing their data. Based on the literature and site visits with selected Corporation for National and 

Community Service (CNCS) grantees (see the “About the Series” box for more information), we 

describe how organizations use data to determine the ways in which an intervention being implemented is effective and the ways it is not, to help organizations improve program implementation, 

and to provide evidence to funders about effectiveness. Furthermore, this guide (1) highlights factors that organizations should consider when assessing whether an existing data system is meeting 

organizational needs, and (2) provides guidance for obtaining a system or modifying an existing 

one to meet those needs. This will help organizations enhance their data collection capabilities and 

make the most of the valuable data they collect.

Making the Most of Data

Daniel Friend and Karen Needels

WAYS DATA CAN HELP 

ORGANIZATIONS

Organizations can use their data in a wide range 

of activities to help improve programs and 

provide evidence of their effectiveness. Based on 

information gathered during site visits to CNCS 

grantees, we observed that organizations typically 

collect data to demonstrate they are meeting the 

performance requirements set by their funders. In 

addition, organizations might take on the increased 

burden of data collection to achieve their goals, but 

realize few benefits from it. These organizations 

may be losing an opportunity to capitalize on the 

data they have worked hard to collect and use them 

for program improvement. Although unlocking this potential can be challenging, as we will 

describe, the potential benefits may be great. 

Based on conversations with practitioners during site 

visits, we list several data usage activities that organization personnel can consider implementing. Some of 

them may be related to funding requirements, and 

others can be important when organizations are considering scaling their interventions. For example, when 

expanding an intervention into new locations, it may 

be especially important to use data to monitor implementation and to identify ways to improve processes 

MATHEMATICA-MPR.COM
2

for providing services in the new locations. Alternatively, it may be very important for organizations to 

conduct research about the intervention’s effectiveness 

to determine whether the expansion is successful at 

helping participants achieve good outcomes. 

• Performance monitoring and reporting.3 As we learned through site visits, and as 

discussed in some of the resources in the Further 

Reading section, this was the most common use 

of data. Personnel can monitor organization milestones or program outputs (enrollments, intervention completion, the numbers and types of services 

provided, etc.), participant outcomes, or costs. These 

could be reported internally to organization personnel (for example, at weekly personnel meetings 

or in regular reports to organizational leadership) or 

externally to broader audiences and funders. 

• Continuous quality improvement 

(CQI).4 This is an ongoing process or cycle that 

uses data to evaluate the strengths and challenges 

of an intervention and to create and test ideas for 

its improvement (the Further Reading section 

contains an introduction to CQI). Organization 

personnel can use the performance monitoring and 

reporting data, as well as other implementation 

data (such as interviews with personnel), to inform 

the CQI process. Organizations may consider 

tracking additional data to monitor the implementation and success of their ideas about improvement. For example, a data system may have to 

include a decision log or keep track of personnel 

satisfaction results to monitor whether program 

improvements are well received by the personnel.

• Research and evidence of effectiveness. In addition to efforts related to program 

implementation, organizations can use data for 

a variety of research purposes, such as assessing 

participant outcomes, to expand the evidence 

about the effectiveness of their programs. These 

research efforts can range from (1) those that 

are fairly simple to conduct, such as comparing 

measures of how participants are faring after 

their involvement with the program to how they 

were doing beforehand; to (2) those that are 

more complex and provide persuasive evidence 

about the effectiveness of services, such as 

through a randomized controlled trial like those 

used in the medical field to test the effectiveness 

of drugs. To conduct this research, programs can 

implement simple surveys (one upon enrolling 

into the program and one after completing it) 

or work with a local evaluator, who can help 

organizations identify a comparison group or 

conduct rigorous randomized designs. Whether 

organizations conduct research themselves or 

work with a local evaluator, a data system can 

be used to track survey completion or responses 

and even to conduct random assignment or 

keep track of a comparison group. 

CHOOSING (OR IMPROVING) 

A DATA SYSTEM

Organization leaders may discover that their 

existing data system does not have the functionality 

required to collect the data needed to support their 

intervention improvement or research efforts. In 

this instance, leadership needs to decide whether 

it is possible to find a new system or enhance their 

current one. The Further Reading section contains 

a resource on developing and enhancing data 

systems.5

 Based on prior experience and conversations with organization leadership and practitioners, when choosing or upgrading a data system, 

organizations should aim to have one that: 

Meets current and future needs. When 

evaluating a current or potential data system, organization leaders must consider how it meets their current data collection needs. In addition to maximizing 

the types of data the organization collects, a data 

system has to be able to accommodate future needs 

and how they may evolve over time. Organization 

leadership may need to consider how they may scale 

the delivery of the intervention and whether the data 

system can be easily adapted to capture new data, 

support use at a larger scale (for example, more sites 

and users), or be easily replicated. 

Maximizes integration and minimizes 

redundancy. When assessing new systems or 

enhancements, organization leaders might consider prioritizing systems that integrate with existing systems. 

This integration could take many forms, such as easily 

importing data into an existing system. This could 

remove redundancies, such as double-entry or reliance 

on additional paper-based forms, reduce burden on the 

personnel, and increase the use of the system.

Is easy to access. Regardless of whether an 

organization is implementing a new system or 

enhancing a current one, the system should allow 

personnel to enter the data easily and access them 

readily. Ensuring that the system is user-friendly for 

the persons entering the data will help them consistently and accurately capture the data they need. This 

ease of use fosters trust in the data quality, which 

encourages continued use. Further, the system must 

be user-friendly for the personnel accessing and 

WHAT IS A DATA 

SYSTEM?

A data system is generally database-driven 

computer software 

designed to capture 

or collect pertinent 

program information. 

It could be a formal 

management information system, a case 

management system, 

or a custom database 

(for example, a system 

created in a program 

such as Microsoft 

Access). This system 

should be able to collect data on program 

milestones (such as 

enrollment), program 

implementation 

(information on how 

personnel deliver the 

program), and costs. 

This system could 

also include research 

elements, like tracking 

consent to participate 

or pre- and post-intervention assessments.

Data systems should:

• Be scalable and 

dynamic so that they 

can grow and adapt 

as the program grows 

and changes 

• Avoid redundancy and 

integrate well with 

any current systems 

and processes

• Provide an easy way 

to access the data

• Provide a way to 

interpret the data, 

including well-defined 

data points and a data 

dictionary
3

interpreting the data. This could take the form of 

system-generated, easily interpretable reports, such 

as tables or graphs, or data that are easily exportable 

in an analysis-ready format (for example, Microsoft 

Excel). If the data are not easily accessible, additional 

organization personnel may be required to retrieve, 

analyze, and interpret the data contained within the 

system; such added requirements often detract from 

system use. As organization leaders think about ease 

of access, they will also need to consider data security needs, especially when working with partners in 

delivering or assessing the intervention.

Reflects organizational capacity and 

resources. Organization leaders often face a 

choice between developing their own system (homegrown) or purchasing from a third party manufacturer (off-the-shelf). It is also possible to purchase 

third-party software and customize it yourself or pay 

the developer to customize it for you. If an organization chooses to develop its own system or do 

in-house customization of an off-the-shelf system, 

its leadership must ensure that they have the appropriate personnel. Organization leaders may want to 

consider starting with an off-the-shelf system until 

they understand all of their data needs to inform the 

development of a home-grown one in the future.

Organizations might need database 

developers and/or programming experts 

to undertake this task. They might also need 

personnel who can create training and support materials, such as user manuals. If personnel lack these 

qualifications, purchasing an off-the-shelf system is 

an option because it might come with the needed 

programming personnel and training/user support. 

However, off-the-shelf systems might not be customizable to the extent of a homegrown system. They also 

might entail additional licensing or usage costs.

WORKFORCE NEEDS 

AND TRAINING

Organization leadership should provide training to 

and monitoring of their frontline personnel, who 

regularly use the data system.6 Some resources in the 

Further Reading section discuss how training is an 

essential implementation factor and helps personnel 

implement the intervention successfully. As noted 

above, organization leaders should consider the 

capabilities of their current personnel when evaluating their data system. Similarly, leaders should also 

consider the qualifications of the personnel who use 

the system day to day. By reviewing the qualifications of current frontline personnel, organization 

leadership can design training and support methods 

to encourage the consistent and accurate use of the 

data system. Based on site visits, we identified several 

ways to support such consistent and accurate use:

Potential Pros and Cons of "Homegrown" 

vs. "Off-the-shelf" Systems

HomegrownPros • Fully customizable 

to match needs

Cons • Needs continuous programming and database support personnel

• Must develop own user 

manuals and trainings

Off-theshelfPros • Previously developed 

and tested product

• Built in user-supports

Cons • Licensing and/or 

customization costs

• Potential lack of 

customizability

• If system is not userfriendly, a remedy 

might be problematic

POTENTIAL 

CHALLENGES WITH 

DATA SYSTEMS

• Lack of integration 

with existing systems

• Redundancy with 

other systems 

or processes

• Questionable 

data quality

• Difficulty accessing 

and interpreting 

the data

Hands-on training. Personnel need formal, standardized training on how to use the data 

system. This training likely should include a mix of lecture, practical examples, and hands-on 

practice within the actual system. This may also include reviewing the data dictionary to ensure 

a common understanding of the data points in the system. The training may also have to be 

repeated to account for personnel turnover. 

Regular monitoring. Once trained, personnel must be supervised and monitored to ensure they 

are using the system consistently and accurately. The system may include user or usage data that 

can help program personnel track who is using the system, when they are using it, and what data 

they are entering. Programs should set up a process to review these usage data regularly and 

provide feedback to personnel.

Additional trainings. Monitoring and supervision may identify personnel who need additional 

training or support on using the system. In these instances, a booster or refresher training or 

more supervised use (for example, job shadowing) may be needed. In addition, as new features 

are added, personnel should receive additional trainings and updated support materials. 
4

FURTHER READING

1 National Implementation Research Network, “Implementation drivers and data systems” 

(https://nirn.fpg.unc.edu/module-1/implementation-drivers/organizational)

2 National Implementation Research Network, “Decision support data systems” 

(https://nirn.fpg.unc.edu/module-2/decision-support-data-system)

3 US Department of Education, “Performance monitoring: Collecting and Using Data to Measure 

Progress, Improve Results” (https://www2.ed.gov/about/inits/ed/implementation-support-unit/

tech-assist/performance-management-collecting-data.pdf )

4 Office of Adolescent Health, “Continuous Quality Improvement” (https://www.hhs.gov/ash/

oah/sites/default/files/cqi-intro.pdf )

5 Mathematica, “Developing and enhancing data systems” (https://www.mathematica-mpr.com/

download-media?MediaItemId=%7B3AB264C7-53C9-4652-A328-8DA5E211E314%7D)

6 National Implementation Research Network, “Implementation drivers: training” 

(https://nirn.fpg.unc.edu/module-2/training)

REDCap, Free, web-based customizable data system: (https://www.project-redcap.org/)

Follow us on:

Mathematica® is a registered trademark of Mathematica Policy Research, Inc.

Scan this QR code 

to visit our website.

FOR MORE INFORMATION

For more information, contact Scott Richman, director of the Scaling Evidence-Based 

Models project, at srichman@mathematica-mpr.com.

ABOUT THE SERIES

The Corporation for National and Community Service (CNCS) supports the scaling of effective 

interventions that it funds. CNCS engaged Mathematica Policy Research to conduct the Scaling 

Evidence-Based Models project (contract GS10F0050L/CNSHQ16F0049). As part of that project, 

Mathematica developed a series of guides to help practitioners assess their scaling efforts critically, 

collect evidence on the effectiveness of their interventions, and increase the likelihood of effective 

scaling of successful interventions.

Each guide provides a succinct, but non-exhaustive, overview of a topic relevant to practitioners. The 

guides are based on implementation science research, as well as information collected during site visits 

conducted by Mathematica staff with three CNCS-funded grantees during fall 2018. As part of the effort, 

Mathematica staff reviewed program documents (such as manuals and grant applications) and conducted 

in-depth interviews with organizational leaders and frontline personnel, partner personnel, AmeriCorps 

members, and other stakeholders. More in-depth information on the topic of this guide can be found 

through references in the Further Reading section.