Using Paradata to Monitor Survey Production, Cost, and Quality 

within an Adaptive Total Design Framework 

David Cunningham Hunter, Susan Mitchell, Lisa Carley-Baxter, Michael Keating

RTI International 

dbc@rti.org, smitchell@rti.org, lcbaxter@rti.org, mkeating@rti.org 

 

Abstract 

Adaptive Total Design (ATD) aims to improve the use of paradata for monitoring survey production, cost, and 

quality. ATD: 

identifies major contributors to total survey error; 

provides means for monitoring costs and errors during implementation; 

allocates resources to maximally control errors and costs; and 

monitors results of embedded experiments to inform later phases or survey rounds. 

The paradata used include record of calls data, cost and productivity data, sample frame and sample information, 

data quality measures, interviewer characteristics, and interviewer observations. 

A multidisciplinary team within RTI has created an ATD approach that collects a universal set of paradata across 

surveys. This approach allows for conservation of costs by minimizing the need for custom programming and allows 

for comparability across surveys. An ATD “dashboard” presents the paradata for review and evaluation by project 

staff. RTI chose Google Apps for Business, a browser-based cloud computing suite of communication and 

collaboration products, for developing and sharing charts that facilitate ATD while data collection is underway. 

This paper provides an overview of RTI’s ATD approach, describes criteria for choosing software for use in 

developing and presenting ATD charts, gives examples of charts RTI has developed, and outlines practical

considerations for developing and implementing ATD.

Introduction 

The increasing reluctance of sample persons to participate in surveys and growing problems related to contacting 

households and gaining cooperation pose challenges to researchers’ management of data collection quality, 

timeliness, and costs. Moreover, actual field conditions are difficult to predict so that the design of a survey cannot 

always be set optimally prior to the start of data collection. Even more generally, findings from one study may not 

lead to optimal design decisions on another study due to the uniqueness of the essential survey conditions. 

Responsive survey design addresses these problems by altering the survey design or data collection procedures 

while data collection is underway in order to improve cost efficiency and minimize survey error (Groves & 

Heeringa, 2006). Some large national surveys are already implementing responsive designs to address various 

sources of error – for example the National Survey of Family Growth is implementing dashboards to monitor 

nonresponse and cost (Groves et al, 2008). 

Responsive designs are often monitored through the use of survey responses and paradata. Paradata are all of the 

data variables that are collected during survey administration. They can range from simple summary measures like 

response rates to something as specific as the elapsed time between call attempts. A uniform definition of paradata 

does not exist. For our Adaptive Total Design (ATD) approach, we adopted the most inclusive definition that 

includes information from the interviewers and external sources, as well as information collected by the survey 

systems. 

A key aspect of responsive designs is routine (e.g., daily, weekly) monitoring of key indicators. Such indicators are 

informative about the data collection process and the properties of key survey estimates; they also help in decisions 

related to the cost-error trade-off of different approaches. Examples of indicators to monitor include response rates 

1 
by subgroups, calls per case, mode of interview (in multi-mode surveys), and interviewer hours per completed 

interview. This monitoring, along with a review of survey-specific measures, help survey managers determine if and 

when they need to implement a change in their survey design or approach. These decisions may be aided by a 

graphical user interface that can plot various data over the duration of the data collection period that is either in real 

time or updated daily. 

 

ATD is RTI’s approach to responsive design in surveys. RTI developed ATD to achieve the highest level of data 

accuracy by minimizing the total survey error of the survey estimates (Biemer, Peytchev, & Peytcheva, 2010). ATD 

is different from other approaches to responsive design because it focuses on total survey error, not just cost, 

production, and response rates. Total survey error comprises six different error sources: specification, nonresponse, 

the sampling frame, the measurement process, data processing, and sampling (Biemer & Lyberg, 2003). This paper 

focuses on ATD tools for reducing nonresponse and measurement error, as well as monitoring cost and production. 

Further ATD work will focus on reducing the other sources of error and to include model-based metrics that predict 

outcomes. 

Using ATD, metrics associated with survey errors, production, and costs are graphed and displayed on dashboards 

that are created and reviewed at regular intervals while data collection is underway. This approach provides a 

snapshot of the progress and quality of data collection that can reveal deviations from expectations, protocols, and 

prior rounds in a convenient and accessible way. When survey errors or cost problems are identified, survey 

managers intervene to reallocate resources or address problems before they become entrenched. ATD is also used to 

monitor the results of any embedded experiments in data collection (such as an experiment with different incentive 

amounts or types) to inform the best approach for later waves. 

Background 

RTI recently funded an initiative to improve the use of paradata for monitoring survey quality, production, and cost. 

This 2-year initiative involved a multidisciplinary team of statisticians, survey researchers, web designers, and 

programmers. Year 1 focused on research and planning. Year 2 focused on implementation. 

Year 1 research included developing an understanding of responsive design principles and the current state of the 

field. We gathered information about RTI’s current practice of responsive design, including data elements and 

system capabilities. We then moved on to planning, which involved identifying paradata that are needed to inform 

ATD. An important theme of our planning was the conceptualization of a system that could be used across projects, 

regardless of mode or study population. 

In the implementation phase, we began by developing a list of the charts we wanted and then mapped paradata to 

specific charts. We then evaluated several software programs for developing the charts that make up the ATD 

dashboard. We evaluated 12 different software packages based on information from product websites, experience 

with the products, and, in selected cases, the installation of trial packages for testing. We evaluated the packages on 

cost/benefit value, customizability possibilities without programmer support, ability of charts to be dynamic, size of 

data files required when producing charts, and data processing time. We narrowed our selection to one software 

suite: Google Apps for Business. The primary reasons for our choice were the scalability, low cost, portability to 

mobile devices, portability to devices using different operating systems (e.g., Microsoft Windows and Apple iOS), 

user friendliness of the applications, and high levels of possible customization. 

Google Apps for Business 

In February 2011, the White House Chief Information Officer instituted a “Cloud First” policy to federal agencies, 

noting that cloud computing can address inefficiencies, improve resource utilization, and increase service 

responsiveness (Kundra, 2011). Google is a major player in this field and will probably continue to gain in market 

share. Google Apps for Business is still relatively new, but its adoption is gaining traction among businesses, state 

governments, and the federal government. Examples include the General Services Administration and the National 

2 
Oceanic and Atmospheric Administration. Currently, more than 4 

3 

million businesses use Google Apps for 

Business.1

Google Apps for Business is a browser-based cloud computing suite of communication and collaboration products. 

Similar to Microsoft Office, Google Apps for Business includes an e-mail client, word processor, spreadsheet, 

presenter, and drawing tool for organizational charts and flowcharts. Google Apps for Business is a subscriptionbased service that costs $50 per user per year.2

 It includes a variety of additional collaboration tools, including chat 

(text, voice, and video), secure online video distribution, the ability to create code-free websites, and Google Groups 

message boards. The integration of these collaboration tools with traditional productivity tools creates a very 

powerful platform for users. The user experience of the suite is very similar to the consumer versions of Google 

Apps (e.g., Gmail, Google Docs, and Google Sites), but Google Apps for Business also includes very customizable 

administrative controls and enhanced data security. 

There are several advantages to using Google Apps for Business to assist in managing survey data collection:

 No software installation is required. A user needs only an Internet connection, a browser, and login 

credentials. 

 Administrators have micro-level control over who can share, edit, and view project documents. Different 

pages can be set up based on each individual user’s permissions. 

 Google Apps for Business provides a universal distribution point for respondent data and paradata, which

are easily portable to smartphones and tablet computers, making it appealing to multi-site studies, field 

studies, or other studies with dispersed personnel. It is also available across operating system platforms, 

giving agencies and survey managers flexibility in the devices they choose to use. 

 Google Apps for Business includes one-to-one and group communications software, which can be used to 

communicate among any authorized user or set of users. 

 Google Apps for Business is easily scalable, so small and large projects can take advantage of all of the 

features at the same price of $50 per user per year. 

 The latest security and software updates are made automatically without burdening users and without the 

risk of a user not updating security or anti-virus patches. 

 If a data collector or field manager loses or breaks his or her device, no data in the Google Cloud are lost. 

Because the data reside in the cloud, the data will still be there when the new device is set up. Furthermore, 

because data are not stored on devices, the risk of sensitive or confidential data being accessed by a nonauthorized user with access to an authorized device is very minimal. 

 There are very sophisticated security controls that allow users to take advantage of Google Apps for 

Business features while mitigating the risk of data security breaches. 

 Google Apps for Business applications are user friendly, similar to common software already used by most 

government and research contracting staff. Therefore, there is little need for training and support. 

Because cloud data do not reside on an organization’s internal servers, there may be concern that these data will be 

less secure in the cloud. Google Apps for Business offers security qualifications and has measures in place to protect 

data security, including the following: 

 Federal Information Security Management Act of 2002 (FISMA) Certified.3 Google Apps for Business 

is FISMA certified by the General Services Administration and has the authority to operate at the moderate 

level. 

 SSAE 16 Type 2 Audit.4 An independent third-party auditor reviewed and verified that Google Apps for 

Business meets industry standard security controls, including logical security, privacy, data center physical 

security, incident management and availability, change management, and organization and administration. 

 

1

For a list of examples of businesses and governments that use Google Apps for Business, please visit the following 

link: http://www.google.com/apps/intl/en/customers/index.html

2

 Source: http://www.google.com/apps/intl/en/business/smb/index.html#utm_campaign=en&utm_source=en-ha-naus-sk&utm_medium=ha&utm_term=%2Bapps%20for%20%2Bbusiness 

3More information on Google Apps for Business’s data security can be found at the following link: 

http://www.google.com/apps/intl/en/business/infrastructure_security.html 

4

 Source: http://www.ssae-16.com/ 
 Certified by the Cloud Security Alliance.5

 Google Apps for Business is certified by the Cloud Security 

Alliance, an organization that promotes best practices for providing security assurance within cloud 

computing. 

 Two-Factor Authentication. This entails an additional layer of protection when a user logs in to Google 

Apps for Business. Upon entering the correct username and password, the user is prompted for a 

verification code. This code is randomly generated by Google and changes every 60 seconds. Users can 

access this code using an application on their smartphone, a text message on their phone, or voice message 

on their phone.

 Walled Garden. The walled garden approach in Google Apps for Business allows users to interact only 

with other users in the same domain. Within a research organization, a domain could be a project. The 

system can be set up so that users can be allowed to e-mail only other people on the same project. This 

gives further assurances that individuals do not accidentally forward project-sensitive materials to 

unauthorized e-mail addresses. 

 

 

 

4 

Google Docs is part of the Google Apps for Business suite. We have chosen one tool within Google Docs, Google 

Charts, to apply ATD concepts to data collection management. Google Charts can be shared through Google Sites, a 

code-free way to create internal project websites. This gives users with no HTML programming experience the 

ability to create sophisticated-looking websites without writing one line of code. In our experience, a wide variety of 

charts covering quality, production, and cost can be developed in this manner. Beyond the codeless options, 

developers can take advantage of Google Code to further customize websites. Sites can be used to create restrictedaccess project pages. These web pages can display project-level statistics and goals in a simple format. The project 

pages can also be used for communications from survey managers to data collectors. If an individual prefers to use 

standard Microsoft Office products, they can still upload documents to Google Apps for Business manually or use 

Google Cloud Connect for Microsoft Office to automatically put the documents in the cloud. In addition, Google’s 

Secure Data Connector can push metadata from data servers to Google Apps for Business applications. 

Example Dashboard and Charts using Google Apps for Business 

ATD links questionnaire data with paradata in a common database and arrays the resulting information under five 

headings: (1) Production and Cost, (2) Timing, (3) Record of Calls, (4) Sample Status, and (5) Data Quality. The 

system is web-based, allowing access by multiple users. A dashboard provides the ability to quickly choose the 

metric in which the user is interested and then drill down very quickly to a specific process, unit, or geographic area. 

Charts can be summarized by elements such as sample geography so that progress can be compared and resources 

can be redirected to areas and/or groups where production is lagging. Exhibit 1 presents a sample dashboard 

showing what the user might see when he/she logs in to a project website to access reports. 

 

5

 Source: https://cloudsecurityalliance.org/about/
Exhibit 1. Dashboard Prototype 

5 

By default in the ATD model, each chart on the dashboard will show the user a study-level summary for the metric. 

If the user sees something of interest or concern, he/she can click on the chart to drill down to a lower level. The 

user can create charts down to the case level, assuming that he/she has the paradata at that level. The user can also 

choose which chart he/she wants to show up on the dashboard page as the default in any given category. 

On the left side of the dashboard screen shown in Exhibit 1, the user sees the list of categories into which the charts 

are divided. These categorizations are fully customizable. If the user clicks on one of the categories, he/she will see 

a list of charts under that topic. The user can then click on any of the links to access the respective chart. 

Alternatively, the user can access a chart by clicking on the image of a chart on the right-hand side of the screen. 

Exhibit 2 is an example of what the user would see after clicking on the Production and Cost chart image. The 

number of completes are represented on the Y axis, dates are represented on the X axis, the blue line represents the 

number of cases completed each day of data collection, and the red line represents the 7-day moving average for 

production. This chart enables the user to see how daily production affects the average. If the average production is

not as high as desired, the user can easily identify which days of the week are more productive and subsequently 

staff more data collectors on those days. 
Exhibit 2. Number of Completes with 7-day Moving Average 

6 

If the user clicks on the Timing chart image, he/she would see Exhibit 3, a line chart that shows survey timing data 

over time. The Y axis represents survey length in number of minutes, the X axis represents day of data collection, 

the blue line represents the average survey length in minutes across all data collectors, and the red line represents 

interviewer Cindy’s average survey length. Cindy’s low average might be a good sign or a bad sign. It might 

indicate short-cutting or some other data quality concern, or perhaps Cindy is just very efficient. The survey 

manager can drill down to Cindy’s section timing and review Cindy’s record of calls to determine if there is reason 

to be concerned. 

Exhibit 3. Overall Survey Length and Outlier Interviewers 
Within the Record of Calls category is the chart shown in Exhibit 4, a pie chart that breaks down cases by broad 

status. Each status is represented by a different color. By placing the cursor over a pie section, the user can see the 

count of cases as well as the percentage of cases in that category. In this example, we have 885 completes, which 

represent 8.7% of all cases. This chart can help the user quickly identify types of cases that might need more 

attention (e.g., noncontacts). 

7 

Exhibit 4. Percent of Cases by Broad Status 

Exhibit 5 presents an example of a Sample Status chart, a Geomap that shows response rates by state. Dark red 

represents the lowest rates, and dark green represents the highest rates. For a national survey, this chart could be 

used to easily identify a state where attention might be needed on increasing response rates. A user could then drill 

down to the response rate by interviewer, demographic area, or subpopulation to determine what cases are 

contributing to low response rates. Clusters of low-responding states can also be observed, indicating potential 

problems within regions that may be related to staffing or resources. By moving the cursor over the state, the user 

can display the actual response rate and the percentage improvement in response from the prior week and month. 
Exhibit 5. Response Rates by State 

 

8 

Exhibit 6 is an example of a stacked bar chart showing Data Quality. The Y axis represents the proportion of item 

nonresponse to number of days worked, and the bars represent item nonresponse by interviewer. Each key 

questionnaire item is represented by a different color. If the user rolls his/her mouse over a bar, he/she will see the 

interviewer ID and the item nonresponse rate pop up. This chart can be used to easily identify interviewers with 

higher than average nonresponse rates. 
Exhibit 6. Item Nonresponse by Interviewer 

9 

Another type of ATD chart is an annotated timeline chart. Exhibit 7 is an example that shows production over time. 

The Y axis represents production, the X axis represents the date, the blue line represents the number of completes 

each day, and the red line represents the 7-day moving average. The letters represent interventions or events that 

might affect production. Survey managers can drill down to see the number of interviews completed by mode and 

further still to observe how interview completion responded to data collection events. Displayed together 

graphically, this chart tells a story about how productivity changes over time and in response to interventions. 
Exhibit 7. Annotated Timeline of Production by Day 

10 

As these examples show, ATD offers the advantage of continuous data collection monitoring in a user-friendly 

format so that problems can be identified early and appropriate remedial action can be taken. At its core, ATD relies 

on an integrated system that combines paradata and survey data in a consistent format so that charts can be 

preprogrammed and used across multiple surveys without the need for expensive customization. 

Considerations When Implementing an ATD Approach 

There are several considerations to take into account when implementing an ATD approach with a graphical 

interface: 

Make sure that paradata are easy to set up for each study, standardized across projects, linked or at least 

linkable, and easily accessed and presented to the primary audience. 

Make decisions about which approach makes sense theoretically as well as practically (e.g., system 

limitations, the problem of too much data). 

Remain flexible because the development process is iterative. 

Begin with a simple, generic system that is economical to set up and use. 

Identify financial resources needed to implement one’s vision. 

Identify a realistic timeline for implementation. 

Get buy-in from stakeholders, including agency directors, clients, contractors, data collection managers, 

and survey organization senior management. 

Develop ways to get staff buy-in on the new approach so the charts will be used appropriately to minimize 

survey error. 

Develop appropriate training for staff to develop and use charts so that implementation is easy and charts 

are fully used. 

The first step above is a critical underpinning that may require system redesigning; otherwise, duplication of effort 

would likely occur while producing standard charts and educating users, which would offset some of the efficiencies 

gained by using standardized charts. To avoid wasting effort, it is advisable not to create a chart just because it is 

possible to do so or because it looks flashy. The user should purposefully plan which types of charts he/she wants 
based on which paradata are available, what the needs of the users are, and which data provide actionable 

information. 

11 

Next Steps 

The ATD initiative at RTI is still underway. We are currently finalizing standard charts that can be used across field 

studies and telephone studies. As presently configured, the ATD system is designed to support household and list 

sample surveys in computer-assisted telephone interviewing (CATI) or computer-assisted personal interviewing 

(CAPI) modes. Expansion to other modes and survey types is planned for the future. 

We are also in the process of expanding our charts in order to apply additional elements of the ATD approach, 

including reducing other elements of total survey error such as the measurement process and data processing. We 

are also planning to develop predictive models of survey outcomes and to explore Google Apps for Business for 

communications. For example, senior field managers can attach comments to a chart asking for a data collection 

supervisor to examine a potential problem and report back on what he/she finds. We are also exploring the use of 

Google e-mail with field management personnel. 

Finally, we will develop an education program for chart developers and end users (who are often the same people) 

on setting up and using the charts. This education program will include an overview of ATD and how it enhances 

current practices, identification of the relevance of the standard charts to a given project, a protocol for monitoring 

outcomes, and appropriate interventions and actions when potential problems are discovered. 

References 

Biemer, P. P., & Lyberg, L. E. (2003). Introduction to survey quality. Wiley Series in Survey Methodology. 

Hoboken, NJ: John Wiley & Sons, Inc. 

Biemer, P., Peytchev, A., & Peytcheva, E. (2010). A total survey quality approach. Presentation made at RTI 

International. Research Triangle Park, NC. 

Groves, R., & Heeringa, S. (2006). Responsive design for household surveys: Tools for actively controlling survey 

errors and costs. Journal of the Royal Statistical Society Series A: Statistics in Society, 169(3), 439–457. 

Groves, R. M., Kirgis, N., Pacer, J., Peytcheva, E., Lepkowski, J., & Wagner, J. (2008). Quality profile 2008: A set 

of potential quality indicators in the National Survey of Family Growth. Prepared for the National Center for Health 

Statistics. 

Kundra, V. (2011). Federal cloud computing strategy. Retrieved March 16, 2012 from 

http://www.cio.gov/documents/Federal-Cloud-Computing-Strategy.pdf.