1 

Riddles Wrapped in Mysteries Inside Enigmas: 

Issues with Getting and Using Administrative Data for 

Impact Evaluations1

Alisha Creel 

Ronald Z. Szoc 

ICF International 

530 Gaither Road, Rockville, MD 20850 

Proceedings of the 2013 Federal Committee on Statistical Methodology (FCSM) Research Conference 

Abstract 

Evaluators turn to administrative data as an objective rather than perceptual source of information that can be used 

to help triangulate evaluation findings along with non-administrative sources. Administrative data are generally 

designed for purposes other than evaluation, however, may only partially meet evaluators’ needs, and bring their 

own challenges. The Office for Juvenile Justice and Delinquency Prevention funded nine programs in eight states to 

reduce drinking among underage Airmen at Air Force bases and their surrounding communities through its

Enforcing Underage Drinking Laws (EUDL) discretionary grant program. Using an environmental approach, the 

program interventions included enforcement aimed at reducing the social availability of alcohol; compliance checks 

of local liquor establishments; impaired driving enforcement; local policy development; community-based 

awareness/media campaigns; and offering alternative activities that did not include drinking alcohol. In this paper, 

we report on our experiences using five administrative data sources to assess the impact of these programs. Grantees 

provided monthly data retrospectively for two years prior to program implementation and prospectively for up to 

two years after the beginning of program implementation. Some of the micro-level issues that we faced included 

data that were incomplete for some months, data stored in forms that prohibited the kinds of aggregation that we 

needed for the purposes of the evaluation, and transferring and summarizing data from repositories that were 

designed for other purposes. The macro-level issues included lack of cooperation from data source gate-keepers, 

lack of access due to concerns with violating HIPAA rules, and the length of time required to collect the 

retrospective data. Additionally, reconciling data from different sources was occasionally problematic. Based on 

these experiences, we offer a number of lessons learned and solutions for dealing with these issues for future 

evaluations that use administrative data. 

Introduction 

Administrative data are appealing to evaluators as they are often readily available and avoid some of the pitfalls of 

self-reported behavior – they can serve as more “objective” measures of certain outcomes. In designing the 

evaluation we report on here, we identified the key data metrics that would let us assess the impact of programmatic 

efforts to reduce underage drinking among Airmen. We wanted "objective behavioral measures" on underage 

drinking and related consequences that were the most robust and the least susceptible to bias, as contrasted with 

“subjective” self-reported alcohol-related behaviors and negative consequences. The most readily available and, we 

thought, the least burdensome measures would come from administrative data from civilian and military sources. 

The reality differed markedly from our initial assumptions in that administrative data was not as readily available as 

we expected. Some of the micro-level issues that we faced included data that were incomplete for some months, data 

stored in paper forms or electronic formats that prohibited the kinds of aggregation that we needed for the purposes 

of the evaluation, and transferring and summarizing data from repositories that were designed for other purposes. 

The macro-level issues included lack of cooperation from data source gate-keepers, lack of access due to gatekeepers’ concerns with violating HIPAA rules or Air Force regulations, and the length of time required to collect the 

retrospective data. Additionally, reconciling data from different sources was occasionally problematic. In spite of all 

 

1

 Paper prepared for presentation at the Federal Committee on Statistical Methodology Annual Meeting in Washington, D.C. November 4-6, 

2013. The opinions and conclusions presented herein are those of the authors and do not necessarily reflect the opinions of the Office of Juvenile 

Justice and Delinquency Prevention, the U.S. Air Force, or of the Department of Defense. 
2 

of these issues, we did manage to use much of the available data to assess the overall impact of the program. We 

learned several lessons along the way that other evaluators may find useful when they attempt to use administrative 

data in impact evaluations. 

In the remainder of this paper, we describe the EUDL program, the Air Force bases and their respective

communities, examine the issues with regard to data acquisition and usefulness for analysis, and offer several 

lessons learned. 

Genesis of the EUDL Demonstration Grants for Communities with Air Force Bases 

Since the fall of 2006, the Office of Juvenile Justice and Delinquency Prevention (OJJDP) within the federal 

Department of Justice has awarded discretionary grants to nine communities in eight states as part of OJJDP’s 

Enforcing Underage Drinking Laws (EUDL) initiative.2 These communities are near United States Air Force

(USAF) bases and the grants fund Air Force / Community partnerships to reduce drinking by underage Airmen. In 

2006, five communities in four states received funding. OJJDP funded two additional states each in the fall of 2009 

and 2012. ICF International (ICF) has served as the external evaluator of the EUDL programs implemented in these 

nine communities. 

Central to each of the funded programs was the establishment and operation of a community coalition made up of 

organizations and personnel from the Air Force community and the civilian community. From the civilian sector, 

agencies in the coalition often included elected officials, local police departments, human service agencies such as 

health and wellness clinics, alcohol beverage control departments, and voluntary organizations such as drunkdriving prevention groups. From the Air Force base (AFB), organizations in the coalition often included the base 

Wing Commander, the Alcohol and Drug Abuse Prevention and Treatment (ADAPT) program, Security Forces, 

housing, Health Awareness Centers (HAWC), and Morale, Welfare and Recreation (MWR) personnel. These 

community coalitions engaged in such prevention activities as community education, training and technical 

assistance, and alternative activities for young military personnel. Of the nine AFB/community sites, six have 

finished their interventions, one is in its final year, and two have recently had their work plans approved and have 

begun their programmatic activities. 

The focus of this paper is on the first seven grantees and the issues surrounding the acquisition of administrative data 

for the evaluation. In the sections below, we first discuss the grant program, then present the overall evaluation 

design and discuss the data requirements and the sources of administrative data. Taking a case study approach, we 

then examine the extent to which we were able to obtain administrative data or not and some of the issues that we 

discovered along the way, and we end with lessons learned and recommendations for other evaluators who may find 

themselves in similar circumstances. The results of the evaluation of the first five sites have been reported elsewhere 

and will not be discussed here (see, for example, Spera, et al., 2012, and other references in the bibliography) 

Table 1 shows the seven communities awarded EUDL grants, the AFBs served, and the history of the AFBs in each 

community. The AFBs in this demonstration grant program have been situated in their respective communities for a

very long time (Mueller, 1989). 

 

2

 EUDL has been in existence since 1998. The program makes funds available to states via block grants and uses discretionary grants to fund 

specific types of local programs. For example, discretionary grants have funded randomized trials and demonstration programs in rural areas. The 

Air Force and community coalition program is another example. For more information on the EUDL initiative, please see: 

http://www.ojjdp.gov/programs/ProgSummary.asp?pi=17, accessed on January 31, 2014. 
3 

Table 1: Seven Demonstration Sits with a Brief History 

Community AFB AFB History 

Phoenix, 

Arizona 

Luke AFB Established in 1941. 

Tucson, 

Arizona 

Davis-Monthan AFB Was a landing field in 1925, the Tucson Municipal Airport in 1927, and 

an AFB in 1941. 

Honolulu, 

Hawaii 

Hickam AFB Established in 1935 and fully occupied by 1937. 

Yuba County, 

California 

Beale AFB Began as an Army post in 1942, and was converted to an AFB in 1951. 

Great Falls, 

Montana 

Malmstrom AFB Began as the Great Falls Army Base in 1942, becoming part of the Air 

Force in 1948.3

Cheyenne, 

Wyoming 

F.E. Warren AFB First established in 1867 as Fort Russell in the same year as the city of 

Cheyenne (then called "Crow Creek Crossing") in Wyoming. A 

presidential proclamation renamed it "Fort Francis E. Warren". It became 

Francis E. Warren AFB in 1947 after World War II. 

Warrensburg 

and Knob 

Noster, MO 

Whiteman AFB Began in 1942 as Sedalia Army Air Field. As part of the demobilization 

efforts after World War II, Sedalia was closed until 1951 when it was 

selected to be a new bombardment wing of the Strategic Air Command. It 

was renamed Whiteman AFB in 1955.

Alcohol Programs in the Air Force 

Air Force Instruction (AFI) 44-121 provides guidance for the Alcohol and Drug Abuse Prevention and Treatment 

(ADAPT) program begun in the 1990’s and still operational today. The objective of the program is to identify Air 

Force members who exhibit various behaviors associated with drug or alcohol abuse or dependence and intervene 

with treatment. Additionally, the ADAPT program at each AFB conducts various activities targeting the prevention 

of such abuse. 

In 2004, the wing commander at F. E. Warren AFB in Wyoming, Col. Evan Hoapili, along with Wyoming’s First

Lady and a non-profit organization called "Facing Alcohol Concerns Through Education" developed the “0-0-1-3” 

policy and program to dramatically decrease the extent of alcohol abuse among all Airmen. The name signified: (0) 

Zero instances of underage drinking; (0) zero DUI/DWI infractions; (1) one drink per hour at social occasions; and 

(3) a maximum of three drinks per night (F.E. Warren AFB, 2005). Alongside the enforcement of the policy, F.E. 

Warren AFB also instituted alcohol-free activities to reinforce the idea that having a good time and enjoying oneself 

is possible without any alcohol. The program was very successful and led to its evolution and expansion in the form 

of the “Culture of Responsible Choices” (CoRC) initiative Air Force-wide in 2006. 

Thus, the EUDL demonstration grants are efforts that occur in addition to substance abuse programs currently 

maintained by the U.S. Air Force, such as ADAPT, 0-0-1-3, and the Culture of Responsible Choices (CoRC). 

Environmental Strategies to Reduce Underage Drinking 

The EUDL initiative uses an environmental-strategies approach in the design of interventions at the community 

level. Environmental strategies are those strategies aimed at the ‘environment’ or the macro level in which people 

live as contrasted with strategies aimed at the individual or micro level. Empirical research on the application of 

environmental strategies to reduce alcohol consumption can be traced back to the early 1990's when communities 

began to take a macro or systems approach to preventing underage drinking. These macro-level strategies included 

formulating alcohol policies, restricting access, targeting establishments that serve alcohol to be more cognizant of 

and compliant with liquor laws, and addressing cultures or social networks that perpetuate dangerous attitudes or 

behaviors with respect to drinking. 

 

3

 The Air Force in its current form was brought into being in 1947 with the National Security Act of that year. Prior to that, it was a component of 

the United States Army starting in 1907. Undergoing various name changes over the years, it was called the “United States Army Air Force”, 

interestingly enough, from 1941 to 1947 when it became a separate branch of the Department of Defense, which the National Security Act also 

brought into existence. So it is not unusual for these AFBs to have started out as Army installations. See

http://en.wikipedia.org/wiki/History_of_the_United_States_Air_Force, accessed on January 31, 2014. 
4 

At the core of the EUDL initiative is the notion of a community coalition that aims to collaboratively design 

interventions that target active duty members living on-base as well as in the community. In some locations, up to 

70% of the military forces reside in the local community or “outside the gates” rather than in military housing 

(Department of Defense, 2010).4

 The demonstration sites spent the first six to twelve months of the grant 

developing their community coalitions and working with Pacific Institute for Research and Evaluation (PIRE), the 

EUDL technical assistance contractor, to create a detailed work plan describing the implementation of their 

interventions. While specific interventions designed at each site varied and were specific to the individual 

circumstances and challenges within each community, the grant solicitation specified that each site implement best 

practices that included: (a) development and deployment of community-based awareness media campaigns to reduce 

drinking, including binge drinking; (b) compliance checks of alcohol retail establishments; (c) strategies to reduce 

the social availability of alcohol to underage persons; (d) policy changes related to underage drinking; (e) driving 

while intoxicated (DWI) enforcement focused on underage persons; (e) shoulder tap operations where an underage 

person attempts to get some 21 years old or older to purchase liquor for him or her; and (f) diversion to social or 

other activities that do not include drinking. 

Each grantee, then, had a great deal of discretion in terms of the specific activities conducted, the scheduling of 

those activities, and how grant resources are allocated and used, as long as they were within grant guidelines. 

Evaluation Design 

The evaluation design can be described as an interrupted time series, with each AFB/community demonstration site 

paired with a similar AFB/community site as a comparison. The demonstration site implemented the EUDL program 

using grant funds, while the comparison sites received no funding and did not implement the EUDL program. We 

defined the intervention starting at the time the work plan was approved, typically within one year of grant award,

although the time can be greater or less than this. Figure 1 below graphically depicts a logic model for the 

AFB/community EUDL demonstration program. 

Figure 1. Air Force Base / Community EUDL Demonstration Program Logic Model 

 

4

 This trend will continue. DoD has a goal of privatizing 98% of all domestic military housing (GAO, 2009). 
5 

The evaluation concerns itself with answering three fundamental research questions: (1) Did the activities 

implemented by the demonstration sites have an impact on underage drinking and alcohol-related misconducts; (2) 

If so, what were the impacts in each AFB and community, and (3) How do these results compare with the 

comparison sites? 

The key outcomes for the evaluation were changes in the following behavioral indicators: 

 binge drinking and prevalence of underage drinking; 

 DUI arrests; 

 alcohol-related traffic accidents; 

 rate of failure of compliance checks among establishments selling/serving alcohol; 

 alcohol-related emergency room visits; and 

 crimes against people involving alcohol such as assaults and acts of domestic violence. 

As may be apparent, many of these measures would be obtained from local law enforcement agencies. While 

developing relationships with local law enforcement was part of each demonstration grantee's activities, this was not 

the case with the comparison AFB/community sites. The comparison sites were "passive sites" in that there were no 

formal overtures or other communications about the evaluation with these sites. With no formal explicit role for the 

comparison sites, there was also no reason or rationale available to persuade local law enforcement to supply arrest 

and other data to the evaluators. Thus, the comparison sites had one administrative source that could use, as 

explained later in this paper. 

Figure 2 provides a graphical overview and timeline for the evaluation design. 

Figure 2. Evaluation Design and Timeline 

Before Grant- First Year- Second Year- Third Year Fourth Year- Fifth Year- 

       

                  

Demonstration Sites 

                  

Comparison Sites 

Grantee Funding Timeline 

Retrospective Data Collection Prospective Data Collection 

Evaluation Contract Timeline 

Legend:  = intervention;  = data collection. 

The figure shows that data collection had two components: a retrospective data collection activity focusing on the 

two years before grant award and the time after grant award but before work plan approval, and a prospective data 

collection component focusing on the data once programmatic intervention activities began in earnest. Additionally, 

the evaluation timeline goes beyond the end of the grant end dates, in order to examine the persistence of any impact 

effects. 
6 

Core Metrics for the Evaluation 

For the evaluation design, we designated a number metrics that could be aggregated and reported monthly - at least 

in theory. The three types of metrics were: 

 Metrics collected or aggregated by each grantee and reported to us monthly5

 Metrics gathered from the Air Force-wide database, Substance Use Assessment Tool (SUAT) 

 Metrics gathered from the 2006, the 2008, and subsequent Air Force Community Assessment Surveys. 

These anonymous surveys are administered to over 300,000 active duty Air Force personnel, spouses of Air 

Force active duty personnel, and the Air Force Reserves. The instrument contains hundreds of questions in 

which respondents are asked to self-report behaviors of various kinds (e.g., drinking, tobacco use). Because 

they constitute primary data collection, these will not be part of the results reported here. 

Table 2 lists the individual measures, and the source for each. Note that all of these come from various 

administrative sources. All of the measures involved de-identified data. We did not request any personally

identifiable data from any sources. 

Table 2. List of Metrics and Data Sources for Administrative Data 

Common Measures Reporting System (EUDL Sites Only) 

Metric Source 

Airmen DWI/DUI 

 Number on-base and off-base 

 Under 21years of age and 21+ 

Security Forces (Air Force military police 

equivalent) for on-base 

City, county, state law enforcement for offbase 

Traffic accidents involving Airmen 

 Number on-base and off-base 

 Under 21 years of age and 21+ 

Security Forces (Air Force military police 

equivalent) for on-base 

City, county, state law enforcement for offbase 

Compliance checks 

 Number completed 

 Number failed 

City, county, state law enforcement 

Alcohol-related emergency room visits by Airmen 

 Under 21 years of age and 21+ 

Local hospitals and clinics 

Crimes involving Airmen where alcohol is a factor 

 For all crimes, incidents involving Airmen under 21years 

of age and 21+ 
7 

Table 2. List of Metrics and Data Sources for Administrative Data 

Air Force Alcohol and Drug Abuse Prevention and Treatment (ADAPT) Program Metrics (EUDL and 

Comparison Sites) 

Metric Source 

Airmen DWI/DUI Reason for Referral 

 Number on-base and off-base 

 Under 21years of age and 21+ 

Substance Use Assessment Tool (SUAT) 

Other Reasons for Referral to ADAPT 

 Domestic violence or other crimes against people or 

pets/family maltreatment 

 Drunk and disorderly conduct 

 Duty related incident 

 Crimes against property 

 Public intoxication 

 Injury 

 Contributing to the delinquency of a minor 

 Open container violation 

Substance Use Assessment Tool (SUAT) 

Number of underage referrals to ADAPT Substance Use Assessment Tool (SUAT) 

Alcohol related traffic accidents involving Airmen 

 On-base and off-base 

Substance Use Assessment Tool (SUAT) 

Binge drinking 

 Number of underage Airmen identified as alcohol 

dependent 

Substance Use Assessment Tool (SUAT) - 

SADD Scale score 

Airmen reasons for drinking alcohol 

 < 21 year of age 
8 

Table 3 – Demonstration Sites: Air Force Bases and the Communities Around Them 

State Community Air Force Base Urban or Rural Mission

AZ Phoenix Luke AFB Urban Combat Operations & Training 

AZ Tucson Davis-Monthan AFB Urban Combat & Support Operations 

CA Yuba County Beale AFB Rural Combat Support Operations. 

HI Honolulu Hickam AFB Urban Combat Support 

MT Great Falls Malmstrom AFB Rural Operational Missile Base 

WY Cheyenne F.E. Warren AFB Rural Operational Missile Base 

MO Warrensburg 

 /Knob Noster Whiteman AFB Rural Air Combat Command / Bomber Wing 

Table 4 displays selected demographic characteristics of the AFBs and the surrounding communities. The proportion 

of problem drinking at pre-test was taken from the Air Force Community Assessment survey data from 2006 for the 

first five AFBs, and from 2008 for the last two. The percentages reported for the AFBs are consistent and 

commensurate with national prevalence rates (SAMHSA, 2009). Among the EUDL sites, Whiteman AFB in 

Missouri had the highest proportion of personnel under the age of 21, and Hickam in Hawaii had the lowest. FE 

Warren AFB/Cheyenne had the lowest number of liquor establishments as measured by Residents Per Outlet (RPO) 

and Whiteman AFB/Warrensburg the highest. 

Table 4 – Selected Characteristics of Air Force Bases and the Nearby Communities 

Characteristic AZ/ 

Phoenix 

Luke 

AFB 

AZ/ 

Tucson 

Davis 

Monthan 

AFB 

CA/

Yuba 

County 

Beale 

AFB 

HI/ 

Honolulu 

Hickam 

AFB 

MT/ 

Great 

Falls 

Malmstro

m AFB 

WY/ 

Cheyenne 

FE 

Warren 

AFB 

MO/ 

Warrensburg 

Whiteman 

AFB 

Number Assigned 

Personnel6

4,586 5,500 3,116 3,767 2,929 3,361 3,600 

Number Under 21 

Years of Age 

217 272 169 109 291 336 377

Percent Under 21 4.7% 4.9% 5.4% 2.9% 9.9% 10.0% 10.5% 

Proportion Problem 

Drinking at Pretest(E1-E4) 7

21.5% 22.3% 18.9% 20.0% 24.6% 10.1% 14.7% 

Community 

Population 

3,600,000 1,023,320 73,000 375,000 56,690 94,000 52,595 

Number of Liquor 

Outlets8

12,932 3,154 276 1,364 156 161 432

Residents Per 

Outlet 

278 324 264 275 363 584 122

The proportion of junior enlisted who scored as “Problem drinkers” varied from a low of 10.1% at Beale AFB, 

which is in a rural community, to a high of 24.6% at Malmstrom AFB, also in a rural community. 

Data Availability and Acquisition 

Table 5 summarizes the data availability and acquisition for each of the seven demonstration sites by type of data. 

The top row lists the AFBs. The row labeled "Project Year" denotes the grant year. A negative number indicates 

retrospective data collection, so that, for example, "-2" signifies data for the year that was 2 years before the grant 

 

6

 The number of personnel assigned to an Air Force Base varies constantly. The numbers shown here are the monthly averages for the year in 

which the demonstration sites submitted grant proposals for this intervention. 

7

 The proportion of problem drinkers is measured by a cutoff score of 8 or higher on the AUDIT scale, which was part of the 2006 and 2008 Air 

Force Community Assessments. 

8

 Source is the Alcoholic Beverage Control entity or its equivalent in each state. The data for Great Falls Montana come from the Great Falls 

Police Department. 
9 

award. This first five sites in the table received their demonstration grants in 2006. The last (i.e., right-most) two 

sites received the grants in 2009. 

Table 5: Data availability by AFB/Community Site and by Data Type

AZ - Luke AZ-DavisMonthan CA-Beale HI-Hickam MTMalmstrom WY-Warren MO-Whiteman 

Project Year -2 -1 1 2 3 -2 -1 1 2 3 -2 -1 1 2 3 -2 -1 1 2 3 -2 -1 1 2 3 -2 -1 1 2 3 -2 -1 1 2 3

Metric

DUI/DWI                                   

Traffic 

Accidents                      

Compliance 

Checks                          

ER Visits                   

Crimes                            

SUAT 

DUI                            

Other Referral 

Reasons                            

Underage 

Referrals                              

Traffic 

Accidents 

Binge 

Drinking 

Reasons for 

Drinking                    

Legend:  fully available,  partially available, system not ready or data not requested, data not available 

All the sites were able to collect at least some form of DUI/DWI data from local law enforcement agencies. The next 

most available data came from compliance check activities. The compliance checks, most of the time, consisted of 

an underage person - in many cases an Airman from the neighboring AFB - attempting to purchase liquor from an 

establishment. If the establishment sold liquor to the covert underage buyer, it was cited, and in some jurisdictions 

the owners had to appear in court. The two gaps in compliance checks occurred at Beale AFB, where the coalition 

was not able to get cooperation from the local police to conduct compliance checks, and at F.E. Warren AFB, where 

no compliance checks were conducted (or data collected and maintained) prior to the grant. 

The most problematic data metrics were traffic accidents and emergency room (ER) visits. Those sites that were 

able to provide traffic accident data spent a significant amount of time abstracting the data from police reports and 

manually entering them in order to provide them in electronic form. With respect to data on ER visits involving 

Airmen and alcohol, the local area hospitals and clinics refused to provide data at three of the seven sites, citing 

HIPAA privacy rules, even though the request was for data in aggregate form, such as the number of underage 

Airmen seeking ER services in any one month. Those sites with only partial ER visit data availability (F.E. Warren 

AFB and Whiteman AFB) were missing data for some months and for some clinics and hospitals. 

Crime data availability was mixed. Apart from Luke AFB, which was not able to provide any data on the crime 

metrics, the other sites were able to provide the data. However, we realized, after receipt of the data, that many of 

the crime metrics were essentially low base-rate phenomena: simply put, they do not occur often enough to be of 

much use in detecting impact. The crimes with greater frequencies of occurrence by Airmen, and thus more useful, 

were: drunk and disorderly conduct, public intoxication, and minor in possession. 

The bottom half of Table 5 lists the metrics available from SUAT, which is a screening tool that all Airmen referred 

to ADAPT must complete. The left most blocks of cells are blacked out because the SUAT screening tool was

implemented in late 2004 and early 2005, one year before the first five sites received their grants. Thus, SUAT data 
10

were not available earlier than 2005. The SUAT data were complete for referrals due to DUI, due to various other 

reasons for referral, and for referral because the Airmen were younger than 21 years of age. Virtually no referrals 

occurred due to traffic accidents, and indicators of binge-drinking were not readily available. The "reasons for 

drinking" metrics were problematic. It appeared that Airmen simply did not respond to these indicators in 

completing the screening tool. These latter metrics were dropped for the evaluation of the right-most two sites in the 

Table. 

It is instructive to examine the reasons behind the lack of availability or usefulness, described in Table 6. 

Table 6: Reasons Limiting Data Availability or Utility

Data Type Reasons Limiting Availability or Utility Sites Most Affected

DUI/DWI Comingling of data by age-groups (under 21, and 21+) 

Comingling of data for civilians and military 

Monthly data not available, only annual 

AZ-Luke 

AZ-Davis-Monthan 

HI-Hickam 

Traffic Accidents No data provided by local law enforcement because of resource 

constraints 

Lack of cooperation by some jurisdictions in community 

Manual reprocessing to provide data in the form needed 

Monthly data not available, only annual 

Data only available after citation format was changed to include 

check box for "Airmen" 

AZ-Luke 

AZ-Davis-Monthan 

HI-Hickam 

MO-Whiteman 

Compliance Checks Refusal to conduct compliance checks by law enforcement CA-Beale 

ER Visits HIPAA Privacy Policy concerns 

Unable to distinguish between Airmen and civilians 

Unable to distinguish between under-age and of-age 

Some hospitals/clinics did not provide but others did

AZ-LUKE 

AZ-Davis Monthan 

WY-Warren 

MO-Whiteman 

Crimes Not reported by local law enforcement 

Low base-rate phenomenon - low frequency of occurrence 

Some crimes greater frequency than others 

Some crimes not reported at all 

Lack of correspondence with data reported by Security Forces 

All 

SUAT 

DUI Reported on-base DUIs only 

DUIs do not correspond to DUIs from local law enforcement 

WY-Warren 

MO-Whiteman 

All other AFBs 

Referral Reasons No issues - all sites had data 

Underage Referrals Low frequency of occurrence All 

Traffic Accidents No referrals due to traffic accidents All 

Binge Drinking No data reported All 

Reasons for Drinking Low frequency of response All 

Those metrics that were the most robust, DUIs and compliance checks, were associated with proactive law 

enforcement operations. The majority of DUI arrests, for example, was a result of DUI checkpoints, rather than 

traffic stops by individual law enforcement officers. However, most law enforcement agencies did not keep track of 

the data that would have been most useful for the evaluation in order to distinguish between Airmen and civilians 

and underage and of-age persons. In most instances, the most usable DUI data came about because of manual 

processing of arrest records by program staff or by law enforcement personnel. For this same reasons, traffic 

accident data was difficult to obtain - there was too much of it, and it was not in the most appropriate form to be 

usable for assessing impact. 

Of course the difficulties in obtaining data was specific to each site, and the success (or lack of thereof) of obtaining 

data could be attributed, to some extent, to the web of relationships among demonstration program staff, Air Force 

leadership (especially the Wing Commander), and the various agencies and stakeholders within each 

AFB/community site. In general, the sites that were the most successful in obtaining data for the evaluation were the 

ones that had or have the best relationships among all the stakeholders. These sites were: Malmstrom AFB/Great 
11

Falls, MT; Whiteman AFB/Warrensburg MO; and to a lesser extent, F.E. Warren/Cheyenne, WY, and Hickam 

AFB/Honolulu, HI. 

Based on the above tables, as well as qualitative data gathered from site visits to two of the demonstration sites (F.E. 

Warren AFB/Cheyenne, WY; Whiteman AFB/Warrensburg, MO), we can summarize some the challenges involved 

in utilizing these data sources. 

Table 7 summarizes the challenges and possible responses to those challenges. 

Table 7: Acquisition Challenges by Data Sources and Possible Responses

Data Source Data Challenges Response 

Local law enforcement 

agencies 

 Number of jurisdictions involved 

 Number of different data 

systems 

 Target formats of the data 

 Information not ordinarily 

maintained 

 Get buy-in from all law-enforcement 

jurisdictions in evaluation "catchment 

area" 

 Do mapping of current data captured, 

and data desired to identify gaps 

 Change data collection forms (e.g., 

traffic citations) to accommodate 

administrative data 

Local hospitals and clinics  HIPAA Privacy Concerns 

 Information not ordinarily 

reported to outsiders 

 Build relationships with hospital/clinic 

administrators 

 Ask for data in aggregate form with no 

PII 

 Accommodate/alleviate lingering 

HIPAA concerns 

 If ER visits are too low to be good 

metrics, consider eliminating this metric 

ADAPT Referrals (SUAT)  Timeliness 

 Congruency with data from 

other sources 

 Missing data for some metrics, 

such as binge drinking 

 Obtain SUAT data semi-annually or 

annually, but with monthly totals 

 Do not expect congruence with local law 

enforcement data 

 Focus on DUI, drunk and disorderly, 

public intoxication, and open container 

violations only. 

Security Forces  Judge Advocate General (JAG) 

concerns 

 Congruence with local law 

enforcement data 

 Collaboration with local law 

enforcement 

 Meet and negotiate with JAG officers to 

ameliorate concerns 

 Assess the working relationship between 

Security Forces and local law 

enforcement and foster collaboration and 

daily communication between the data 

As Table 7 illustrates, much of the success of acquiring and using administrative data at this very local communitybased level depended on relationships and collaboration. Additionally, Table 6 reinforces the notion that one can 

(and sometimes must) drop metrics that do not contribute to assessing change as a result of the intervention. 

Lessons and Conclusions 

When we originally planned the evaluation, we made several assumptions. The first was that retrospective data 

would be relatively easy to collect because all of the data was being collected by various entities already. We also 

assumed that, because much of the data would be coming from law enforcement agencies, collecting data during and 

after the intervention would also be relatively easy. Law enforcement agencies constantly report crime data to other 

agencies (e.g., the Uniform Crime Reports) and to the public (e.g., maps of police patrol districts and crimes). 

Finally, we assumed that local law enforcement at these communities would already be working with Security

Forces because of the large military presence. 

None of these assumptions were completely correct. 
12

Part of this was due to the fact that information needed to make the data usable for the evaluation was not collected, 

or was buried in the text of the arrest record, rather than being a field in a database that could act as a filter. Much of 

the data we required had to distinguish between military and civilian status and underage for alcohol (i.e., younger 

than 21 years of age) versus of-age. All of the law enforcement data systems did not distinguish between military 

and civilian.9

 For many offenses, adult status is defined as being the age of 18 or older, so there was less attention 

paid to the 21 years of age threshold, as well. 

Virtually all of the sites had to perform some manual processing of data from law enforcement agencies. Typically, 

this involved going through computerized reports of certain types of police citations, identifying those citations that 

would be relevant to the data required, going to the original arrest reports, and summarizing the relevant infractions 

for purposes of the evaluations. At one of the sites, the EUDL Project Coordinator conducted this manual processing 

on his own time. Another site hired a full-time staff person for the project whose duties consisted largely of culling 

arrest and citation reports from the local police department and summarizing the results for the evaluation. 

Air Force data systems, such as those used by Security Forces, typically used age-categories, such as 18-25 years of 

age. Obtaining data for underage Airmen entailed special processing. 

Those sites that were able to obtain data for the evaluations typically had very strong relationships among the major 

stakeholders: the Air Force Base command leadership, the EUDL Program project team, and local law enforcement. 

Where one of these was lacking, data collection efforts suffered. 

Among the lessons that we learned are the following: 

 Validate all assumptions before starting data collection. 

 When planning the evaluation and identifying metrics to use, perform a rigorous gap analysis between the 

data required to assess impact, and the type and format of administrative data available. In some cases, the 

gaps may be filled, either by slight or modest changes in procedures or in record-keeping. 

 Do not under-estimate the amount of time or resources required to obtain retrospective administrative data. 

Several sites took almost one year to obtain two years' worth of such data; other sites never succeeded. 

 Those metrics that represent low-base rate phenomena may not be worth collecting at all, because the 

frequency of occurrence is too low. In our experience, the data we could obtain for such crimes as alcoholrelated domestic violence was almost non-existent. 

 There may be objections from data-source providers, based on interpretations of certain laws. For example, 

some hospitals and clinics refused to provide any data because of HIPAA Privacy Act concerns. In one 

case, a JAG officer interpreted some Air Force regulations as limiting the distribution of data to parties 

outside of the Air Force, such as external evaluators. 

 Over time, however, you will be able to identify the most complete and robust administrative measures to

use in evaluating the impact of a program. 

We applied these lessons to the second group of grantees who received their grants in 2009 (F.E. Warren

AFB/Cheyenne, and Whiteman AFB/Warrensburg). We reduced the number of crimes for which we wanted data;

we eliminated metrics from the SUAT data; and we worked with the grantees to help them confront many of these 

issues early on. We are also further refining the metrics for the most recent group of grantees, awarded in 2012. We 

expect the data collection efforts to be even more improved with them. 

 

9

 As noted earlier, one site (Davis-Monthan AFB/ Tucson, AZ) influenced the Tucson Police Department to add a check box to the police citation 

form that indicated whether the person involved was military or not. A second site (F.E. Warren AFB/Cheyenne) was in the process of adding 

something similar. 
13

Bibliography

Barlas, F. M., Higgins, W. B., Pflieger, J. C. & Diecker, K. (2013). 2011 Health Related Behaviors Survey of Active 

Duty Military Personnel. TRICARE Management Activity and United States Coast Guard, Washington, 

DC. 

Barlas, F., Spera, C., Szoc, R.Z., Prabhakaran, J. & Cambridge, M.H. (2011, invited). Examining the Influence of 

the Enforcing Underage Drinking Laws (EUDL) Program on Alcohol-Related Outcomes in Five 

Communities Surrounding Air Force Bases. Community Anti-Drug Coalitions of America Annual 

Conference, Washington DC. 

Creel, A. H. & Szoc, R. Z. (2013a). Serving Two Masters: Making Evaluation Findings Useful for the National 

Program and the Local Grantees. Paper presented at the Eastern Evaluation Research Society Conference, 

April 4-6, 2013. 

Creel, A. H. & Szoc, R. Z. (2013b). Evaluation of the Enforcing Underage Drinking Laws (EUDL) Program on 

Alcohol-Related Attitudes, Behaviors and Consequences in the Air Force: Survey Findings. Poster 

presented at the American Public Health Association Meeting and Exposition, November 2-6, 2013. 

Department of Defense (1980). “Alcohol and Drug Abuse of DoD Personnel," Directive no. 1010.4 (Washington, 

D.C.: 25 August 1980). 

Department of Defense (2010). Office of the Undersecretary of Defense, Installations and Environments, FAQs. 

Accessed May 21, 2010 at: http://www.acq.osd.mil/housing/faqs.htm#3. 

F.E. Warren Air Force Base. (2005). F.E. Warren 0-0-1-3 program teaches responsible drinking. Press release, April 

7, 2005. Accessed May 21, 2010 at: http://www.wfli.org/News/FE%20Warren 

%200013%20program%20teaches%20responsible%20drinking.htm

ICF International. (2012). 2009-2010 Department of Defense Survey of Health Related Behaviors Among the Guard 

and Reserve Force: Online Data Collection Pilot Test. TRICARE Management Activity, Washington, DC. 

General Accountability Office (GAO). (2009). Military Housing Privatization DoD Faces New Challenges Due To 

Significant Growth At Some Installations And Recent Turmoil In The Financial Markets. GAO Report 09-

352, May, 2009. 

Mueller, Robert (1989). Active Air Force Bases Within the United States of America on 17 September 1982. USAF 

Reference Series, Maxwell AFB, Alabama: Office of Air Force History. 

Spera, C., Barlas, F., Thomas, R., Szoc, R., & Cambridge, M. (2011). Relationship of Military Deployment

Recency, Frequency, Duration, and Combat Exposure to Alcohol Use in the Air Force. Journal of Studies 

on Alcohol and Drugs, 5-14.

Spera, C., Barlas, F.M., Szoc, R. Z., Prabhakaran, J. & Cambridge, M.H. (2012). Examining the influence of the 

Enforcing Underage Drinking Laws (EUDL) program on alcohol-related outcomes in five communities 

surrounding Air Force bases. Addictive Behaviors, 37(4), 513-516. 

Spera, C., Creel, A. H., Szoc, R., Barlas, F. (2012). Examining the Influence of the Enforcing Underage Drinking 

Laws (EUDL) Program on Alcohol-Related Outcomes in the Air Force. Symposium paper to be presented 

at the Society for Prevention Research 20th Annual Meeting, Washington, DC, May 31, 2012. 

Spera, C., Franklin, K., Uekawa, K., Kunz, J., Szoc, R., Thomas, R., & Cambridge, M. (2010). Reducing drinking 

among junior enlisted Air Force members in five communities: Early findings of the EUDL Program’s 

influence on self-reported drinking behaviors. Journal of Studies on Alcohol and Drugs, May, 2010. 

Spera, C., Franklin, K., Uekawa, K., Kunz, J.F., Szoc, R.Z., Thomas, R.K. & Cambridge, M.H. (2011). Reducing 

Drinking Among Underage Air Force Members in Five Communities. OJJDP Bulletin. 

Spera, C., Szoc, R., Kunz, J., Franklin, K., & Schreck, A. (2008). Enforcing Underage Drinking Laws (EUDL) at 

Air Force Bases: Year 1 Findings. National Leadership Conference, Department of Justice, Nashville, TN. 

Spera, C., Szoc, R., Kunz, J., Franklin, K., & Schreck, A. (2008). Enforcing Underage Drinking Laws (EUDL) at 

Air Force Bases: Year 1 Findings. National Leadership Conference, Department of Justice, Nashville, TN. 
14 

Spera, C., Szoc, R., Thomas, R., & Barlas, F. (2009). Enforcing Underage Drinking Laws (EUDL) at Air Force 

Bases: Year 2 Findings. National Leadership Conference, Department of Justice, Dallas, TX. 

Spera, C., Szoc, R., Thomas, R., & Barlas, F. (2010). Enforcing Underage Drinking Laws (EUDL) at Air Force 

Bases: Year 3 Findings. National Leadership Conference, Department of Justice, Anaheim, CA. 

Substance Abuse and Mental Health Services Administration (SAMHSA). (2009). Results from the 2008 National 

Survey on Drug Use and Health: National Findings (Office of Applied Studies, NSDUH Series H-36, HHS 

Publication No. SMA 09-4434). Rockville, MD. 

Szoc, R. & Barlas, F. The Raw and the Dirty, the Cooked and the Clean: Strategies and Resources for Collecting 

EUDL-Relevant Data in the Field. Invited Workshop presented at the OJJDP National Conference, August 

10, 2011, Orlando, FL. 

Szoc, R. Z. & Creel, A. H. The Role of Community Coalitions in Successful Environmental Approaches to 

Decreasing Underage Drinking: A Case Study Approach. Poster presented at the American Public Health 

Association Meeting and Exposition, November 2-6, 2013. 

Trest, W. A. (1998). Air Force Roles and Missions: A History. Air Force History and Museums Program, 

Washington, DC.