Sexual Violence Data: Actions Needed to Improve Clarity and Address Differences Across Federal Data Collection Efforts
Released: 2016-08-18T00:00:00

Letter

Background

The federal government has implemented a number of initiatives to address sexual violence or mitigate its effects. For example, the Department of Justice’s (DOJ) Office on Violence Against Women (OVW), created in 1995 in order to help implement the Violence Against Women Act (VAWA), sponsors grant programs for local law enforcement agencies, prosecutors and judges, health care providers, and other organizations that assist victims of sexual violence by providing, for example, forensic medical services in sexual violence cases in rural areas and specialized counseling services for victims from underserved populations. Another office within DOJ, the Office for Victims of Crime (OVC), convened crime victim advocates and experts in 2013 as part of the Vision 21 Initiative and recommended in their report, among other things, that federal agencies collaborate and expand the collection and analysis of data on all forms of criminal victimization. The Department of Health and Human Services’ (HHS) Family Violence Prevention and Services Program supports two national resource centers on domestic violence and special-issue and culturally-specific resource centers. In addition, HHS’s College Sexual Assault Policy and Prevention Initiative was launched in 2016 and is intended to provide support for organizations that are implementing policies and practices at postsecondary schools to prevent sexual assault on their campuses. The Department of Education’s Office for Civil Rights (OCR) issued guidance to colleges and universities, one in 2011 and another in 2014, concerning the responsibilities of those institutions under Title IX of the Education Amendments of 1972 with regard to addressing sexual violence against students. OCR’s guidance sets standards for the grievance procedures institutions must adopt and publish to promptly and equitably resolve complaints brought by students alleging sex discrimination (including acts of sexual violence and sexual harassment), and recommends preventive education and training programs designed to reduce the occurrence of sexual violence on campus and improve institutions’ responses to sexual violence on campus when it does occur. In 2004, Congress passed a law that required the Secretary of Defense to develop, among other things, a comprehensive policy for the Department of Defense (DOD) on the prevention of sexual assaults involving servicemembers. In response to that statutory requirement, DOD established its sexual-assault prevention and response program in 2005, and in 2008, DOD published its first sexual assault prevention strategy.

Several of the federal government’s responses to sexual violence involve data collection on the occurrence of sexual violence. For example, the Prison Rape Elimination Act (PREA) of 2003 directed DOJ to carry out studies of the incidence and effects of prison rape. The Jeanne Clery Disclosure of Campus Security Policy and Campus Crime Statistics Act, as amended by VAWA in 2013, requires that all institutions of higher education that participate in federal student financial assistance programs disclose statistics on certain crimes, including those related to sexual violence, to the Department of Education (Education). Since 2005, National Defense Authorization Acts have directed DOD to, among other things, collect and report information on sexual assaults against service members.

Under the Paperwork Reduction Act (PRA), the Office of Information and Regulatory Affairs (OIRA) in the Office of Management and Budget (OMB) is charged with improving the efficiency and effectiveness of federal information resources, which includes functions relating to statistical policy and coordination. Specifically, with regard to statistics, OMB’s responsibilities include:

Oversight and approval of data collection: OMB reviews statistical information collections as part of its responsibility under the PRA to approve all federal agency information collections that will be administered to 10 or more people to ensure adherence with PRA standards for minimizing information collection burdens and maximizing the practical utility of information collected by federal agencies, including eliminating unnecessary duplication.

Guidance and standards: OMB develops and oversees governmentwide policies, principles, standards, and guidelines for collecting and disseminating statistical information.

Coordination: OMB coordinates the activities of the federal statistical system, including ensuring the integrity, objectivity, and utility of federal statistics.

Oversight of budgets: OMB ensures that statistical agencies’ budget proposals are consistent with systemwide priorities for maintaining and improving the quality of federal statistics.

Other entities also provide guidance to agencies that conduct statistical work. For example, the National Academy of Sciences’ Committee on National Statistics (CNSTAT) publishes Principles and Practices for a Federal Statistical Agency for newly appointed cabinet secretaries at the beginning of each presidential administration. Principles and Practices outlines basic principles for statistical agencies to carry out their missions effectively, as well as practices designed to help implement them.

Different entities use federal data on sexual violence, including, for example victim advocacy groups, other special interest groups, and other federal agencies. Officials at victim advocacy groups we spoke with publish reports on topics related to sexual violence and lobby Congress for laws and programs designed to address the needs of victims. Other groups include law enforcement associations and campus safety groups that provide training and educational materials for law enforcement and campus safety personnel. Federal agencies also use data on sexual violence, for example to inform grant making decisions regarding research and program development.

Four Federal Agencies Manage Ten Efforts to Collect Data on Sexual Violence, which Differ in Target Population, Terminology, Measurements, and Methodology

Federal Agencies Collect Sexual Violence Data on the General Population and on Segments of the Population

Four federal agencies manage at least 10 data collection efforts that include data on sexual violence, among other things. Some of these data collection efforts focus on a target population that the agency serves. For example, Education’s Clery Act data collection effort obtains information on the occurrence of sexual violence at institutions of higher education. DOD’s Defense Sexual Assault Incident Database (DSAID) and the Workplace and Gender Relations Survey of Active Duty Members (WGRA) include and collect data on sexual violence involving military service members. Others, such as the FBI’s Uniform Crime Reporting Program (UCR) data collection efforts compile data from law enforcement agencies on the general population. Those data collection efforts that include information from the general population differ in terms of the ages of respondents or individuals from whom reports of sexual violence are taken. For example, the National Intimate Partner and Sexual Violence Survey (NISVS) collects data from individuals who are 18 and older while the National Crime Victimization Survey (NCVS) collects data on household members who are 12 and older; both the Uniform Crime Reporting Program-Summary Reporting System (UCR-SRS) and the Uniform Crime Reporting Program-National Incident-Based Reporting System (UCR-NIBRS) include data from law enforcement agencies on criminal incidents involving people of all ages. Table 1 includes information about each of the 10 data collections discussed in this report, including their respective target populations.

Data collection efforts that are focused on target populations—such as, the military population, institutions of higher education, and the incarcerated population—provide information on the problem of sexual violence within those groups and thus may be helpful for informing policy affecting those groups. For example, Education officials told us that Education’s Office of Federal Student Aid administers inquiries to specific campuses if Clery Act data show unusually high incidences of certain crimes, including rape. The Bureau of Justice Statistics (BJS) reports that the National Inmate Survey (NIS) and the Survey of Sexual Victimization (SSV) data provide helpful information for understanding and addressing the problem of sexual violence in prisons, jails, and juvenile correctional facilities. In March 2015, we reported on the importance of using military data on sexual violence to inform program decisionmaking.

The 5 data collection efforts whose target population is a segment of the national population are the result of specific congressional mandates, while the 5 data collection efforts that focus on the general population are discretionary initiatives arising from broad agency missions. For example, BJS is mandated under the terms of the Prison Rape Elimination Act (PREA) of 2003 to collect data on sexual violence in prisons, jails, and other detention facilities. BJS conducts both the NIS and SSV in response to that mandate. In contrast, according to HHS’s Centers for Disease Control and Prevention (CDC) officials, the agency launched NISVS as part of its public health mission, and with support from the National Institute of Justice and DOD as a result of requests from organizations in the field of sexual violence prevention.

The extent to which the data collection efforts focus on sexual violence also varies. Some of the data collection efforts collect information solely or primarily on the occurrence of sexual violence, such as the SSV or DSAID. Other data collection efforts have a larger focus. For example, the UCR-SRS, UCR-NIBRS, and NCVS include information on a broad spectrum of crimes, and the National Electronic Injury Surveillance System–All Injury Program (NEISS-AIP) includes information on a wide variety of types of injuries.

Data Collection Efforts Differ in the Terms They Use to Describe Sexual Violence, and Measurements and Definitions for Some Efforts are Inconsistent

Data collection efforts use a range of terms to describe sexual violence in publicly-available agency documentation. Specifically, the 10 data collection efforts use a total of 23 different terms to describe sexual violence. Table 2 shows the terms that data collection efforts use to describe sexual violence.

Given the variation in terminology, data collection efforts may characterize the same sex act using different terms. For example, regarding sexual violence involving vaginal penetration of a victim, 6 data collection efforts include this act of sexual violence in their measurement of “rape,” 2 include it in their measurements of “nonconsensual sexual acts” or “staff sexual misconduct,” 2 include it in their measurements of “sexual assault” or “assault-sexual,” 1 includes it in its measurement of “sexual coercion,” 1 includes it in its measurement of “penetrative sexual assault,” and 1 includes it in its measurement of “sexual assault with an object.” See tables 5 through 7 in app. II for additional information on acts of sexual violence included in measurements of sexual violence by data collection effort.

It is also the case that one data collection effort may use multiple terms to characterize a particular act of sexual violence, depending on the contextual factors that may be involved, such as whether the perpetrator used physical force. For example, if a victim is penetrated vaginally, NISVS may characterize that particular act as either “rape” or “sexual coercion,” and the decision as to which term is most appropriate is based on the contextual factors surrounding the act. As such, NISVS characterizes vaginal penetration of a victim as “rape” if the act involves the use of physical force or threats to physically harm the victim. On the other hand, NISVS characterizes this same act as “sexual coercion” if the act occurs after the victim is verbally pressured in a nonphysical way, for example if the perpetrator uses their influence or authority.

Based on our analysis, data collection efforts rarely use the same terminology to describe sexual violence; however, when they do, there are some differences in the particular acts of sexual violence and contextual factors that they include in their measurements of those terms. For example, 4 of the 6 data collection efforts that use the term “rape” consider whether actual physical force was used and the other two do not. Three of the 6 that use the term “rape” consider whether the threat of physical force was used and the other 3 do not. See tables 8 through 10 in app. II for additional information on contextual factors included in measurements of sexual violence by data collection effort.

In general, measurements of sexual violence closely relate to definitions of sexual violence in federal data collection efforts. For 5 of the data collection efforts we reviewed, the acts of sexual violence and contextual factors that are included in the measurements generally align with the acts of sexual violence and contextual factors that are included in the definitions.

However, for the remaining 5 data collection efforts, some of the acts of sexual violence or contextual factors are included in both their measurements and definitions and others are not. Further, these data collection efforts do not have publicly-available descriptions of what is included in their respective measurements to allow persons using the data to understand the differences. Specifically, Clery Act data include attempted rapes in its measurement of rape, but do not include attempted rapes in its definition of rape. NEISS-AIP includes acts of sexual violence involving penetration of a victim with an object and acts of sexual violence involving a victim being made to penetrate someone else with an object in its measurement of assault-sexual, but does not explicitly include these acts of sexual violence in its definition of assault-sexual or in the description of assault-sexual in the NEISS-AIP coding manual. Additionally, NCVS includes the contextual factors of “victim unable to consent or refuse” and “victim alcohol/drug facilitated” in its measurements of rape and sexual assault, but does not include these contextual factors in its definitions of rape and sexual assault. Similarly, SSV includes attempted nonconsensual sexual acts in its measurement of nonconsensual sexual acts, but does not include attempts in its definition of nonconsensual acts. NIS includes the act of victim penetration with an object in its measurements of nonconsensual sexual acts and staff sexual misconduct, but does not include it in its definitions of nonconsensual sexual acts and staff sexual misconduct.

The National Academy of Sciences’ Principles and Practices for a Federal Statistical Agency states that data releases from a statistical program should include the methods and assumptions used for data collection and reporting. Similarly, OMB guidelines regarding the federal Information Quality Act call for agencies that disseminate government information to ensure its utility, objectivity, and integrity, which refers to information reproducibility and transparency. Additionally, federal internal control standards state that an agency’s information requirements should consider the expectations of both internal and external users and that reliable internal and external information sources should provide data that faithfully represent what they purport to represent. Education officials told us that they are updating The Handbook for Campus Safety and Security Reporting and expect to issue the updated handbook in summer 2016. This may provide an opportunity for Education to eliminate discrepancies between the Clery Act’s sexual violence measurements and definitions. Regarding NEISS-AIP, CDC officials told us that the definition of “assault-sexual” is intended to include the range of sexual assault experiences that victims presenting to the emergency department may report. BJS officials told us that it is not possible to enumerate every act of sexual violence that is included under NCVS’s terms of sexual violence and that data users can make their own determinations about what acts of sexual violence are included in the measurements. BJS officials also told us that the definitions included in the NIS and SSV summary reports are intended for the general public, and acknowledged that the reports present ambiguity regarding which acts of sexual violence and contextual factors are included in the measurements. CDC and BJS officials told us that researchers who are interested in descriptions of what is included in the measurements for NEISS-AIP, NIS, and SSV could access coding information, which is available at the University of Michigan’s National Archive of Criminal Justice Data of the Interuniversity Consortium for Political and Social Research. However, a layperson who is not a researcher may not know how to access this information.

If data users are seeking to understand what sexual acts and contextual factors are included in a data collection effort’s measurement of sexual violence, they may read the definitions of terms contained in reports of those data collections. If the definitions of the terms are different from what the data collection effort includes in its measurement of those terms, data users may lack clarity about what acts of sexual violence and contextual factors the efforts are including in their measurements of sexual violence.

Federal Sexual Violence Data Collection Methodologies Differ by Context, Data Source, Units of Measurement, and Time Frames

Context

Federal agencies generally collect data on sexual violence within one of two contexts—criminal justice or public health. For the purposes of this report, “criminal justice” describes data collection efforts that refer to acts of sexual violence as “crimes” or “offenses.” “Public health” describes data collection efforts that seek to understand the health implications of acts of sexual violence. Of the 10 data collection efforts within our scope, 7 collect data primarily in a criminal justice context, and 2 collect data primarily in a public health context, and 1 data collection effort combines both contexts. Table 11 in app. II outlines which data collection efforts fall into each category.

According to agency officials, context can determine how each data collection effort is designed and how the data are collected—and context may inform what is included in the measurements of sexual violence in the data collection efforts. The data collection efforts of BJS and FBI included in this report have a criminal justice focus, intended to collect information on crimes, victims, or trends. CDC officials stated that their data collection efforts—NEISS-AIP and NISVS—which both have a public health focus, are more concerned with assessing the health impacts of victimization and informing violence prevention efforts and are less concerned with categorizing incidents as crimes. In some public health surveys, interviewers begin by asking basic health and lifestyle questions to establish a rapport with the interviewee and introduce the concepts of public health and experiences rather than crimes and criminal events. According to BJS and CDC officials, some studies with a focus on criminal justice may ask questions about sexual violence as crimes that have occurred, whereas some studies with a public health focus may ask questions about violent sexual experiences by describing specific acts of sexual violence while avoiding criminal terminology. For instance, NISVS does not use the word rape when questioning interviewees, whereas NCVS asks respondents if anyone has been attacked for example by rape, attempted rape, or other type of sexual attack. NISVS program documentation states that the term rape may carry a stigma or have different meanings to different people, so the survey poses multiple questions about behaviorally specific sexual acts, without using the term rape.

Data Sources

Federal sexual violence data primarily come from two sources: information reported to authorities and information obtained from victim surveys. Some data collection efforts compile information reported to relevant authorities. Other federal data collection efforts obtain their data from surveys where agencies attempt to identify victims from a larger population and invite them to share information about their experiences with sexual violence. Table 12 in app. II outlines which data collections fall into each category.

Information reported to authorities may originate from situations in which a victim or observer reports an alleged act of sexual violence to law enforcement, campus, or prison authorities or to military officials. Data collection efforts vary in how and to whom the information is submitted. For DSAID, restricted and unrestricted reports of sexual violence are reported by victims to sexual assault response coordinators or victim advocates who input information on the incident into DSAID. Whereas for Clery Act data, all institutions of higher education that receive federal student financial aid are required to report campus security data (including information on sexual violence) to the Department of Education.

Some data collection efforts obtain information on sexual violence through surveys. Each survey uses different methods to collect data from their subjects. For instance, NISVS employs a random digit dialing telephone survey while the NCVS uses a mix of face-to-face and telephone interviewing.

Both types of data sources involve tradeoffs. With respect to information reported to authorities, according to agency documentation and a senior official from a law enforcement special interest group, data collection efforts that provide information on crimes reported to authorities are useful for administrative and funding decisions related to law enforcement. For example, the Bureau of Justice Assistance uses UCR- SRS data, in part, to determine how much grant funding should be awarded to state, local, and tribal governments through the Edward Byrne Memorial Justice Assistance Grant program, which made $255.7 million in funding available to states, territories, and localities in fiscal year 2015. However, one limitation is that these data may underestimate the scope of the problem, since acts of sexual violence are historically underreported to authorities.

As such, obtaining information through a survey may identify more instances of sexual violence than efforts that rely on information reported to authorities. However, surveys also have their limitations. Surveys are subject to variable response rates over time, and different surveys may have different response rates, which may affect the resulting estimates and the validity of the data. For example, response rates of the data collection efforts included in our review range from 24 percent for WGRA in 2012 to 33 percent for NISVS in 2011 to 84 percent for NCVS in 2014. Survey results may also be subject to response biases, for example the tendency for a respondent to provide untruthful but socially acceptable responses or the tendency for individuals who either have or have not experienced the action (e.g., sexual violence) to not participate in the survey (which can lead to nonresponse bias). Also, according to BJS officials, obtaining information directly from victims creates a burden on survey respondents and interview subjects, and research and officials from an entity that uses federal data on sexual violence stated that such surveys may face difficulties in getting people to discuss victimization experiences with strangers during interviews. Furthermore, administrative costs associated with surveys and interviews can affect the practical frequency of data collection.

Units of Measurement

Different federal data collection efforts measure and report different aspects of the occurrence of sexual violence. Some data collection efforts report the number of incidents that involved an act of sexual violence, some report the number of unique victims of sexual violence, and some report information about the number of times an act of sexual violence occurred. On the surface it may appear that the number of incidents that involve an act of sexual violence and the number of times an act of sexual violence occurred are synonymous, but that is not necessarily the case. Multiple offenses could occur in the same incident, and one incident could involve multiple victims. For example, a perpetrator could both rob and sexually assault someone in the same incident, or a perpetrator could carry out sexually violent acts against multiple victims in the same incident.

Table 3 outlines the units of measurement that each data collection effort reports.

The 4 data collection efforts that report the number of times an act of sexual violence occurred—NCVS, NIS, UCR-SRS, and UCR-NIBRS— may not record the exact number of acts of sexual violence from each individual incident in some cases. These data collection efforts count one act of sexual violence per victim per incident, meaning they would capture acts against multiple victims in a single incident but not multiple acts against each individual victim.

The data source each federal data collection effort uses is an important determinant of the units of measurement that each study uses in reporting its results. Data collection efforts that use information reported to authorities generally measure the number of incidents or reports that involve sexual violence. Those efforts may or may not publish a count of the number of acts of sexual violence that occurred. Such efforts are generally not set up to count the number of unique victims across incidents; they may report a count of victims, but have no mechanism to avoid the double-counting of victims who experience multiple incidents and are thus not measuring the same quantity that surveys that seek to report a number of unique victims measure. By contrast, data collection efforts that use information obtained from victim surveys generally measure and report the number of victims rather than the number of incidents that involved sexual violence. These efforts also collect data on the number of separate times each individual respondent has been a victim of sexual violence. However, the agencies operating those studies are cognizant of the challenges associated with asking respondents about multiple incidents, particularly with respect to the respondent’s ability to accurately recall multiple experiences of sexual violence. For example, a DOD official stated that WGRA asks questions about how many times a crime has been committed against the respondent, but DOD does not report a total number of times that an act of sexual violence occurred because respondents’ recall of multiple events may be subject to memory biases and confirming details of each act may be overly burdensome in a survey.

Time Frames

Federal agencies collect sexual violence data for different periods of time and report the data at different frequencies. Table 13 in app. II outlines the time frames for each federal data collection effort.

The data collection efforts cover different intervals of time for which an act of sexual violence occurred. For instance, NISVS asks whether each respondent has experienced sexual violence during the previous 12 months and during the respondent’s lifetime. NCVS asks whether each respondent has experienced sexual violence during the previous 6 months. By contrast, the data collection efforts that compile reports from authorities may capture information at the point in time when the event was reported to those authorities.

Some data collection efforts release their results annually, whereas others do so less often. For example, NISVS collected data annually except in 2014 and issues reports periodically. NISVS most recently reported results to the public in 2014 using data collected in 2011. Most data collection efforts that compile reports from authorities release their results annually.

One expert we interviewed stated that the period of time for which data on sexual violence are collected and how often data are publicly reported affects each data collection effort’s results regarding the occurrence of sexual violence, an observation also found in academic literature and a nongovernmental report. For example, data on lifetime experience of sexual violence may yield larger numbers of rape and sexual assault than data on experiences of sexual violence in the last 6 or 12 months. According to officials from 2 entities that use federal data on sexual violence, data that are reported annually may be more useful for trend analysis than data reported less frequently.

Differences in Data Collection Efforts May Hinder the Understanding of the Occurrence of Sexual Violence; Agencies Have Taken Steps to Explain and Lessen Differences, but Efforts Have Been Fragmented and Limited in Scope

The differences across the data collection efforts may hinder understanding of the extent of sexual violence, and agencies have taken steps to clarify differences and harmonize the data collection efforts. However, these efforts have been fragmented and more could be done to increase understanding of the problem of sexual violence.

Differences in Data Collection Efforts Collectively May Hinder Understanding of the Extent of Sexual Violence

Collectively, the differences across federal data collection efforts lead to differing estimates of sexual violence, for example rape, in the United States, as shown in selected data collection efforts on the general population in table 4.

According to research and individuals we spoke with who are familiar with the data, differences in federal data on sexual violence may confuse the public. A National Academy of Sciences’ Panel that studied the incidence of rape in the United States reported in 2014 that the data collection efforts’ different purposes and methodologies produce different results, which creates confusion for the public, law enforcement, policymakers, researchers, and victim advocacy groups. Additionally, officials from four entities that use federal data told us that they believe the public does not understand data from federal sources on sexual violence. Officials from three entities that use federal data on sexual violence stated that they—and the media—cite a range of sources and may not always, or adequately, explain the details of the data collection efforts. In addition, the public may not take the time to understand the differences among the data collection efforts. For example, an official from one entity told us that the entity frequently uses the results from one particular data collection effort to educate the public on sexual violence, but the official was not aware of certain methodological details and limitations of the data. Also, as previously discussed, some data collection efforts’ measurements and definitions do not align and information on what is included in these measurements is not publicly available, which may lead to confusion for data users. Further, officials from the federal agencies and entities we spoke with that use federal data on sexual violence emphasized that the differences across the data collection efforts are such that the results are not comparable.

Officials we spoke with who use the data stated that differences in measurements, definitions, and methodology across the data collection efforts can lead to confusion. Officials at one entity stated that they found challenges in using federal data on sexual violence because varying measurements and definitions across the data collection efforts make it difficult to compare data. However, even in instances where the acts of sexual violence and contextual factors that are included in measurements and definitions are similar across data collection efforts, other differences create challenges. For example, officials at the National Center for Campus Public Safety stated that whereas the Clery Act data program and UCR-SRS use the same definition for rape, the methodologies are different—the Clery Act data program collects information on allegations made in “good faith,” whereas the UCR-SRS includes only information on incidents resulting in a police report—which results in different estimates of rape and may lead to confusion for users who try to compare the data.

Because there is wide variation in the results, entities that use federal data on sexual violence have a choice of which data to use, and entities reported using data that best suited their needs. For example, officials from one entity told us that they use NCVS data because it includes information on incidents not reported to the police and is user-friendly, and officials from another entity told us they use NCVS data because it has a larger sample size than other data collection efforts. Officials from another entity stated that they use NISVS because it includes the most expansive set of acts of sexual violence and contextual factors in its measurement of “rape” and estimates lifetime prevalence rates.

Agencies Have Taken Some Steps to Clarify Differences across Sexual Violence Data Collection Efforts, but Efforts to Harmonize the Data Collection Efforts Have Been Fragmented

Federal agencies have acknowledged that differences exist among data collection efforts, for example in terms of methodology, context, and data sources, which has led some agencies to take steps to identify and explain differences across the data collection efforts. In addition, some federal agencies have taken steps to lessen the differences among data collection efforts by focusing on harmonization—that is, coordination of practices to enhance data collection to achieve a shared goal. However, such efforts have been fragmented—that is, they are limited in scope and generally involve two data collection efforts at a time.

Two ongoing efforts are intended to clarify the differences across two data collection efforts:

BJS and FBI coauthored a statement describing the differences between UCR and NCVS. In 1995, BJS and FBI coauthored statements entitled “The Nation’s Two Crime Measures,” which describe similarities and differences between the UCR program and NCVS. The statement was updated in September 2014. BJS and FBI post similar but different versions of the statement on their websites. BJS’s statement provides a side-by-side description of FBI’s UCR program and BJS’s NCVS including, for example, information on historical background, data sources, and time frames for data collection and reporting. Both statements also include a section on comparing UCR and NCVS data, which describes the data collection efforts’ similarities (e.g., they both have somewhat similar subsets of serious crimes, such as rape, robbery, aggravated assault, burglary, theft, and motor vehicle theft ) and key differences (e.g., definitions of certain crimes). The statements conclude with a description of the two data collection efforts’ strengths (e.g., UCR provides data on the number of crimes reported to law enforcement and NCVS provides data on the number and types of crimes not reported to law enforcement).

CDC and BJS have discussed publishing a statement that compares sexual violence statistics in NISVS and NCVS. At a meeting in November 2015, CDC and BJS discussed coauthoring a statement about NISVS and NCVS that would describe the differences and similarities of the two data collection efforts.

There are five efforts underway or recently implemented that are intended to increase harmonization across the data collection efforts, including:

Education adopted FBI’s UCR-SRS definition of rape for use in the Clery Act data. Education, in its 2014 rule implementing the VAWA 2013 reauthorization, changed the definition of rape that is used in the Clery Act data to match UCR-SRS’ definition. The definition used by both UCR-SRS and the Clery Act data is “Penetration, no matter how slight, of the vagina or anus with any body part or object or oral penetration by a sex organ of another person, without the consent of the victim.” Education began collecting data using the new definition of rape for calendar year 2014.

BJS sponsored the National Academy of Sciences’ CNSTAT Panel on Estimating the Incidence of Rape and Sexual Assault in BJS Household Surveys. In March 2011, BJS charged the panel to “assess the quality and relevance of statistics on rape and sexual assault from NCVS and other surveys contracted for by other federal agencies as well as surveys conducted by private organizations,” examining issues such as the “legal definitions in use by the states for these crimes, best methods for representing the definitions in survey instruments so that their meaning is clear to respondents, and best methods for obtaining as complete reporting as possible of these crimes in surveys, including methods whereby respondents may report anonymously.” The panel, which was comprised of experts in research and sexual assault response, held five in-person meetings and issued 15 recommendations in a final report published in 2014. For example, the panel recommended that BJS’s definitions of sexual violence be expanded to include victimizations when the victim does not have the capacity to consent to the sexual actions of the offender and that this research be conducted in a coordinated manner because many of the issues to be investigated are interrelated. The panel also recommended that the survey questionnaire should have a neutral context, such as a health survey. BJS officials told us that some of their current work addresses some of the panel’s recommendations. For example, BJS is currently conducting a methodological comparison of NCVS with a public health approach that includes a five-city comparison study and consultation with CDC. The five-city comparison study involves an expanded scope of sexual violence, which includes questions regarding consent, and BJS is testing a range of behavior-specific questions. BJS officials told us that they are planning to issue a report on the progress of the project in spring 2016.

BJS and FBI commissioned a National Academy of Sciences’ CNSTAT Panel on Modernizing the Nation’s Crime Statistics. Commissioned in 2013, the panel will assess and make recommendations for the development of a modern set of crime measures in the United States and the best means for obtaining them. The review will focus, among other things, on full and accurate measurement of criminal victimization events and their attributes, considering types of crime (and their definitions), including the current scope of crime types covered by existing FBI and BJS data collections; gaps in knowledge of contemporary crime; development of international crime classification frameworks that should be considered in increasing international comparability; and the optimal scope of crime statistics to serve the needs of the full array of data users and stakeholders—-federal agencies, other law enforcement agencies, Congress, other actors in the justice system (such as the courts and corrections officials), researchers, and the general public. Panel membership includes academics in the field of criminal justice and statistics and stakeholders who use and provide the data that the government collects. According to the chair of the panel, as part of the first phase of work, the panel has developed an initial conceptualization for classifying all types of crime, including rape and sexual assault. The recommended classification and its justification appear in the panel’s first report, which was released in May 2016. The panel has also begun the second phase of its work, which will suggest the means for gathering data for the comprehensive crime classification, including information from non-BJS or FBI sources, and recommend how crime data collection should proceed in practice. The panel plans to consider possible coordination among agencies to produce more comprehensive reports on data, instead of one agency producing one report and another agency producing a separate but related report. The panel intends to finish its second phase of work in 2016 and issue a final report in early 2017.

CDC, partnering with BJS, plans to convene a Technical Expert Panel to examine ways to improve NISVS. As part of OMB’s review of a CDC information collection proposal, OMB requested that CDC and BJS officials convene a panel of experts in survey methods to improve NISVS’s methodology, including increasing the response rate and minimizing non-response bias. The agencies identified and recruited a panel of experts and plan to meet in spring 2016.

CDC provided DOD with an adapted dataset from NISVS. In 2010, NISVS included two random samples of active duty women and wives of active duty men in addition to a random sample of the general U.S. population. Using identical survey methods, data were collected in the first two quarters of 2010. According to DOD officials, CDC provided a subset of NISVS data to DOD so that DOD received information on crimes that fell under military law to enable DOD to do an “apples to apples” comparison of CDC and DOD data. CDC officials informed us that they are working on a follow-up military population study in 2016.

However, these various federal efforts to clarify and harmonize sexual violence data have been fragmented. CNSTAT’s Principles and Practices for a Federal Statistical Agency (2013) calls for federal agencies that produce similar federal statistics, with different missions, to coordinate and collaborate with each other to meet current information needs and provide new or more useful data than a single system can provide. While the guidance applies primarily to the 13 federal statistical agencies, OMB officials stated that the report provides best practices for all federal data collection activities. The guidance encourages collaborative interagency efforts and highlights the importance of agencies developing standard definitions as a way to maximize the value and comparability of data.

However, the coordination that has occurred across the agencies that collect data on sexual violence has been limited. Specifically, coordination has been bilateral—generally involving only 2 of the 10 data collection efforts at a time and limited in scope. Agency officials expressed skepticism that broader harmonization efforts could benefit federal data on sexual violence, stating that each data collection effort is designed— through target population, measurements and definitions, and methodology—to fulfill a certain purpose. Agency officials told us that changes to any of these differences may undermine the specific purpose of each data collection effort. However, harmonization does not necessarily entail making the data collection efforts identical; instead, it could entail agencies considering how they could make their efforts more complementary and, as appropriate, more alike without compromising their programmatic needs.

The Paperwork Reduction Act (PRA), among other things, established a process for OMB to oversee agency information collection efforts in order to improve the quality and use of federal information while reducing collection burdens, including through the coordination of federal statistics. Per its authority under the PRA, OMB has convened interagency working groups to assess differences across data collection efforts and determine which of those differences are beneficial and which are unnecessary. For example, OMB has convened the following interagency groups:

The Interagency Working Group for Research on Race and Ethnicity was formed in 2014 to exchange research findings, identify implementation issues, and collaborate on a shared research agenda to improve federal statistics on race and ethnicity.

The Interagency Working Group on Measuring Relationships in Federal Household Surveys, which was established in 2010, convenes representatives from a variety of federal agencies involved in the collection, dissemination, or use of household relationship data to address the challenges in measuring household relationships, including same sex couples.

The Federal Interagency Forum on Child and Family Statistics was formally established in 1997 to develop priorities for collecting enhanced data on children and youth, improve the reporting and dissemination of information on the status of children to the policy community and the general public, and produce more complete data on children at the state and local levels.

We asked OMB if there are plans to convene a similar group for harmonizing data on sexual violence. OMB staff stated that they did not have plans to form an interagency group on the topic, but instead they plan to invest limited resources strategically by engaging with BJS on its redesign of NCVS and with CDC on its information quality of NISVS. OMB staff also stated that their plans are to ensure that both agencies are taking advantage of the insights gained as each agency undergoes redesign and technical consultations in the next couple of years. However, other data collection efforts, in addition to NCVS and NISVS, also influence policy decisions on sexual violence. Depending upon the outcome of the work being conducted by BJS and CDC, OMB may encourage other data collection efforts to adapt or adopt insights gained as appropriate to their respective programmatic missions. In the absence of broader harmonization efforts, agency sexual violence data continue to be inconsistent and incomparable, leading to confusion about the data and lack of clarity about the scope of the problem of sexual violence in the United States.

Conclusions

Differences in data collection efforts—particularly in terms of what is included in measurements and definitions of sexual violence and methodologies—collectively can lead to confusion. Without publicly- available information on which acts of sexual violence and contextual factors are included in the measurements of sexual violence, data users may lack clarity about what each data collection effort’s results represent. Additionally, entities that use federal data may misunderstand the data and develop policies that may not be based on the full extent of the problem. In the absence of collaboration among agencies that manage data collection efforts, it is unclear which differences enhance and which impair the overall understanding of sexual violence, and as a result, policy makers and the public lack coordinated information by which to address the problem.

Recommendations for Executive Action

To enhance the clarity and transparency of sexual violence data that is reported to the public, we recommend that the Secretary of Education direct the Assistant Secretary for the Office of Postsecondary Education, the Secretary of Health and Human Services direct the Director of CDC, and the Attorney General direct the Director of BJS to make information on the acts of sexual violence and contextual factors that are included in their measurements of sexual violence publicly available. This effort could entail revising their definitions of key terms used to describe sexual violence so that the definitions match the measurements of sexual violence.

To help lessen confusion among the public and policy makers regarding federal data on sexual violence, we recommend that the Director of OMB establish a federal interagency forum on sexual violence statistics. The forum should consider the broad range of differences across the data collection efforts to assess which differences enhance or hinder the overall understanding of sexual violence in the United States.

Agency Comments and Our Evaluation

We provided a copy of our report to DOD, Education, HHS, DOJ and OMB for their review and comment. The agencies provided technical comments, which we incorporated as appropriate. Education, HHS, and DOJ also provided written comments, which are reprinted in appendices IV, V, and VI, respectively. The OMB liaison to GAO provided us with comments via email, which are summarized below.

DOJ, Education, and HHS agreed with our recommendation that, in order to enhance the clarity and transparency of sexual violence data, they should make information on the acts of sexual violence and contextual factors that are included in their measurements of sexual violence publicly available. In their written comments, DOJ and Education described actions they have recently taken or plan to take to implement the recommendation. DOJ stated that beginning in calendar year 2017, BJS will provide the exact computer code used to construct its measures of sexual violence as well as additional information on how sexual violence is defined and measured. Education stated that in June 2016, the department released an updated version of The Handbook for Campus Safety and Security Reporting, which provides additional details on what acts of sexual violence and contextual factors are included in the data collection effort’s sexual violence measurements. HHS stated that the department is committed to improving the quality of the data and the clarity of the descriptions and definitions of sexual violence.

In an email responding to our recommendation that OMB establish a federal interagency forum on sexual violence statistics, OMB stated that it did not believe convening a forum at this time was the most strategic use of resources. OMB stated that other interagency groups it has convened were typically about statistical methods or measurement issues that would affect a wide swath of government and for which OMB guidance or a best practice working paper would be forthcoming. OMB noted that there are only four agencies involved in collecting sexual violence data, and regarded none to be conducting or far enough along in its research for OMB to develop guidance or identify best practices at this time. OMB does, however, plan to follow closely and participate in CDC’s and BJS’s ongoing technical work, and will consider convening or sharing information across agencies when that work is further along. We understand the importance of allowing time for a data collection effort to mature before providing guidance or best practices. However, considering that 7 of the 10 data collection efforts have been in place for more than 10 years, and several have been in place for multiple decades, we disagree with OMB’s assertion that none of the data collection efforts are far enough along for OMB to provide guidance and best practices. DOJ and Education also commented on the recommendation to OMB. DOJ stated that BJS welcomes OMB efforts to coordinate data collection and reporting on sexual violence and stands ready to participate in an interagency forum. Education stated that efforts to “harmonize” definitions should not be pursued solely to achieve symmetry for its own sake, but that the focus should be on the needs of each individual program. We agree that the data collection efforts should continue to meet the needs of individual agencies; however, considering the number of federal data collection efforts, the range of differences across them, and the potential for causing confusion, it would be beneficial for agencies to discuss these differences and determine whether they are, in fact, necessary.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Director of OMB, the Attorney General, the Secretaries of Defense, Education, and Health and Human Services, and other interested parties. The report will also be available at no charge on the GAO web site at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-8777 or goodwing@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII.

Appendix I: Objectives, Scope and Methodology

Our objectives for this report were to address the following questions: (1) What are the federal efforts underway to collect data on sexual violence, and how, if at all, do these efforts differ? (2) How do any differences across the data collection efforts affect the understanding of sexual violence, and to what extent are federal agencies addressing any challenges posed by the differences?

To address the first question, we identified federal efforts to collect data on sexual violence, for which the data: provided information on the extent to which acts of sexual violence occur in the United States in a particular year (for example, the number of times a rape or sexual assault has occurred or the number of victims of rape and sexual assault); were collected recently (i.e., 2010 or after); were collected periodically (i.e., at least once every 2 years); were reported publicly; and were not focused primarily on minors.

To identify efforts that met these criteria, we reviewed past GAO reports and a Federal Bureau of Investigation (FBI) list of federal agencies that may collect crime data. We asked officials at those agencies if they had any data collection efforts that met our criteria. Additionally, we asked experts in the field (for example, academic researchers) and officials from victim advocacy groups and other special interest groups about any additional federal data collection efforts they were aware of that met our selection criteria. We initially identified experts and entities that use federal data on sexual violence by conducting background research. Then, we interviewed experts and officials from the entities identified in our research, and from these contacts we identified additional entities that use federal data on sexual violence. In all, we spoke with officials from three victim advocacy groups, five other special interest groups (for example, law enforcement associations and campus safety groups), three other federal agencies, and two academic experts. Based on this work, we identified 10 data collection efforts that met our criteria across four federal agencies, including the Departments of Defense, Education, Health and Human Services, and Justice.

To identify and describe differences across the data collection efforts, we obtained information on the purpose, scope, and methodology of each data collection effort. We obtained this information through a review of documents, such as user manuals and program descriptions. We also conducted interviews with senior agency officials, senior officials at entities that use federal data on sexual violence, and academic experts. Using documentary and testimonial information, we compared the similarities and differences of the data collection efforts with respect to target population; context in which data were collected; source of the data; unit of measurement; time frames (for example, when data are collected and how often data are reported by the federal agency); and terminology and measurements of sexual violence. To compare similarities and differences of terminology of sexual violence, we used agency documents to identify for each data collection effort the terms used to describe sexual violence. To identify for each data collection effort what acts of sexual violence and contextual factors are included in measurements of sexual violence, we reviewed agency documentation and interviewed agency officials.

To identify how the differences affect understanding of sexual violence, we obtained and reviewed federal reports and interviewed and reviewed relevant documentation from agency officials, experts, and officials from entities that use federal data on sexual violence. We asked these officials and experts whether, in their experience, any of the differences made it difficult for people who may use the data (e.g., Congress, policy makers, academics, the general public) to understand the extent to which sexual violence occurs in the United States. We also asked them to identify any difficulties or challenges, of which they were aware, that have resulted from the differences across federal efforts to collect data on sexual violence. Because these officials and experts were not selected as a representative sample, the information obtained from these interviews applies solely to this set of officials and experts, and cannot be generalized to others.

We also reviewed articles, conference papers, and government and nongovernment reports that discuss differences across federal sexual violence data collection efforts. To identify articles, a research librarian conducted a search of several bibliographic databases, such as ProQuest, Embase, and Scopus, using terms such as “rape data” or “sexual assault statistics,” among others. The search looked for peer- reviewed articles, books, and conference papers published during or after 2005. This search yielded 36 publications, 16 of which were relevant to our research objective on the impact of the differences across the data collection efforts and 20 which were not. In reviewing the identified publications, we found and reviewed an additional 9 articles and reports that were pertinent. See app. III for a list of articles and reports that we reviewed and determined to be relevant for our analysis. The librarian- assisted literature search was conducted in August 2015, and we reviewed literature from that search and identified additional sources from August 2015 to May 2016.

To describe the extent to which federal agencies are addressing any challenges posed by differences across the data collection efforts, we interviewed senior agency officials and academic experts and obtained relevant documentation. We asked agency officials about what, if any, steps their agency has taken, or planned to take, to address some of the difficulties or challenges that may have resulted from the differences across the data collection efforts. We also asked agency officials as well as officials from entities that use federal data on sexual violence if they were aware of any additional steps being taken by other federal agencies, state agencies, or other national entities, etc., to address some of the difficulties or challenges that may have resulted from the differences across the data collection efforts.

We conducted this performance audit from March 2015 to July 2016, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Descriptive Tables of Federal Data Collection Efforts

Tables 5 through 10 provide information on the acts of sexual violence and the contextual factors included in the measurements of sexual violence by federal data collection efforts identified by GAO.

Appendix III: List of Relevant Articles and Reports

National Academy of Sciences, Committee on National Statistics, Estimating the Incidence of Rape and Sexual Assault. Washington, DC, US: National Academies Press, Washington, DC, 2014. . “Overcoming Challenges Related to Data Collection and Measurement.” Forced Migration Review, vol. 27, no. Jan (2007): 28-29,.

Addington, L. and C. Rennison. “Rape Co-Occurrence: Do Additional Crimes Affect Victim Reporting and Police Clearance of Rape?” Journal of Quantitative Criminology, vol. 24, no. 2 (2008): 205-226.

Bachman, R. “Measuring Rape and Sexual Assault: Successive Approximations to Consensus.” A paper commissioned for the National Academy of Sciences, June 6, 2012.

Campbell, R, A. E. Adams, and D. Patterson. “Methodological Challenges of Collecting Evaluation Data from Traumatized Clients/Consumers - A Comparison of Three Methods.” American Journal of Evaluation, vol. 29, no. 3 (2008): 369-381.

Chen, Y. and S. E. Ullman. “Women’s Reporting of Sexual and Physical Assaults to Police in the National Violence Against Women Survey.” Violence Against Women, vol. 16, no. 3 (2010): 262-79.

Clay-Warner, J. and J. McMahon-Howard. “Rape Reporting: ‘Classic Rape’ and the Behavior of Law.” Violence and Victims, vol. 24, no. 6 (2009): 723-743.

Cohen, M.A. and A. R. Piquero. “New Evidence on the Monetary Value of Saving a High Risk Youth,” Journal of Quantitative Criminology, vol. 25 (2008).

Cohn, A. M., H. M. Zinzow, H. S. Resnick, and D. G. Kilpatrick. “Correlates of Reasons for Not Reporting Rape to Police: Results from a National Telephone Household Probability Sample of Women with Forcible Or Drug-Or-Alcohol Facilitated/Incapacitated Rape.” Journal of Interpersonal Violence, vol. 28, no. 3 (2013): 455-473.

Cook, S. L., C. A. Gidycz, M. P. Koss, and M. Murphy. “Emerging Issues in the Measurement of Rape Victimization.” Violence Against Women, vol. 17, no. 2 (2011): 201-18.

Delisi, M., A. Koloski, M. Sweeny, E. Hachmeister, M. Moore, A. Drury. “Murder by Numbers: Monetary Costs Imposed by a Sample of Homicide Offenders,” The Journal of Forensic Psychiatry & Psychology, vol.21, no. 4 (2010).

Fang, H., M. T. French, and K. E. McCollister. “The Cost of Crime to Society: New Crime-Specific Estimates for Policy and Program Evaluation.” Drug Alcohol Dependence, vol. 108, no.1-2 (2010).

Du Mont, J., K. Miller, and T.L. Myhr, “The Role of “Real Rape” and “Real Victim” Stereotypes in the Police Reporting Practices of Sexually Assaulted Women.” Violence Against Women, vol. 9, no. 466 (2003).

Fisher, B. S. “The Effects of Survey Question Wording on Rape Estimates: Evidence from a Quasi-Experimental Design.” Violence Against Women, vol. 15, no. 2 (2009): 133-47.

Gardella, J. H., C. A. Nichols-Hadeed, J. M. Mastrocinque, J. T. Stone, C. A. Coates, C. J. Sly, and C. Cerulli. “Beyond Clery Act Statistics: A Closer Look at College Victimization Based on Self-Report Data.” Journal of Interpersonal Violence, vol. 30, no. 4 (2015).

Lynch, J. P. “Clarifying Divergent Estimates of Rape from Two National Surveys.” The Public Opinion Quarterly, vol. 60, no. 3 (1996) 410-430.

Lynch, J. P. and J. L. Lauritsen. “Modernizing the Nation’s Crime Statistics.” The Criminologist, vol. 40, no. 2 (2015).

Miller, T.R., M. A. Cohen, and B. Wiersema, “Victim Costs and Consequences: A New Look.” A final summary report presented to the National Institute of Justice, January 1996.

Murphy, S. B., K. M. Edwards, S. Bennett, S. J. Bibeau, and J. Sichelstiel. “Police Reporting Practices for Sexual Assault Cases in which ‘the Victim does Not Wish to Pursue Charges’.” Journal of Interpersonal Violence, vol. 29, no. 1 (2014): 144-156.

Palermo, T. and A. Peterman. “Undercounting, Overcounting and the Longevity of Flawed Estimates: Statistics on Sexual Violence in Conflict.” Bulletin of the World Health Organization, vol. 89, no. 12 (2011): 924-925.

Saltzman, L. E., K. C. Basile, R. R. Mahendra, M. Steenkamp, E. Ingram, and R. Ikeda. “National Estimates of Sexual Violence Treated in Emergency Departments.” Annals of Emergency Medicine, vol. 49, no. 2 (2007): 210-7.

Simon, T. R., M. Kresnow, and R. M. Bossarte. “Self-Reports of Violent Victimization among U.S. Adults.” Violence and Victims, vol. 23, no. 6 (2008): 711-726.

Violence Against Women: A Statistical Overview, Challenges and Gaps in Data Collection and Methodology and Approaches for Overcoming Them. A report of the expert group meeting organized by the UN Division for the Advancement of Women in collaboration with the Economic Commission for Europe and World Health Organization, April 11-14, 2005.

Weiss, K. G. “‘You Just Don’t Report that Kind of Stuff’: Investigating Teens’ Ambivalence Toward Peer-Perpetrated, Unwanted Sexual Incidents.” Violence and Victims, vol. 28, no. 2 (2013): 288-302.

Wolitzky-Taylor, K. B., H. S. Resnick, J. L. McCauley, A. B. Amstadter, D. G. Kilpatrick, and K. J. Ruggiero. “Is Reporting of Rape on the Rise? A Comparison of Women with Reported Versus Unreported Rape Experiences in the National Women’s Study-Replication.” Journal of Interpersonal Violence, vol. 26, no. 4 (2011): 807-832.

Appendix IV: Comments from the Department of Education

Appendix V: Comments from the Department of Health and Human Services

Appendix VI: Comments from the Department of Justice

Appendix VII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making key contributions to this report were Kristy Love, Assistant Director; Meghan Squires, Analyst-in-Charge; Tim Young; Kirsten Leikem; Janelle House; David Alexander; and David Plocher. Diana Maurer, Tom Jessor, Janet Temko-Blinder, Tovah Rom and Eric Hauswirth also provided valuable assistance.