Higher Education: Approaches and Strategies Used in College Campus Surveys on Sexual Violence
Released: 2020-05-21T00:00:00

Letter

Background

Campus Climate Surveys on Sexual Violence

Campus climate surveys on sexual violence are designed to collect information on the incidence and characteristics of sexual violence on college campuses as well as related student attitudes and behaviors. The topics covered by campus climate surveys can vary, depending on the questions included on the survey instrument. For example, these surveys may include questions about incidents of sexual violence, such as the number of incidents of sexual assault, intimate partner violence, or stalking, among other topics. There are two different methods that colleges can use to administer these surveys: In a census survey, all members of a group, such as the student body of a college, are surveyed. This type of survey can be used when the group that is the focus of the survey is small, when substantial resources and time are available to obtain enough responses to the survey, or when there is reason to provide all members of the group the opportunity to participate.

In a sample survey, a portion of the group is selected using statistical methods to provide accurate information about the larger group. Administering a survey to a sample of students can reduce the time and resources needed to obtain enough survey responses to produce accurate data. Sample-based surveys are appropriate when it is not practical or desirable to survey every member of a group.

With both sample and census-based surveys, collecting data that accurately represents the experiences of respondents requires taking a variety of steps when designing, administering, and analyzing the survey, such as weighting or analyzing the completed responses to ensure that they represent the larger group.

A 2014 White House Task Force report recommended conducting these surveys as an initial step in a college’s plan to address campus sexual assault. The report also suggested follow-up actions for colleges to consider, such as providing training for college officials and creating partnerships with community sexual assault support services. In addition, there have been efforts to compare campus climate survey results across colleges.

Some states have also enacted laws requiring colleges in their state to administer campus climate surveys. These state laws may vary in the nature of the survey requirements, such as the types of colleges covered by these requirements and whether a particular survey instrument must be used. For example, Louisiana requires public colleges in the state to administer these surveys, while New York requires all colleges located within the state to do so. Additionally, Louisiana requires that schools use a standard survey instrument developed by the state, while New York allows colleges to select their own survey instrument. However, there is currently no federal requirement for colleges to conduct campus climate surveys on sexual violence.

Federal Efforts Related to Addressing Campus Sexual Violence

Education, Justice, and HHS currently engage in a variety of efforts to address sexual violence on college campuses, including overseeing relevant federal laws and funding prevention and response activities. Education and Justice oversee colleges’ compliance with Title IX of the Education Amendments of 1972 (Title IX), which prohibits discrimination on the basis of sex in any education program or activity that receives federal financial assistance. Title IX prohibits sex discrimination— including sexual harassment and sexual violence—that effectively denies victims equal access to recipients’ educational opportunities or benefits. Under Education’s regulations, colleges receiving federal financial assistance from Education, such as those participating in federal student aid programs, must establish procedures for resolving Title IX complaints, and take steps to ensure that members of the college community are aware of their rights under Title IX. In addition, these colleges must designate at least one employee to coordinate their efforts to comply with and carry out their responsibilities under Title IX. According to Education guidance, the Title IX coordinator is responsible for coordinating the college’s response to all complaints involving possible sex discrimination, including monitoring outcomes, identifying and addressing any patterns, and assessing effects on the campus climate.

Education also oversees the Clery Act, which requires colleges that participate in student financial assistance programs under Title IV of the Higher Education Act, as amended, to collect statistics on certain crimes that occur on or near their campuses, including specified sex offenses, publish those statistics in an annual security report, and annually report them to Education. Colleges must also include a policy statement in their annual security reports describing their sexual violence prevention and awareness programs for students and employees. In addition, Justice and HHS have funded grants for campus sexual violence prevention and response efforts. HHS has also developed a technical assistance document for planning and implementing sexual violence prevention strategies on college campuses.

Federal Data Sources Related to Campus Sexual Violence

Education and Justice also manage key efforts to collect data related to campus sexual violence (see table 1). For example, Education oversees the Campus Safety and Security Survey, which collects information from colleges that participate in student financial aid programs on reported criminal incidents, including specified sex offenses, which occur on or near campuses that the colleges own or control, as required by the Clery Act. Colleges are required to include data on specified crimes that are reported to local police or campus security authorities and that occurred (1) on campus (including the subset of crimes that occurred in on-campus student housing facilities), (2) on public property within or immediately adjacent to campus, and (3) in or on non-campus buildings or property the college owns or controls. The survey collects data on the following offenses related to sexual violence: rape, fondling, incest, statutory rape, domestic violence, dating violence, and stalking. Education publishes the data on a public website.

Justice collects data on crimes, including sex crimes, through the Bureau of Justice Statistics’ National Crime Victimization Survey (NCVS). The NCVS captures data on a range of offenses related to sexual violence: completed rape, attempted rape, threatened rape, sexual assault other than rape or attempted rape, unwanted sexual contact with or without force (e.g., grabbing, fondling), verbal threat of sexual assault other than rape, and stalking. The NCVS collects data through in-person interviews and phone calls with a nationally representative sample of households on the frequency, characteristics, and consequences of criminal victimization in the United States. In particular, the NCVS collects information about crimes reported and not reported to the police.

Although the NCVS includes certain group residences, such as college residence halls, in its sample of households, the resulting data may not fully represent the sexual victimization experiences of college students residing on campus because the sample is primarily comprised of households. Research has found that individuals living in group residences may be at higher risk of sexual violence. Research has also noted concerns with how the NCVS is administered. Specifically, interviews are conducted in person at respondents’ homes or over the phone. As a result, victims may be less likely to honestly answer sensitive questions, such as those related to sexual violence, as their responses might be overheard by other members of their household or the offender.

Stakeholders Said Campus Climate Surveys Provide Insights into Campus Sexual Violence, but Colleges Face Challenges Administering Them

Stakeholders we interviewed, including survey developers, other researchers, and federal, state, and college officials, considered campus climate surveys a useful tool for learning more about the incidence of campus sexual violence and identifying areas for improvement to address it. However, stakeholders also noted that colleges face a variety of challenges with developing and conducting surveys, such as limited access to needed survey expertise and low response rates, which can affect the reliability of campus climate survey results.

Campus Climate Surveys Can Provide More Information about Students’ Experiences with Sexual Violence and Their Awareness of Related Policies, Procedures, and Services

Surveys Can Provide More Comprehensive Information on Incidents of Sexual Violence

Nearly all stakeholders said that campus climate surveys provide an opportunity to learn more about the incidents of sexual violence occurring on individual campuses, such as those that students may not have previously reported to campus authorities or law enforcement. According to Justice officials and one researcher, campus climate surveys, which collect data directly from victims, can help overcome limitations in law enforcement data that rely on victims reporting to authorities (see sidebar). For example, the three campus climate surveys we reviewed are designed to capture information on incidents of sexual violence that students have experienced regardless of whether the incidents were previously reported to campus authorities or law enforcement.

Underreporting of Traditional Crime Statistics According to a National Research Council panel, rape and sexual assault are generally underreported to law enforcement, which can affect traditional crime statistics for these incidents. For example, according to a 2014 Department of Justice report, National Crime Victimization Survey (NCVS) data showed the rate of rape and sexual assault for female college students was 6.1 per 1,000 (the 95 percent confidence interval ranges from 5.0 to 7.2 percent) for the period 1995–2013, and an estimated 80 percent of rape and sexual assault incidents went unreported to police (the 95 percent confidence interval ranges from 75 to 85 percent). College students responding to the NCVS who indicated they did not report incidents of rape and sexual assault to police cited a variety of reasons, such as considering the assault to be a personal matter, fear of reprisal, or not considering the victimization important enough to report. Similarly, Clery Act data are based on reports made to campus security authorities and law enforcement. According to one federally funded pilot study, data from campus climate surveys at nine colleges suggested that the majority of rapes are not represented in a college’s Clery numbers.

In contrast, Clery Act data collected through Education’s Campus Safety and Security Survey provides information only on incidents that are reported to campus security authorities or law enforcement and that occurred on or near campuses that the colleges own or control. This can result in campus climate surveys identifying a larger number of campus sexual violence incidents than federal Clery Act data. For instance, a pilot study of campus climate surveys at nine colleges found that undergraduate students attending these colleges experienced an estimated 2,380 incidents of rape during the 2014-2015 academic year, of which an estimated 770 occurred on campus. In contrast, Clery Act data documented 40 reported rape incidents for these colleges during the 2014 calendar year.

Several stakeholders we spoke with also said that campus climate surveys can provide information on a broader range of sexual violence incidents than federal crime statistics data, such as the National Crime Victimization Survey and Clery Act data, which collect information specifically on criminal offenses. For example, one researcher we spoke with noted that campus climate surveys can collect information about sexual harassment, which is not included in federal Clery Act crime statistics. The three surveys we reviewed collect information on a range of sexual violence incidents, including sexual assault, coerced sexual contact, stalking, intimate partner violence, and sexual harassment.

Behavioral Questions Campus climate surveys on sexual violence ask students about a variety of topics that are often sensitive. One challenge of conducting these surveys is that students’ understanding of what behaviors are considered “sexual assault” or “rape” may differ and the words used to describe sexual violence will determine what is measured by the survey, such as incidents of rape. Research has found, for example, that surveys asking directly whether students experienced specific types of sexual violence can produce inaccurate data. To improve the quality of information collected by campus climate surveys, researchers ask students about specific behaviors and events that describe the incident rather than referring to it using a label such as “sexual assault” or “rape.” campus climate survey instruments we reviewed included questions that asked for additional context on incidents of sexual violence reported by students, such as the victim’s relationship to the perpetrator.

Researchers we interviewed also noted that campus climate surveys can include behaviorally specific questions to identify conduct that survey respondents might not categorize as sexual violence (see sidebar). Each of the three surveys we reviewed used behaviorally specific questions to describe behaviors that may constitute sexual violence for survey respondents, without using specific terms, such as rape. For example:

One survey we reviewed asks, “Since the beginning of the current academic year, has an intimate partner threatened to hurt you and you thought you might really get hurt?” instead of asking whether the respondent has experienced “intimate partner violence.”

Another survey we reviewed asks, “How many times have one or more people left you unwanted messages (including text or voice messages)?” instead of asking if the respondent has experienced “stalking.”

Seven of the nine researchers we spoke with considered behavioral questions to be a best practice for collecting data on sexual violence, including one that noted the general public may not be aware of the definitions of rape or other types of unwanted sexual contact or behaviors. However, one researcher we spoke with expressed concern that the wording of behavioral questions can be imprecise.

Surveys May Help Colleges Identify Specific Areas for Improvement

Each of the three campus climate surveys we reviewed included questions regarding student knowledge of the administering college’s policies and resources related to preventing and responding to sexual violence on campus. According to nearly all of the stakeholders we interviewed, these data can help colleges identify areas for improvement. In particular, about half of these stakeholders noted that campus climate survey results can help colleges address barriers to reporting. For example, officials from one college we spoke with reported increasing their efforts to educate students about where to go if they experienced sexual assault based on gaps in awareness identified through survey results. Further, information from campus climate surveys also helped one state identify how it could better assist colleges, such as by providing training on intimate partner violence, according to a state official.

Several stakeholders reported that campus climate surveys may also help colleges assess their performance on reducing sexual violence. For example, two researchers said that these surveys can help colleges see where improvements were made and where additional action might be needed. Another researcher we spoke with noted that colleges are very interested in using campus climate surveys to establish baseline data and are beginning to understand the usefulness of having data on sexual violence prevalence.

Colleges Face Challenges Administering Surveys and Analyzing Results

While campus climate surveys can provide additional information on campus sexual violence, stakeholders reported that colleges face a variety of challenges with developing and conducting surveys, as well as analyzing the results.

Surveys Can Be Costly and Require Technical Expertise to Administer

Although some survey instruments are free, about half of the stakeholders we interviewed offered that some institutions, particularly smaller colleges, may not have the resources to effectively administer surveys on their own. Stakeholders cited costs associated with hiring contractors or relying on faculty and staff to administer and analyze the results of a survey. For example, one researcher we spoke with said that administering a survey can require having people available to respond to student questions about the survey. Officials from one college we spoke with said they relied on faculty volunteers to analyze survey results over a school break, due to a limited survey budget. About half of the stakeholders also noted that providing incentives to students can help increase survey response rates, yet incentives can also be the most expensive part of a college’s survey budget. For example, one college reported that the $10 incentives offered to students who completed the survey constituted the college’s largest survey expense. Given the potentially high costs of these surveys, officials in one state we spoke with reported that the state provided funding to help its colleges administer surveys, analyze results, prepare reports, and translate the survey results into action.

About half of the stakeholders also reported that some colleges may not have technical expertise readily available to conduct a campus climate survey on sexual violence. As previously noted, colleges can administer campus climate surveys to a sample of students (sample approach) or all students at a college (census approach). According to federal guidance, a sample approach can reduce the amount of follow-up needed to encourage survey completion; however, expertise is needed to create a sampling frame that includes all, or nearly all, of a target population, and then to accurately select a sample of that population to survey that still represents the target population. Several stakeholders said colleges may face challenges in creating a representative sample of their students, in particular. For example:

One researcher noted that colleges may not collect sufficient demographic data or have adequate funding to create a representative sample of their students.

Another researcher observed that for both sample and census surveys, colleges may also lack the expertise to ensure, through statistical methods such as non-response bias analysis or weighting responses, that respondents are representative of the student body.

Justice officials noted that properly administering campus climate surveys requires personnel with adequate statistical expertise, as well as support from college administration. Research shows that statistical methods like testing for non-response bias and weighting responses are an important consideration when developing estimates on prevalence since non- response bias can potentially limit the extent to which the results can be generalized to the entire student population. One college that had not conducted a campus climate survey also noted that doing so would be a challenge due to limited expertise with conducting surveys on sensitive topics, such as sexual violence. However, two of the selected colleges that conducted campus climate surveys reported working with a third party to ensure more reliable results.

Surveys May Not Yield Reliable Results Due to Low Response Rates

Response rates are a key factor in producing reliable survey results, and most stakeholders reported that obtaining a sufficient number of responses from students can be a challenge. Achieving a sufficient response rate can help ensure that the survey results are representative of the target population, so that the results can be used with confidence to inform decisions. However, our prior work on federal sexual violence data found surveys are subject to variable response rates over time, and different surveys may have different response rates, which may affect the resulting estimates and the validity of the data. The seven selected colleges that conducted surveys reported response rates ranging from less than 10 percent to more than 60 percent. Additionally, officials we interviewed in two of the selected states reported that their survey response rates were not high enough to generalize or draw meaningful conclusions regarding campus sexual violence, as originally intended. Officials in these states said they primarily included the data in required state reports, with limitations noted as needed.

Most stakeholders noted that survey design or administration factors can affect response rates. About half of the stakeholders noted that keeping the survey short is critical to ensuring more students complete it, but some topics of interest to the colleges may not be covered as a result. For example, at one college, officials included survey questions about sexual assault and sexual harassment, but did not pursue questions about stalking due to concerns about survey length. One researcher also told us that long and complicated surveys may not work well on smart phones, which is how many students take these surveys. As for survey administration, most stakeholders noted that choices on how to administer the survey can also affect the response rates for surveys. For example, two researchers we spoke with said that it is better to leave surveys open to respondents for a longer time period to increase the response rate. In addition, a researcher and one state official noted that technical issues can interfere with obtaining a high response rate, such as sending survey invitations to university email accounts that students may not check regularly.

Survey Results May Not Be Comparable Across Colleges

About half of the stakeholders stated that differences in survey instruments and methodology may make it difficult for colleges to compare their results with the results of other colleges.

Variation in questions and definitions. The surveys we reviewed varied in the wording used to ask respondents about their knowledge of institutional policies for reporting sexual violence. According to one researcher, differences in the wording of questions and structure of questionnaires can affect comparability across surveys. Officials in one state also reported that colleges used different definitions of key terms on their campus climate surveys, which made it challenging to reach general conclusions across colleges. Similarly, another researcher stated that differences in the definitions of terms used in colleges’ campus climate surveys make accurate comparisons difficult.

Variation in time periods. The surveys we reviewed ask respondents about incidents of sexual violence occurring over different time periods, which may also limit the comparability of survey results across colleges. According to one researcher we spoke with, colleges using different survey instruments should not compare prevalence estimates with results from other surveys that ask about incidents of sexual violence for different time periods. For example, one survey instrument we reviewed asks students about their experiences with sexual violence during the current academic year. In contrast, the two other survey instruments we reviewed ask students about their experiences with sexual violence since first enrolling at college, which covers a longer time period for seniors than first-year students. Time periods may also affect the accuracy of the data collected. One researcher we spoke with, for example, stated that survey questions that cover longer time periods can introduce bias, such as the telescoping effect, whereby respondents recall certain events as being more recent than they actually are. Additionally, longer time periods may yield larger numbers of incidents, since more individuals may experience the behavior over time.

To address these comparability challenges, some colleges have used the same survey instruments as other colleges. For example, two colleges included in our review participated in a survey effort among multiple colleges that was designed to allow for comparisons across participating schools. To make these comparisons, a third party administered the survey at participating colleges using a survey instrument with standardized questions and a standardized methodology to enable the measurement of prevalence, and then analyzed the results.

In summary, while all stakeholders noted the value of conducting campus climate surveys, about half of them generally cautioned against requiring colleges to administer them in light of the associated challenges previously discussed. Officials at one college that voluntarily conducted a campus climate survey using a one-time grant stated they would have to use funds from faculty and staff salaries if they were required to conduct a survey in the future. Additionally, an official from one college that had not conducted a campus climate survey noted that high turnover in the Title IX coordinator position would make it difficult for the college to sustain a survey effort over time. Further, another college that has not conducted a campus climate survey to examine the incidence of sexual violence noted it would be difficult to design a standard survey instrument that would apply across all colleges, such as those that primarily serve students who take courses online.

Selected Colleges Have Used Various Survey Design, Administration, and Outreach Strategies to Increase Understanding of Campus Sexual Violence

The seven selected colleges that conducted campus climate surveys used various survey design, administration, and outreach strategies to learn about the incidence of campus sexual violence. Most of these colleges also chose to publicly report some survey results.

Survey Design

Choosing a survey instrument. These seven colleges considered several factors when choosing a survey instrument:

Rigor. Officials from each of the seven colleges that conducted a campus climate survey said it was important to use a rigorous survey instrument, such as one that survey developers have validated or colleges have widely adopted. One college official explained that using a validated instrument provided assurances that helped secure a timely approval from the college’s institutional review board.

Flexibility. Officials from five colleges said they valued the flexibility of using a survey instrument that could be modified based on the specific characteristics and needs of their colleges. For example, officials from one college said that the chosen instrument enabled administrators to use gender-inclusive language and ask questions about incidents of sexual violence from the perspective of the perpetrator in addition to the victim.

Comparability. Officials from four colleges noted that comparability was a consideration when selecting a survey instrument, including the potential to compare survey results across colleges that share similar characteristics or at their own colleges over time. However, as previously discussed, stakeholders noted that differences across survey instruments can limit the comparability of survey results.

Cost. Officials from four colleges said the cost of conducting campus climate surveys informed their selection of a survey instrument. For example, officials at one college said they used a free, publicly available survey instrument because the college lacked the resources to pay for an instrument.

Length. Officials from four colleges identified survey length as another factor they considered. Officials from three of these colleges specifically noted that longer surveys may result in lower response rates. In addition, officials from one of these colleges stated that because longer surveys collect more data, the college would need more time to analyze the results. An official from another college expressed concern that longer surveys with multiple follow-up questions about incidents of sexual violence risk re-traumatizing victims.

Modifying the survey instrument. Six of the seven colleges modified their survey instruments to some extent. Officials at five of the six colleges reported adding questions to their survey instruments. For example, two colleges reported adding questions to comply with a state survey requirement, while another college reported adding follow-up questions to collect information on events prior to an incident of sexual violence. Officials at three of the five colleges reported limiting the number of questions they added to keep the survey short. Officials at two of these colleges noted that lengthening the survey could result in fewer students completing it. Officials at one of these colleges cited additional fees that the vendor charged for such modifications as another factor in their decision to limit the number of questions they added. Officials from two colleges also said they modified the language in the survey instruments to reflect the names of specific offices and programs on their campuses. In contrast, officials at one college reported making no changes to their survey instrument because they planned to use the original survey as a baseline against which to compare future survey results.

Survey Administration

Identifying the survey population. Six of the seven selected colleges that conducted a campus climate survey distributed their surveys via email to all students in the target population (i.e., a census approach), and one worked with a third party to select a representative sample of students to receive the survey. As previously discussed, surveying a sample of students can reduce the amount of follow-up work needed to obtain sufficient responses to provide information about the student body as a whole. However, officials at four of the seven colleges cited other considerations for choosing a census approach. Specifically, officials from three of these colleges said that a census approach provided every student the opportunity to share their experiences and perspectives through the climate survey. Officials from two of these colleges further explained that administering the survey to a sample of students could give the appearance they were excluding students, some of whom might be victims of sexual violence, from participating in the survey. Another college reported using a census approach because it lacked the resources needed to develop a representative survey sample.

Determining survey timing and frequency. All seven of the selected colleges that conducted a climate survey administered at least one survey during the spring semester. Officials from three of these colleges said that administering climate surveys in the spring ensures that first-year students have spent time on campus prior to taking the survey. However, officials at four colleges said that competing demands for students’ time, such as other surveys and final exams, are a tradeoff to administering these surveys in the spring. As a result, students may experience “survey fatigue”—that is, they may be less likely to respond to or complete the survey. The seven selected colleges administered surveys with varying frequencies. For example, one college reported administering its survey biennially in accordance with a state requirement, while two others administered their surveys less frequently (e.g., every 4 years) to avoid survey fatigue and low response rates.

Protecting confidentiality. Six of the colleges reported taking steps to preserve the confidentiality of survey respondents. For example, officials from five colleges explained that in order to maintain respondents’ confidentiality they had to redirect students who completed the survey to a separate webpage to claim their incentive or enter a drawing. Officials from four colleges reported using a third-party vendor to help protect students’ confidentiality or, at a minimum, signal that the college had no direct role in collecting or storing student responses. For example, to protect students’ confidentiality, officials from three of these colleges said their vendors provided summary data, rather than student-level data, and did not report results with a low number of respondents. Officials from three of the six colleges reported consulting their institutional review boards to help ensure that the colleges protected respondents’ confidentiality. Officials from another college reported limiting how often they administered campus climate surveys to head off potential student concerns that they were being “tracked” during their time on campus.

Survey Outreach

Offering survey incentives. As part of their outreach efforts, six of the seven colleges offered incentives to students who completed surveys, which some research suggests can increase web-based survey participation rates (see fig. 1). For example, one college offered a $20 gift card to survey respondents, which college administrators considered critical to achieving a higher response rate. This comports with a study funded by Justice that found incentives between $20 and $30 appear to help maximize survey participation, whereas a $40 incentive does not clearly offer any additional advantage. To manage the cost of incentives, two colleges offered a limited number of incentives to students via lottery drawings. Officials from one of these colleges said it funded its lottery for five $200 gift cards with proceeds from an on-campus student event. Another college offered a coupon for a free drink at a campus coffee shop to the first 300 survey respondents. An official from the college that did not offer incentives in its most recent climate survey said that incentives would help improve response rates for future surveys.

While incentives can help increase survey participation, two colleges noted that offering incentives may require additional precautions to prevent abuse. For example, one college had to put its survey on hold to fix a technical error that enabled a student to collect additional incentives by completing the survey multiple times. Another college with experience offering survey incentives reported that it received calls and emails from other colleges requesting assistance with preventing such abuses.

Marketing the survey. Each of the seven selected colleges that conducted a climate survey used email to invite students to respond to the survey and various marketing efforts to encourage survey participation (see fig. 1). Officials from five of the colleges reported following up with email reminders. For example, one college reported adding the incentive dollar amount to the subject lines and another reported varying the gender of those who sent follow-up emails and the timing of them to increase student responses. In addition, five colleges reported using social media to advertise their climate surveys. Recognizing the importance of gaining institutional buy-in, officials at all seven colleges said they engaged college administrators or faculty in their marketing efforts. For example, officials at one college said that deans of its various schools were asked to send emails encouraging students to take the survey. The officials credited this particular email strategy for doubling the college’s survey response rate. Officials from five colleges also reported involving student leaders and influencers in their marketing efforts, such as creating t-shirts for students to wear that included information about the survey; having students publish an op-ed in the campus newspaper promoting the survey; and asking student leaders to share information about the survey with student organizations.

Survey Reporting

Six of the seven selected colleges publicly reported at least some of the results of their surveys. Five colleges, for example, published survey results on their respective websites, and another created a campus poster with an infographic illustrating key survey results. Two of these colleges also presented the results during meetings with different student populations, such as fraternities and sororities and lesbian, gay, bisexual, transgender, and queer/questioning students. Officials from four colleges expressed that they felt a responsibility to be transparent. However, according to an official at one college, a potential drawback to making survey results publicly available is that the results could create or reinforce negative perceptions of a college’s climate regarding campus sexual violence. Finally, officials from the one college that had not publicly disclosed any survey results explained that, due to a lack of resources and in-house expertise, they did not feel sufficiently confident in their analysis of the survey results to publish them.

Federal Agencies Have Provided Information to Colleges about Developing and Implementing Campus Climate Surveys, Among Other Efforts

Since the issuance of the White House Task Force to Protect Students from Sexual Assault report in 2014, federal agencies have created and disseminated informational resources for colleges interested in conducting campus climate surveys. For example, Justice’s Bureau of Justice Statistics and Office on Violence Against Women funded the development of a publicly available survey instrument and a validation study from 2014 to 2016, to provide colleges and researchers with access to a free and reliable survey instrument to collect school-level data on campus climate and sexual victimization. In 2017, Justice also collaborated with HHS’s Centers for Disease Control and Prevention to provide funding and project planning assistance for a pilot study to develop and test a campus climate survey for use at two Historically Black Colleges and Universities. According to Justice and HHS officials, this survey instrument was based on the validated Justice survey instrument, with some modifications made to the campus climate questions. In October 2019, Justice officials told us the agency had decided not to proceed with funding for the study due to concerns that modifications to the original validated survey instrument would result in data that are not comparable to data from the validation study.

In addition, Justice has developed technical assistance materials for colleges interested in conducting a campus climate survey. For example, from 2016 to 2017 Justice’s Office on Violence Against Women issued documents outlining lessons learned from the Justice survey validation study, talking points to help college administrators and students communicate about climate surveys, and a frequently asked questions sheet on campus climate surveys. These documents covered a range of topics, including the goals of a campus climate survey, best practices for developing survey content, and tips for choosing survey participants and protecting their confidentiality, among others. Justice’s campus climate survey, validation study, and technical assistance documents are publicly available on Justice’s website.

Justice’s campus climate survey and validation study are also available through the Center for Changing Our Campus Culture, an online clearinghouse developed and maintained by a nonprofit organization with funding from Justice’s Office on Violence Against Women. The clearinghouse provides resources for colleges on addressing sexual assault, domestic violence, dating violence, and stalking. For example, the clearinghouse includes documents outlining (1) selected research initiatives and resources on campus climate surveys, (2) suggested campus sexual assault policies and procedures, and (3) steps college institutional review boards and administrators can take to oversee research on sexual violence while maintaining participant confidentiality. Most stakeholders we interviewed were aware of federal information and resources available to assist colleges in conducting campus climate surveys. For example, officials at two of the colleges reported using Justice’s survey instrument for their campus climate surveys, with officials from one college noting they selected the instrument because it had been validated as a reliable instrument. An official from another college reported using Justice’s validation study during the survey instrument selection process, to better understand the strengths and weakness of survey instruments and potential sources of bias in the data collected.

In addition to the resources provided by Justice, Education has offered information to colleges regarding the prevention of campus sexual violence. For example, Education’s 2015 Title IX Resource Guide encouraged Title IX coordinators to help colleges develop a method, appropriate to their college, for surveying students about the campus climate. Additionally, to address Title IX concerns or complaints, Education may enter into voluntary resolution agreements with colleges. These agreements describe the changes colleges agree to make to ensure their procedures for preventing and responding to sex discrimination comply with the law. According to agency officials, Education may include campus climate surveys as part of these voluntary agreements, on a case-by-case basis.

Justice and HHS have also funded campus sexual assault prevention and response grants. For example, Justice’s Office on Violence Against Women provides grant funding to colleges to help improve responses to sexual assault and other types of domestic and sexual violence through its Grants to Reduce Sexual Assault, Domestic Violence, Dating Violence, and Stalking on Campus Program. According to a Justice official, colleges receiving these grants are allowed, with prior approval, to use a small percentage of the grant funds to conduct campus climate surveys for program improvement purposes, but it is not a requirement of the program. Additionally, HHS’s Office on Women’s Health provided funding for the College Sexual Assault Policy and Prevention Initiative from 2016 to 2019 to organizations that partnered with colleges to provide technical assistance and support in developing sexual assault policies and prevention strategies. According to HHS officials, grantees were encouraged to conduct campus climate surveys to establish baseline data for their partner campuses.

HHS officials also reported providing grantees with information on different campus climate survey instrument options, including a free, publicly available survey instrument. One college we spoke with reported partnering with one of these HHS grantees to conduct baseline and follow-up campus climate surveys and to develop comprehensive campus prevention strategies. For example, officials from the college and the grantee told us they used funds from the grant to help the college establish memoranda of understanding with community-based organizations, such as the local women’s crisis center, to support students living off-campus who may have experienced sexual violence.

Agency Comments

We provided a draft of this report to the Departments of Education and Justice for review and comment. The Departments of Education and Justice provided technical comments, which we incorporated as appropriate. We also provided relevant report sections to the Department of Health and Human Services, and to third parties, including survey developers and states included in our review, for technical comments. The Department of Health and Human Services, survey developers, and state officials provided technical comments, which we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Education, the Attorney General, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II.

Appendix I: Objectives, Scope, and Methodology

This report examines (1) what key stakeholders view as the strengths and limitations of using campus climate surveys to examine the incidence and characteristics of sexual violence on college campuses, (2) what approaches selected colleges have used to survey their students about the incidence of sexual violence on campus, and (3) what role federal agencies play in helping colleges develop and implement climate surveys.

To inform our examination of stakeholders’ views on the strengths and limitations of campus climate surveys, we reviewed three commonly used survey instruments that included questions regarding the incidence of sexual violence, including sexual assaults, coerced sexual contact, stalking, and intimate partner violence. Each of these survey instruments is also available, online or by request, for any college to use for free. For each of the surveys, we reviewed survey questions and methodological reports, and conducted interviews with representatives from the organizations involved in developing them.

The Association of American Universities (AAU), an association of 65 research universities, developed its survey instrument in conjunction with the research firm Westat. AAU administered its survey to participating colleges in spring 2015 and spring 2019.

The Administrator Research Campus Climate Collaborative (ARC3), an organization of sexual assault researchers, university administrators, and student and legal affairs professionals, developed and tested its campus climate survey from 2014 to 2015. The final survey instrument was made available to colleges in 2015. According to the survey developers, there is no comprehensive list of schools that have conducted the ARC3 survey.

The Department of Justice (Justice) survey instrument was initially developed by the White House Task Force to Protect Students from Sexual Assault in 2014, and later refined and tested by Justice in collaboration with RTI International, a research organization. The survey instrument, also known as the Campus Climate Survey Validation Study, is publicly available online. According to Justice officials, there is no comprehensive list of schools that have conducted the Justice survey.

Additionally, we reviewed two key federal data sources on campus sexual violence: Clery Act data from Education’s Campus Safety and Security Survey and the National Crime Victimization Survey (NCVS) from Justice. We identified these data sources based on a review of prior GAO work and interviews with Education and Justice officials. We examined documentation for these data sources and interviewed the responsible agency officials to determine the type of data they collect on campus sexual violence, the methods for collecting this information, and their limitations. We determined these data sources were sufficiently reliable for our purposes.

To inform all three objectives, we also interviewed a total of 25 stakeholders with relevant expertise, including representatives of four organizations involved in developing the three surveys we reviewed and five additional researchers who have studied campus sexual violence; officials from 10 colleges; officials from four states; and federal officials from Education and Justice. We refer to the representatives of these organizations and entities collectively as “stakeholders” in our report. When discussing stakeholder views, we group them into the following categories: “several” (between four and nine), “about half” (between 10 and 14), “most” (between 15 and 19), and “nearly all” (20 or greater). In instances where we report on the views of specific groups, such as colleges, researchers, or state officials, we refer to the individual group and enumerate the number of group members. During these interviews, we gathered information on issues related to designing and conducting campus climate surveys and analyzing and communicating survey results. We also discussed federal information and resources available to help colleges develop and implement campus climate surveys. Findings from our interviews summarize selected stakeholders’ views regarding campus climate surveys on sexual violence. These findings do not represent the views of all researchers on these topics and do not represent the experiences of all colleges developing or implementing these surveys.

To identify researchers with a variety of perspectives, we reviewed research on sexual violence and conducted targeted web searches. We then selected individuals or organizations with experience conducting research on campus sexual violence or developing and administering a campus climate survey on sexual violence. We also spoke with representatives of the organizations responsible for developing the three climate survey instruments we reviewed.

We used multiple approaches to identify the 10 selected colleges included in our review since there is no central repository of information on whether colleges have conducted a campus climate survey on sexual violence.

Colleges that conducted a campus climate survey. Based on targeted web searches, we identified colleges that had conducted a campus climate survey and then grouped them according to which of the three survey instruments they used. We analyzed data from the Department of Education’s Integrated Postsecondary Education Data System to identify the characteristics of these colleges, including sector (i.e., public, private not-for-profit, and private for-profit), program length (i.e., 2-year and 4-year), size, and geographic location.

Colleges that had not conducted a campus climate survey. We also used data from the Integrated Postsecondary Education Data System to help identify colleges that had not conducted campus climate surveys on sexual violence. Specifically, we grouped colleges into categories by sector and program length and randomized the lists within each category. To select specific colleges, we started with the college in each category at the top of the randomized list and conducted targeted web searches in an effort to ensure the college had not publicly reported conducting a campus climate survey.

We conducted outreach to the Title IX coordinators at each of the selected colleges via email or telephone to confirm whether or not the college had conducted a campus climate survey on sexual violence. In total, we selected 10 colleges, including seven that have conducted campus climate surveys that examine the incidence of sexual violence on their campuses and three that have not. We selected these colleges to ensure variation in size, sector (i.e., public, private not-for-profit, and private for-profit), program length (i.e., 2-year and 4-year), geographic location, survey instrument used, and whether the college was located in a state that as of January 1, 2017 had a statutory requirement in effect for at least some colleges in their state to conduct a campus climate survey (see table 2 for selected colleges by program length, sector, and use of campus climate survey). We interviewed Title IX coordinators and other knowledgeable officials regarding the selected colleges’ experiences with conducting campus climate surveys and their perspectives on the strengths and limitations of these surveys.

As part of our efforts to obtain a variety of perspectives, we also conducted semi-structured interviews with officials from four states regarding the use of campus climate surveys in their states. We selected three states (Louisiana, New York, and Washington) that as of January 1, 2017, had a statutory requirement in effect for at least some colleges in their state to conduct a campus climate survey, and one state (Ohio) that recommended colleges conduct such surveys. To identify states that required or recommended that colleges conduct campus climate surveys, we used several approaches to develop a preliminary list, including consulting with researchers, reviewing annual reports from 2014 to 2019 prepared by the National Conference of State Legislatures on state higher education legislation, and conducting targeted web searches. Based on these reviews, we judgmentally selected four states to ensure a diversity of state experiences with requiring or recommending campus climate surveys. We also confirmed applicable state requirements with state officials. The selected states differed in the nature of the survey requirement or recommendation, such as the types of colleges covered (e.g., public or public and private) and how frequently the survey was required or recommended to be administered.

To supplement information gathered from our interviews, we also identified and reviewed studies and reports that examined the design and use of campus climate surveys. We conducted a targeted search of various databases to identify studies on leading survey practices. We selected studies for additional review based on their relevance to our objectives, and, using a standard review instrument, assessed the quality and rigor of each study’s findings and methods. Our report includes information about leading survey design and implementation practices from those studies we found were appropriate through this review process.

To examine the role that federal agencies play in helping colleges develop and implement climate surveys, we reviewed Justice and Education resources available to help colleges conduct campus climate surveys. Additionally, we reviewed information on campus sexual violence prevention grants provided by the Department of Health and Human Services. We also reviewed relevant federal laws and regulations, as well as federal guidance and other documentation pertaining to campus sexual violence and campus climate surveys.

We conducted this performance audit from July 2018 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Debra Prescott (Assistant Director), Maria Gadel (Analyst-in-Charge), Jonathan Adams, Will Colvin, Caitlin Cusati, Kirsten Lauber, and Erica Vilay made key contributions to this report. Additional assistance was provided by MacKenzie Cooper, Sarah Cornetto, Holly Dye, Monika Gomez, Dana Hopings, Connor Kincaid, Sheila R. McCoy, Mimi Nguyen, and Almeta Spencer.