Abstinence Education: Efforts to Assess the Accuracy and Effectiveness of Federally Funded Programs
Published: 2006-10-03T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Reducing the incidence of sexually transmitted diseases and unintended pregnancies is one objective of the Department of Health and Human Services (HHS). HHS provides funding to states and organizations that provide abstinence-until-marriage education as one approach to address this objective. GAO was asked to describe the oversight of federally funded abstinence-until-marriage education programs. GAO is reporting on (1) efforts by HHS and states to assess the scientific accuracy of materials used in these programs and (2) efforts by HHS, states, and researchers to assess the effectiveness of these programs. GAO reviewed documents and interviewed HHS officials in the Administration for Children and Families (ACF) and the Office of Population Affairs (OPA) that award grants for these programs.
section_title: What GAO Found
paragraphs: Efforts by HHS and states to assess the scientific accuracy of materials used in abstinence-until-marriage education programs have been limited. This is because HHS's ACF--which awards grants to two programs that account for the largest portion of federal spending on abstinence-until-marriage education--does not review its grantees' education materials for scientific accuracy and does not require grantees of either program to review their own materials for scientific accuracy. In contrast, OPA does review the scientific accuracy of grantees' proposed educational materials. In addition, not all states that receive funding from ACF have chosen to review their program materials for scientific accuracy. In particular, 5 of the 10 states that GAO contacted conduct such reviews. Officials from these states reported using a variety of approaches in their reviews. While the extent to which federally funded abstinence-until-marriage education materials are inaccurate is not known, in the course of their reviews OPA and some states reported that they have found inaccuracies in abstinence-until-marriage education materials. For example, one state official described an instance in which abstinence-until-marriage materials incorrectly suggested that HIV can pass through condoms because the latex used in condoms is porous. HHS, states, and researchers have made a variety of efforts to assess the effectiveness of abstinence-until-marriage education programs; however, a number of factors limit the conclusions that can be drawn about the effectiveness of abstinence-until-marriage education programs. ACF and OPA have required their grantees to report on various outcomes that the agencies use to measure the effectiveness of grantees' abstinence-until-marriage education programs. In addition, 6 of the 10 states in GAO's review have worked with third-party evaluators to assess the effectiveness of abstinence-until-marriage education programs in their states. Several factors, however, limit the conclusions that can be drawn about the effectiveness of abstinence-until-marriage education programs. Most of the efforts to evaluate the effectiveness of abstinence-until-marriage education programs included in GAO's review have not met certain minimum scientific criteria--such as random assignment of participants and sufficient follow-up periods and sample sizes--that experts have concluded are necessary in order for assessments of program effectiveness to be scientifically valid, in part because such designs can be expensive and time-consuming to carry out. In addition, the results of efforts that meet the criteria of a scientifically valid assessment have varied and two key studies funded by HHS that meet these criteria have not yet been completed. When completed, these HHS-funded studies may add substantively to the body of research on the effectiveness of abstinence-until-marriage education programs.

section_title: Letter
paragraphs: Preventing sexually transmitted diseases (STD) and unintended  pregnancies among adolescents is an important public health challenge.  Although pregnancy and birth rates among female adolescents in the  United States have been declining since the early 1990s, the rates continue  to be high when compared with those in other industrialized nations. The  Centers for Disease Control and Prevention (CDC) reports that about  141,000 children were born to girls 17 years old and younger in the United  States in 2003. CDC also reports that STDs disproportionately affect  adolescents, with adolescents and young adults ages 15 to 24 acquiring  almost half of the estimated 19 million new infections each year.
Reducing the incidence of STDs and unintended pregnancies among  adolescents is an important objective for the Department of Health and  Human Services (HHS), which identifies as one of its goals the need to  reduce major threats to the health and well-being of Americans. Among  the efforts it supports to reduce the incidence of STDs and unintended  pregnancies among adolescents, HHS funds abstinence-until-marriage  education programs. Abstinence-until-marriage education programs, also  referred to as abstinence-only education programs, teach adolescents to  abstain from sexual activity until marriage in order to avoid risks of  unintended pregnancy, STDs, and related health problems. The content of  federally funded abstinence-until-marriage programs is required to be  consistent with several principles, such as teaching that a mutually faithful  monogamous relationship in the context of marriage is the expected  standard of human sexual activity, and that abstinence from sexual  activity is the only certain way to avoid STDs. Abstinence-until-marriage  education programs are delivered by a variety of entities, including  schools, human service agencies, faith-based organizations, youth  development groups, and pregnancy crisis centers. Instructors can  incorporate a variety of educational materials into their abstinence-until- marriage education programs, including textbooks, student manuals,  brochures, slide presentations, and videos.
The three main federally funded abstinence-until-marriage programs are  the Abstinence Education Program (State Program), which is administered  by HHS’s Administration for Children and Families (ACF); the Community- Based Abstinence Education Program (Community-Based Program),  which is also administered by ACF; and the Adolescent Family Life (AFL)  Program, which is administered by HHS’s Office of Population Affairs  (OPA) within the Office of Public Health and Science. Funding provided  by HHS for the three abstinence-until-marriage programs increased from  about $73 million in fiscal year 2001 to about $158 million in fiscal year  2005.
Recent studies have raised concerns about the accuracy of educational  materials used in abstinence-until-marriage education programs and about  the effectiveness of these programs. These studies have reported that  some of the materials used in abstinence-until-marriage education  programs contain, for example, scientifically inaccurate information about  anatomy and physiology as they relate to reproductive health as well as  misleading information about contraceptive failure rates and STDs. State  and federal agencies have also documented inaccuracies in abstinence- until-marriage educational materials. Further, studies examining the  effectiveness of these programs have reported varied results. For example,  some researchers have reported that abstinence-until-marriage education  programs have resulted in adolescents reporting less frequent sexual  intercourse or fewer sexual partners, while other researchers have  reported that these types of programs did not affect the frequency of  sexual intercourse or were ineffective in delaying the initiation of sexual  intercourse.
You asked us to describe certain aspects of the oversight of federally  funded abstinence-until-marriage education programs. Our objectives  were to report on (1) efforts by HHS and states to assess the scientific  accuracy of materials used in abstinence-until-marriage education  programs and (2) efforts by HHS, states, and researchers to assess the  effectiveness of abstinence-until-marriage education programs. You also  asked us to describe how HHS selected a contractor for the abstinence- until-marriage technical assistance contract that was awarded in  September 2002. This information is provided in appendix I.
To describe the efforts by HHS and states to assess the scientific accuracy  of materials used in abstinence-until-marriage education programs, we  reviewed published reports, program announcements, Federal Register  notices, agency Web sites, and other documents related to abstinence- until-marriage education. We focused our review on efforts related to the  three main federally funded abstinence-until-marriage education programs  administered by HHS, as well as efforts to review the accuracy of scientific  facts included in abstinence-until-marriage education materials. We did  not assess the criteria used to determine the scientific accuracy of  education materials or the quality of the reviews. We interviewed officials  from ACF, the Health Resources and Services Administration (HRSA),  OPA, and CDC. We also interviewed officials from the 10 states that  received the largest share of federal funding (together accounting for   51 percent of the total funding in fiscal year 2005) through the State  Program for abstinence-until-marriage education.
To describe efforts by HHS, states, and researchers to assess the  effectiveness of abstinence-until-marriage education programs, we  focused on efforts that examined the extent to which these programs  achieved their program goals. In general, these goals include teaching  adolescents to abstain from sexual activity until marriage in order to avoid  unintended pregnancies, STDs, and related health problems. As part of our  review, we compared these efforts to the design characteristics that  experts have identified as important for a scientifically valid study of  program effectiveness. We reviewed journal articles and other published  reports, agency budget submissions, program announcements, agency and  grantee performance reports, Federal Register notices, agency Web sites,  and other documents related to abstinence-until-marriage education. (For  a more detailed description of our literature review methodology, see   app. II). We also interviewed officials from ACF, HRSA, OPA, CDC, the  National Institutes of Health (NIH), the Office of the Assistant Secretary  for Planning and Evaluation (ASPE), and 10 states that received the largest  share of federal funding for abstinence-only education through the State  Program in fiscal year 2005. We also interviewed individuals from the  National Campaign to Prevent Teen Pregnancy, The Brookings Institution,  ETR Associates, The Heritage Foundation, and Advocates for Youth, and  researchers from Case Western Reserve University and Columbia  University to obtain general information regarding the state of the  research on abstinence-until-marriage education. We focused our review  on efforts to assess the scientific accuracy of materials and the  effectiveness of the programs during fiscal year 2006, and also reviewed  the administration of the programs back to fiscal year 2001. We also  attended conferences organized by ACF and OPA to learn about training  that is provided to grantees on scientific accuracy and program  evaluations.
To describe how HHS selected a contractor for the abstinence-until- marriage technical assistance contract that was awarded in September  2002, we reviewed the Request for Proposals and other related contract  documents. We also interviewed officials at HRSA, ACF, and the National  Abstinence Clearinghouse about the technical assistance contract. We  performed our work from October 2005 through September 2006 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: Recent statistics from CDC show that many high school students engage in  sexual behavior that places them at risk for unintended pregnancy and  STDs. In 2005, 46.8 percent of high school students reported that they have  ever had sexual intercourse, with 14.3 percent of students reporting that  they had had sexual intercourse with four or more persons. The likelihood  of ever having sexual intercourse varied by grade, with the highest rate  among 12th grade students (63.1 percent) and the lowest rate among 9th  grade students (34.3 percent). CDC also has reported that the prevalence  of certain STDs—including the rate of chlamydia infection, the most  frequently reported STD in the United States—peaks in adolescence and  young adulthood. According to CDC, in 2004 the chlamydia rates among  adolescents 15 to 19 years old (1,579 cases per 100,000 adolescents) and  young adults 20 to 24 years old (1,660 cases per 100,000) were each more  than twice the rates among all other age groups.
HHS’s current strategic plan includes the objectives to reduce the  incidence of STDs and unintended pregnancies and to promote family  formation and healthy marriages. These two objectives support HHS’s  goals to reduce the major threats to the health and well-being of  Americans and to improve the stability and healthy development of  American children and youth, respectively. Abstinence-until-marriage  education programs are one of several types of programs that support  these objectives.
The three main federal abstinence-until-marriage education programs— the State Program, the Community-Based Program, and the AFL  Program—provide grants to support the recipients’ own efforts to provide  abstinence-until-marriage education at the local level. These programs  must comply with the statutory definition of abstinence education (see  table 1).
subsections: section_title: The State Program
paragraphs: The State Program, administered by ACF, provides funding to its  grantees—states—for the provision of abstinence-until-marriage education  to those most likely to have children outside of marriage. States that  receive grants through the State Program have discretion in how they use  their funding to provide abstinence-until-marriage education. Some require  that organizations apply for funds and use them to administer abstinence- until-marriage education programs. Others may directly administer such  programs. At their discretion, states may also provide mentoring,  counseling, and adult supervision to adolescents to promote abstinence  from sexual activity until marriage.
The Personal Responsibility and Work Opportunity Reconciliation Act of  1996 established the State Program, and states were awarded grants  beginning in fiscal year 1998. Funds are allotted to each state that  submits the required annual application based on the ratio of the number  of low-income children in the state to the total number of low-income  children in all states. States are required to match every $4 they receive in  federal money with $3 of nonfederal money and are required to report  annually on the performance of the abstinence-until-marriage education  programs that they support or administer. In fiscal year 2005, 47 states, the  District of Columbia, and 3 insular areas were awarded funding.
section_title: The Community-Based Program
paragraphs: The Community-Based Program, which is also administered by ACF, is  focused on funding public and private entities that provide abstinence- until-marriage education for adolescents from 12 to 18 years old, with the  purpose of creating an environment within communities that supports  adolescent decisions to postpone sexual activity until marriage. The  Community-Based Program provides grants for school-based programs,  adult and peer mentoring, and parent education groups. The Community- Based Program first awarded grants in fiscal year 2001. Grantees of the  Community-Based Program are selected through a competitive process  and are evaluated according to several criteria, such as the extent to which  they have demonstrated that a need exists for abstinence-until-marriage  education for a targeted population or in a specific geographic location.  Grantees are required to report to ACF, on a semiannual basis, on the  performance of their programs. For fiscal year 2005, 63 grants were  awarded to organizations and other entities.
section_title: The AFL Program
paragraphs: The AFL Program supports programs that provide abstinence-until- marriage education. The primary purpose of these programs is to find  effective means of reaching preadolescents and adolescents before they  become sexually active in order to encourage them to abstain from sexual  activity and other risky behaviors. Under the AFL Program, OPA awards  competitive grants to public or private nonprofit organizations or  agencies, including community-based and faith-based organizations, to  facilitate abstinence-until-marriage education in a variety of settings,  including schools and community centers. Established in 1981, the AFL  Program began awarding grants in fiscal year 1982. AFL Program grantees  include school districts, youth development groups, and medical centers.  Grant applicants are evaluated based on several criteria, such as the extent  to which they provide a clear statement of mission, goals, measurable  objectives, and a reasonable method for achieving their objectives.  Grantees are required to conduct evaluations of certain aspects of their  programs and report annually on their performance. As of August 2006,  OPA funded 58 abstinence-until-marriage education programs, and most of  these were focused on reaching young adolescents from the ages of 9 to  14.
section_title: Funding for Abstinence- until-Marriage Education Has Increased
paragraphs: Funding provided by HHS for abstinence-until-marriage education  programs has increased steadily since 2001 (see table 2). For the three  main programs combined—the State Program, the Community-Based  Program, and the AFL Program—the amount of agency funding increased  from about $73 million in fiscal year 2001 to about $158 million in fiscal  year 2005. Nearly all of this increase was for the Community-Based  program; funding under this program increased by about $84 million from  fiscal years 2001 through 2005. In fiscal year 2005, agency funding for the  Community-Based Program constituted the largest share of the total  funding (about 66 percent) for the three main programs combined.
Within each of the three main abstinence-until-marriage education  programs, the amount of individual grants varied. In fiscal year 2005, the  State Program’s annual grants ranged from $57,057 to $4,777,916 and the  median annual grant amount was $569,675. That same year, the  Community-Based Program’s annual grants ranged from $213,276 to  $800,000 and the median grant amount was $642,250. In fiscal year 2006,  the AFL Program’s annual grants ranged from $95,676 to $300,000 and the  median grant amount was $225,000.
section_title: Federal Agency Responsibilities Related to Abstinence-until-Marriage Education
paragraphs: Five organizational units located within HHS—ACF, OPA, CDC, ASPE, and  NIH—have responsibilities related to abstinence-until-marriage education.  ACF and OPA administer the three main federal abstinence-until-marriage  education programs. CDC supports abstinence-until-marriage education at  the national, state, and local levels. CDC, ASPE, and NIH are sponsoring  research on the effectiveness of abstinence-until-marriage programs.
ACF is responsible for federal programs that promote the economic and  social well-being of families, children, individuals, and communities. ACF  administers and provides oversight of both the State Program and the  Community-Based Program by, among other things, awarding grants,  providing training and technical assistance to grantees, and requiring  annual performance reporting from grantees. ACF has been responsible  for the State Program since June 2004 and the Community-Based Program  since October 2005. HRSA previously administered these programs.
OPA has responsibility for advising the Secretary of HHS on a wide range  of reproductive health topics, including adolescent pregnancy and family  planning. The office is also responsible for administering programs that  provide services for pregnant and parenting teens and prevention  programs, such as abstinence-until-marriage education programs. OPA  administers and provides oversight of the AFL Program by awarding  grants, providing training and technical assistance to grantees, and  requiring annual performance reporting from grantees.
CDC is primarily responsible for the prevention and control of infectious  and chronic diseases, including STDs. CDC provides funding to state and  local education agencies in their efforts to support comprehensive school  health education and HIV/STD prevention education programs, and CDC  officials told us that some of these are focused on abstinence. CDC also  provides funding to several state education agencies to implement various  abstinence projects, such as collaboration-building among agencies to  increase the impact of their efforts to encourage abstinence. Further, CDC  develops tools to assist state and local education agencies with their  health education programs. CDC provides funding to several national  organizations to build the capacity of abstinence-until-marriage education  providers. Organizations’ activities include, but are not limited to, the  development and distribution of educational materials. CDC is also  sponsoring research on the effectiveness of an abstinence-until-marriage  education program.
ASPE advises the Secretary of HHS in several areas, including policy  development in health, human services, data, and science. ASPE is  responsible for the development of policy analyses and it conducts  research and evaluation studies in several areas, including the health of  children and adolescents. ASPE is currently sponsoring research on the  effectiveness of abstinence-until-marriage education programs.
NIH is the primary federal agency that conducts and supports medical and  behavioral research among various populations, including children and  adolescents. NIH is currently sponsoring research on the effectiveness of  abstinence-until-marriage education programs.
section_title: Federal and State Efforts to Assess the Scientific Accuracy of Materials Used in Abstinence-until- Marriage Education Programs Have Been Limited
paragraphs: Efforts by HHS and states to assess the scientific accuracy of materials  used in abstinence-until-marriage education programs have been limited.  ACF—which awards grants to two programs that account for the largest  portion of federal spending on abstinence-until-marriage education—does  not review its grantees’ education materials for scientific accuracy and  does not require grantees of either program to review their own materials  for scientific accuracy. In addition, not all states funded through the State  Program have chosen to review their program materials for scientific  accuracy. In contrast to ACF, OPA has reviewed the scientific accuracy of  grantees’ proposed educational materials and corrected inaccuracies in  these materials.
subsections: section_title: ACF Does Not Review Program Materials for Scientific Accuracy and Does Not Require Grantees to Do So, though Some State Grantees Have Conducted Such Reviews
paragraphs: There have been limited efforts to review the scientific accuracy of  educational materials used in ACF’s State and Community-Based  Programs—the two programs that account for the largest portion of  federal spending on abstinence education. ACF does not review materials  for scientific accuracy in either reviewing grant applications or in  overseeing grantees’ performance. Prior to fiscal year 2006, State Program  and Community-Based Program applicants were not required to submit  copies of their proposed educational materials with their applications.  While ACF required grantees of the Community-Based Program—but not  the State Program—to submit their educational materials with their fiscal  year 2006 applications, ACF officials told us that grantee applications and  materials are only reviewed to ensure that they address all aspects of the  scope of the Community-Based Program, such as the A-H definition of  abstinence education. Further, documents provided to us by ACF  indicate that the agency does not review grantees’ educational materials  for scientific accuracy as a routine part of its oversight activities. In  addition, ACF also does not require its grantees to review their own  materials for scientific accuracy. Similarly, when HRSA was responsible  for the State and Community-Based Programs, the agency did not review  materials used by grantees for scientific accuracy or require grantees to  review their own materials.
Not all grantees of the State Program have chosen to review the scientific  accuracy of their educational materials. Officials from 5 of the 10 states in  our review reported that their states have chosen to conduct such  reviews. Officials in these states identified a variety of reasons why  their states reviewed abstinence-until-marriage educational materials,  including program requirements, state education laws and guidelines, and  past lawsuits, to ensure that materials used in abstinence-until-marriage  programs were accurate. For example, Michigan’s Revised School Code  states that materials and instruction in the sex education curricula,  including information on abstinence, “shall not be medically inaccurate,”  and Ohio’s fiscal year 2007 abstinence-until-marriage education program  guidance states that abstinence-until-marriage educational materials  “should be medically accurate in all assertions.”
The five states we contacted that review abstinence-until-marriage  educational materials for scientific accuracy have used a variety of  approaches in their reviews. Some states contracted with medical  professionals—such as nurses, gynecologists, and pediatricians—to serve  as medical advisors who review program materials and use their expertise  to determine what is and is not scientifically accurate. Some states have  created checklists or worksheets to guide their staff conducting the review  and document findings of inaccuracy or verification of a statement. All five  states use medical professionals in conducting these reviews. One of the  states requires that all statistics or scientific statements cited in a  program’s materials are sourced to CDC or a peer-reviewed medical  journal. Officials from this state told us that if statements in these  materials cannot be attributed to these sources, the statements are  required to be removed until citations are provided and materials are  approved. Officials from this state told us they have also supplemented  their review of program materials with on-site classroom observations to  assess the scientific accuracy of the information presented to students.
Officials from two of the five states reported that they have found  inaccuracies as a result of their reviews. For example, one state official  stated that because information is constantly evolving, state officials have  had to correct out-of-date scientific information. In addition, this official  cited an instance where materials incorrectly suggested that HIV can pass  through condoms because the latex used in condoms is porous. In  addition, this official provided documentation that the state has had to  correct a statement indicating that when a person is infected with the  human papillomavirus, the virus is “present for life” because, in almost all  cases, this is not true. State officials who have identified inaccuracies told  us that they informed their grantees of inaccuracies so that they could  make corrections in their individual programs. One state official added  that she contacted the authors of the materials to report an inaccuracy.
Some of the educational materials that states have reviewed are materials  that are commonly used in the Community–Based Program. Officials from  four of the five states that review materials for scientific accuracy told us  that they have each reviewed at least one of the five curricula most  commonly used in the Community-Based Program because programs in  their state were using them: Choosing the Best, WAIT Training, Sex Can  Wait, A.C. Green’s Game Plan Abstinence Program, and Worth the Wait.  Based on ACF documents, we found that there were 58 different curricula  used by grantees of the Community-Based Program in fiscal year 2005.  However, more than half of the grantees of the Community-Based Program  reported using at least one of these five curricula.
While there has been limited review of materials used in the State and  Community-Based Programs, grantees of these programs have received  some technical assistance designed to improve the scientific accuracy of  their materials. For example, ACF officials reported that the agency  provided a conference for grantees of the Community-Based Program in  February 2006 that included a presentation focused on medical accuracy,  including a discussion of state legislative proposals that would require  medical accuracy in abstinence-until-marriage education, and how to  identify reliable data. In addition, in 2002, HRSA awarded a contract to the  National Abstinence Clearinghouse requiring, among other things, that the  contractor develop and implement a program to provide medically  accurate information and training to grantees of the State and Community- Based Programs. (See app. I for a description of HRSA’s process for  awarding this contract). The portion of the contract that focused on  providing medically accurate information to grantees was subcontracted  to the Medical Institute for Sexual Health (Medical Institute), which has  conducted presentations at regional educational conferences to provide  grantees with medical and scientific information, such as updated  information on condoms and STD transmission. The Medical Institute has  also provided consultative services to grantees by responding to medical  and scientific questions.
section_title: OPA Reviews Materials Used by AFL Program Grantees for Scientific Accuracy
paragraphs: In contrast to ACF, OPA reviews for scientific accuracy the educational  materials used by AFL Program grantees. Specifically, OPA reviews its  grantees’ proposed educational materials for scientific accuracy before  they are used. Agency officials stated that they began to review these  materials while litigation concerning the AFL Program was ongoing. OPA  continued to review these materials as part of a 1993 settlement to this  lawsuit. The settlement agreement expired in 1998, though the agency has  continued to review grantees’ proposed educational materials for accuracy  as a matter of policy. OPA officials told us that grant applicants submit  summaries of materials they propose to use, though the materials are not  reviewed for scientific accuracy until after grantees have been selected.  OPA officials said that after grants are awarded, a medical education  specialist (in consultation with several part-time medical experts) reviews  the grantees’ printed materials and other educational media, such as  videos. OPA officials explained that the medical education specialist must  approve all materials before they are used. On many occasions, OPA  grantees have proposed using—and therefore OPA has reviewed— materials commonly used in the Community-Based Program. For example,  an OPA official told us that the agency had reviewed three of the  Community-Based Program’s commonly used curricula—Choosing the  Best, Sex Can Wait, and A.C. Green’s Game Plan Abstinence Program— and is also currently reviewing another curriculum commonly used by  Community-Based Program grantees, WAIT Training.
OPA officials stated that the medical education specialist has occasionally  found and addressed inaccuracies in grantees’ proposed educational  materials. OPA officials stated that these inaccuracies are often the result  of information being out of date because, for example, medical and  statistical information on STDs changes frequently. OPA has addressed  these inaccuracies by either not approving the materials in which they  appeared or correcting the materials through discussions with the  grantees and, in some cases, the authors of the materials. In fiscal year  2005, OPA disapproved of a grantee using a specific pamphlet about STDs  because the pamphlet contained statements about STD prevention and  HIV transmission that were considered incomplete or inaccurate. For  example, the pamphlet stated that there was no cure for hepatitis B, but  the medical education specialist required the grantee to add that there was  a preventive vaccine for hepatitis B. In addition, OPA required that a  grantee correct several statements in a true/false quiz—including  statements about STDs and condom use—in order for the quiz to be  approved for use. For example, the medical education specialist changed a  sentence from “The only 100% effective way of avoiding STDs or unwanted  pregnancies is to not have sexual intercourse.” to “The only 100% effective  way of avoiding STDs or unwanted pregnancies is to not have sexual  intercourse and engage in other risky behaviors.”
While OPA and some states have reviewed their grantees’ abstinence-until- marriage education materials for scientific accuracy, these types of  reviews have the potential to affect abstinence-until-marriage education  providers more broadly. Such efforts may create an incentive for authors  of abstinence-until-marriage education materials to ensure they are  accurate. Thus, some authors of abstinence-until-marriage education  materials have recently updated materials in their curricula following  reports that questioned their accuracy. For example, one of the most  widely used curricula used by grantees of the Community-Based  Program—WAIT Training—has been recently updated and provides the  updated information on its Web site. A representative from WAIT Training  stated that the company recently revised its curriculum, in part, in  response to a congressional review that found inaccuracies in its  abstinence-until-marriage education materials.
section_title: A Variety of Efforts Have Been Made to Assess the Effectiveness of Abstinence-until- Marriage Education Programs, but a Number of Factors Limit the Conclusions That Can Be Drawn
paragraphs: HHS, states, and researchers have made a variety of efforts to assess the  effectiveness of abstinence-until-marriage education programs; however, a  number of factors limit the conclusions that can be drawn about the  effectiveness of these programs. ACF and OPA have required their  grantees to report on various outcomes used to measure the effectiveness  of grantees’ abstinence-until-marriage education programs, though the  reporting requirements for each of the three abstinence-until-marriage  programs differ. In addition, to assess the effectiveness of the State and  Community-Based Programs, ACF has analyzed national data on  adolescent birth rates and the proportion of adolescents who report  having had sexual intercourse. Other organizational units within HHS— ASPE, CDC, and NIH—are funding studies designed to assess the  effectiveness of abstinence-until-marriage education programs in delaying  sexual initiation, reducing pregnancy and STD rates, and reducing the  frequency of sexual activity. Despite these efforts, several factors limit the  conclusions that can be drawn about the effectiveness of abstinence-until- marriage education programs. Most of the efforts to evaluate the  effectiveness of abstinence-until-marriage education programs that we  describe in our review have not met certain minimum criteria that experts  have concluded are necessary in order for assessments of program  effectiveness to be scientifically valid, in part because such designs can be  expensive and time-consuming to carry out. In addition, the results of  some efforts that meet the criteria of a scientifically valid assessment have  varied, and two key studies that meet these criteria have not yet been  completed.
subsections: section_title: HHS, States, and Researchers Have Made a Variety of Efforts to Assess the Effectiveness of Abstinence-until-Marriage Education Programs
paragraphs: Efforts of HHS, states, and researchers to assess the effectiveness of  abstinence-until-marriage education programs have included ACF and  OPA requiring grantees to report data on outcomes of their abstinence- until-marriage education programs; ACF analyzing national data on  adolescent behavior and birth rates; and other HHS agencies, states, and  researchers funding or conducting studies to assess the effectiveness of  abstinence-until-marriage education programs.
subsections: section_title: ACF and OPA Have Required Grantee Reporting of Data on Outcomes
paragraphs: ACF has made efforts to assess the effectiveness of abstinence-until- marriage education programs funded by the State Program and the  Community-Based Program. One of ACF’s efforts has been to require  grantees of both programs to report data on outcomes, though the two  programs have different requirements for the outcomes grantees must  report. For the State Program, as of fiscal year 2006, grantees must report  annually on four measures of the prevalence of adolescent sexual behavior  in their states, such as the rate of pregnancy among adolescents aged 15 to  17 years, and compare these data to program targets over 5 years. To  report on these four measures, states may choose the data sources they  will use. States must also develop and report on two additional  performance measures that are related to the goals of their programs.  (See table 3 for a list of ACF’s fiscal year 2006 reporting requirements for  the State Program.)
As of fiscal year 2006, ACF requires Community-Based Program grantees  to develop and report on outcome measures designed to demonstrate the  extent to which grantees’ community-based abstinence education  programs are accomplishing their program goals. ACF requires grantees  of the Community-Based Program to contract with third-party evaluators,  who are responsible for both helping grantees develop the outcome  measures and monitoring grantee performance against the measures, but  because this is a new requirement established for fiscal year 2006 grantees,  ACF has not yet received the results of these evaluations. In addition to  outcome reporting, ACF requires grantees of the Community-Based  Program to report on program “outputs,” which measure the quantity of  program activities and other deliverables, such as the number of  participants who are served by the abstinence-until-marriage education  programs. According to ACF officials, the agency requires grantees of both  the State Program and the Community-Based Program to report on  program outcomes in order to monitor grantees’ performance, target  training, and technical assistance, and help grantees improve service  delivery. (See table 3 for a list of ACF’s fiscal year 2006 reporting  requirements for the Community-Based Program.)
ACF’s fiscal year 2006 reporting requirements for grantees of the State  Program are the same as HRSA’s when it administered the State Program.  In contrast, ACF’s fiscal year 2006 reporting requirements for the  Community-Based Program differ from HRSA’s reporting requirements for  the program. For example, for Community-Based Program grants awarded  in fiscal year 2001, HRSA required grantees to report on the effectiveness  of their programs, as measured by program participation as well as  behavioral and biological outcomes. These performance measures were  modified for fiscal year 2002, in part HHS officials explained, because of  concerns expressed by members of the abstinence-education community  that the original performance measures did not accurately reflect the  efforts of the grantees of the Community-Based Program. For grants  awarded from fiscal years 2002 through 2004, HRSA required grantees of  the Community-Based Program to report on a combination of program  outputs, such as the proportion of adolescents who completed an  abstinence-until-marriage education program, and measures of adolescent  intentions, such as the proportion of adolescents who committed to  abstaining from sexual activity until marriage. For grants awarded in  fiscal year 2005, when ACF assumed responsibility for the Community- Based Program from HRSA, grantees were not required to report on any  specific performance measures.
OPA has also made efforts to assess the effectiveness of the AFL Program.  Specifically, OPA requires grantees of the AFL Program to develop and  report on outcome measures that are used to help demonstrate the extent  to which grantees’ programs are having an effect on program  participants. According to OPA officials, the agency recommends that  grantees report on outcome measures, such as participants’ knowledge of  the benefits of abstinence and their reported intentions to abstain from  sexually activity, reported beliefs in their ability to remain abstinent, and  reported parental involvement in their lives. To collect data on these  outcome measures and any others, OPA requires all grantees funded in  fiscal year 2004 and beyond to administer, at a minimum, a standardized  questionnaire—developed by OPA—to their program participants, both  when participants begin an abstinence-only education program and after  the program’s completion. The standardized questionnaire includes  questions intended to obtain information on participants’ reported  involvement in extracurricular activities, behaviors linked to health risks,  attitudes and intentions about abstinence, and opinions about the  consequences of premarital sexual activity. Like ACF, OPA requires its  grantees to contract with independent evaluators, such as colleges or  universities, which are responsible for evaluating the effectiveness of  grantees’ individual abstinence-until-marriage education programs. In  addition to evaluating the extent to which grantees are meeting their goals,  OPA officials stated that the independent evaluators may also provide  input to grantees of the AFL Program on other aspects of the program to  improve their service delivery. Unlike ACF, OPA requires that the third- party evaluations incorporate specific methodological characteristics,  such as control groups or comparison groups and sufficient sample  sizes. In addition, OPA requires that the evaluations for grantees funded  in fiscal year 2004 and beyond account for baseline and follow-up data  obtained from the standardized questionnaires.
OPA’s requirement that grantees use a standardized set of questionnaires,  with data from these questionnaires used in evaluations, differs from  OPA’s previous requirements. Previously, grantees of the AFL Program  were not required to use a standardized method for collecting data that  could be used to assess the effectiveness of their programs; instead,  grantees chose their own data collection instruments. As a result, an OPA  official explained, the collected data varied from one project to another.  OPA officials said that the agency developed the standardized  questionnaire to ensure uniformity in the data collected and allow the  agency to more effectively aggregate the data reported in evaluations of  individual abstinence-until-marriage education programs.
OPA officials told us that they plan to aggregate information from certain  questions in the standardized set of questionnaires in order to report on  certain performance measures as part of the agency’s annual performance  reports. The measures include the extent of parental involvement in  adolescents’ lives and the extent to which adolescents understand the  benefits of abstinence. An agency official stated that the agency expects to  begin receiving data from grantees that are using these questionnaires in  January 2007. OPA did not previously have long-term measures of the  performance of the AFL Program. Its current measures were developed in  collaboration with the Office of Management and Budget (OMB) in  response to an OMB review in 2004 that found that the AFL Program did  not have any annual performance measures for measuring progress  toward long-term goals.
In addition to requiring their grantees to report on outcomes used to  assess program effectiveness, both ACF and OPA have provided technical  assistance and training to their grantees in order to support grantees’ own  program evaluation efforts. For example, in November 2005 the two  agencies sponsored an evaluation conference for abstinence-until- marriage grantees that included presentations about evaluations and their  methodology. Similarly, ACF’s Office of Planning, Research, and  Evaluation sponsors annual evaluation conferences, and an ACF official  told us that a recent conference placed “a significant emphasis” on the  evaluation of abstinence-until-marriage education programs. In addition,  HHS officials told us that ACF, along with ASPE, is funding a multiyear  project that is designed to identify gaps in abstinence education evaluation  and technical assistance needs, develop materials on abstinence education  evaluation, deliver technical assistance and capacity-building activities  related to program evaluation, and develop research reports related to  abstinence education. OPA officials also told us that they attempt to help  ensure grantees’ progress and effectiveness by offering various technical  assistance workshops and conferences. For example, in May 2006 OPA  provided a 2-day training conference to its grantees on the importance of  program evaluations and administering evaluation instruments. In  addition, OPA officials stated that the agency contracts with evaluation  consultants, who review grantees’ evaluation tools and activities. OPA  officials explained that these consultants provide in-depth technical  assistance to grantees on how to improve grantees’ evaluations.
section_title: ACF Uses National Data on Adolescent Behavior as a Measure of the Overall Effectiveness of Its Two Programs
paragraphs: Requiring outcome reporting from state and community-based grantees is  not ACF’s only effort to assess the effectiveness of its two programs. ACF  also analyzes trends in adolescent behavior, as reflected in national data  on birth rates among teens and the proportion of surveyed high school  students reporting that they have had sexual intercourse. ACF uses these  national data as a measure of the overall effectiveness of its State and  Community-Based Programs, comparing the national data to program  targets. In its annual performance reports, the agency summarizes the  progress being made toward lowering the rate of births to unmarried  teenage girls and the proportion of students (grades 9-12) who report  having ever had sexual intercourse.
ACF’s use of national data to assess the effectiveness of the State and  Community-Based Programs represents a change from how HRSA  assessed the overall effectiveness of these programs. Whereas ACF  compares national data on adolescent behavior to program targets, HRSA  aggregated data from its state and community-based grantees. HRSA’s  state grantees were allowed to select the data sources used to gauge their  progress against certain performance measures. For example, in its annual  performance reports on the State Program, HRSA reported information on  the percentage of its state grantees meeting target rates for reducing the  proportion of adolescents who have engaged in sexual intercourse, the  incidence of youths aged 15 to 19 who have contracted selected STDs, and  the rate of births among youths aged 15 to 17. To determine their progress  in meeting their target rates, some state grantees, for example, reported  national data from the Youth Risk Behavior Surveillance System, while  other grantees reported state-collected data. After ACF assumed  responsibility for the State and Community-Based Programs from HRSA,  ACF began using national data on adolescent behavior as a measure of the  programs’ effectiveness. According to ACF officials, the agency changed  how it assessed its programs out of concern over the quality of the data  state grantees were using in their performance reporting and because the  agency wanted to use parallel measures of effectiveness for both  programs. For example, according to state performance reports for fiscal  year 2001 that we reviewed, two reports did not include adolescent  pregnancy rates that year because the states did not collect data on  abortions among this population. In addition, ACF officials told us that  they decided not to use national data on STDs as a measure of program  effectiveness because the goal of reducing STD rates is not as central to  the State and Community-Based Programs as reducing sexual activity and  birth rates among teens. However, one official stated that reducing STDs is  an important “by-product” of the programs.
section_title: Some States Have Made Additional Efforts to Assess the Effectiveness of Abstinence- until-Marriage Education Programs
paragraphs: Some states have made additional efforts to assess the effectiveness of  abstinence-until-marriage education programs, although they are not  required by ACF. Specifically, we found that 6 of the 10 states in our  review that receive funding through ACF’s State Program have made  efforts to conduct evaluations of selected abstinence-until-marriage  programs in their state. All 6 of the states worked with third-party  evaluators, such as university researchers or private research firms, to  perform the evaluations, which in general measure self-reported changes  in program participants’ behavior and attitudes related to sex and  abstinence as indicators of program effectiveness. To obtain this  information, the third-party evaluators have typically relied on surveys  administered to program participants at the start of a program, its  conclusion, and during a follow-up period anywhere from 3 months to  almost 3 years after the conclusion. The third-party evaluations for 4 of the  6 states in our review have been completed as of February 2006, and the  results of these studies have varied.
Among the 4 states that have completed third-party evaluations, 3 states  require the abstinence programs in their state to measure reported  changes in participants’ behavior as an indicator of program  effectiveness—both at the start of the program and after its completion.  The 3 states require their programs to track participants’ reported  incidence of sexual intercourse. In addition, 2 states require their  programs to track biological outcomes, such as pregnancies, births, or  STDs. In addition, 6 of the 10 states in our review require their programs to  track participants’ attitudes about abstinence and sex, such as the number  of participants who make pledges to remain abstinent.
Some states also provide technical assistance to the abstinence-until- marriage programs they support in their state. This assistance is designed  to help programs evaluate and improve their effectiveness. Officials from 5  of the 10 states in our review either told us or provided documentation  that they provide technical assistance on evaluations to abstinence  programs in their state. One state official said that the abstinence-until- marriage programs supported by the state were found to be ill-prepared to  conduct evaluations themselves, and that she now requires these programs  to dedicate a portion of their grants to contract with a third-party or state  evaluator to assist them in program-level evaluations. Officials from  another state told us that they contract with a private organization of  public health professionals in order to provide evaluation consultation and  technical assistance for the abstinence-until-marriage programs the state  supports.
section_title: ASPE, CDC, and NIH Are Funding Studies Designed to Assess the Effectiveness of Abstinence-until-Marriage Education Programs
paragraphs: In addition to ACF and OPA, other organizational units within HHS have  made efforts to assess the effectiveness of abstinence-until-marriage  education programs. ASPE is currently sponsoring a study of the  Community-Based Program and a study of the State Program. For the  former program, ASPE has contracted with Abt Associates to help design  the study, and an ASPE official told us that once the agency selects an  appropriate design, it will competitively award a contract to conduct the  study. For the latter program, ASPE has contracted with Mathematica  Policy Research, Inc. (Mathematica), which is in the process of examining  the impact of five programs funded through the State Program on  participants’ attitudes and behaviors related to abstinence and sex. As of  August 2006, Mathematica has published two reports on findings from its  study—an interim report documenting the experiences of schools and  communities that receive abstinence-until-marriage education funding,  and a report on the first-year impacts of selected state abstinence-until- marriage education programs. Mathematica’s final report, which has not  been completed, will examine the impact of the State Program on  behavioral outcomes, including abstinence, sexual activity, risk of STDs,  risk of pregnancy, and drug and alcohol use. An ASPE official told us that  the agency expects a final report to be published in 2007.
Like ASPE, CDC has made its own effort to assess the effectiveness of  abstinence-until-marriage education. CDC is sponsoring a study to  evaluate the effectiveness of two middle school curricula—one that  complies with abstinence education program requirements and one that  teaches a combination of abstinence and contraceptive information and  skills. In CDC’s study, five middle schools chosen at random will receive  a program consisting of abstinence-until-marriage education exclusively;  five schools will receive comprehensive sex education, which also  includes information on contraception; and five schools will be assigned to  a control group. The study will examine the relative effectiveness of the  programs on behavioral outcomes such as reported sexual risk behaviors  and changes in attitudes related to abstinence and sex. CDC plans to  recruit approximately 1,500 seventh grade students into its study and will  follow them over a 2-year period. The agency expects to complete the  study in 2009.
NIH has funded studies comparing the effectiveness of education  programs that focus only on abstinence with the effectiveness of sex  education programs that teach both abstinence and information about  contraception. As of August 2006, NIH is funding five studies, which in  general are comparing the effects of these two types of programs on the  sexual behavior and related attitudes among groups of either middle  school or high school students. For example, in one NIH study,  researchers are using groups of seventh and eighth grade adolescents to  assess the impact of a variety of programs on, among other issues,  adolescents’ reported sexual activities, knowledge, and beliefs. For this  study, researchers are comparing these outcomes among students who  received abstinence-until-marriage education; students who received a  combination of abstinence and contraceptive education; and students who  participated in a general health class, who serve as a comparison group.
NIH expects both this study and its other four studies to be competed in  2006.
section_title: Other Researchers Have Also Made Efforts to Assess the Effectiveness of Abstinence- until-Marriage Education Programs
paragraphs: In addition to the efforts of researchers working on behalf of HHS and  states, other researchers—such as those affiliated with universities and  various advocacy groups—have made efforts to study the effectiveness of  abstinence-until-marriage education programs. This work includes studies  of the outcomes of individual programs and reviews of other studies on  the effectiveness of individual abstinence-until-marriage education  programs. In general, research studies on the effectiveness of individual  abstinence-until-marriage education programs have examined the extent  to which they changed participants’ demonstrated knowledge, declared  intentions, and reported behavior related to sexual activity and abstinence.  For example, some studies examined the impact of abstinence-until- marriage education programs on participants’ knowledge of concepts  taught in the programs, as well as participants’ declared attitudes about  abstinence and teen sex. Some studies examined the impact of these  programs on such outcomes as participants’ declared commitment to  abstain from sex until marriage, participants’ understanding of the  potential consequences of having intercourse, and participants’ reported  ability to resist pressures to engage in sexual activity. Some of the studies  we reviewed examined the impact of abstinence-until-marriage programs  on participants’ sexual behavior, as measured, for example, by the  proportion of participants who reported having had sexual intercourse and  the frequency of sexual intercourse reported by participants. In general,  the efforts to study and build a body of research on the effectiveness of  most abstinence education programs have been under way for only a few  years, in part because grants under the two programs that account for the  largest portion of federal spending on abstinence education—the State  Program and the Community-Based Program—were not awarded until  1998 and 2001, respectively.
section_title: Several Factors Limit the Conclusions That Can Be Drawn about the Effectiveness of Abstinence-until-Marriage Education Programs
paragraphs: Most of the efforts of HHS, states, and other researchers to evaluate the  effectiveness of abstinence-until-marriage education programs included in  our review have not met certain minimum criteria that experts have  concluded are necessary in order for assessments of program  effectiveness to be scientifically valid. For example, most of the efforts  included in our review did not include experimental or quasi-experimental  designs, nor did they measure behavioral or biological outcomes. In  addition, the results of some assessment efforts that meet the criteria of a  scientifically valid assessment have varied, and two key studies that meet  these criteria have not yet been completed.
subsections: section_title: Experts Have Developed Criteria to Evaluate Efforts to Assess Abstinence-until- Marriage Education Programs
paragraphs: In an effort to better assess the merits of the studies that have been  conducted on the effectiveness of sexual health programs—including  abstinence-until-marriage education programs—scientific experts have  developed criteria that can be used to gauge the scientific rigor of these  evaluations. For example, in 2001, the National Campaign to Prevent Teen  Pregnancy—an organization focused on reducing teen pregnancy— published a report by a panel of scientific experts that assessed the  evidence reported on abstinence-until-marriage education programs in  peer-reviewed journals and other literature. The panel developed criteria  that an evaluation of a program’s effectiveness must meet in order for the  program’s results to be considered scientifically valid. In addition, in 2004,  former U.S. Surgeon General David Satcher convened a panel of experts to  discuss, among other things, best practices for evaluating the effectiveness  of sexual health education programs—including abstinence-until-marriage  education programs. This panel published a report in 2006 that describes  similar scientific criteria that assessments of program effectiveness need  to meet in order for their results to be scientifically valid. Further, experts  we interviewed agreed that these criteria are important for ensuring that  the results of a study support valid conclusions. In general, these panels,  as well as the experts we interviewed, agreed that scientifically valid  studies of a program’s effectiveness should include the following  characteristics:    An experimental design that randomly assigns individuals or schools to  either an intervention group or control group, or a quasi-experimental  design that uses nonrandomly assigned but well-matched comparison  groups. According to the panel of scientific experts convened by the  National Campaign to Prevent Teen Pregnancy, experimental designs or  quasi-experimental designs with well-matched comparison groups have at  least three important strengths that are typically not found in other  studies, such as those that use aggregated data: they evaluate specific  programs with known characteristics, they can clearly distinguish between  participants who did and did not receive an intervention, and they control  for other factors that may affect study outcomes. Therefore, experimental  and quasi-experimental study designs have a greater ability to assess the  causal impact of specific programs than other types of studies. According  to scientific experts, studies that include experimental or quasi- experimental designs should also collect follow-up data for a minimum  number of months after subjects receive an intervention. Experts  reported that follow-up periods are important in order to identify the  effects of a program that are not immediately apparent or to determine  whether these effects diminish over time. In addition, experts have  reported that studies should have a sample size of at least 100 individuals  for study results to be considered scientifically valid.
Studies should assess or measure changes in biological outcomes or  reported behaviors instead of attitudes or intentions. According to  scientific experts, biological outcomes—such as pregnancy rates, birth  rates, and STD rates—and reported behaviors—such as reported initiation  and frequency of sexual activity—are better measures of the effectiveness  of abstinence-until-marriage programs, because adolescent attitudes and  intentions may or may not be indicative of actual behavior. For example,  adolescents may report that they intend to abstain from sexual intercourse  but may not actually do so.
section_title: Most Efforts to Assess the Effectiveness of Abstinence- until-Marriage Education Programs Have Not Used an Experimental or Quasi- Experimental Design
paragraphs: Many of the efforts by HHS, states, and other researchers that we  identified in our review lack at least one of the characteristics of a  scientifically valid study of program effectiveness. That is, most of the  efforts to assess the effectiveness of these programs have not used  experimental or quasi-experimental designs with sufficient follow-up  periods and sample sizes to make their conclusions scientifically valid. For  example, ACF—and before it, HRSA—used, according to ACF officials,  grantee reporting on outcomes in order to monitor grantees’ performance,  target training and technical assistance, and help grantees improve service  delivery. However, because the outcomes reported by grantees have not  been produced through experimentally or quasi-experimentally designed  studies, such information cannot be causally attributed to any particular  abstinence-until-marriage education program. While ACF requires its fiscal  year 2006 grantees of the Community-Based Program to contract with  third-party evaluators to select and monitor outcomes for their programs,  ACF is not specifically requiring these grantees to use experimental or  quasi-experimental designs. Therefore, it is not clear whether these  evaluations will include such designs. Similarly, ACF’s use of national data  on adolescent behavior and birth rates to assess its State and Community- Based Programs is of limited value because these data do not distinguish  between those who participated in abstinence-until-marriage education  programs and those who did not. Consequently, these national data sets,  which represent state-reported vital statistics and a nationwide survey of  high school students, cannot be used to causally link declines in birth rates  and adolescent sexual activity to the effects of specific abstinence-until- marriage education programs.
Similarly, the efforts we identified by states and researchers to assess the  effectiveness of abstinence-until-marriage education programs often did  not include experimental or quasi-experimental designs. None of the state  evaluations we reviewed that have been completed included randomly  assigned control groups. For instance, one state evaluation that we  reviewed only included students who volunteered to participate in the  study. This evaluation report stated that the absence of a randomly  assigned control group in the evaluation did not allow the evaluators to  determine whether observed changes in participants’ reported sexual  behavior—as indicated through surveys administered at the beginning and  end of a program—could be attributed to the abstinence-until-marriage  education program. Similarly, some of the journal articles that we  reviewed described studies to assess the effectiveness of abstinence-until- marriage programs that did not include experimental or quasi- experimental designs needed to support scientifically valid conclusions  about the programs’ effectiveness. In these studies, researchers  administered questionnaires to study participants before and after they  completed an abstinence-until-marriage education program and assessed  the extent to which the responses of participants changed. These studies  did not compare the responses of study participants with a group that did  not participate in an abstinence-until-marriage education program. In  addition, some of the studies used insufficient follow-up periods, thereby  limiting the conclusions that can be drawn about the effectiveness of the  abstinence-until-marriage education programs being studied. For example,  two journal articles that we reviewed described studies that measured the  effectiveness of abstinence-until-marriage programs in delaying the  initiation of sexual activity from 1 to 2 months after completion of the  program. Scientific experts consider this follow-up period too short to  assess whether the programs had a valid effect.
According to scientific experts, HHS, states, and other researchers face a  number of challenges in designing experimental or quasi-experimental  studies of program effectiveness. According to these experts, experimental  or quasi-experimental studies can be expensive and time-consuming to  carry out, and many grantees of abstinence-until-marriage education  programs have insufficient time and funding to support these types of  studies. Moreover, it can be difficult for researchers assessing abstinence- until-marriage education programs to convince school districts to  participate in randomized intervention and control groups, in part because  of sensitivities to surveying attitudes, intentions, and behaviors related to  abstinence and sex. For example, in a third-party evaluation of its  program, one grantee of the State Program originally planned to  administer follow-up surveys 1 year after participants finished their  abstinence education program, but the evaluators decided not to conduct  this follow-up because of confidentiality concerns and the difficulty of  locating students. In addition, the contractors hired to design ASPE’s  study of the effectiveness of the Community-Based Program have reported  difficulties finding school districts that are willing to participate in  randomly assigned intervention and control groups receiving either  abstinence-until-marriage education or comprehensive sex education. An  ASPE official told us that although a “randomized approach” is the best  design for assessing the effectiveness of a program, the approach is also  the most difficult to conduct.
section_title: Most Efforts to Assess the Effectiveness of Abstinence- until-Marriage Education Programs Have Not Measured Behavioral or Biological Outcomes
paragraphs: Another factor that limits the conclusions that can be drawn about the  effectiveness of abstinence-until-marriage education programs is the fact  that most efforts in our review to study the effectiveness of these  programs did not measure changes in behavioral or biological outcomes  among participants. Instead, most of the efforts we identify in our review  used reported intentions and attitudes in order to assess the effectiveness  of abstinence-until-marriage programs. For example, neither ACF’s  community-based grantees nor OPA’s AFL grantees are required to report  on behavioral or biological outcomes, such as rates of intercourse or  pregnancy. Similarly, the journal articles we reviewed were more likely to  use reported attitudes and intentions—such as study participants’ reported  attitudes about premarital sexual activity or their reported intentions to  remain abstinent until marriage—rather than their reported behaviors or  biological outcomes to assess the effectiveness of abstinence-until- marriage programs. For example, in one journal article we reviewed,  participants were asked to rate the likelihood that they would have sexual  intercourse as unmarried teenagers; another journal article described a  study in which participants rated the likelihood that they would have  sexual intercourse in the next year, before finishing high school, and  before marriage.
Experts, as well as state and HHS officials, have reported that it can be  difficult to obtain scientifically valid information on biological outcomes  and sexual behaviors. Specifically, experts have reported that when  measuring an abstinence-until-marriage education program’s affect on  biological outcomes—such as reducing pregnancy or birth rates—it is  necessary to have large sample sizes in order to determine whether a small  change in biological outcomes is the result of the abstinence-until- marriage education program. In addition, state and federal officials told us  that they have experienced difficulties obtaining information on sexual  behaviors because of the sensitive nature of the information they were  trying to collect. For example, one state official told us that her state’s  effort to evaluate abstinence-until-marriage education programs was only  able to measure changes in participants’ reported attitudes, instead of  behaviors, because the evaluators needed to obtain consent from the  parents of the program participants in order to ask them about their  sexual behavior. The state official explained that the requirement to obtain  consent from parents raised issues of self-selection, and therefore state  officials decided to ultimately halt the study and only report on the  attitudes that they had measured. In another example, ACF’s fiscal year  2006 budget justification reports that ACF has had some difficulty in  obtaining reliable data from state grantees, in part because questions  about teenage sexual behavior are sensitive. OPA officials also  acknowledged that many communities will not allow grantees to ask  program participants questions about their sexual behavior because the  communities believe such questions are too intrusive. One OPA official  said that such restrictions affect the agency’s ability to measure behavioral  outcomes, explaining that OPA cannot measure what it cannot ask about.
section_title: Results of Some Scientifically Valid Assessment Efforts Have Varied, and Other Key Studies Have Not Been Completed
paragraphs: Among the assessment efforts we identified are some studies that meet the  criteria of a scientifically valid effectiveness study. However, results of  these studies have varied, and this limits the conclusions that can be  drawn about the effectiveness of abstinence-until-marriage education  programs. Some researchers have reported that abstinence-until-marriage  education programs have resulted in adolescents reporting having less  frequent sexual intercourse or fewer sexual partners. For example, in one  study of middle school students, participants in an abstinence-until- marriage education program who had sexual intercourse during the  follow-up period were 50 percent less likely to report having two or more  sexual partners when compared with their nonparticipant peers. In  contrast, other studies have reported that abstinence-until-marriage  education programs did not affect the reported frequency of sexual  intercourse or number of sexual partners. For example, one study of  middle school students found that participants of an abstinence-until- marriage program were not less likely than nonparticipants at the 1 year  follow-up to report less frequent sexual intercourse or fewer sexual  partners. In addition to these varied findings, one study found that an  abstinence-until-marriage program was effective in delaying the initiation  of sexual intercourse in the short term but not long term. Experts with  whom we spoke emphasized that there are still too few scientifically valid  studies completed to date that can be used to determine conclusively  which, if any, abstinence-until-marriage programs are effective.
Additionally, among the assessment efforts we identified are some studies  that experts anticipate will meet the criteria of a scientifically valid  effectiveness study but are not yet completed. One of these key studies is  the final Mathematica report, contracted by ASPE, on the State Program.  The final report was originally slated for publication in 2005, but an ASPE  official stated that the final report has been delayed until 2007 so that  researchers can extend the follow-up period to improve their response  rate and the reliability of the information they collect. Another key study is  CDC’s research on middle school programs, which is not expected to be  completed until 2009.
Experts and federal officials we interviewed stated that they expect the  results of these two federally funded studies to add substantively to the  body of research on the effectiveness of abstinence-until-marriage  education programs. One expert with whom we spoke said that she  expects the final Mathematica report on participants’ behaviors to provide  the groundwork for the field. Another expert we interviewed stated that  the CDC study was very well-designed and she expects the results to  contribute to the development of effective abstinence-until-marriage  education curricula.
section_title: Conclusions
paragraphs: There have been various efforts—by HHS, states, and others—to assess  the scientific accuracy of educational materials used in abstinence-until- marriage education programs and the effectiveness of these programs.  However, efforts to evaluate both the accuracy and effectiveness of  abstinence-until-marriage education programs have been, in various ways,  limited. ACF, which administers the two programs that account for the  largest portion of federal spending on abstinence-until-marriage education,  does not review or require its grantees to review program materials for  scientific accuracy. In addition, not all grantees of the State Program have  chosen to review their materials. Because of these limitations, ACF cannot  be assured that the materials used in its State and Community-Based  Programs are accurate. Moreover, OPA, which reviews all grantees’  proposed abstinence-until-marriage educational materials, and states that  review educational materials have found inaccuracies in some educational  materials used by abstinence-until-marriage programs.
Similarly, most of the efforts described in our review to assess the  effectiveness of abstinence-until-marriage programs have not met  minimum scientific criteria needed to draw valid conclusions about their  effectiveness. Specifically, most efforts by agencies, states, and other  researchers have not included experimental or quasi-experimental designs  that can establish whether changes in behaviors or biological outcomes  can be causally linked to specific abstinence-until-marriage education  programs. While these types of studies are time-consuming and expensive,  experts said that they are the only definitive way to draw valid conclusions  about the effectiveness of these programs. In addition, among the  assessment efforts we identified are some studies funded by HHS that  experts anticipate will meet the criteria of a scientifically valid  effectiveness study but are not yet completed. When completed, these  HHS-funded studies may add substantively to the body of research on the  effectiveness of abstinence-until-marriage education programs.
section_title: Recommendation for Executive Action
paragraphs: To address concerns about the scientific accuracy of materials used in  abstinence-until-marriage education programs, we recommend that the  Secretary of HHS develop procedures to help assure the accuracy of such  materials used in the State and Community-Based Programs. To help  provide such assurances, the Secretary could consider alternatives such as  (1) extending the approach currently used by OPA to review the scientific  accuracy of the factual statements included in abstinence-until-marriage  education to materials used by grantees of ACF’s Community-Based  Program and requiring grantees of ACF’s State Program to conduct such  reviews or (2) requiring grantees of both programs to sign written  assurances in their grant applications that the materials they propose  using are accurate.
section_title: Agency Comments and Our Evaluation
paragraphs: HHS provided written comments on a draft of this report. (See app. III.)
In its written comments, HHS stated that it will consider requiring grantees  of both ACF programs to sign written assurances in grant applications that  the materials they use are accurate. Regarding accuracy, HHS’s written  comments also noted that all applicants for federal assistance attest on the  application form--Standard Form 424--that all data in their applications are  “true and correct,” and that in the view of HHS, this applies to information  presented in curricula funded by federal grants. However, as we stated in  the draft report, grantees of the State Program are not required to submit  curricula as a part of their applications; therefore, the attestation in  Standard Form 424 would not apply to curricula used by those grantees. In  addition, as stated in the draft report, some states have reviewed materials  used in abstinence-until-marriage education programs, but these reviews  occurred after they received funding from ACF. Further, while grantees of  the Community-Based Program were required to submit copies of their  curricula and a Standard Form 424 in fiscal year 2006 as part of their  applications, none of the materials specifically require an assurance of  scientific accuracy. Further, OPA and states have found inaccuracies in  educational materials used by abstinence-until-marriage programs.
HHS’s written comments also stated that ACF requires that curricula  conform to HHS’s standards grounded in scientific literature. HHS’s  comments refer to the curriculum standards for this program that detail  what types of information must be included in abstinence-until-marriage  curricula, and the comments stated that the curricula must provide  supporting references for this information. Further, HHS’s comments  stated that ACF staff review the curricula to ensure compliance with these  standards. The draft report stated this. However, a requirement that  curricula include certain types of information does not necessarily ensure  the accuracy of the scientific facts included in the abstinence-until- marriage materials. For example, while education materials may include  information on failure rates associated with contraceptives or STD  infections, this information may be outdated or otherwise inaccurate or  incomplete. HHS’s written comments also stated that if it finds inaccurate  statements during the review process or at any time during the grant  period, ACF works with grantees to take corrective action. To ensure  completeness, we have added this statement to the report. Further, HHS  stated that 2 inaccuracies cited in the draft report had been corrected  before our work began. We believe HHS is referring to inaccuracies  identified by OPA during its review of materials for scientific accuracy and  this reinforces the need for review of materials used by ACF’s grantees.
As HHS noted in its written comments, we did not define the term  scientific accuracy. HHS stated that it disagreed with certain findings of  the report because it was difficult to precisely determine the criteria we  employed in making the recommendation as to scientific accuracy. As we  stated in the scope and methodology section of the draft report, the  objective of our work was to focus on efforts by HHS and states to review  the accuracy of scientific facts included in abstinence-until-marriage  education materials. Performing an independent assessment of the criteria  used by these entities to determine the scientific accuracy of education  materials or the quality of the reviews was beyond the scope of the work.
Regarding effectiveness, HHS’s written comments also described a  number of actions it is taking to determine program effectiveness and  improve the quality of programs and research. Specifically, HHS’s  comments described (1) studies undertaken or funded by ASPE, CDC, and  NIH; (2) technical assistance provided by OPA and ACF; (3) grantee  evaluation requirements; and (4) ACF and OPA requirements for the  amount of grant funds to be spent on evaluations. All of this information  was included in our draft report. HHS’s comments also described a new  effort funded by ACF and ASPE that is designed to build capacity for  quality research in the field of abstinence education. We added  information on this effort to the report. HHS’s written comments also  describe evaluations that resulted from an Abstinence Education  Evaluation Conference sponsored by ACF and OPA. While this conference  was described in the draft report, we added more detail regarding the  content of the conference. HHS’s written comments also describe OPA’s  efforts to assess the effectiveness of the AFL Program. We had included  this information in the draft report.
HHS’s written comments stated that it may be too soon to draw  conclusions about the effectiveness of ACF’s and OPA’s programs, in part,  because key studies have not been completed. We agree and discussed this  in the draft report. As we noted in the draft report, key studies funded by  HHS that experts anticipate will meet the criteria of a scientifically valid  effectiveness study are not yet completed, but when completed these HHS  funded studies may add substantively to the body of research on the  effectiveness of abstinence-until-marriage education programs.
In addition, the comments stated that having an inadequate amount of  scientifically valid and conclusive evaluation studies is not unique to  abstinence-until-marriage education programs, and a recent ASPE review  of comprehensive sex education programs found mixed results on their  effectiveness. However, the scope of our report was focused on  abstinence-until-marriage education programs, and we did not review  comprehensive sex education programs or make any comparisons  between the two types of programs.
HHS also provided technical comments, which we incorporated into the  report as appropriate.
As agreed with your office, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after its  date. We will then send copies of this report to the Secretary of HHS and  to other interested parties. In addition, this report is available at no charge  on GAO’s Web site at http://www.gao.gov. We will also make copies  available to others upon request.
If you or your staff have any questions about this report, please call me at  (202) 512-3407 or crossem@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix IV.
section_title: Appendix I: HRSA’s Technical Assistance Contract for Abstinence Education
paragraphs: The Health Resources and Services Administration (HRSA) awarded a  contract to the National Abstinence Clearinghouse (NAC) in 2002 to  provide assistance with its Community-Based Abstinence Education  Program (Community-Based Program) and Abstinence Education Program  (State Program). NAC is a nonprofit educational organization whose  mission is to promote the appreciation for and practice of sexual  abstinence until marriage through the distribution of age appropriate,  factual, and medically referenced materials. The purpose of the contract  was (1) to develop national criteria for the review of abstinence-until- marriage educational materials and to create a directory of approved  materials; (2) to provide medical accuracy training to grantees; and (3) to  provide technical support to grantees, such as assistance with program  evaluation.
We are reporting on the steps that HRSA took to award the contract to  NAC in response to concerns that have been raised by a congressional  requester. In general, these concerns centered on the extent to which the  selection process was competitive and whether HRSA identified the  potential for an organizational conflict of interest.
HRSA awarded the contract to address three concerns it had with the  Community-Based Program during 2001, the first year of its  implementation. First, HRSA officials needed guidance to determine  whether abstinence-until-marriage education materials conformed to the  definitional requirements of the Social Security Act. Second, many  grantees lacked the medical background and training to ensure that they  would provide medically accurate, science-based information in their  programs. Third, grantees also lacked experience with the technical  management of federal grants, including how to conduct evaluations of  their programs.
subsections: section_title: HRSA Used Full and Open Competition Procedures to Award the Contract to NAC
paragraphs: HRSA used full and open competition procedures to award the contract to  NAC. In doing so, HRSA (1) publicly solicited proposals from potential  contractors; (2) conducted technical evaluations of both the original  proposals and the revised proposals for those considered to be in the  competitive range; and (3) determined that NAC’s proposal represented  the best overall value to the government. This process, which took place  from May 2002 through September 2002, resulted in HRSA awarding NAC  the contract with a potential value of nearly $2.7 million.
HRSA issued a notice on May 20, 2002, on the FedBizOpps Web site, the  government point of entry for notifying potential contractors of federal  contract opportunities, indicating its intent to publicly request proposals  from prospective contractors in June 2002. On June 20, 2002, HRSA  posted the solicitation on the FedBizOpps Web site indicating that the  abstinence contract would be awarded using full and open competition  procedures, that is, all responsible prospective contractors would be  provided the opportunity to compete. The solicitation, which was a  Request for Proposals (RFP), described the contract objectives, which  included (1) the development of national criteria for the review of  abstinence-until-marriage educational materials and the development of a  directory of approved materials; (2) the provision of medical accuracy  training to grantees; and (3) the provision of technical support to grantees,  such as assistance with program evaluation. The RFP stated that HRSA  intended to award a cost-reimbursement contract with fixed fee for a   1-year base period and 2 option years. This was a best value procurement;  that is, HRSA reserved the right in the RFP to select for award the  proposal that HRSA determined offered the best value to the government,  even if it did not offer the lowest cost. Further, the RFP stated that the  technical evaluation of the prospective contractors’ proposals would  receive paramount consideration in the selection of the contractor.  According to the RFP, this evaluation would include an assessment of the  prospective contractor’s technical approach, the organizational experience  and expertise of the prospective contractor, the plans for personnel and  management of the work, and the prospective contractor’s statement and  understanding of the project purpose. Other factors, such as the estimated  cost, past performance under other contracts for similar services, and the  subcontracting plan would also be considered in the selection process.  Five prospective contractors submitted proposals to HRSA by July 31,  2002, when proposals were due.
HRSA established a review committee to conduct the technical evaluation  of the five proposals. This committee included three voting members and a  nonvoting chairperson. The Director of HRSA’s Community-Based and  State Programs and two analysts from other programs within the  Department of Health and Human Services (HHS) served as the voting  members, and the chairperson of the review committee was a project  officer of HRSA’s Community-Based Program. The committee members  conducted the technical evaluation of the proposals, according to the  criteria in the RFP, as described above. Three proposals with the highest  technical scores were determined to be in the competitive range, with  NAC’s proposal receiving the highest technical score. HRSA requested in  writing that the competitive range offerors address certain technical and  cost issues and submit revised proposals to HRSA by September 17, 2002.  For example, HRSA requested that one of the prospective contractors  other than NAC clearly describe its proposed management of day-to-day  tasks of the contract and provide justification for several labor and travel  expenditures. HRSA did not have oral discussions with the competitive  range offerors. HRSA’s review committee evaluated the revised proposals  and again gave NAC’s revised proposal the highest technical score.
Although NAC’s estimated cost was not the lowest among the proposals in  the competitive range, HRSA determined that NAC had proposed a  realistic cost estimate for the contract. Accordingly, and in light of the  NAC proposal’s high technical rating and the RFP’s evaluation criteria  giving paramount consideration to the technical evaluation, HRSA  determined that NAC’s proposal represented the best value to the  government. HRSA awarded a contract to NAC on September 27, 2002.  The contract had a 1-year base period of performance with an estimated  value of $854,681, and included 2 option years for a total potential value of  $2,673,784. According to a HRSA official, this cost-reimbursement contract  did not include a fee. All of the prospective contractors were made aware  that a debriefing to explain the selection decision and contract award  would be provided at their request. One prospective contractor requested  and received a debriefing from HRSA. No protests were filed with the  agency challenging the award of the contract to NAC. There were no bid  protests filed with GAO.
section_title: HRSA Identified No Actual or Potential Organizational Conflicts of Interest
paragraphs: HRSA officials told us that they did not identify any actual or potential  organizational conflicts of interest during the acquisition process. As  defined in the Federal Acquisition Regulation (FAR), an organizational  conflict of interest arises where    because of other activities or relationships, a person is unable or  potentially unable to provide impartial assistance or advice to the  government; or the person’s objectivity in performing the contract work is or might be  otherwise impaired; or   a person has an unfair competitive advantage.
An organizational conflict of interest may result when factors create an  actual or potential conflict of interest during performance of a contract, or  when the nature of the work to be performed under one contract creates  an actual or potential conflict of interest involving a future acquisition.  Under the FAR, contracting officers are required to analyze planned  acquisitions to identify and evaluate potential organizational conflicts of  interest as early in the acquisition process as possible, and to take steps to  avoid, neutralize, or mitigate significant potential conflicts of interest  before a contract is awarded.
According to HRSA’s contracting officer, HRSA did not identify any actual  or potential organizational conflicts of interest. In reaching this  conclusion, the contracting officer told us that he reviewed the statement  of work, including the background and objectives of the proposed  contract, the stated purpose of the contact, the criteria established to  evaluate the proposals, the past performance of the competitors, and  NAC’s proposal. HRSA’s contracting officer also told us that he did not  formally document his assessment of organizational conflict of interest.
section_title: Appendix II: Methodology for Identifying and Reviewing Research Studies
paragraphs: To identify research studies that examine the effectiveness of abstinence- until-marriage education programs among adolescents and young adults,  we searched two reference database systems, PubMed and ProQuest. We  used the following keywords to search for research studies that were  published from January 1, 1998, through May 22, 2006: “virginity,”  “abstinence education,” “abstinence and curriculum,” “abstinence only,”  “teen pregnancy and prevention,” and “abstinence until marriage.” We  reviewed the research article titles that were generated from the PubMed  and ProQuest searches and identified articles that appeared to focus on  the evaluation of the effectiveness of abstinence-until-marriage education  programs. In cases where we could not determine, based on the title,  whether a study appeared to focus on an abstinence-until-marriage  education program evaluation, we reviewed a summary of the article to  obtain more information about the research study. We also examined  previous summaries of the literature to identify additional research  studies.
We then selected research studies for inclusion in our literature review if  they met three criteria. First, the study evaluated a group-based,  abstinence-until-marriage education program. We did not select studies  that evaluated one-on-one interactions, such as education programs  focused exclusively on parent-child interactions, or that evaluated media  campaigns. We reviewed the description of each education program and  curriculum, as described in the study, to determine whether an abstinence- until-marriage education program was being evaluated. Education  programs that were described as including detailed contraceptive  information in their curricula, for example, were not classified as  abstinence-until-marriage programs. Second, the study targeted  adolescents and young adults in the United States, for example, by  indicating that participants in the evaluation were high school or middle  school students. Third, the study was a quantitative rather than a  qualitative evaluation of an abstinence-until-marriage education program.  We selected 13 research studies for inclusion in our literature review.
We reviewed the selected research studies to obtain detailed information  about their methodologies and outcome variables. For example, we  determined whether each study used an experimental or quasi- experimental design and whether the outcome measures included  attitudes, behavioral intentions, behaviors such as initiation of sexual  intercourse, or a combination of these.
section_title: Appendix III: Comments from the Department of Health and Human Services
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Kristi Peterson, Assistant  Director; Kelly DeMots; Pam Dooley; Krister Friday; Julian Klazkin; and  Amy Shefrin made key contributions to this report.