Abstinence Education: Assessing the Accuracy and Effectiveness of Federally Funded Programs
Published: 2008-04-23T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Among the efforts of the Department of Health and Human Services (HHS) to reduce the incidence of sexually transmitted diseases and unintended pregnancies, the agency provides funding to states and organizations that offer abstinence-until-marriage education. GAO was asked to testify on the oversight of federally funded abstinence-until-marriage education programs. This testimony is primarily based on Abstinence Education: Efforts to Assess the Accuracy and Effectiveness of Federally Funded Programs, GAO-07-87 (Oct. 3, 2006). In this testimony, GAO discusses efforts by (1) HHS and states to assess the scientific accuracy of materials used in abstinence-until-marriage education programs and (2) HHS, states, and researchers to assess the effectiveness of abstinence-until-marriage education programs. GAO also discusses a Public Health Service Act requirement regarding medically accurate information about condom effectiveness. GAO focused on the three main federally funded abstinence-until-marriage programs and reviewed documents and interviewed HHS officials in the Administration for Children and Families (ACF) and the Office of Population Affairs (OPA). To update certain information, GAO contacted officials from ACF and OPA.
section_title: What GAO Found
paragraphs: Efforts by HHS and states to assess the scientific accuracy of materials used in abstinence-until-marriage education programs have been limited. As of October 2006, HHS's ACF--which awards grants under two programs that account for the largest portion of federal spending on abstinence education--did not review its grantees' education materials for scientific accuracy, nor did it require grantees of either program to do so. Not all states that receive funding from ACF had chosen to review their program materials for scientific accuracy. OPA reviewed the scientific accuracy of grantees' proposed education materials, and any inaccuracies found had to be corrected before those materials could be used. The extent to which federally funded abstinence-until-marriage education materials are inaccurate was not known, but OPA and some states reported finding inaccuracies. GAO recommended that the Secretary of HHS develop procedures to help assure the accuracy of abstinence-until-marriage education materials. An ACF official reported that ACF is currently implementing a process to review the accuracy of Community-based grantees' curricula and has required those grantees to sign assurances that the materials they propose using are accurate. The official also reported that, in the future, state grantees will have to provide ACF with descriptions of their strategies for reviewing the accuracy of their programs. As of August 2006, HHS, states, and researchers had made a variety of efforts to assess the effectiveness of abstinence-until-marriage education programs, but a number of factors limit the conclusions that can be drawn about the programs' effectiveness. ACF and OPA have required their grantees to report on various outcomes used to measure program effectiveness. To assess the effectiveness of its grantees' programs, ACF has analyzed national data on adolescent birth rates and the proportion of adolescents who report having had sexual intercourse. Additionally, 6 of the 10 states in GAO's review worked with third-party evaluators to assess the effectiveness of abstinence-until-marriage programs in their states. However, the conclusions that can be drawn are limited because most of the efforts to evaluate program effectiveness have not met certain minimum criteria that experts have concluded are necessary for such assessments to be scientifically valid. Additionally, the results of some efforts that do meet such criteria have varied. While conducting work for its October 2006 report, GAO identified a legal matter that required the attention of HHS. Section 317P(c)(2) of the Public Health Service Act requires certain educational materials to contain medically accurate information about condom effectiveness. GAO concluded that this requirement would apply to abstinence education materials prepared and used by federal grant recipients, depending on their substantive content, and recommended that HHS adopt measures to ensure that, where applicable, abstinence education materials comply with this requirement. The fiscal year 2007 program announcement for the Community-based Program provides information about the applicability of this requirement, and future State and Community-based Program announcements are to include this information.

section_title: Letter
paragraphs: I am pleased to be here today as you examine federally funded abstinence- until-marriage education programs. Reducing the incidence of sexually  transmitted diseases (STD) and unintended pregnancies among  adolescents has been an important objective of the Department of Health  and Human Services (HHS). Among its efforts to do so, HHS funds  abstinence-until-marriage education programs. These programs are  delivered by a variety of entities, including schools, human service  agencies, and faith-based organizations. Studies have raised concerns  about the accuracy of the educational materials that are incorporated into  these programs, as well as the effectiveness of the programs themselves.  My remarks today are primarily based on our October 2006 report on the  oversight of federally funded abstinence-until-marriage programs,  Abstinence Education: Efforts to Assess the Accuracy and Effectiveness  of Federally Funded Programs (GAO-07-87). In that report, we  recommended that the Secretary of Health and Human Services develop  procedures to help assure the accuracy of such materials. Today, I will  discuss findings from our report on (1) efforts by HHS and states to assess  the scientific accuracy of materials used in abstinence-until-marriage  programs, and (2) efforts by HHS, states, and researchers to assess the  effectiveness of abstinence-until-marriage education programs as well as  updates on selected information. I will also discuss a legal matter that  came to our attention during the course of our work regarding the  applicability of section 317P(c)(2) of the Public Health Service Act to  Abstinence Education programs. We recommended in a letter dated  October 18, 2006, that HHS adopt measures to ensure that, where  applicable, abstinence-until-marriage education materials comply with the  requirement that educational materials specifically designed to address  STDs contain medically accurate information about condom effectiveness  in preventing the STDs the materials were designed to address.
For our assessment of the accuracy and effectiveness of abstinence-until- marriage education programs, we focused our review on the three main  federally funded abstinence-until-marriage programs: the Abstinence  Education Program (State Program), the Community-Based Abstinence  Education Program (Community-Based Program), and the Adolescent  Family Life (AFL) Program. The State Program and the Community-Based  Program are both administered by HHS’s Administration for Children and  Families (ACF); AFL is administered by HHS’s Office of Population Affairs  (OPA). According to HHS, funding for the three abstinence-until-marriage  programs was about $165 million in fiscal year 2007.
In order to describe the efforts to assess the scientific accuracy of  program materials, we reviewed published reports, program  announcements, Federal Register notices, agency Web sites, and other  documents related to abstinence-until-marriage education. We did not  assess the criteria used to determine the scientific accuracy of education  materials or the quality of the reviews. We interviewed officials from ACF  and OPA. We also interviewed officials from the 10 states that received the  largest share of federal funding (together accounting for 51 percent of the  total funding in fiscal year 2005) through the State Program for abstinence- until-marriage education.
To describe efforts by HHS, states, and researchers to assess the  effectiveness of abstinence-until-marriage education programs, we  focused on efforts that examined the extent to which these programs  achieved their program goals. In general, these goals include teaching  adolescents to abstain from sexual activity until marriage in order to avoid  unintended pregnancies, STDs, and related health problems. As part of our  review, we compared these efforts to the design characteristics that  experts have identified as important for a scientifically valid study of  program effectiveness. We reviewed journal articles and other published  reports, agency budget submissions, program announcements, agency and  grantee performance reports, Federal Register notices, agency Web sites,  and other documents related to abstinence-until-marriage education. We  also interviewed officials from ACF, OPA, the Centers for Disease Control  and Prevention (CDC), the National Institutes of Health (NIH), the Office  of the Assistant Secretary for Planning and Evaluation (ASPE), and   10 states that received the largest share of federal funding for abstinence- only education through the State Program in fiscal year 2005. We focused  our review on efforts to assess the scientific accuracy of materials and the  effectiveness of the programs during fiscal year 2006. We conducted this  work from October 2005 through September 2006 and during April 2008 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
To assess the applicability of section 317P(c)(2) of the Public Health  Service Act to abstinence-until-marriage education programs, we reviewed  the statute, pertinent legislative history, and relevant program guidance. In  addition, we solicited the views of HHS officials on this issue.
In summary, we found that efforts by HHS and states to assess the  scientific accuracy of materials used in abstinence-until-marriage  education programs had been limited. ACF did not review its grantees’  education materials for scientific accuracy and did not require that  grantees of either the State Program or the Community-Based Program do  so. In addition, not all states that received funding through ACF’s State  Program chose to review their program materials for scientific accuracy.  Five of the 10 states in our review conducted such reviews. In contrast to  ACF, OPA did review the scientific accuracy of AFL grantees’ proposed  educational materials and any inaccuracies found had to be corrected  before the materials could be used. While we reported that the extent to  which federally funded abstinence-until-marriage education materials are  inaccurate was not known, in the course of their reviews OPA and some  states reported that they had found some inaccuracies in abstinence-until- marriage education materials. For example, one state official described an  instance in which abstinence-until-marriage materials incorrectly  suggested that HIV can pass through condoms because the latex used in  condoms is porous. To address concerns about the scientific accuracy of  materials used in abstinence-until-marriage programs, we recommended  that the Secretary of HHS develop procedures to help assure the accuracy  of such materials, and HHS agreed to consider this recommendation. In  April 2008, an ACF official reported that, in response to our  recommendation, ACF began requiring in fiscal year 2007 that community- based grantees sign written assurances that the materials they propose  using are accurate. This official also reported that, starting in fiscal year  2008, grantees of the State Program will also be required to sign these  written assurances. In addition, this official reported that ACF is  implementing a process to review the accuracy of the proposed curricula  of fiscal year 2007 Community-based grantees. The ACF official reported  that the curricula will be reviewed by a research analyst to ensure that all  statements are referenced to source documents, and then by a healthcare  professional who will compare the information in the curricula to  information in the source documents. The official also reported that, in the  future, ACF will require states to provide the agency with descriptions of  their strategies for reviewing the accuracy of their abstinence-until- marriage education programs.
HHS, states, and researchers have made a variety of efforts to assess the  effectiveness of abstinence-until-marriage education programs; however, a  number of factors limit the conclusions that can be drawn about the  effectiveness of abstinence-until-marriage education programs. To assess  the effectiveness of their abstinence-until-marriage education programs,  ACF and OPA have required their grantees to report on various outcomes.  For example, as of fiscal year 2006, states that received funding through  the State Program were required to report annually on four measures of  the prevalence of adolescent sexual behavior in their state, such as the  rate of pregnancy among adolescents aged 15 to 17 years. To assess the  effectiveness of both its State and Community-Based Programs, ACF also  analyzed trends in adolescent behavior, as reflected in national data on  birth rates among teens and the proportion of surveyed high school  students reporting that they have had sexual intercourse. OPA required  grantees of the AFL Program to develop and report on outcome measures  that demonstrated the extent to which grantees’ programs are having an  effect on program participants. Further, 6 of the 10 states in our review  that received funding through the State Program worked with third-party  evaluators to assess the effectiveness of abstinence-until-marriage  education programs in their states. Several factors, however, limit the  conclusions that can be drawn about the effectiveness of abstinence-until- marriage education programs. Most of the efforts to evaluate the  effectiveness of abstinence-until-marriage education programs that we  described in our report did not meet certain minimum criteria—such as  random assignment of participants and sufficient follow-up periods and  sample sizes—that experts have concluded are necessary in order for  assessments of program effectiveness to be scientifically valid.
During the course of our work on abstinence-until-marriage education, we  identified a legal matter that required the attention of HHS. Section  317P(c)(2) of the Public Health Service Act requires educational materials  specifically designed to address STDs to contain medically accurate  information about condom effectiveness in preventing the diseases the  educational materials are designed to address. We concluded that this  requirement would apply to abstinence-until-marriage education materials  prepared by and used by federal grant recipients, depending upon the  substantive content of those materials. In other words, in materials  otherwise meeting the statutory criteria, HHS’ grantees are required to  include information on condom effectiveness, and that information must  be medically accurate. At the time of our review, an ACF official reported  that materials prepared by abstinence-until-marriage education grantees  were not subject to section 317P(c)(2). Therefore, we recommended in a  letter dated October 18, 2006, that HHS reexamine its position and adopt  measures to ensure that, where applicable, abstinence-until-marriage  education materials comply with this requirement. The fiscal year 2007  Community-Based Program announcement states that mass produced  materials that as their primary purpose are specifically about STDs are  required to contain medically accurate information regarding the  effectiveness or lack of effectiveness of condoms in preventing the STDs  the educational materials are designed to address. An ACF official also  told us that future State and Community-Based Program announcements  would include this language.
subsections: section_title: Background
paragraphs: Statistics reported by CDC show that many high school students engage in  sexual behavior that places them at risk for unintended pregnancy and  STDs. In 2005, 46.8 percent of high school students reported that they have  had sexual intercourse, with 14.3 percent of students reporting that they  had had sexual intercourse with four or more persons. CDC also has  reported that the prevalence of certain STDs—including the rate of  chlamydia infection, the most frequently reported STD in the United  States—peaks in adolescence and young adulthood.
At the time of our 2006 report, HHS’s strategic plan included the objectives  to reduce the incidence of STDs and unintended pregnancies and to  promote family formation and healthy marriages. These two objectives  supported HHS’s goals to reduce the major threats to the health and well- being of Americans and to improve the stability and healthy development  of American children and youth. Abstinence-until-marriage education  programs were one of several types of programs that supported these  objectives. The State Program, the Community-Based Program, and the  AFL Program provide grants to support the recipients’ own efforts to  provide abstinence-until-marriage education at the local level. These  programs must comply with the statutory definition of abstinence  education (see table 1).
The State Program, administered by ACF, provides funding to its  grantees—states—for the provision of abstinence-until-marriage education  to those most likely to have children outside of marriage. States that  receive grants through the State Program have discretion in how they use  their funding to provide abstinence-until-marriage education. Funds are  allotted to each state that submits the required annual application based  on the ratio of the number of low-income children in the state to the total  number of low-income children in all states. States are required to match  every $4 they receive in federal money with $3 of nonfederal money and  are required to report annually on the performance of the abstinence-until- marriage education programs that they support or administer. In fiscal  year 2007, 40 states, the District of Columbia, and 3 insular areas were  awarded funding.
The Community-Based Program, which is also administered by ACF, is  focused on funding public and private entities that provide abstinence- until-marriage education for adolescents from 12 to 18 years old. The  Community-Based Program provides grants for school-based programs,  adult and peer mentoring, and parent education groups. For fiscal year  2007, 59 grants were awarded to organizations and other entities. Grantees  are required to report to ACF, on a semiannual basis, on the performance  of their programs.
The AFL Program also supports programs that provide abstinence-until- marriage education. Under the AFL Program, OPA awards competitive  grants to public or private nonprofit organizations or agencies, including  community-based and faith-based organizations, to facilitate abstinence- until-marriage education in a variety of settings, including schools and  community centers. In fiscal year 2007, OPA awarded funding to 36  grantees. Grantees are required to conduct evaluations of certain aspects  of their programs and report annually on their performance.
Five organizational units located within HHS—ACF, OPA, CDC, ASPE, and  NIH—have responsibilities related to abstinence-until-marriage education.  ACF and OPA administer the three main federal abstinence-until-marriage  education programs. CDC supports abstinence-until-marriage education at  the national, state, and local levels. CDC, ASPE, and NIH are sponsoring  research on the effectiveness of abstinence-until-marriage programs.
section_title: Federal and State Efforts to Assess the Scientific Accuracy of Materials Used in Abstinence-until- Marriage Education Programs Have Been Limited
paragraphs: In October 2006 we reported that efforts by HHS and states to assess the  scientific accuracy of materials used in abstinence-until-marriage  education programs had been limited. ACF—whose grants to the State  and Community-Based Programs accounted for the largest portion of  federal spending on abstinence-until-marriage education—did not review  its grantees’ education materials for scientific accuracy and did not require  grantees of either program to review their own materials for scientific  accuracy. In addition, not all states funded through the State Program  chose to review their program materials for scientific accuracy. In contrast  to ACF, OPA reviewed the scientific accuracy of grantees’ proposed  educational materials and corrected inaccuracies in these materials.
subsections: section_title: ACF Neither Reviewed Nor Required Grantees to Review Program Materials for Scientific Accuracy, Although Some State Grantees Had Conducted Such Reviews
paragraphs: As of October 2006, there had been limited efforts to review the scientific  accuracy of educational materials used in ACF’s State and Community- Based Programs—the two programs that accounted for the largest portion  of federal spending on abstinence-until-marriage education. ACF did not  review materials for scientific accuracy in either reviewing grant  applications or in overseeing grantees’ performance. Prior to fiscal year  2006, State Program and Community-Based Program applicants were not  required to submit copies of their proposed educational materials with  their applications. While ACF required grantees of the Community-Based  Program—but not the State Program—to submit their educational  materials with their fiscal year 2006 applications, ACF officials told us that  grantee applications and materials were only reviewed to ensure that they  addressed all aspects of the scope of the Community-Based Program, such  as the A-H definition of abstinence education. Further, documents  provided to us by ACF indicated that the agency did not review grantees’  educational materials for scientific accuracy as a routine part of its  oversight activities. In addition, ACF also did not require its grantees to  review their own materials for scientific accuracy.
While not all grantees of the State Program had chosen to review the  scientific accuracy of their educational materials, officials from 5 of the   10 states in our review reported that their states chose to do so. These five  states used a variety of approaches in their reviews. For example, some  states contracted with medical professionals—such as nurses,  gynecologists, and pediatricians—to serve as medical advisors who review  program materials and use their expertise to determine what is and is not  scientifically accurate. One of the states required that all statistics or  scientific statements cited in a program’s materials be sourced to CDC or a  peer-reviewed medical journal. Officials from this state told us that if  statements in these materials could not be attributed to these sources, the  statements were required to be removed until citations were provided and  materials were approved.
As a result of their reviews, officials from two of the five states reported  that they had found inaccuracies. One state official cited an instance  where materials incorrectly suggested that HIV can pass through condoms  because the latex used in condoms is porous. State officials who have  identified inaccuracies told us that they informed their grantees of  inaccuracies so that they could make corrections in their individual  programs. Some of the educational materials that states reviewed were  materials that were commonly used in the Community–Based Program.
While there had been limited review of materials used in the State and  Community-Based Programs, grantees of these programs had received  some technical assistance designed to improve the scientific accuracy of  their materials. For example, ACF officials reported that the agency  provided a conference for grantees of the Community-Based Program in  February 2006 that included a presentation focused on medical accuracy.
section_title: OPA Reviewed Materials Used by AFL Program Grantees for Scientific Accuracy
paragraphs: As of 2006, in contrast to ACF, OPA reviewed for scientific accuracy the  educational materials used by AFL Program grantees, and it did so before  those materials were used. OPA officials said that after grants were  awarded, a medical education specialist (in consultation with several part- time medical experts) reviewed the grantees’ printed materials and other  educational media, such as videos. OPA officials explained that the  medical education specialist must approve all proposed materials before  they are used. On many occasions, OPA grantees had proposed using— and therefore OPA has reviewed—materials commonly used in the  Community-Based Program. For example, an OPA official told us that the  agency had reviewed three of the Community-Based Program’s commonly  used curricula and was also currently reviewing another curriculum  commonly used by Community-Based Program grantees.
OPA officials stated that the medical education specialist had occasionally  found and addressed inaccuracies in grantees’ proposed educational  materials. OPA officials stated that these inaccuracies were often the  result of information being out of date because, for example, medical and  statistical information on STDs changes frequently. OPA addressed these  inaccuracies by either not approving the materials in which they appeared  or correcting the materials through discussions with the grantees and, in  some cases, the authors of the materials. In fiscal year 2005, OPA  disapproved of a grantee using a specific pamphlet about STDs because  the pamphlet contained statements about STD prevention and HIV  transmission that were considered incomplete or inaccurate. For example,  the pamphlet stated that there was no cure for hepatitis B, but the medical  education specialist required the grantee to add that there was a  preventive vaccine for hepatitis B. In addition, OPA required that a grantee  correct several statements in a true/false quiz—including statements about  STDs and condom use—in order for the quiz to be approved for use. For  example, the medical education specialist changed a sentence from “The  only 100% effective way of avoiding STDs or unwanted pregnancies is to  not have sexual intercourse.” to “The only 100% effective way of avoiding  STDs or unwanted pregnancies is to not have sexual intercourse and  engage in other risky behaviors.”
While OPA and some states had reviewed their grantees’ abstinence-until- marriage education materials for scientific accuracy, these types of  reviews have the potential to affect abstinence-until-marriage education  providers more broadly, perhaps creating an incentive for the authors of  such materials to ensure they are accurate. As of October 2006, the  company that produced one of the most widely used curricula used by  grantees of the Community-Based Program had updated its curriculum. A  representative from that company stated that this had been done, in part,  in response to a congressional review that found inaccuracies in its  abstinence-until-marriage materials.
To address concerns about the scientific accuracy of materials used in  abstinence-until-marriage education programs, we recommended that the  Secretary of HHS develop procedures to help assure the accuracy of such  materials used in the State and Community-Based Programs. We  recommended that in order to provide such assurance, the Secretary could  consider alternatives such as (1) extending the approach currently used by  OPA to review the scientific accuracy of the factual statements included in  abstinence-until-marriage education to materials used by grantees of  ACF’s Community-Based Program and requiring grantees of ACF’s State  Program to conduct such reviews or (2) requiring grantees of both  programs to sign written assurances in their grant applications that the  materials they propose using are accurate. In its written comments on a  draft of our report, HHS stated that it would consider requiring grantees of  both ACF programs to sign such written assurances to the accuracy of  their materials. In April 2008, an ACF official reported that, in response to  our recommendation, ACF began requiring in fiscal year 2007 that  community-based grantees sign written assurances that the materials they  propose using are accurate. This official also reported that, starting in  fiscal year 2008, grantees of the State Program will also be required to sign  these written assurances. In addition, this official reported that ACF is  implementing a process to review the accuracy of the proposed curricula  of fiscal year 2007 Community-based grantees. The ACF official reported  that the curricula will be reviewed by a research analyst to ensure that all  statements are referenced to source documents, and then by a healthcare  professional who will compare the information in the curricula to  information in the source documents. The official also reported that, in the  future, ACF will require states to provide the agency with descriptions of  their strategies for reviewing the accuracy of their abstinence-until- marriage education programs.
section_title: A Variety of Efforts Were Made to Assess the Effectiveness of Abstinence-until- Marriage Education Programs, but a Number of Factors Limit the Conclusions That Can Be Drawn
paragraphs: HHS, states, and researchers have made a variety of efforts to assess the  effectiveness of abstinence-until-marriage education programs; however, a  number of factors limit the conclusions that can be drawn. ACF and OPA  have required their grantees to report on various outcomes used to  measure the effectiveness of grantees’ abstinence-until-marriage education  programs. To assess the effectiveness of the State and Community-Based  Programs, ACF has analyzed national data on adolescent birth rates and  the proportion of adolescents who report having had sexual intercourse.  As of October 2006, other organizational units within HHS were funding  studies designed to assess the effectiveness of abstinence-until-marriage  education programs in delaying sexual initiation, reducing pregnancy and  STD rates, and reducing the frequency of sexual activity. Despite these  efforts, several factors limit the conclusions that can be drawn about the  effectiveness of abstinence-until-marriage education programs. Most of the  efforts to evaluate the effectiveness of abstinence-until-marriage education  programs that we reviewed have not met certain minimum criteria that  experts have concluded are necessary in order for assessments of program  effectiveness to be scientifically valid, in part because such designs can be  expensive and time-consuming to carry out. In addition, the results of  some efforts that meet the criteria of a scientifically valid assessment have  varied.
subsections: section_title: HHS, States, and Researchers Have Made a Variety of Efforts to Assess the Effectiveness of Abstinence-until-Marriage Education Programs
paragraphs: ACF has made efforts to assess the effectiveness of abstinence-until- marriage education programs funded by the State Program and the  Community-Based Program. One of ACF’s efforts has been to require  grantees of both programs to report data on outcomes, though the two  programs have different requirements for the outcomes grantees must  report. As of fiscal year 2006, State Program grantees were required to  report annually on four measures of the prevalence of adolescent sexual  behavior in their states, such as the rate of pregnancy among adolescents  aged 15 to 17 years, and compare these data to program targets over   5 years. States also were required to develop and report on two additional  performance measures that were related to the goals of their programs.  Also as of fiscal year 2006, ACF required Community-Based Program  grantees to develop and report on outcome measures designed to  demonstrate the extent to which grantees’ community-based abstinence- until-marriage education programs were accomplishing their program  goals. In addition to outcome reporting, ACF required grantees of the  Community-Based Program to report on program “outputs,” which  measure the quantity of program activities and other deliverables, such as  the number of participants who are served by the abstinence-until- marriage education programs.
As of October 2006, OPA also had made efforts to assess the effectiveness  of the AFL Program. Specifically, OPA required grantees of the AFL  Program to develop and report on outcome measures, such as  participants’ knowledge of the benefits of abstinence and their reported  intentions to abstain from sexual activity, that were used to help  demonstrate the extent to which grantees’ programs were having an effect  on program participants. To collect data on outcome measures, OPA  required grantees to administer, at a minimum, a standardized  questionnaire to their program participants, both when participants begin  an abstinence-only education program and after the program’s completion.  OPA officials told us that they were planning to aggregate information  from certain questions in the standardized set of questionnaires in order to  report on certain performance measures as part of the agency’s annual  performance reports; the agency expected to begin receiving data from  grantees that were using these questionnaires in January 2007.
To help grantees measure the effectiveness of their programs, both ACF  and OPA required that grantees use independent evaluators and have  provided assistance to grantees in support of their program evaluation  efforts. ACF and OPA required their grantees to contract with third-party  evaluators, such as university researchers or private research firms, who  were responsible for helping grantees develop the outcome measures they  were required to report on and monitoring grantee performance against  those measures. Unlike ACF, OPA required that these third-party  evaluations incorporate specific methodological characteristics, such as  control groups of individuals that did not receive the program and  sufficient sample sizes to ensure that any observed differences between  the groups were statistically valid. Both ACF and OPA have provided  technical assistance and training to their grantees in order to support  grantees’ own program evaluation efforts.
ACF also analyzed trends in adolescent behavior, as reflected in national  data on birth rates among teens and the proportion of surveyed high  school students reporting that they have had sexual intercourse. ACF  used these national data as a measure of the overall effectiveness of its  State and Community-Based Programs, comparing the national data to  program targets. In its annual performance reports, the agency has  summarized the progress being made toward lowering the rate of births to  unmarried teenage girls and the proportion of students (grades 9-12) who  report having ever had sexual intercourse.
Some states have made additional efforts to assess the effectiveness of  abstinence-until-marriage education programs. Specifically, we found that  6 of the 10 states in our review that received funding through ACF’s State  Program had made efforts to conduct evaluations of selected abstinence- until-marriage programs in their state. All 6 of the states worked with  third-party evaluators, such as university researchers or private research  firms, to perform the evaluations, which in general measured self-reported  changes in program participants’ behavior and attitudes related to sex and  abstinence as indicators of program effectiveness. Four of these states had  completed third-party evaluations as of February 2006, and the results of  these studies varied. Among those 4 states, 3 states required the  abstinence programs in their state to measure reported changes in  participants’ behavior as an indicator of program effectiveness—both at  the start of the program and after its completion. The 3 states required  their programs to track participants’ reported incidence of sexual  intercourse. Additionally, 2 of the 4 states required their programs to track  biological outcomes, such as pregnancies, births, or STDs. In addition, 6 of  the 10 states in our review required their programs to track participants’  attitudes about abstinence and sex, such as the number of participants  who make pledges to remain abstinent.
Besides ACF and OPA, other organizational units within HHS have made  efforts to assess the effectiveness of abstinence-until-marriage education  programs. As of 2006, ASPE was sponsoring a study of the Community- Based Program and a study of the State Program. The study of the State  Program was conducted by Mathematica Policy Research, Inc.  (Mathematica) and completed in 2007. It examined the impact of five  programs funded through the State Program on participants’ attitudes and  behaviors related to abstinence and sex. Like ASPE, CDC has made its  own effort to assess the effectiveness of abstinence-until-marriage  education by sponsoring a study to evaluate the effectiveness of two  middle school curricula—one that complies with abstinence-until- marriage education program requirements and one that teaches a  combination of abstinence and contraceptive information and skills. The  agency expects to complete the study in 2009. Likewise, NIH has funded  studies comparing the effectiveness of education programs that focus only  on abstinence with the effectiveness of sex education programs that teach  both abstinence and information about contraception. As of October 2006,  NIH was funding five studies, which in general were comparing the effects  of these two types of programs on the sexual behavior and related  attitudes among groups of either middle school or high school students.
In addition to the efforts of researchers working on behalf of HHS and  states, other researchers—such as those affiliated with universities and  various advocacy groups—have made efforts to study the effectiveness of  abstinence-until-marriage education programs. This work includes studies  of the outcomes of individual programs and reviews of other studies on  the effectiveness of individual abstinence-until-marriage education  programs. In general, research studies on the effectiveness of individual  programs have examined the extent to which they changed participants’  demonstrated knowledge of concepts taught in the programs, declared  intentions to abstain from sex until marriage, and reported behavior  related to sexual activity and abstinence. As of October 2006, the efforts to  study and build a body of research on the effectiveness of most  abstinence-until-marriage education programs had been under way for  only a few years, in part because grants under the two programs that  account for the largest portion of federal spending on abstinence-until- marriage education—the State Program and the Community-Based  Program—were not awarded until 1998 and 2001, respectively.
section_title: Several Factors Limit the Conclusions That Can Be Drawn about the Effectiveness of Abstinence-until-Marriage Education Programs
paragraphs: Most of the efforts of HHS, states, and other researchers to evaluate the  effectiveness of abstinence-until-marriage education programs included in  our review have not met certain minimum criteria that experts have  concluded are necessary in order for assessments of program  effectiveness to be scientifically valid. In an effort to better assess the  merits of the studies that have been conducted on the effectiveness of  sexual health programs—including abstinence-until-marriage education  programs—scientific experts have developed criteria that can be used to  gauge the scientific rigor of these evaluations. The reports of two panels of  experts, as well as the experts we interviewed in the course of our  previous work, generally agreed that scientifically valid studies of a  program’s effectiveness should include the following characteristics:    An experimental design that randomly assigns individuals or schools to  either an intervention group or control group, or a quasi-experimental  design that uses nonrandomly assigned but well-matched comparison  groups. According to the panel of scientific experts convened by the  National Campaign to Prevent Teen Pregnancy, experimental designs or  quasi-experimental designs with well-matched comparison groups have at  least three important strengths that are typically not found in other  studies, such as those that use aggregated data: they evaluate specific  programs with known characteristics, they can clearly distinguish between  participants who did and did not receive an intervention, and they control  for other factors that may affect study outcomes. According to scientific  experts, studies that include experimental or quasi-experimental designs  should also collect follow-up data for a minimum number of months after  subjects receive an intervention. In addition, experts have reported that  studies should have a sample size of at least 100 individuals for study  results to be considered scientifically valid.
Studies should assess or measure changes in biological outcomes or  reported behaviors instead of attitudes or intentions. According to  scientific experts, biological outcomes—such as pregnancy rates, birth  rates, and STD rates—and reported behaviors—such as reported initiation  and frequency of sexual activity—are better measures of the effectiveness  of abstinence-until-marriage programs, because adolescent attitudes and  intentions may or may not be indicative of actual behavior.
Many of the efforts by HHS, states, and other researchers that we  identified in our review lack at least one of the characteristics of a  scientifically valid study of program effectiveness. Most of the efforts to  assess the effectiveness of these programs have not used experimental or  quasi-experimental designs with sufficient follow-up periods and sample  sizes. For example, ACF used, according to ACF officials, grantee  reporting on outcomes in order to monitor grantees’ performance, target  training and technical assistance, and help grantees improve service  delivery. However, because the outcomes reported by grantees have not  been produced through experimentally or quasi-experimentally designed  studies, such information cannot be causally attributed to any particular  abstinence-until-marriage education program. Further, none of the state  evaluations we reviewed that had been completed included randomly  assigned control groups. Similarly, some of the journal articles that we  reviewed described studies to assess the effectiveness of abstinence-until- marriage programs that also lacked at least one of the characteristics of a  scientifically valid study of program effectiveness. In these studies,  researchers administered questionnaires to study participants before and  after they completed an abstinence-until-marriage education program and  assessed the extent to which the responses of participants changed.  These studies did not compare the responses of study participants with a  group that did not participate in an abstinence-until-marriage education  program.
Like the lack of an experimental or quasi-experimental design, not  measuring changes in behavioral or biological outcomes among  participants limits the conclusions that can be drawn about the  effectiveness of abstinence-until-marriage education programs. Most of the  efforts we identified in our review used reported intentions and attitudes  in order to assess the effectiveness of abstinence-until-marriage programs.  For example, as of 2006, neither ACF’s community-based grantees nor  OPA’s AFL grantees were required to report on behavioral or biological  outcomes, such as rates of intercourse or pregnancy. Similarly, the journal  articles we reviewed were more likely to use reported attitudes and  intentions—such as study participants’ reported attitudes about premarital  sexual activity or their reported intentions to remain abstinent until  marriage—rather than their reported behaviors or biological outcomes to  assess the effectiveness of abstinence-until-marriage programs.
According to scientific experts, HHS, states, and other researchers face a  number of challenges in applying either of these criteria. According to  these experts, experimental or quasi-experimental studies can be  expensive and time-consuming to carry out, and many grantees of  abstinence-until-marriage education programs have insufficient time and  funding to support these types of studies. Moreover, it can be difficult for  researchers assessing abstinence-until-marriage education programs to  convince school districts to participate in randomized intervention and  control groups, in part because of sensitivities to surveying attitudes,  intentions, and behaviors related to abstinence and sex. Similarly, experts,  as well as state and HHS officials, have reported that it can be difficult to  obtain scientifically valid information on biological outcomes and sexual  behaviors. For example, experts have reported that when measuring a  program’s effect on biological outcomes—such as reducing pregnancy  rates or birth rates—it is necessary to have large sample sizes in order to  determine whether a small change in such outcomes is the result of an  abstinence-until-marriage education program.
Among the assessment efforts we identified are some studies that meet the  criteria of a scientifically valid effectiveness study. However, results of  these studies varied, and this limits the conclusions that can be drawn  about the effectiveness of abstinence-until-marriage education programs.  Some researchers have reported that abstinence-until-marriage education  programs have resulted in adolescents reporting having less frequent  sexual intercourse or fewer sexual partners. For example, in one study of  middle school students, participants in an abstinence-until-marriage  education program who had sexual intercourse during the follow-up  period were 50 percent less likely to report having two or more sexual  partners when compared with their nonparticipant peers. In contrast,  other studies have reported that abstinence-until-marriage education  programs did not affect the reported frequency of sexual intercourse or  number of sexual partners. For example, one study of middle school  students found that participants of an abstinence-until-marriage program  were not less likely than nonparticipants at the 1 year follow-up to report  less frequent sexual intercourse or fewer sexual partners. Experts with  whom we spoke emphasized that there were still too few scientifically  valid studies completed as of 2006 that could be used to determine  conclusively which, if any, abstinence-until-marriage programs are  effective.
We identified two key studies that experts anticipated would meet the  criteria of a scientifically valid effectiveness study. Experts and federal  officials we interviewed stated that they expected the results of these two  federally funded studies to add substantively to the body of research on  the effectiveness of abstinence-until-marriage education programs. One of  these key studies—the final Mathematica report, contracted by ASPE, on  the State Program—has been completed. In this report, the researchers  found that youth who participated in the abstinence-until-marriage  education programs were no more likely than control group youth to have  abstained from sex, and among those who reported having had sex, they  had similar numbers of sexual partners and had initiated sex at the same  average age. The youth in abstinence-until-marriage education programs  also were no more likely to have engaged in unprotected sex than control  group youth. The second key study we identified is CDC’s research on  middle school programs, which is still ongoing. In addition, since October  2006, a third key report was released, presenting the 2007 analysis of the  National Campaign to Prevent Teen and Unplanned Pregnancy of the  available research on abstinence-until-marriage education programs. This  report stated that studies of abstinence programs have not produced  sufficient evidence of effectiveness, and that efforts should be directed  toward further evaluation of these programs.
section_title: Statutory Requirement to Include Information on Condom Effectiveness Would Apply to Certain Abstinence-until- Marriage Education Materials
paragraphs: During the course of our work on abstinence-until-marriage education, we  identified a federal statutory provision—section 317P(c)(2) of the Public  Health Service Act—relevant to the grants provided by HHS’s State  Program, Community-Based Program, and AFL Program. This provision  requires that educational materials prepared by HHS’s grantees, among  others, that are specifically designed to address STDs, contain medically  accurate information regarding the effectiveness or lack of effectiveness  of condoms in preventing the diseases the materials are designed to  address.
At the time of our review, an ACF official reported that materials prepared  by abstinence-until-marriage education grantees were not subject to  section 317P(c)(2). However, we concluded that this requirement would  apply to abstinence-until-marriage education materials prepared by and  used by federal grant recipients, depending upon the substantive content  of those materials. In other words, in materials specifically designed to  address STDs, HHS’s grantees are required to include information on  condom effectiveness, and that information must be medically accurate.  Therefore, we recommended in a letter dated October 18, 2006, that HHS  reexamine its position and adopt measures to ensure that, where  applicable, abstinence education materials comply with this requirement.
In a letter to us dated January 16, 2007, ACF responded that it would take  steps to “make it clear to grantees that when they mass produce materials  that as a primary purpose are specifically about STDs those materials are  required by section 317P(c)(2) of the Public Health Service Act to contain  medically accurate information regarding the effectiveness or lack of  effectiveness of condoms in preventing the sexually transmitted disease  the materials are designed to address.” The fiscal year 2007 Community- Based Program announcement states that mass produced materials that as  their primary purpose are specifically about STDs are subject to this  requirement. The announcement also states that mass produced materials  are considered to be specifically designed to address STDs if more than   50 percent of the content is related to STDs. An ACF official also told us  that future State and Community-Based Program announcements would  include this language.
Mr. Chairman, this completes my prepared remarks. I will be happy to  answer questions you or other Committee Members may have.
section_title: Contact and Acknowledgments
paragraphs: For further information regarding this testimony, please contact   Marcia Crosse at (202) 512-7114 or crossem@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found on  the last page of this statement. Major contributors to this report were  Kristi Peterson, Assistant Director; Kelly DeMots; Cathleen Hamann;   Helen Desaulniers; and Julian Klazkin.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. It may be reproduced and distributed in its entirety without further  permission from GAO. However, because this work may contain copyrighted images or  other material, permission from the copyright holder may be necessary if you wish to  reproduce this material separately.