Abstinence Education: Efforts to Assess the Accuracy and Effectiveness of Federally Funded Programs
Released: 2006-11-16T00:00:00

Letter

Background

Recent statistics from CDC show that many high school students engage in sexual behavior that places them at risk for unintended pregnancy and STDs. In 2005, 46.8 percent of high school students reported that they have ever had sexual intercourse, with 14.3 percent of students reporting that they had had sexual intercourse with four or more persons. The likelihood of ever having sexual intercourse varied by grade, with the highest rate among 12th grade students (63.1 percent) and the lowest rate among 9th grade students (34.3 percent). CDC also has reported that the prevalence of certain STDs—including the rate of chlamydia infection, the most frequently reported STD in the United States—peaks in adolescence and young adulthood. According to CDC, in 2004 the chlamydia rates among adolescents 15 to 19 years old (1,579 cases per 100,000 adolescents) and young adults 20 to 24 years old (1,660 cases per 100,000) were each more than twice the rates among all other age groups.

HHS’s current strategic plan includes the objectives to reduce the incidence of STDs and unintended pregnancies and to promote family formation and healthy marriages. These two objectives support HHS’s goals to reduce the major threats to the health and well-being of Americans and to improve the stability and healthy development of American children and youth, respectively. Abstinence-until-marriage education programs are one of several types of programs that support these objectives.

The three main federal abstinence-until-marriage education programs— the State Program, the Community-Based Program, and the AFL Program—provide grants to support the recipients’ own efforts to provide abstinence-until-marriage education at the local level. These programs must comply with the statutory definition of abstinence education (see table 1).

The State Program

The State Program, administered by ACF, provides funding to its grantees—states—for the provision of abstinence-until-marriage education to those most likely to have children outside of marriage. States that receive grants through the State Program have discretion in how they use their funding to provide abstinence-until-marriage education. Some require that organizations apply for funds and use them to administer abstinence- until-marriage education programs. Others may directly administer such programs. At their discretion, states may also provide mentoring, counseling, and adult supervision to adolescents to promote abstinence from sexual activity until marriage.

The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 established the State Program, and states were awarded grants beginning in fiscal year 1998. Funds are allotted to each state that submits the required annual application based on the ratio of the number of low-income children in the state to the total number of low-income children in all states. States are required to match every $4 they receive in federal money with $3 of nonfederal money and are required to report annually on the performance of the abstinence-until-marriage education programs that they support or administer. In fiscal year 2005, 47 states, the District of Columbia, and 3 insular areas were awarded funding.

The Community-Based Program

The Community-Based Program, which is also administered by ACF, is focused on funding public and private entities that provide abstinence- until-marriage education for adolescents from 12 to 18 years old, with the purpose of creating an environment within communities that supports adolescent decisions to postpone sexual activity until marriage. The Community-Based Program provides grants for school-based programs, adult and peer mentoring, and parent education groups. The Community- Based Program first awarded grants in fiscal year 2001. Grantees of the Community-Based Program are selected through a competitive process and are evaluated according to several criteria, such as the extent to which they have demonstrated that a need exists for abstinence-until-marriage education for a targeted population or in a specific geographic location. Grantees are required to report to ACF, on a semiannual basis, on the performance of their programs. For fiscal year 2005, 63 grants were awarded to organizations and other entities.

The AFL Program

The AFL Program supports programs that provide abstinence-until- marriage education. The primary purpose of these programs is to find effective means of reaching preadolescents and adolescents before they become sexually active in order to encourage them to abstain from sexual activity and other risky behaviors. Under the AFL Program, OPA awards competitive grants to public or private nonprofit organizations or agencies, including community-based and faith-based organizations, to facilitate abstinence-until-marriage education in a variety of settings, including schools and community centers. Established in 1981, the AFL Program began awarding grants in fiscal year 1982. AFL Program grantees include school districts, youth development groups, and medical centers. Grant applicants are evaluated based on several criteria, such as the extent to which they provide a clear statement of mission, goals, measurable objectives, and a reasonable method for achieving their objectives. Grantees are required to conduct evaluations of certain aspects of their programs and report annually on their performance. As of August 2006, OPA funded 58 abstinence-until-marriage education programs, and most of these were focused on reaching young adolescents from the ages of 9 to 14.

Funding for Abstinence- until-Marriage Education Has Increased

Funding provided by HHS for abstinence-until-marriage education programs has increased steadily since 2001 (see table 2). For the three main programs combined—the State Program, the Community-Based Program, and the AFL Program—the amount of agency funding increased from about $73 million in fiscal year 2001 to about $158 million in fiscal year 2005. Nearly all of this increase was for the Community-Based program; funding under this program increased by about $84 million from fiscal years 2001 through 2005. In fiscal year 2005, agency funding for the Community-Based Program constituted the largest share of the total funding (about 66 percent) for the three main programs combined.

Within each of the three main abstinence-until-marriage education programs, the amount of individual grants varied. In fiscal year 2005, the State Program’s annual grants ranged from $57,057 to $4,777,916 and the median annual grant amount was $569,675. That same year, the Community-Based Program’s annual grants ranged from $213,276 to $800,000 and the median grant amount was $642,250. In fiscal year 2006, the AFL Program’s annual grants ranged from $95,676 to $300,000 and the median grant amount was $225,000.

Federal Agency Responsibilities Related to Abstinence-until-Marriage Education

Five organizational units located within HHS—ACF, OPA, CDC, ASPE, and NIH—have responsibilities related to abstinence-until-marriage education. ACF and OPA administer the three main federal abstinence-until-marriage education programs. CDC supports abstinence-until-marriage education at the national, state, and local levels. CDC, ASPE, and NIH are sponsoring research on the effectiveness of abstinence-until-marriage programs.

ACF is responsible for federal programs that promote the economic and social well-being of families, children, individuals, and communities. ACF administers and provides oversight of both the State Program and the Community-Based Program by, among other things, awarding grants, providing training and technical assistance to grantees, and requiring annual performance reporting from grantees. ACF has been responsible for the State Program since June 2004 and the Community-Based Program since October 2005. HRSA previously administered these programs.

OPA has responsibility for advising the Secretary of HHS on a wide range of reproductive health topics, including adolescent pregnancy and family planning. The office is also responsible for administering programs that provide services for pregnant and parenting teens and prevention programs, such as abstinence-until-marriage education programs. OPA administers and provides oversight of the AFL Program by awarding grants, providing training and technical assistance to grantees, and requiring annual performance reporting from grantees.

CDC is primarily responsible for the prevention and control of infectious and chronic diseases, including STDs. CDC provides funding to state and local education agencies in their efforts to support comprehensive school health education and HIV/STD prevention education programs, and CDC officials told us that some of these are focused on abstinence. CDC also provides funding to several state education agencies to implement various abstinence projects, such as collaboration-building among agencies to increase the impact of their efforts to encourage abstinence. Further, CDC develops tools to assist state and local education agencies with their health education programs. CDC provides funding to several national organizations to build the capacity of abstinence-until-marriage education providers. Organizations’ activities include, but are not limited to, the development and distribution of educational materials. CDC is also sponsoring research on the effectiveness of an abstinence-until-marriage education program.

ASPE advises the Secretary of HHS in several areas, including policy development in health, human services, data, and science. ASPE is responsible for the development of policy analyses and it conducts research and evaluation studies in several areas, including the health of children and adolescents. ASPE is currently sponsoring research on the effectiveness of abstinence-until-marriage education programs.

NIH is the primary federal agency that conducts and supports medical and behavioral research among various populations, including children and adolescents. NIH is currently sponsoring research on the effectiveness of abstinence-until-marriage education programs.

Federal and State Efforts to Assess the Scientific Accuracy of Materials Used in Abstinence-until- Marriage Education Programs Have Been Limited

Efforts by HHS and states to assess the scientific accuracy of materials used in abstinence-until-marriage education programs have been limited. ACF—which awards grants to two programs that account for the largest portion of federal spending on abstinence-until-marriage education—does not review its grantees’ education materials for scientific accuracy and does not require grantees of either program to review their own materials for scientific accuracy. In addition, not all states funded through the State Program have chosen to review their program materials for scientific accuracy. In contrast to ACF, OPA has reviewed the scientific accuracy of grantees’ proposed educational materials and corrected inaccuracies in these materials.

ACF Does Not Review Program Materials for Scientific Accuracy and Does Not Require Grantees to Do So, though Some State Grantees Have Conducted Such Reviews

There have been limited efforts to review the scientific accuracy of educational materials used in ACF’s State and Community-Based Programs—the two programs that account for the largest portion of federal spending on abstinence education. ACF does not review materials for scientific accuracy in either reviewing grant applications or in overseeing grantees’ performance. Prior to fiscal year 2006, State Program and Community-Based Program applicants were not required to submit copies of their proposed educational materials with their applications. While ACF required grantees of the Community-Based Program—but not the State Program—to submit their educational materials with their fiscal year 2006 applications, ACF officials told us that grantee applications and materials are only reviewed to ensure that they address all aspects of the scope of the Community-Based Program, such as the A-H definition of abstinence education. Further, documents provided to us by ACF indicate that the agency does not review grantees’ educational materials for scientific accuracy as a routine part of its oversight activities. In addition, ACF also does not require its grantees to review their own materials for scientific accuracy. Similarly, when HRSA was responsible for the State and Community-Based Programs, the agency did not review materials used by grantees for scientific accuracy or require grantees to review their own materials.

Not all grantees of the State Program have chosen to review the scientific accuracy of their educational materials. Officials from 5 of the 10 states in our review reported that their states have chosen to conduct such reviews. Officials in these states identified a variety of reasons why their states reviewed abstinence-until-marriage educational materials, including program requirements, state education laws and guidelines, and past lawsuits, to ensure that materials used in abstinence-until-marriage programs were accurate. For example, Michigan’s Revised School Code states that materials and instruction in the sex education curricula, including information on abstinence, “shall not be medically inaccurate,” and Ohio’s fiscal year 2007 abstinence-until-marriage education program guidance states that abstinence-until-marriage educational materials “should be medically accurate in all assertions.”

The five states we contacted that review abstinence-until-marriage educational materials for scientific accuracy have used a variety of approaches in their reviews. Some states contracted with medical professionals—such as nurses, gynecologists, and pediatricians—to serve as medical advisors who review program materials and use their expertise to determine what is and is not scientifically accurate. Some states have created checklists or worksheets to guide their staff conducting the review and document findings of inaccuracy or verification of a statement. All five states use medical professionals in conducting these reviews. One of the states requires that all statistics or scientific statements cited in a program’s materials are sourced to CDC or a peer-reviewed medical journal. Officials from this state told us that if statements in these materials cannot be attributed to these sources, the statements are required to be removed until citations are provided and materials are approved. Officials from this state told us they have also supplemented their review of program materials with on-site classroom observations to assess the scientific accuracy of the information presented to students.

Officials from two of the five states reported that they have found inaccuracies as a result of their reviews. For example, one state official stated that because information is constantly evolving, state officials have had to correct out-of-date scientific information. In addition, this official cited an instance where materials incorrectly suggested that HIV can pass through condoms because the latex used in condoms is porous. In addition, this official provided documentation that the state has had to correct a statement indicating that when a person is infected with the human papillomavirus, the virus is “present for life” because, in almost all cases, this is not true. State officials who have identified inaccuracies told us that they informed their grantees of inaccuracies so that they could make corrections in their individual programs. One state official added that she contacted the authors of the materials to report an inaccuracy.

Some of the educational materials that states have reviewed are materials that are commonly used in the Community–Based Program. Officials from four of the five states that review materials for scientific accuracy told us that they have each reviewed at least one of the five curricula most commonly used in the Community-Based Program because programs in their state were using them: Choosing the Best, WAIT Training, Sex Can Wait, A.C. Green’s Game Plan Abstinence Program, and Worth the Wait. Based on ACF documents, we found that there were 58 different curricula used by grantees of the Community-Based Program in fiscal year 2005. However, more than half of the grantees of the Community-Based Program reported using at least one of these five curricula.

While there has been limited review of materials used in the State and Community-Based Programs, grantees of these programs have received some technical assistance designed to improve the scientific accuracy of their materials. For example, ACF officials reported that the agency provided a conference for grantees of the Community-Based Program in February 2006 that included a presentation focused on medical accuracy, including a discussion of state legislative proposals that would require medical accuracy in abstinence-until-marriage education, and how to identify reliable data. In addition, in 2002, HRSA awarded a contract to the National Abstinence Clearinghouse requiring, among other things, that the contractor develop and implement a program to provide medically accurate information and training to grantees of the State and Community- Based Programs. (See app. I for a description of HRSA’s process for awarding this contract). The portion of the contract that focused on providing medically accurate information to grantees was subcontracted to the Medical Institute for Sexual Health (Medical Institute), which has conducted presentations at regional educational conferences to provide grantees with medical and scientific information, such as updated information on condoms and STD transmission. The Medical Institute has also provided consultative services to grantees by responding to medical and scientific questions.

OPA Reviews Materials Used by AFL Program Grantees for Scientific Accuracy

In contrast to ACF, OPA reviews for scientific accuracy the educational materials used by AFL Program grantees. Specifically, OPA reviews its grantees’ proposed educational materials for scientific accuracy before they are used. Agency officials stated that they began to review these materials while litigation concerning the AFL Program was ongoing. OPA continued to review these materials as part of a 1993 settlement to this lawsuit. The settlement agreement expired in 1998, though the agency has continued to review grantees’ proposed educational materials for accuracy as a matter of policy. OPA officials told us that grant applicants submit summaries of materials they propose to use, though the materials are not reviewed for scientific accuracy until after grantees have been selected. OPA officials said that after grants are awarded, a medical education specialist (in consultation with several part-time medical experts) reviews the grantees’ printed materials and other educational media, such as videos. OPA officials explained that the medical education specialist must approve all materials before they are used. On many occasions, OPA grantees have proposed using—and therefore OPA has reviewed— materials commonly used in the Community-Based Program. For example, an OPA official told us that the agency had reviewed three of the Community-Based Program’s commonly used curricula—Choosing the Best, Sex Can Wait, and A.C. Green’s Game Plan Abstinence Program— and is also currently reviewing another curriculum commonly used by Community-Based Program grantees, WAIT Training.

OPA officials stated that the medical education specialist has occasionally found and addressed inaccuracies in grantees’ proposed educational materials. OPA officials stated that these inaccuracies are often the result of information being out of date because, for example, medical and statistical information on STDs changes frequently. OPA has addressed these inaccuracies by either not approving the materials in which they appeared or correcting the materials through discussions with the grantees and, in some cases, the authors of the materials. In fiscal year 2005, OPA disapproved of a grantee using a specific pamphlet about STDs because the pamphlet contained statements about STD prevention and HIV transmission that were considered incomplete or inaccurate. For example, the pamphlet stated that there was no cure for hepatitis B, but the medical education specialist required the grantee to add that there was a preventive vaccine for hepatitis B. In addition, OPA required that a grantee correct several statements in a true/false quiz—including statements about STDs and condom use—in order for the quiz to be approved for use. For example, the medical education specialist changed a sentence from “The only 100% effective way of avoiding STDs or unwanted pregnancies is to not have sexual intercourse.” to “The only 100% effective way of avoiding STDs or unwanted pregnancies is to not have sexual intercourse and engage in other risky behaviors.”

While OPA and some states have reviewed their grantees’ abstinence-until- marriage education materials for scientific accuracy, these types of reviews have the potential to affect abstinence-until-marriage education providers more broadly. Such efforts may create an incentive for authors of abstinence-until-marriage education materials to ensure they are accurate. Thus, some authors of abstinence-until-marriage education materials have recently updated materials in their curricula following reports that questioned their accuracy. For example, one of the most widely used curricula used by grantees of the Community-Based Program—WAIT Training—has been recently updated and provides the updated information on its Web site. A representative from WAIT Training stated that the company recently revised its curriculum, in part, in response to a congressional review that found inaccuracies in its abstinence-until-marriage education materials.

A Variety of Efforts Have Been Made to Assess the Effectiveness of Abstinence-until- Marriage Education Programs, but a Number of Factors Limit the Conclusions That Can Be Drawn

HHS, states, and researchers have made a variety of efforts to assess the effectiveness of abstinence-until-marriage education programs; however, a number of factors limit the conclusions that can be drawn about the effectiveness of these programs. ACF and OPA have required their grantees to report on various outcomes used to measure the effectiveness of grantees’ abstinence-until-marriage education programs, though the reporting requirements for each of the three abstinence-until-marriage programs differ. In addition, to assess the effectiveness of the State and Community-Based Programs, ACF has analyzed national data on adolescent birth rates and the proportion of adolescents who report having had sexual intercourse. Other organizational units within HHS— ASPE, CDC, and NIH—are funding studies designed to assess the effectiveness of abstinence-until-marriage education programs in delaying sexual initiation, reducing pregnancy and STD rates, and reducing the frequency of sexual activity. Despite these efforts, several factors limit the conclusions that can be drawn about the effectiveness of abstinence-until- marriage education programs. Most of the efforts to evaluate the effectiveness of abstinence-until-marriage education programs that we describe in our review have not met certain minimum criteria that experts have concluded are necessary in order for assessments of program effectiveness to be scientifically valid, in part because such designs can be expensive and time-consuming to carry out. In addition, the results of some efforts that meet the criteria of a scientifically valid assessment have varied, and two key studies that meet these criteria have not yet been completed.

HHS, States, and Researchers Have Made a Variety of Efforts to Assess the Effectiveness of Abstinence-until-Marriage Education Programs

Efforts of HHS, states, and researchers to assess the effectiveness of abstinence-until-marriage education programs have included ACF and OPA requiring grantees to report data on outcomes of their abstinence- until-marriage education programs; ACF analyzing national data on adolescent behavior and birth rates; and other HHS agencies, states, and researchers funding or conducting studies to assess the effectiveness of abstinence-until-marriage education programs.

ACF and OPA Have Required Grantee Reporting of Data on Outcomes

ACF has made efforts to assess the effectiveness of abstinence-until- marriage education programs funded by the State Program and the Community-Based Program. One of ACF’s efforts has been to require grantees of both programs to report data on outcomes, though the two programs have different requirements for the outcomes grantees must report. For the State Program, as of fiscal year 2006, grantees must report annually on four measures of the prevalence of adolescent sexual behavior in their states, such as the rate of pregnancy among adolescents aged 15 to 17 years, and compare these data to program targets over 5 years. To report on these four measures, states may choose the data sources they will use. States must also develop and report on two additional performance measures that are related to the goals of their programs. (See table 3 for a list of ACF’s fiscal year 2006 reporting requirements for the State Program.)

As of fiscal year 2006, ACF requires Community-Based Program grantees to develop and report on outcome measures designed to demonstrate the extent to which grantees’ community-based abstinence education programs are accomplishing their program goals. ACF requires grantees of the Community-Based Program to contract with third-party evaluators, who are responsible for both helping grantees develop the outcome measures and monitoring grantee performance against the measures, but because this is a new requirement established for fiscal year 2006 grantees, ACF has not yet received the results of these evaluations. In addition to outcome reporting, ACF requires grantees of the Community-Based Program to report on program “outputs,” which measure the quantity of program activities and other deliverables, such as the number of participants who are served by the abstinence-until-marriage education programs. According to ACF officials, the agency requires grantees of both the State Program and the Community-Based Program to report on program outcomes in order to monitor grantees’ performance, target training, and technical assistance, and help grantees improve service delivery. (See table 3 for a list of ACF’s fiscal year 2006 reporting requirements for the Community-Based Program.)

ACF’s fiscal year 2006 reporting requirements for grantees of the State Program are the same as HRSA’s when it administered the State Program. In contrast, ACF’s fiscal year 2006 reporting requirements for the Community-Based Program differ from HRSA’s reporting requirements for the program. For example, for Community-Based Program grants awarded in fiscal year 2001, HRSA required grantees to report on the effectiveness of their programs, as measured by program participation as well as behavioral and biological outcomes. These performance measures were modified for fiscal year 2002, in part HHS officials explained, because of concerns expressed by members of the abstinence-education community that the original performance measures did not accurately reflect the efforts of the grantees of the Community-Based Program. For grants awarded from fiscal years 2002 through 2004, HRSA required grantees of the Community-Based Program to report on a combination of program outputs, such as the proportion of adolescents who completed an abstinence-until-marriage education program, and measures of adolescent intentions, such as the proportion of adolescents who committed to abstaining from sexual activity until marriage. For grants awarded in fiscal year 2005, when ACF assumed responsibility for the Community- Based Program from HRSA, grantees were not required to report on any specific performance measures.

OPA has also made efforts to assess the effectiveness of the AFL Program. Specifically, OPA requires grantees of the AFL Program to develop and report on outcome measures that are used to help demonstrate the extent to which grantees’ programs are having an effect on program participants. According to OPA officials, the agency recommends that grantees report on outcome measures, such as participants’ knowledge of the benefits of abstinence and their reported intentions to abstain from sexually activity, reported beliefs in their ability to remain abstinent, and reported parental involvement in their lives. To collect data on these outcome measures and any others, OPA requires all grantees funded in fiscal year 2004 and beyond to administer, at a minimum, a standardized questionnaire—developed by OPA—to their program participants, both when participants begin an abstinence-only education program and after the program’s completion. The standardized questionnaire includes questions intended to obtain information on participants’ reported involvement in extracurricular activities, behaviors linked to health risks, attitudes and intentions about abstinence, and opinions about the consequences of premarital sexual activity. Like ACF, OPA requires its grantees to contract with independent evaluators, such as colleges or universities, which are responsible for evaluating the effectiveness of grantees’ individual abstinence-until-marriage education programs. In addition to evaluating the extent to which grantees are meeting their goals, OPA officials stated that the independent evaluators may also provide input to grantees of the AFL Program on other aspects of the program to improve their service delivery. Unlike ACF, OPA requires that the third- party evaluations incorporate specific methodological characteristics, such as control groups or comparison groups and sufficient sample sizes. In addition, OPA requires that the evaluations for grantees funded in fiscal year 2004 and beyond account for baseline and follow-up data obtained from the standardized questionnaires.

OPA’s requirement that grantees use a standardized set of questionnaires, with data from these questionnaires used in evaluations, differs from OPA’s previous requirements. Previously, grantees of the AFL Program were not required to use a standardized method for collecting data that could be used to assess the effectiveness of their programs; instead, grantees chose their own data collection instruments. As a result, an OPA official explained, the collected data varied from one project to another. OPA officials said that the agency developed the standardized questionnaire to ensure uniformity in the data collected and allow the agency to more effectively aggregate the data reported in evaluations of individual abstinence-until-marriage education programs.

OPA officials told us that they plan to aggregate information from certain questions in the standardized set of questionnaires in order to report on certain performance measures as part of the agency’s annual performance reports. The measures include the extent of parental involvement in adolescents’ lives and the extent to which adolescents understand the benefits of abstinence. An agency official stated that the agency expects to begin receiving data from grantees that are using these questionnaires in January 2007. OPA did not previously have long-term measures of the performance of the AFL Program. Its current measures were developed in collaboration with the Office of Management and Budget (OMB) in response to an OMB review in 2004 that found that the AFL Program did not have any annual performance measures for measuring progress toward long-term goals.

In addition to requiring their grantees to report on outcomes used to assess program effectiveness, both ACF and OPA have provided technical assistance and training to their grantees in order to support grantees’ own program evaluation efforts. For example, in November 2005 the two agencies sponsored an evaluation conference for abstinence-until- marriage grantees that included presentations about evaluations and their methodology. Similarly, ACF’s Office of Planning, Research, and Evaluation sponsors annual evaluation conferences, and an ACF official told us that a recent conference placed “a significant emphasis” on the evaluation of abstinence-until-marriage education programs. In addition, HHS officials told us that ACF, along with ASPE, is funding a multiyear project that is designed to identify gaps in abstinence education evaluation and technical assistance needs, develop materials on abstinence education evaluation, deliver technical assistance and capacity-building activities related to program evaluation, and develop research reports related to abstinence education. OPA officials also told us that they attempt to help ensure grantees’ progress and effectiveness by offering various technical assistance workshops and conferences. For example, in May 2006 OPA provided a 2-day training conference to its grantees on the importance of program evaluations and administering evaluation instruments. In addition, OPA officials stated that the agency contracts with evaluation consultants, who review grantees’ evaluation tools and activities. OPA officials explained that these consultants provide in-depth technical assistance to grantees on how to improve grantees’ evaluations.

ACF Uses National Data on Adolescent Behavior as a Measure of the Overall Effectiveness of Its Two Programs

Requiring outcome reporting from state and community-based grantees is not ACF’s only effort to assess the effectiveness of its two programs. ACF also analyzes trends in adolescent behavior, as reflected in national data on birth rates among teens and the proportion of surveyed high school students reporting that they have had sexual intercourse. ACF uses these national data as a measure of the overall effectiveness of its State and Community-Based Programs, comparing the national data to program targets. In its annual performance reports, the agency summarizes the progress being made toward lowering the rate of births to unmarried teenage girls and the proportion of students (grades 9-12) who report having ever had sexual intercourse.

ACF’s use of national data to assess the effectiveness of the State and Community-Based Programs represents a change from how HRSA assessed the overall effectiveness of these programs. Whereas ACF compares national data on adolescent behavior to program targets, HRSA aggregated data from its state and community-based grantees. HRSA’s state grantees were allowed to select the data sources used to gauge their progress against certain performance measures. For example, in its annual performance reports on the State Program, HRSA reported information on the percentage of its state grantees meeting target rates for reducing the proportion of adolescents who have engaged in sexual intercourse, the incidence of youths aged 15 to 19 who have contracted selected STDs, and the rate of births among youths aged 15 to 17. To determine their progress in meeting their target rates, some state grantees, for example, reported national data from the Youth Risk Behavior Surveillance System, while other grantees reported state-collected data. After ACF assumed responsibility for the State and Community-Based Programs from HRSA, ACF began using national data on adolescent behavior as a measure of the programs’ effectiveness. According to ACF officials, the agency changed how it assessed its programs out of concern over the quality of the data state grantees were using in their performance reporting and because the agency wanted to use parallel measures of effectiveness for both programs. For example, according to state performance reports for fiscal year 2001 that we reviewed, two reports did not include adolescent pregnancy rates that year because the states did not collect data on abortions among this population. In addition, ACF officials told us that they decided not to use national data on STDs as a measure of program effectiveness because the goal of reducing STD rates is not as central to the State and Community-Based Programs as reducing sexual activity and birth rates among teens. However, one official stated that reducing STDs is an important “by-product” of the programs.

Some States Have Made Additional Efforts to Assess the Effectiveness of Abstinence- until-Marriage Education Programs

Some states have made additional efforts to assess the effectiveness of abstinence-until-marriage education programs, although they are not required by ACF. Specifically, we found that 6 of the 10 states in our review that receive funding through ACF’s State Program have made efforts to conduct evaluations of selected abstinence-until-marriage programs in their state. All 6 of the states worked with third-party evaluators, such as university researchers or private research firms, to perform the evaluations, which in general measure self-reported changes in program participants’ behavior and attitudes related to sex and abstinence as indicators of program effectiveness. To obtain this information, the third-party evaluators have typically relied on surveys administered to program participants at the start of a program, its conclusion, and during a follow-up period anywhere from 3 months to almost 3 years after the conclusion. The third-party evaluations for 4 of the 6 states in our review have been completed as of February 2006, and the results of these studies have varied.

Among the 4 states that have completed third-party evaluations, 3 states require the abstinence programs in their state to measure reported changes in participants’ behavior as an indicator of program effectiveness—both at the start of the program and after its completion. The 3 states require their programs to track participants’ reported incidence of sexual intercourse. In addition, 2 states require their programs to track biological outcomes, such as pregnancies, births, or STDs. In addition, 6 of the 10 states in our review require their programs to track participants’ attitudes about abstinence and sex, such as the number of participants who make pledges to remain abstinent.

Some states also provide technical assistance to the abstinence-until- marriage programs they support in their state. This assistance is designed to help programs evaluate and improve their effectiveness. Officials from 5 of the 10 states in our review either told us or provided documentation that they provide technical assistance on evaluations to abstinence programs in their state. One state official said that the abstinence-until- marriage programs supported by the state were found to be ill-prepared to conduct evaluations themselves, and that she now requires these programs to dedicate a portion of their grants to contract with a third-party or state evaluator to assist them in program-level evaluations. Officials from another state told us that they contract with a private organization of public health professionals in order to provide evaluation consultation and technical assistance for the abstinence-until-marriage programs the state supports.

ASPE, CDC, and NIH Are Funding Studies Designed to Assess the Effectiveness of Abstinence-until-Marriage Education Programs

In addition to ACF and OPA, other organizational units within HHS have made efforts to assess the effectiveness of abstinence-until-marriage education programs. ASPE is currently sponsoring a study of the Community-Based Program and a study of the State Program. For the former program, ASPE has contracted with Abt Associates to help design the study, and an ASPE official told us that once the agency selects an appropriate design, it will competitively award a contract to conduct the study. For the latter program, ASPE has contracted with Mathematica Policy Research, Inc. (Mathematica), which is in the process of examining the impact of five programs funded through the State Program on participants’ attitudes and behaviors related to abstinence and sex. As of August 2006, Mathematica has published two reports on findings from its study—an interim report documenting the experiences of schools and communities that receive abstinence-until-marriage education funding, and a report on the first-year impacts of selected state abstinence-until- marriage education programs. Mathematica’s final report, which has not been completed, will examine the impact of the State Program on behavioral outcomes, including abstinence, sexual activity, risk of STDs, risk of pregnancy, and drug and alcohol use. An ASPE official told us that the agency expects a final report to be published in 2007.

Like ASPE, CDC has made its own effort to assess the effectiveness of abstinence-until-marriage education. CDC is sponsoring a study to evaluate the effectiveness of two middle school curricula—one that complies with abstinence education program requirements and one that teaches a combination of abstinence and contraceptive information and skills. In CDC’s study, five middle schools chosen at random will receive a program consisting of abstinence-until-marriage education exclusively; five schools will receive comprehensive sex education, which also includes information on contraception; and five schools will be assigned to a control group. The study will examine the relative effectiveness of the programs on behavioral outcomes such as reported sexual risk behaviors and changes in attitudes related to abstinence and sex. CDC plans to recruit approximately 1,500 seventh grade students into its study and will follow them over a 2-year period. The agency expects to complete the study in 2009.

NIH has funded studies comparing the effectiveness of education programs that focus only on abstinence with the effectiveness of sex education programs that teach both abstinence and information about contraception. As of August 2006, NIH is funding five studies, which in general are comparing the effects of these two types of programs on the sexual behavior and related attitudes among groups of either middle school or high school students. For example, in one NIH study, researchers are using groups of seventh and eighth grade adolescents to assess the impact of a variety of programs on, among other issues, adolescents’ reported sexual activities, knowledge, and beliefs. For this study, researchers are comparing these outcomes among students who received abstinence-until-marriage education; students who received a combination of abstinence and contraceptive education; and students who participated in a general health class, who serve as a comparison group.

NIH expects both this study and its other four studies to be competed in 2006.

Other Researchers Have Also Made Efforts to Assess the Effectiveness of Abstinence- until-Marriage Education Programs

In addition to the efforts of researchers working on behalf of HHS and states, other researchers—such as those affiliated with universities and various advocacy groups—have made efforts to study the effectiveness of abstinence-until-marriage education programs. This work includes studies of the outcomes of individual programs and reviews of other studies on the effectiveness of individual abstinence-until-marriage education programs. In general, research studies on the effectiveness of individual abstinence-until-marriage education programs have examined the extent to which they changed participants’ demonstrated knowledge, declared intentions, and reported behavior related to sexual activity and abstinence. For example, some studies examined the impact of abstinence-until- marriage education programs on participants’ knowledge of concepts taught in the programs, as well as participants’ declared attitudes about abstinence and teen sex. Some studies examined the impact of these programs on such outcomes as participants’ declared commitment to abstain from sex until marriage, participants’ understanding of the potential consequences of having intercourse, and participants’ reported ability to resist pressures to engage in sexual activity. Some of the studies we reviewed examined the impact of abstinence-until-marriage programs on participants’ sexual behavior, as measured, for example, by the proportion of participants who reported having had sexual intercourse and the frequency of sexual intercourse reported by participants. In general, the efforts to study and build a body of research on the effectiveness of most abstinence education programs have been under way for only a few years, in part because grants under the two programs that account for the largest portion of federal spending on abstinence education—the State Program and the Community-Based Program—were not awarded until 1998 and 2001, respectively.

Several Factors Limit the Conclusions That Can Be Drawn about the Effectiveness of Abstinence-until-Marriage Education Programs

Most of the efforts of HHS, states, and other researchers to evaluate the effectiveness of abstinence-until-marriage education programs included in our review have not met certain minimum criteria that experts have concluded are necessary in order for assessments of program effectiveness to be scientifically valid. For example, most of the efforts included in our review did not include experimental or quasi-experimental designs, nor did they measure behavioral or biological outcomes. In addition, the results of some assessment efforts that meet the criteria of a scientifically valid assessment have varied, and two key studies that meet these criteria have not yet been completed.

Experts Have Developed Criteria to Evaluate Efforts to Assess Abstinence-until- Marriage Education Programs

In an effort to better assess the merits of the studies that have been conducted on the effectiveness of sexual health programs—including abstinence-until-marriage education programs—scientific experts have developed criteria that can be used to gauge the scientific rigor of these evaluations. For example, in 2001, the National Campaign to Prevent Teen Pregnancy—an organization focused on reducing teen pregnancy— published a report by a panel of scientific experts that assessed the evidence reported on abstinence-until-marriage education programs in peer-reviewed journals and other literature. The panel developed criteria that an evaluation of a program’s effectiveness must meet in order for the program’s results to be considered scientifically valid. In addition, in 2004, former U.S. Surgeon General David Satcher convened a panel of experts to discuss, among other things, best practices for evaluating the effectiveness of sexual health education programs—including abstinence-until-marriage education programs. This panel published a report in 2006 that describes similar scientific criteria that assessments of program effectiveness need to meet in order for their results to be scientifically valid. Further, experts we interviewed agreed that these criteria are important for ensuring that the results of a study support valid conclusions. In general, these panels, as well as the experts we interviewed, agreed that scientifically valid studies of a program’s effectiveness should include the following characteristics: An experimental design that randomly assigns individuals or schools to either an intervention group or control group, or a quasi-experimental design that uses nonrandomly assigned but well-matched comparison groups. According to the panel of scientific experts convened by the National Campaign to Prevent Teen Pregnancy, experimental designs or quasi-experimental designs with well-matched comparison groups have at least three important strengths that are typically not found in other studies, such as those that use aggregated data: they evaluate specific programs with known characteristics, they can clearly distinguish between participants who did and did not receive an intervention, and they control for other factors that may affect study outcomes. Therefore, experimental and quasi-experimental study designs have a greater ability to assess the causal impact of specific programs than other types of studies. According to scientific experts, studies that include experimental or quasi- experimental designs should also collect follow-up data for a minimum number of months after subjects receive an intervention. Experts reported that follow-up periods are important in order to identify the effects of a program that are not immediately apparent or to determine whether these effects diminish over time. In addition, experts have reported that studies should have a sample size of at least 100 individuals for study results to be considered scientifically valid.

Studies should assess or measure changes in biological outcomes or reported behaviors instead of attitudes or intentions. According to scientific experts, biological outcomes—such as pregnancy rates, birth rates, and STD rates—and reported behaviors—such as reported initiation and frequency of sexual activity—are better measures of the effectiveness of abstinence-until-marriage programs, because adolescent attitudes and intentions may or may not be indicative of actual behavior. For example, adolescents may report that they intend to abstain from sexual intercourse but may not actually do so.

Most Efforts to Assess the Effectiveness of Abstinence- until-Marriage Education Programs Have Not Used an Experimental or Quasi- Experimental Design

Many of the efforts by HHS, states, and other researchers that we identified in our review lack at least one of the characteristics of a scientifically valid study of program effectiveness. That is, most of the efforts to assess the effectiveness of these programs have not used experimental or quasi-experimental designs with sufficient follow-up periods and sample sizes to make their conclusions scientifically valid. For example, ACF—and before it, HRSA—used, according to ACF officials, grantee reporting on outcomes in order to monitor grantees’ performance, target training and technical assistance, and help grantees improve service delivery. However, because the outcomes reported by grantees have not been produced through experimentally or quasi-experimentally designed studies, such information cannot be causally attributed to any particular abstinence-until-marriage education program. While ACF requires its fiscal year 2006 grantees of the Community-Based Program to contract with third-party evaluators to select and monitor outcomes for their programs, ACF is not specifically requiring these grantees to use experimental or quasi-experimental designs. Therefore, it is not clear whether these evaluations will include such designs. Similarly, ACF’s use of national data on adolescent behavior and birth rates to assess its State and Community- Based Programs is of limited value because these data do not distinguish between those who participated in abstinence-until-marriage education programs and those who did not. Consequently, these national data sets, which represent state-reported vital statistics and a nationwide survey of high school students, cannot be used to causally link declines in birth rates and adolescent sexual activity to the effects of specific abstinence-until- marriage education programs.

Similarly, the efforts we identified by states and researchers to assess the effectiveness of abstinence-until-marriage education programs often did not include experimental or quasi-experimental designs. None of the state evaluations we reviewed that have been completed included randomly assigned control groups. For instance, one state evaluation that we reviewed only included students who volunteered to participate in the study. This evaluation report stated that the absence of a randomly assigned control group in the evaluation did not allow the evaluators to determine whether observed changes in participants’ reported sexual behavior—as indicated through surveys administered at the beginning and end of a program—could be attributed to the abstinence-until-marriage education program. Similarly, some of the journal articles that we reviewed described studies to assess the effectiveness of abstinence-until- marriage programs that did not include experimental or quasi- experimental designs needed to support scientifically valid conclusions about the programs’ effectiveness. In these studies, researchers administered questionnaires to study participants before and after they completed an abstinence-until-marriage education program and assessed the extent to which the responses of participants changed. These studies did not compare the responses of study participants with a group that did not participate in an abstinence-until-marriage education program. In addition, some of the studies used insufficient follow-up periods, thereby limiting the conclusions that can be drawn about the effectiveness of the abstinence-until-marriage education programs being studied. For example, two journal articles that we reviewed described studies that measured the effectiveness of abstinence-until-marriage programs in delaying the initiation of sexual activity from 1 to 2 months after completion of the program. Scientific experts consider this follow-up period too short to assess whether the programs had a valid effect.

According to scientific experts, HHS, states, and other researchers face a number of challenges in designing experimental or quasi-experimental studies of program effectiveness. According to these experts, experimental or quasi-experimental studies can be expensive and time-consuming to carry out, and many grantees of abstinence-until-marriage education programs have insufficient time and funding to support these types of studies. Moreover, it can be difficult for researchers assessing abstinence- until-marriage education programs to convince school districts to participate in randomized intervention and control groups, in part because of sensitivities to surveying attitudes, intentions, and behaviors related to abstinence and sex. For example, in a third-party evaluation of its program, one grantee of the State Program originally planned to administer follow-up surveys 1 year after participants finished their abstinence education program, but the evaluators decided not to conduct this follow-up because of confidentiality concerns and the difficulty of locating students. In addition, the contractors hired to design ASPE’s study of the effectiveness of the Community-Based Program have reported difficulties finding school districts that are willing to participate in randomly assigned intervention and control groups receiving either abstinence-until-marriage education or comprehensive sex education. An ASPE official told us that although a “randomized approach” is the best design for assessing the effectiveness of a program, the approach is also the most difficult to conduct.

Most Efforts to Assess the Effectiveness of Abstinence- until-Marriage Education Programs Have Not Measured Behavioral or Biological Outcomes

Another factor that limits the conclusions that can be drawn about the effectiveness of abstinence-until-marriage education programs is the fact that most efforts in our review to study the effectiveness of these programs did not measure changes in behavioral or biological outcomes among participants. Instead, most of the efforts we identify in our review used reported intentions and attitudes in order to assess the effectiveness of abstinence-until-marriage programs. For example, neither ACF’s community-based grantees nor OPA’s AFL grantees are required to report on behavioral or biological outcomes, such as rates of intercourse or pregnancy. Similarly, the journal articles we reviewed were more likely to use reported attitudes and intentions—such as study participants’ reported attitudes about premarital sexual activity or their reported intentions to remain abstinent until marriage—rather than their reported behaviors or biological outcomes to assess the effectiveness of abstinence-until- marriage programs. For example, in one journal article we reviewed, participants were asked to rate the likelihood that they would have sexual intercourse as unmarried teenagers; another journal article described a study in which participants rated the likelihood that they would have sexual intercourse in the next year, before finishing high school, and before marriage.

Experts, as well as state and HHS officials, have reported that it can be difficult to obtain scientifically valid information on biological outcomes and sexual behaviors. Specifically, experts have reported that when measuring an abstinence-until-marriage education program’s affect on biological outcomes—such as reducing pregnancy or birth rates—it is necessary to have large sample sizes in order to determine whether a small change in biological outcomes is the result of the abstinence-until- marriage education program. In addition, state and federal officials told us that they have experienced difficulties obtaining information on sexual behaviors because of the sensitive nature of the information they were trying to collect. For example, one state official told us that her state’s effort to evaluate abstinence-until-marriage education programs was only able to measure changes in participants’ reported attitudes, instead of behaviors, because the evaluators needed to obtain consent from the parents of the program participants in order to ask them about their sexual behavior. The state official explained that the requirement to obtain consent from parents raised issues of self-selection, and therefore state officials decided to ultimately halt the study and only report on the attitudes that they had measured. In another example, ACF’s fiscal year 2006 budget justification reports that ACF has had some difficulty in obtaining reliable data from state grantees, in part because questions about teenage sexual behavior are sensitive. OPA officials also acknowledged that many communities will not allow grantees to ask program participants questions about their sexual behavior because the communities believe such questions are too intrusive. One OPA official said that such restrictions affect the agency’s ability to measure behavioral outcomes, explaining that OPA cannot measure what it cannot ask about.

Results of Some Scientifically Valid Assessment Efforts Have Varied, and Other Key Studies Have Not Been Completed

Among the assessment efforts we identified are some studies that meet the criteria of a scientifically valid effectiveness study. However, results of these studies have varied, and this limits the conclusions that can be drawn about the effectiveness of abstinence-until-marriage education programs. Some researchers have reported that abstinence-until-marriage education programs have resulted in adolescents reporting having less frequent sexual intercourse or fewer sexual partners. For example, in one study of middle school students, participants in an abstinence-until- marriage education program who had sexual intercourse during the follow-up period were 50 percent less likely to report having two or more sexual partners when compared with their nonparticipant peers. In contrast, other studies have reported that abstinence-until-marriage education programs did not affect the reported frequency of sexual intercourse or number of sexual partners. For example, one study of middle school students found that participants of an abstinence-until- marriage program were not less likely than nonparticipants at the 1 year follow-up to report less frequent sexual intercourse or fewer sexual partners. In addition to these varied findings, one study found that an abstinence-until-marriage program was effective in delaying the initiation of sexual intercourse in the short term but not long term. Experts with whom we spoke emphasized that there are still too few scientifically valid studies completed to date that can be used to determine conclusively which, if any, abstinence-until-marriage programs are effective.

Additionally, among the assessment efforts we identified are some studies that experts anticipate will meet the criteria of a scientifically valid effectiveness study but are not yet completed. One of these key studies is the final Mathematica report, contracted by ASPE, on the State Program. The final report was originally slated for publication in 2005, but an ASPE official stated that the final report has been delayed until 2007 so that researchers can extend the follow-up period to improve their response rate and the reliability of the information they collect. Another key study is CDC’s research on middle school programs, which is not expected to be completed until 2009.

Experts and federal officials we interviewed stated that they expect the results of these two federally funded studies to add substantively to the body of research on the effectiveness of abstinence-until-marriage education programs. One expert with whom we spoke said that she expects the final Mathematica report on participants’ behaviors to provide the groundwork for the field. Another expert we interviewed stated that the CDC study was very well-designed and she expects the results to contribute to the development of effective abstinence-until-marriage education curricula.

Conclusions

There have been various efforts—by HHS, states, and others—to assess the scientific accuracy of educational materials used in abstinence-until- marriage education programs and the effectiveness of these programs. However, efforts to evaluate both the accuracy and effectiveness of abstinence-until-marriage education programs have been, in various ways, limited. ACF, which administers the two programs that account for the largest portion of federal spending on abstinence-until-marriage education, does not review or require its grantees to review program materials for scientific accuracy. In addition, not all grantees of the State Program have chosen to review their materials. Because of these limitations, ACF cannot be assured that the materials used in its State and Community-Based Programs are accurate. Moreover, OPA, which reviews all grantees’ proposed abstinence-until-marriage educational materials, and states that review educational materials have found inaccuracies in some educational materials used by abstinence-until-marriage programs.

Similarly, most of the efforts described in our review to assess the effectiveness of abstinence-until-marriage programs have not met minimum scientific criteria needed to draw valid conclusions about their effectiveness. Specifically, most efforts by agencies, states, and other researchers have not included experimental or quasi-experimental designs that can establish whether changes in behaviors or biological outcomes can be causally linked to specific abstinence-until-marriage education programs. While these types of studies are time-consuming and expensive, experts said that they are the only definitive way to draw valid conclusions about the effectiveness of these programs. In addition, among the assessment efforts we identified are some studies funded by HHS that experts anticipate will meet the criteria of a scientifically valid effectiveness study but are not yet completed. When completed, these HHS-funded studies may add substantively to the body of research on the effectiveness of abstinence-until-marriage education programs.

Recommendation for Executive Action

To address concerns about the scientific accuracy of materials used in abstinence-until-marriage education programs, we recommend that the Secretary of HHS develop procedures to help assure the accuracy of such materials used in the State and Community-Based Programs. To help provide such assurances, the Secretary could consider alternatives such as (1) extending the approach currently used by OPA to review the scientific accuracy of the factual statements included in abstinence-until-marriage education to materials used by grantees of ACF’s Community-Based Program and requiring grantees of ACF’s State Program to conduct such reviews or (2) requiring grantees of both programs to sign written assurances in their grant applications that the materials they propose using are accurate.

Agency Comments and Our Evaluation

HHS provided written comments on a draft of this report. (See app. III.)

In its written comments, HHS stated that it will consider requiring grantees of both ACF programs to sign written assurances in grant applications that the materials they use are accurate. Regarding accuracy, HHS’s written comments also noted that all applicants for federal assistance attest on the application form--Standard Form 424--that all data in their applications are “true and correct,” and that in the view of HHS, this applies to information presented in curricula funded by federal grants. However, as we stated in the draft report, grantees of the State Program are not required to submit curricula as a part of their applications; therefore, the attestation in Standard Form 424 would not apply to curricula used by those grantees. In addition, as stated in the draft report, some states have reviewed materials used in abstinence-until-marriage education programs, but these reviews occurred after they received funding from ACF. Further, while grantees of the Community-Based Program were required to submit copies of their curricula and a Standard Form 424 in fiscal year 2006 as part of their applications, none of the materials specifically require an assurance of scientific accuracy. Further, OPA and states have found inaccuracies in educational materials used by abstinence-until-marriage programs.

HHS’s written comments also stated that ACF requires that curricula conform to HHS’s standards grounded in scientific literature. HHS’s comments refer to the curriculum standards for this program that detail what types of information must be included in abstinence-until-marriage curricula, and the comments stated that the curricula must provide supporting references for this information. Further, HHS’s comments stated that ACF staff review the curricula to ensure compliance with these standards. The draft report stated this. However, a requirement that curricula include certain types of information does not necessarily ensure the accuracy of the scientific facts included in the abstinence-until- marriage materials. For example, while education materials may include information on failure rates associated with contraceptives or STD infections, this information may be outdated or otherwise inaccurate or incomplete. HHS’s written comments also stated that if it finds inaccurate statements during the review process or at any time during the grant period, ACF works with grantees to take corrective action. To ensure completeness, we have added this statement to the report. Further, HHS stated that 2 inaccuracies cited in the draft report had been corrected before our work began. We believe HHS is referring to inaccuracies identified by OPA during its review of materials for scientific accuracy and this reinforces the need for review of materials used by ACF’s grantees.

As HHS noted in its written comments, we did not define the term scientific accuracy. HHS stated that it disagreed with certain findings of the report because it was difficult to precisely determine the criteria we employed in making the recommendation as to scientific accuracy. As we stated in the scope and methodology section of the draft report, the objective of our work was to focus on efforts by HHS and states to review the accuracy of scientific facts included in abstinence-until-marriage education materials. Performing an independent assessment of the criteria used by these entities to determine the scientific accuracy of education materials or the quality of the reviews was beyond the scope of the work.

Regarding effectiveness, HHS’s written comments also described a number of actions it is taking to determine program effectiveness and improve the quality of programs and research. Specifically, HHS’s comments described (1) studies undertaken or funded by ASPE, CDC, and NIH; (2) technical assistance provided by OPA and ACF; (3) grantee evaluation requirements; and (4) ACF and OPA requirements for the amount of grant funds to be spent on evaluations. All of this information was included in our draft report. HHS’s comments also described a new effort funded by ACF and ASPE that is designed to build capacity for quality research in the field of abstinence education. We added information on this effort to the report. HHS’s written comments also describe evaluations that resulted from an Abstinence Education Evaluation Conference sponsored by ACF and OPA. While this conference was described in the draft report, we added more detail regarding the content of the conference. HHS’s written comments also describe OPA’s efforts to assess the effectiveness of the AFL Program. We had included this information in the draft report.

HHS’s written comments stated that it may be too soon to draw conclusions about the effectiveness of ACF’s and OPA’s programs, in part, because key studies have not been completed. We agree and discussed this in the draft report. As we noted in the draft report, key studies funded by HHS that experts anticipate will meet the criteria of a scientifically valid effectiveness study are not yet completed, but when completed these HHS funded studies may add substantively to the body of research on the effectiveness of abstinence-until-marriage education programs.

In addition, the comments stated that having an inadequate amount of scientifically valid and conclusive evaluation studies is not unique to abstinence-until-marriage education programs, and a recent ASPE review of comprehensive sex education programs found mixed results on their effectiveness. However, the scope of our report was focused on abstinence-until-marriage education programs, and we did not review comprehensive sex education programs or make any comparisons between the two types of programs.

HHS also provided technical comments, which we incorporated into the report as appropriate.

As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its date. We will then send copies of this report to the Secretary of HHS and to other interested parties. In addition, this report is available at no charge on GAO’s Web site at http://www.gao.gov. We will also make copies available to others upon request.

If you or your staff have any questions about this report, please call me at (202) 512-3407 or crossem@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: HRSA’s Technical Assistance Contract for Abstinence Education

The Health Resources and Services Administration (HRSA) awarded a contract to the National Abstinence Clearinghouse (NAC) in 2002 to provide assistance with its Community-Based Abstinence Education Program (Community-Based Program) and Abstinence Education Program (State Program). NAC is a nonprofit educational organization whose mission is to promote the appreciation for and practice of sexual abstinence until marriage through the distribution of age appropriate, factual, and medically referenced materials. The purpose of the contract was (1) to develop national criteria for the review of abstinence-until- marriage educational materials and to create a directory of approved materials; (2) to provide medical accuracy training to grantees; and (3) to provide technical support to grantees, such as assistance with program evaluation.

We are reporting on the steps that HRSA took to award the contract to NAC in response to concerns that have been raised by a congressional requester. In general, these concerns centered on the extent to which the selection process was competitive and whether HRSA identified the potential for an organizational conflict of interest.

HRSA awarded the contract to address three concerns it had with the Community-Based Program during 2001, the first year of its implementation. First, HRSA officials needed guidance to determine whether abstinence-until-marriage education materials conformed to the definitional requirements of the Social Security Act. Second, many grantees lacked the medical background and training to ensure that they would provide medically accurate, science-based information in their programs. Third, grantees also lacked experience with the technical management of federal grants, including how to conduct evaluations of their programs.

HRSA Used Full and Open Competition Procedures to Award the Contract to NAC

HRSA used full and open competition procedures to award the contract to NAC. In doing so, HRSA (1) publicly solicited proposals from potential contractors; (2) conducted technical evaluations of both the original proposals and the revised proposals for those considered to be in the competitive range; and (3) determined that NAC’s proposal represented the best overall value to the government. This process, which took place from May 2002 through September 2002, resulted in HRSA awarding NAC the contract with a potential value of nearly $2.7 million.

HRSA issued a notice on May 20, 2002, on the FedBizOpps Web site, the government point of entry for notifying potential contractors of federal contract opportunities, indicating its intent to publicly request proposals from prospective contractors in June 2002. On June 20, 2002, HRSA posted the solicitation on the FedBizOpps Web site indicating that the abstinence contract would be awarded using full and open competition procedures, that is, all responsible prospective contractors would be provided the opportunity to compete. The solicitation, which was a Request for Proposals (RFP), described the contract objectives, which included (1) the development of national criteria for the review of abstinence-until-marriage educational materials and the development of a directory of approved materials; (2) the provision of medical accuracy training to grantees; and (3) the provision of technical support to grantees, such as assistance with program evaluation. The RFP stated that HRSA intended to award a cost-reimbursement contract with fixed fee for a 1-year base period and 2 option years. This was a best value procurement; that is, HRSA reserved the right in the RFP to select for award the proposal that HRSA determined offered the best value to the government, even if it did not offer the lowest cost. Further, the RFP stated that the technical evaluation of the prospective contractors’ proposals would receive paramount consideration in the selection of the contractor. According to the RFP, this evaluation would include an assessment of the prospective contractor’s technical approach, the organizational experience and expertise of the prospective contractor, the plans for personnel and management of the work, and the prospective contractor’s statement and understanding of the project purpose. Other factors, such as the estimated cost, past performance under other contracts for similar services, and the subcontracting plan would also be considered in the selection process. Five prospective contractors submitted proposals to HRSA by July 31, 2002, when proposals were due.

HRSA established a review committee to conduct the technical evaluation of the five proposals. This committee included three voting members and a nonvoting chairperson. The Director of HRSA’s Community-Based and State Programs and two analysts from other programs within the Department of Health and Human Services (HHS) served as the voting members, and the chairperson of the review committee was a project officer of HRSA’s Community-Based Program. The committee members conducted the technical evaluation of the proposals, according to the criteria in the RFP, as described above. Three proposals with the highest technical scores were determined to be in the competitive range, with NAC’s proposal receiving the highest technical score. HRSA requested in writing that the competitive range offerors address certain technical and cost issues and submit revised proposals to HRSA by September 17, 2002. For example, HRSA requested that one of the prospective contractors other than NAC clearly describe its proposed management of day-to-day tasks of the contract and provide justification for several labor and travel expenditures. HRSA did not have oral discussions with the competitive range offerors. HRSA’s review committee evaluated the revised proposals and again gave NAC’s revised proposal the highest technical score.

Although NAC’s estimated cost was not the lowest among the proposals in the competitive range, HRSA determined that NAC had proposed a realistic cost estimate for the contract. Accordingly, and in light of the NAC proposal’s high technical rating and the RFP’s evaluation criteria giving paramount consideration to the technical evaluation, HRSA determined that NAC’s proposal represented the best value to the government. HRSA awarded a contract to NAC on September 27, 2002. The contract had a 1-year base period of performance with an estimated value of $854,681, and included 2 option years for a total potential value of $2,673,784. According to a HRSA official, this cost-reimbursement contract did not include a fee. All of the prospective contractors were made aware that a debriefing to explain the selection decision and contract award would be provided at their request. One prospective contractor requested and received a debriefing from HRSA. No protests were filed with the agency challenging the award of the contract to NAC. There were no bid protests filed with GAO.

HRSA Identified No Actual or Potential Organizational Conflicts of Interest

HRSA officials told us that they did not identify any actual or potential organizational conflicts of interest during the acquisition process. As defined in the Federal Acquisition Regulation (FAR), an organizational conflict of interest arises where because of other activities or relationships, a person is unable or potentially unable to provide impartial assistance or advice to the government; or the person’s objectivity in performing the contract work is or might be otherwise impaired; or a person has an unfair competitive advantage.

An organizational conflict of interest may result when factors create an actual or potential conflict of interest during performance of a contract, or when the nature of the work to be performed under one contract creates an actual or potential conflict of interest involving a future acquisition. Under the FAR, contracting officers are required to analyze planned acquisitions to identify and evaluate potential organizational conflicts of interest as early in the acquisition process as possible, and to take steps to avoid, neutralize, or mitigate significant potential conflicts of interest before a contract is awarded.

According to HRSA’s contracting officer, HRSA did not identify any actual or potential organizational conflicts of interest. In reaching this conclusion, the contracting officer told us that he reviewed the statement of work, including the background and objectives of the proposed contract, the stated purpose of the contact, the criteria established to evaluate the proposals, the past performance of the competitors, and NAC’s proposal. HRSA’s contracting officer also told us that he did not formally document his assessment of organizational conflict of interest.

Appendix II: Methodology for Identifying and Reviewing Research Studies

To identify research studies that examine the effectiveness of abstinence- until-marriage education programs among adolescents and young adults, we searched two reference database systems, PubMed and ProQuest. We used the following keywords to search for research studies that were published from January 1, 1998, through May 22, 2006: “virginity,” “abstinence education,” “abstinence and curriculum,” “abstinence only,” “teen pregnancy and prevention,” and “abstinence until marriage.” We reviewed the research article titles that were generated from the PubMed and ProQuest searches and identified articles that appeared to focus on the evaluation of the effectiveness of abstinence-until-marriage education programs. In cases where we could not determine, based on the title, whether a study appeared to focus on an abstinence-until-marriage education program evaluation, we reviewed a summary of the article to obtain more information about the research study. We also examined previous summaries of the literature to identify additional research studies.

We then selected research studies for inclusion in our literature review if they met three criteria. First, the study evaluated a group-based, abstinence-until-marriage education program. We did not select studies that evaluated one-on-one interactions, such as education programs focused exclusively on parent-child interactions, or that evaluated media campaigns. We reviewed the description of each education program and curriculum, as described in the study, to determine whether an abstinence- until-marriage education program was being evaluated. Education programs that were described as including detailed contraceptive information in their curricula, for example, were not classified as abstinence-until-marriage programs. Second, the study targeted adolescents and young adults in the United States, for example, by indicating that participants in the evaluation were high school or middle school students. Third, the study was a quantitative rather than a qualitative evaluation of an abstinence-until-marriage education program. We selected 13 research studies for inclusion in our literature review.

We reviewed the selected research studies to obtain detailed information about their methodologies and outcome variables. For example, we determined whether each study used an experimental or quasi- experimental design and whether the outcome measures included attitudes, behavioral intentions, behaviors such as initiation of sexual intercourse, or a combination of these.

Appendix III: Comments from the Department of Health and Human Services

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Kristi Peterson, Assistant Director; Kelly DeMots; Pam Dooley; Krister Friday; Julian Klazkin; and Amy Shefrin made key contributions to this report.