Healthy Marriage and Responsible Fatherhood Initiative: Further Progress Is Needed in Developing a Risk-Based Monitoring Approach to Help HHS Improve Program Oversight
Released: 2008-10-27T00:00:00

Letter

Background

Welfare reform in 1996 made sweeping changes to the national welfare policy, including a new emphasis on marriage as an area of societal and governmental concern. With the passage of the Personal Responsibility and Work Opportunity Reconciliation Act of 1996, which established the Temporary Assistance for Needy Families (TANF) program, Congress wrote into law that marriage is the foundation of a successful society and promotes the interests of children. Congress was, in part, prompted to address this issue because of what it deemed a “crisis in our Nation” in the rate of pregnancies and births to unmarried women. In the legislation, Congress cited the negative consequences to children that result from these pregnancies and births, including greater risk for child abuse and neglect, higher rates of poverty, and lower educational aspirations.

TANF was reauthorized under the Deficit Reduction Act of 2005 (DRA), and signed into law in February 2006. DRA appropriated $150 million a year for 5 years in discretionary grants for the Healthy Marriage and Responsible Fatherhood Initiative (Initiative). While the Initiative was established as part of TANF, the nation’s welfare program, it does not impose income limits for program participants. However, HHS designated a few priority groups for funding under the Initiative, including incarcerated fathers and low-income, unwed, expectant or new parents. In structuring the Initiative, HHS created two distinct grant programs—one relating to Healthy Marriage and one to Responsible Fatherhood—but with common aims. The Healthy Marriage program is aimed at encouraging the formation and maintenance of two-parent households to improve child well-being through healthy marriage promotion, and the Responsible Fatherhood program is designed to strengthen the role of the father as a means of promoting child well-being, specifically within the context of marriage. HHS has stressed that the overarching Initiative is not designed to encourage couples to stay in unhealthy marriages.

In the legislation, Congress prescribed the “allowable” activities for the Initiative (see table 1). Given the broadness of these allowable activities, HHS developed examples of services grantees could provide, such as providing after-school programs for high school students and marriage education courses that incorporate information on financial literacy. Although providing services to victims of domestic violence is not an allowable activity (see table 1), organizations were required by DRA to describe in their grant application how their programs or activities would “address” issues of domestic violence, and commit in their application to consult with experts in domestic violence in developing their programs and activities. The DRA also required that organizations describe in their application what they would do to ensure and how they would inform individuals that participation in programs is voluntary.

In fiscal year 2007 most of the funding, approximately $113 million, was used to support Healthy Marriage and Responsible Fatherhood demonstration grants, while the remaining funds were used for research, technical assistance, administrative costs, and other TANF-related activities (see fig. 1).

As part of the agency’s overall research agenda, HHS has sponsored several impact evaluations of its programs. These evaluations are considered to be the best method of determining the extent to which the program, rather than other factors, is causing specific participant outcomes. Impact evaluations, which are awarded through a competitive bid process to experienced research firms, often are complex, multiyear studies that can be difficult and costly to undertake and require particular attention to both study planning and execution. Moreover, maintaining proper incentives to obtain and sustain the participation of populations that do not have financial and familial stability can be challenging. In previous work, we found that HHS has established a rigorous research agenda that regularly evaluates how well its programs are working. In particular, HHS has a diverse research agenda focused on TANF that includes research on strategies to help low-income individuals gain self- sufficiency.

HHS Awarded Grants to a Range of Public and Private Organizations, but the Awards Process Contributed to Challenges for Some Grantees

HHS awarded grants to a range of public and private organizations, but its awards process later contributed to challenges for these grantees. HHS shortened its awards process to meet a deadline specified in legislation that allowed 7 months to award grants. HHS awarded grants to a diverse set of grantees that provided direct services to program participants in 47 states, the District of Columbia, and American Samoa. However, as part of its awards process, HHS did not fully examine grantees’ programs as described in grantee applications, including the activities they planned to offer, contributing to challenges for some grantees as they were implementing their programs.

HHS Shortened Its Existing Awards Process to Meet DRA Deadline for Awarding Grants

HHS shortened its process to award grants by the end of the fiscal year (September 30). Under DRA, which became law in February 2006, HHS had to award grants in 7 months. Within this time frame, HHS had to perform several tasks related to the awards process. Specifically, HHS staff said they developed the grant announcements and the criteria for selecting grantees under tight time constraints and limited the amount of time organizations could apply for grants to fewer than the 60 days recommended in HHS’s policy manual. HHS officials, who told us they had not expected that more than 1,650 organizations would apply for funding, hired The Dixon Group, a management consulting firm, to receive applications, locate grant application reviewers, and assist with reviewer training. At the same time The Dixon Group was receiving applications, they also were selecting peer reviewers. Approximately 600 peer reviewers served on 40 to 50 review panels for 4 weeks during July and August. While the grant announcements stated that grant application reviewers should be experts, HHS allowed peer review of the applications and The Dixon Group and HHS characterized graduate students, professors, and practitioners as peer reviewers. Further, because individuals who were experts in the field of marriage and fatherhood applied for the grants, it limited the pool of available expert reviewers. We reviewed several of the resumes of the peer reviewers and found that while most had experience as federal reviewers, their professional and volunteer experiences were not always directly relevant to marriage and fatherhood services. For example, one peer reviewer had experience in nursing and another listed experience as a social studies teacher.

To determine which organizations would receive funding, HHS developed guidance that outlined a five-part criteria for most grants, with each criterion worth a specific amount of points. Reviewers scored organizations’ applications using the guidance provided by HHS and by judging how well the applicant responded to each criterion. For example, a major criterion was the applicant’s “approach,” worth 40 points. For this criterion, applicants were asked to describe their approach to recruiting and retaining participants, their proposed activities, and time frames for accomplishing specific milestones. Applicants also were required to demonstrate that their proposed activities were consistent with the needs of their target population and that the rationale for the approach was based on the demonstrated effectiveness of similar activities. Finally, under their approach section, applicants also had to describe how they planned to address issues of domestic violence and ensure voluntary participation. For the “organizational profile” criterion, worth 20 points, organizations had to provide information that demonstrated their qualifications to serve participants, including organizational charts, financial statements, resumes, letters of support, and the qualifications of partnering organizations. As part of other criteria, applicants were asked to provide a budget and budget justification, and information on how they proposed to measure the outcomes of their programs. Applicants could receive up to 5 bonus points if they demonstrated prior experience in developing, implementing, or managing skills-based marriage or fatherhood education programs. See appendix II for a table of the criteria used for each type of grant.

The peer reviewers used these criteria to score applicants, and HHS ranked the applications based on the scores. With some exceptions, applications that received the highest scores were awarded grants. HHS made exceptions to ensure, among other things, that grants were geographically distributed and reflected a diversity of target populations and communities served.

In September 2006, HHS began notifying grantees of their awards, but experienced a setback when they had to reconvene review panels to rescore 31 applications. When scoring some applications, some reviewers incorrectly gave applicants zero points for the “approach” section. According to the grant announcements, if applicants failed to discuss how they would inform individuals that program participation was voluntary, as well as discuss specific issues relating to domestic violence issues, they would receive no points for the “approach” criterion. HHS discovered that reviewers had incorrectly interpreted whether applicants satisfied this portion of the “approach” criterion, and after clarifying the criteria, required that they rereview the applications.

HHS Awarded Grants to a Diverse Set of Grantees

HHS awarded grants to a diverse set of grantees that included 216 different organizations—122 were Healthy Marriage and 94 were Responsible Fatherhood demonstration grants that provided direct services to program participants in 47 states, the District of Columbia, and American Samoa (see app. III). In responding to our survey, grantees selected multiple categories to describe their organizational type. The majority—89 percent of the grantees—classified themselves as nonprofits. However, faith- based, for-profit, and private organizations also received funding. Awards for Healthy Marriage demonstration grants ranged from $225,000 to $2.4 million, and awards for Responsible Fatherhood demonstration grants were for smaller amounts, ranging from $188,000 to $1 million.

Over two-thirds of our survey respondents indicated that their organization had prior experience related to healthy marriage or responsible fatherhood activities. This experience included providing workshops for couples and singles, parenting classes, and relationship workshops for high school students. Some of these organizations also provided a broader array of other services to the community, such as mental health services and counseling services, and substance abuse treatment. Also, at least a dozen of the grantees had provided abstinence services and some Healthy Marriage grantees were previous recipients of grants from HHS for related purposes, including healthy marriage curriculum development and fostering healthy marriage within underserved communities.

HHS’s Grant Awards Process Contributed to Challenges Grantees Had Implementing Programs

HHS’s grant awards process contributed to challenges grantees later faced implementing their programs. HHS was able to announce grant awards by September 30; however HHS did not fully examine grantees’ programs as described in grantee applications. Specifically, we found during 5 of our 14 site visits that grantees, whose program activities had initially been approved by HHS, were later told that those same activities were not allowed under the conditions of their award. For example, during a site visit, one grantee reported that it proposed providing services to unmarried couples in its application and was doing so until HHS informed them that these services were not allowed under the conditions of their award. Another grantee told us that it was providing General Educational Development (GED) education as part of its Healthy Marriage program, but was later notified that the activity was not allowed. These grantees were well into program implementation when they were told to discontinue certain activities. One grantee we visited said it engaged in activities that were not allowable under the grant for a full year before being informed by HHS that the activities were not permissible. The grantee told us that it would have benefited from more timely review and feedback from HHS. In another case, HHS told a grantee that it would have to extend the length of its workshops for participants from 60 minutes to 90 minutes to 8 hours, even though the grant application noted that short, workshops would be provided. To implement this change, the grantee said it would likely incur additional expenses, such as paying facilitators for extra time and spending more for rental space.

HHS told us that it received more applications than expected and this was the first time it awarded these grants. HHS also said it had learned from this experience.

Programs Offer a Range of Similar Activities, but Their Focus and Target Populations Differ

While the range of activities offered and populations served by Healthy Marriage and Responsible Fatherhood programs’ grantees are similar, their focus and target populations differ. Both programs offer a range of similar activities, but a greater percentage of marriage programs provided activities related to marriage and relationship skills and a larger percentage of fatherhood programs provided parenting skills. Grantees for both programs reported that they refer domestic violence victims to specialists when appropriate. Additionally, while both programs target such groups as minority and low-income populations, Healthy Marriage grantees are more likely to target high school or teenaged youths, and Responsible Fatherhood grantees are more likely to target incarcerated parents.

Both Programs Offer a Range of Similar Activities and Refer Domestic Violence Victims to Specialists When Appropriate, but They Differ in Which Activities They Offer Most Frequently

Both programs offer a range of similar activities, and grantees from both programs said they refer victims of domestic violence to specialists in their communities when appropriate (see fig. 2). However, according to our survey, while both programs offer many similar activities, Healthy Marriage programs focus more on those related to marriage and relationship services, whereas Responsible Fatherhood programs are more likely to focus on providing services teaching parenting skills. Specifically, 94 percent of Healthy Marriage grantees, compared to 55 percent of Responsible Fatherhood grantees, reported offering marriage and relationship activities. During our visits to several Healthy Marriage grantees, we often observed activities related to marriage and relationships. For example, we observed a Healthy Marriage workshop where couples took quizzes to determine how well they knew one another and then participated in a discussion about commitment, chemistry, and compatibility. Conversely, 92 percent of Responsible Fatherhood grantees, compared to 47 percent of Healthy Marriage grantees, reported in our survey that they provide services related to teaching parenting skills. For example, a Responsible Fatherhood grantee program we visited included in its curriculum parenting skills training, such as lessons on a child’s developmental needs and how to communicate with children of different ages. In addition, Responsible Fatherhood grantees were more likely than Healthy Marriage grantees to report that they focused on providing programs with specific services to help participants achieve economic stability, including assistance with finding a job. Healthy Marriage grantees also reported that they focus on economic stability activities, but to a lesser extent than Responsible Fatherhood programs. According to HHS, Healthy Marriage grantees can provide these services only within the context of allowed activities (see table 1). For example, Healthy Marriage grantees might discuss financial issues as part of marriage and relationship skills. Depending on the conditions of the award, grantees might provide more than one of the services or activities listed in figure 2.

Both Healthy Marriage and Responsible Fatherhood grantee programs offer services for varying lengths of time and in various settings. Some programs have one intensive session in a lecture setting, while others offer classroom settings that are more interactive and may be offered for 1 or 2 hours 1 night a week for up to 17 weeks. One grantee program we visited offered marriage workshops to participants at weekend retreats with paid lodging, and two Responsible Fatherhood programs we visited included optional home visits by staff. In addition, some grantees run advertisements or sponsor advertising campaigns that discuss the importance of healthy marriage and responsible fatherhood. For example, one advertising campaign designed a billboard that read “a diamond isn’t the only thing that should last forever.”

According to our survey, the majority of grantees—98 percent—deliver their services through classroom instruction using a curriculum (see fig. 3). Many survey respondents said they developed and used their own curriculum (41 percent of Healthy Marriage and 47 percent of Responsible Fatherhood respondents). For example, one grantee we visited said it developed its own Spanish-language curriculum because the few existing Spanish-language curricula for Responsible Fatherhood programs did not meet the specific needs of the Latino population the grantee served. Other grantees adapt commercially available curricula to meet the needs of participants. The most-commonly-used, commercially available curriculum was the Prevention and Relationship Enhancement Program. This curriculum focuses on identifying strengths and weaknesses of a marriage, improving communication skills, and increasing the connection between the partners. Technical assistance providers make information about curricula available to grantees on their Web site. A list of curricula used by multiple grantees is in appendix IV.

Most grantees—about 93 percent—reported in our survey that they include information on domestic violence in their programs. For example, several grantees modified their curriculum to include a discussion of domestic violence with participants. One survey respondent noted that it leads a discussion on domestic violence issues that helps participants self- identify and understand domestic violence. During our site visits, some Healthy Marriage grantees told us that they focus on the characteristics of a healthy relationship. In addition to discussing topics related to relationship health and domestic violence awareness, grantees also distribute informational materials about domestic violence (see fig. 4). For example, during a site visit to a Healthy Marriage grantee, we observed classroom instructors distributing pamphlets on recognizing signs of domestic violence. Handouts include state Directories of Domestic Violence Support Services; handbooks for domestic violence victims, and victims’ rights; and pamphlets on topics ranging from “recipes for safety” to the characteristics of an abusive relationship.

Additionally, most grantees reported in our survey that they have protocols for how staff should handle instances where program participants may be victims of domestic violence, and many grantees train their staff on identifying signs of domestic violence, as well as on teaching program participants the signs of unhealthy relationships. Moreover, most grantees reported that they consult with domestic violence organizations and refer potential domestic violence victims to them. For example, one grantee we visited told us that it consulted with two different domestic violence organizations when designing its Responsible Fatherhood program. The domestic violence organizations helped the grantee develop part of a workshop related to domestic violence and also presented information to program participants. During our site visits, grantees also told us they refer program participants to domestic violence specialists when appropriate. For example, one of the grantees we visited said that when it encountered a potential domestic violence situation, it held a joint meeting with a caseworker, domestic violence expert, and a family services coordinator. Collectively they determined the appropriate referral for the person. The DRA does not include domestic violence services as an allowed activity, but does require that programs have in place mechanisms for addressing domestic violence.

Programs Focus Services on Different Target Populations

Healthy Marriage and Responsible Fatherhood grantee programs focus on providing services to different populations, but they both target low- income and minority populations. According to our survey, 58 percent of Healthy Marriage and 52 percent of Responsible Fatherhood grantees target low-income individuals, and 39 percent of Healthy Marriage and 36 percent of Responsible Fatherhood grantees target minorities (see fig. 5). Healthy Marriage grantee programs target high school or teenaged youths at higher rates than Responsible Fatherhood grantee programs, in part, because education in high schools is one of the Healthy Marriage program’s allowed activities. On the other hand, Responsible Fatherhood programs target incarcerated parents, typically fathers, because HHS designated a portion of the program’s funding for this population. Both grantee programs allow men and women to participate in their programs—even though the Responsible Fatherhood programs were created specifically to target men, they are both open to men and women. An administrative complaint was filed by a legal advocacy organization centering on whether women have equal access to the program and subsequently HHS reminded grantees that the Responsible Fatherhood programs are open to eligible men and women.

Grantees use a variety of methods to attract participants to the program. According to our survey, grantees rely heavily on word of mouth, but they also attract participants through educational handouts and brochures, referrals, and advertisements such as promotion campaigns (see fig. 6). For example, one grantee we visited, which targets Latinos, indicated that while it advertises through a variety of methods including community- based advertising, radio, and door-to-door recruiting, it had difficulty attracting participants. Some grantees told us they devised numerous incentives to better retain participants. For example, one grantee we visited told us it provides food and child care at each session, transportation subsidies, and Wal-Mart and Babies R Us gift cards once participants completed the program.

Participation in the Healthy Marriage and Responsible Fatherhood programs must be voluntary as required by DRA, and according to our survey, grantees used a variety of methods to inform participants that participation was voluntary. Specifically, 95 percent of survey respondents indicated they provide verbal notification that participation is voluntary, while 89 percent indicated that they provide written notification (see fig. 7).

HHS Has a Program Monitoring System, but Lacks Mechanisms to Identify and Target Grantees Not in Compliance with Grant Requirements or Not Meeting Performance Goals

HHS has a program monitoring system, but it lacks the mechanisms to identify and target grantees not in compliance with grant requirements or not meeting performance goals. HHS uses multiple tools to monitor grantee programs, such as site visits and reviews of reports submitted by grantees. However, HHS lacks specific guidance for conducting monitoring site visits. Moreover, HHS’s ability to target grantees in need of assistance is hindered by the lack of an effective Management Information System.

HHS Uses Multiple Tools to Monitor Grantees

To monitor Healthy Marriage and Responsible Fatherhood grantee performance, HHS uses multiple tools including a combination of phone calls, e-mails, grantee progress reports, and site visits. HHS also reviews grantee Single Audit Act reports. HHS is responsible for monitoring the 216 Healthy Marriage and Responsible Fatherhood grantees and according to our survey; almost all grantees reported some contact from HHS staff. According to the grantees we visited, HHS staff contact them at least once a month. Grantees said that HHS staff typically contact them to notify them of opportunities for technical assistance, address errors or issues that arise during review of required programmatic and financial progress reports, and to notify them of upcoming events. In addition, some grantees also initiate communication with HHS to ask questions regarding policy, to request approval for certain activities, or to request budget modifications.

Semiannually, HHS requires grantees to submit both programmatic and financial progress reports, which, among other things, provide HHS with updates on grantees’ progress toward meeting performance goals that grantees established for themselves in their applications, as well as provide information on grantees’ compliance with domestic violence and other HHS policies. For example, some grantees report to HHS on the number of participants they expect to serve. Some grantees also may report on the types of activities and participant satisfaction with programs or services as well as changes in participant behavior before and after programs. They also may report on any problems they may be experiencing, including recruiting challenges. Because grantees can set their own program goals and establish their own measures for these goals, there is considerable variation among the information being collected. Financial progress reports contain information, such as financial statements, that allow HHS to track the use of grant funds. HHS also monitors grantees’ use of funds by tracking grantees’ draw down of funds. Specifically, HHS also is able to compare financial progress reports submitted by grantees with reports from the HHS electronic grant payment management system to monitor grantees’ withdrawal of funds. For example, if HHS observes that a grantee has not withdrawn funds according to its schedule, they will contact the grantee to determine the reason the grantee has not been withdrawing funds. For grantees that received federal funds in excess of $500,000, HHS monitors and reviews audit reports in accordance with the Single Audit Act. According to HHS, its review of grantee Single Audit Act reports covers compliance with audit standards, completeness, timeliness, and other audit considerations.

As part of HHS’s on-site monitoring, at least one HHS staff member will interview grantee staff, review program documents, and in some instances observe programs in operation. For example, when we accompanied HHS during two grantee site visits in March of this year, HHS and one of the grantees discussed challenges the grantee was experiencing with recruiting participants. HHS discovered that the grantee, whose target population included a rural district, was struggling to meet its goal for the number of participants it initially believed it would serve. The HHS official referred the grantee for technical assistance in order to help it improve participant recruitment and retention. HHS officials told us that monitoring site visits was a priority for them and their goal was to visit all grantees within the first 3 years of the award period. As of August 2008, HHS told us that approximately 84 percent of grantees had received a site visit from HHS since September of 2006, when the programs were first funded. Our survey results confirmed that HHS had visited most of the grantees in the first 2 years.

HHS Lacks Guidance for Conducting Site Visits and Other Monitoring Activities

HHS staff lack specific guidance for conducting site visits and other monitoring activities, according to our interviews with HHS staff, visits and interviews with grantees, and file reviews. As a result, the length and types of issues reviewed and documentation examined by HHS during site visits varied depending on who conducted the visit. HHS officials told us that staff responsible for monitoring are to use the legislation, grant announcements, and site visit protocol as guidance to monitor grantee performance. Although legislation and grant announcements provide some general guidance, they do not specifically define what is permitted under each allowed activity. For example, the grant announcement lists marriage education as an allowed activity for some grantees, but does not specifically describe what marriage education activities are permitted under the grant. We also found the site visit protocol provided by HHS was limited to a checklist of topics for HHS to cover during grantee site visits. The checklist did not detail the process, the criteria for conducting monitoring site visits, or the key items to be examined, leaving each monitoring staff member the discretion to determine what information to gather and how best to gather it. Moreover, we found other inconsistencies in how HHS conducts monitoring visits. For example, during some monitoring site visits, HHS staff observed grantees providing services while in other instances they did not. According to HHS officials, HHS staff are required only to observe services if the timing of the visit coincides with services, but they are not required to schedule monitoring site visits to coincide with sessions. Because some HHS officials do not observe grantees providing services, they cannot confirm that the services are in fact being provided or that the funding is being spent as intended.

The lack of sufficient guidance from HHS may have led HHS staff to inconsistently apply HHS policy among some grantees. For example, through our interviews and file review, we found that some monitoring staff members allowed several Healthy Marriage and Responsible Fatherhood grantees to use incentives to retain program participants, while others were told they were not permitted to use similar incentives. From our review of grantee files, we found instances where HHS staff worked with grantees to adjust or lower the goals they developed for themselves to meet second-year targets. Other grantees who did not meet their year-1 performance goals were not permitted to adjust their performance targets. In another example, HHS officials told us that abstinence education was not allowable under the Healthy Marriage program, but we observed during our site visits and review of grantee data several Healthy Marriage grantees operating programs that focused on abstinence education.

HHS’s Ability to Target Grantees Not in Compliance with Grant Requirements or Not Meeting Performance Goals Is Hindered by the Lack of an Effective Management Information System

The lack of an effective management information system that captures key information on individual grantees hinders HHS’s ability to appropriately identify which grantees are not in compliance with grant requirements or are not meeting performance goals. Although it maintains paper files on each grantee, the breadth and detail in these files vary considerably. For example, some HHS staff keep very detailed logs on grantees, while others maintain minimal records. Moreover, the information in these files is not always used to target grantees in need of assistance or to identify how grantees are using their funds. For example, one grantee used grant funds to provide marriage education services not allowed under its grant to participants. Although information such as how grantees are using their funds should be contained in the files, the grantee in this instance was notified months after initiating services that the program was not allowed, causing the grantee to use alternative sources of funding to provide services. Moreover, through our case studies, we found instances where grantees did not receive timely feedback on progress reports, documents that are part of the files HHS maintains on individual grantees. These files provide an early alert to problems grantees may be experiencing and could potentially identify grantees at risk of not meeting performance goals. Despite HHS having this information, some grantees told us that they did not receive timely feedback from HHS, causing them setbacks in implementing program activities.

Without an effective management information system, HHS has not been able to take a strategic approach to conducting grantee site visits and other monitoring activities. Although HHS told us that grantees experiencing challenges should receive priority for site visits, our review of a random sample of grantee files showed that several grantees were having difficulty recruiting participants, yet HHS did not always give them priority for on-site review. Moreover, during our site visits, some grantees told us they were experiencing difficulty meeting participation goals or recruiting the number of participants they indicated to HHS they would serve through their program. These grantees also were not targeted specifically for on-site monitoring. Specifically, the decision of which grantees to visit and in what order was left to the discretion of HHS staff, according to HHS officials. Because grantees that were experiencing challenges did not always receive priority for monitoring site visits and these site visits were scheduled based on HHS staff scheduling preferences, we found that monitoring was not always based on grantee risk or need.

HHS told us it is in the process of developing a database that will help it standardize and combine grantee communications and performance information. According to HHS, the first phase of the web-based management information system has been completed. The system is designed to replace the paper files and, according to HHS, will considerably reduce or eliminate inconsistencies in HHS’s recordkeeping. The management information system will capture performance indicators developed by the grantee and submitted semiannually in grantee programmatic progress reports, such as grantees’ progress toward meeting participant recruitment goals and changes in participant behavior. The new system should allow HHS to better manage and search for grantee information, upload grantee communications, and track data from grantee programmatic progress reports. It is not clear, however, when HHS will be able to include uniform performance indicators that it plans to collect from individual grantees. HHS officials told us that performance indicators have been developed, but are pending implementation while they are currently under review by the Office of Management and Budget. HHS said it anticipates having grantees begin collecting data in autumn or early winter of 2008, the start of the third year of funding for the 5-year initiative. According to HHS, the uniform performance indicators will eventually be part of its planned management information system.

HHS Has Long-term Research Underway Intended to Assess Program Impact

HHS has four multiyear studies of marriage and fatherhood programs underway that are intended to assess the impact of the programs on various populations and understudied groups, the final results of which are expected between 2011 and 2013. Funded partially by the DRA, HHS awarded contracts to three organizations—RTI International; Mathematica Policy Research; and MDRC—that competitively bid to conduct the evaluations, which run over several years and across several marriage or fatherhood programs. Two of the impact studies will exclusively follow grantees funded under the Healthy Marriage and Responsible Fatherhood Initiative, while the other two studies will follow a mix of grantees and healthy marriage programs not funded under the Initiative. In all cases, the programs being studied primarily offer participants skills-based marriage or fatherhood education. The primary focus of HHS’s research is to determine the impact, if any, marriage and fatherhood programs have on couples, families, and fathers as a result of participation in the programs. Impact evaluations are the strongest method for assessing the efficacy of a program because they allow for a comparison between similar groups that differ only with respect to whether they received a service or “treatment.” However, they often are difficult and expensive to conduct because they take years to complete and it often is difficult to retain enough participants to produce meaningful results. Prior research has focused on the impact of marriage services on middle-income families and couples. A review of the literature, sponsored by HHS, on the overall impact of marriage and relationship programs found that, on average, middle-income couples receiving services showed increased relationship satisfaction and improved communication skills. HHS’s research agenda represents the first major federal effort to study the impact of healthy marriage and responsible fatherhood programs on low-income populations and is part of a wider body of research being developed by HHS.

Two of the three healthy marriage studies—the Building Strong Families (BSF) and the Supporting Healthy Marriage (SHM) evaluations—focus on low-income couples who are expecting or have recently had a child. The BSF is following 5,103 low-income unmarried couples across seven marriage programs around the time of the birth of a child using data collected at three stages of participants’ lives. The SHM study is examining the effects of healthy marriage programs on 6,860 married couples across eight marriage programs. The third healthy marriage study—the Community Healthy Marriage Initiative—expands its focus beyond specific target populations to entire communities: the initiative is comparing couples in three different geographic communities with federally funded healthy marriage programs— Milwaukee, Wisconsin; Dallas, Texas; and St. Louis, Missouri—with three demographically similar communities—Cleveland, Ohio; Ft. Worth, Texas; and Kansas City, Missouri—where there are no federally funded healthy marriage programs. The study, which involves 4,200 participants, will explore whether the presence of intensive healthy marriage programs promotes changes in attitudes and behavior toward marriage in the communities being studied. In addition to the three healthy marriage evaluations, HHS also is funding an impact evaluation of Responsible Fatherhood programs. The National Evaluation of the Responsible Fatherhood, Marriage and Family Strengthening Grants for Incarcerated and Re-entering Fathers and Their Partners (MFS-IP) began in 2006, when the first year of Responsible Fatherhood funds became available, and is currently enrolling participants. The MFS-IP, much like the three marriage studies, will explore changes in couple quality and changes in attitudes toward marriage. In addition, the MFS-IP will assess changes in outcomes for employment and economic stability, in line with the parameters of activities allowed under the legislation for Responsible Fatherhood grantees (see fig. 8).

For all four studies, evaluators will collect outcome data for the couples participating in programs at various stages of the study and then compare the results against groups of couples who did not participate in the programs. Because the two groups are, by nature of the study design, similar in every major respect, any differences between the two groups can be attributed to the program. The evaluators for the four studies differed on the methods they used to create these two groups. Two of the four studies, the BSF and the SHM, randomly assign couples to either a group that receives services (the experimental group) or group that does not (the control group). The other two studies are quasi-experimental. This type of study uses methods other than random assignment to create a comparison group, such as selecting a set of individuals who have similar characteristics to the group receiving the program services under study.

To compare these groups in the four studies over time, the evaluators are conducting surveys and interviews, generally 1 year and 3 years after participating in a program, in order to gauge couples’ and families’ outcomes. The surveys ask questions about how couples are communicating after participating in a program; whether they are using the skills they learned in the program; and how they would rate, overall, the quality of their relationship since participating in the program. The evaluators also will administer the same surveys to the couples not participating in Healthy Marriage or Responsible Fatherhood programs in order to make comparisons between the two groups. For example, the BSF study will examine a range of outcomes, including whether marriage services improved marital relationships, reduced marital instability, and improved child well-being.

In general, we found the evaluations to be well-designed and rigorous, however, there are inherent difficulties presented by the Community Healthy Marriage Initiative, which assesses the impact of healthy marriage programs on entire geographic areas. Specifically, it may be difficult to find and study true comparison communities. One positive feature of the study is the collection of baseline data for each of the participating communities; however, it is difficult to determine if the contractors have captured and controlled for the important variables needed to match the communities. In addition, it will be difficult to determine if changes in the community stem from Healthy Marriage program services or some other factors.

Conclusions

Marriage and fatherhood programs have emerged as a national strategy for improving the well-being of children. The federal government has committed $150 million annually for 5 years for these programs and provided for an evaluation the Healthy Marriage and Responsible Fatherhood Initiative to determine how well the Initiative is working for low-income populations. While HHS has made an effort to visit nearly all of the programs in their first 2 years of operations, absent mechanisms for detecting grantee compliance and performance issues, some grantees did not receive monitoring and technical assistance soon enough and had to make modifications to their program well into implementation. Moreover, effective monitoring was hampered by a lack of an effective management information system that captures key information, including uniform performance indicators for grantees, and the lack of consistent and clear monitoring guidance. Without an effective monitoring system or clear and consistent monitoring guidance, grantees may continue to be at risk of noncompliance with HHS policy or of not meeting performance requirements.

Recommendations for Executive Action

In order to improve monitoring and oversight of Healthy Marriage and Responsible Fatherhood grantees, we are recommending that the Secretary of HHS: employ a risk-based approach to monitoring grantees and conducting grantee site visits, using its planned management information system and information from both progress reports and uniform performance indicators to help identify those grantees at risk of not meeting performance goals or not in compliance with grant requirements; and create clear, consistent guidance and policy for monitoring Healthy Marriage and Responsible Fatherhood grantees.

Agency Comments and Our Evaluation

We provided a draft of this report to HHS for its comments; these appear in appendix V. In its comments, HHS concurred with our recommendation that it employ a risk-based approach to monitoring using its planned management information system and performance indicators to help identify grantees for monitoring, saying these tools would further enhance oversight and monitoring efforts currently underway. In its comments, HHS states that it has already developed and implemented this portion of the recommendation, including developing a customized approach to prioritizing site visits and technical assistance. However, HHS caveats that only the first phase of its web-based management information system has been completed and that performance indicators that would help them identify those grantees at risk, are still awaiting approval by OMB. A fully implemented management information system with performance indicators in place will further enhance HHS’s ability to monitor grantees based on risk.

HHS disagreed with the portion of our recommendation that HHS lacks specific guidance for conducting monitoring site visits. In its comments, HHS stated that it developed a clear, comprehensive, and thorough protocol and trained project officers on the critical and essential items that must be covered during grantee site visits. As we stated in our report, this protocol was limited to a checklist of topics to be covered during the site visit and did not describe the process to be followed or criteria to be used to monitor grantees. Moreover, the lack of clarity in this protocol may have contributed to the inconsistencies in how site visits were administered by HHS staff, as noted in our report.

HHS also stated in its response that fiscal oversight or monitoring a grantee’s fiscal compliance can be used as an alternative mechanism to confirm whether grantees are providing services or spending funds as the grant intended. While we agree that monitoring grantee’s fiscal compliance is essential, HHS’s comments do not change our view that observing activities is critical to confirming that grantees are actually providing services as intended by the grant.

Finally, HHS commented on our finding that some grantees were operating programs focused on abstinence education. HHS stated that it is impermissible to use Deficit Reduction Act (DRA) funding for abstinence education, however, grantees may use funding from other sources to provide abstinence education through programs separate from the Healthy Marriage and Fatherhood programs. We visited one such program whose staff told us that they used DRA funding to support their abstinence education program and that abstinence education was not provided as a single lesson, but was the focus of the entire curriculum.

HHS also provided technical changes to a draft of the report, which we incorporated into the report as appropriate.

As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies of this report to the Honorable Michael O. Leavitt, Secretary of Health and Human Services, relevant congressional committees, and other interested parties. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov. Please contact me on (202) 512-7215 if you or your staff have any questions about this report. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI.

Appendix I: Objective, Scope, and Methodology

To gain insight into how Healthy Marriage and Responsible Fatherhood programs are being implemented, we were asked to report on (1) how the Department of Health and Human Services (HHS) awarded grants and the types of organizations that received funding; (2) the activities and services grantees are providing, including those for domestic violence victims; (3) the manner in which HHS monitors and assesses program implementation and use of funds; and (4) how program impact is measured.

To address the objectives, we conducted a Web-based survey of 122 Healthy Marriage and 94 Responsible Fatherhood grantees asking them to provide information about various aspects of their programs. We received a response rate of 98 percent. We also visited 14 grantees in Washington, Oklahoma, New Mexico, Indiana, Oregon, and the District of Columbia. In addition, we conducted telephone interviews with organizations that provide technical assistance to grantees and help other organizations develop fatherhood programs. To further understand the criteria HHS used to award grants and the manner in which HHS monitors and assesses program implementation, we reviewed 50 grantee case files, 40 randomly and 10 deliberately selected, examining documents such as applications, semiannual progress and financial reports, grantee selection panel score sheets, and correspondences between the grantees and agency officials. To determine how program impact is measured, we interviewed organizations that received contracts to conduct impact evaluations of Healthy Marriage and Responsible Fatherhood interventions and assessed their methodological approach to measuring impact.

Survey of Marriage and Fatherhood Programs

To address all of our objectives, we conducted a Web-based survey of all 216 demonstration grantees that provided direct services to participants, 122 Healthy Marriage and 94 Responsible Fatherhood grantees. We asked grantees about various aspects of their programs, including the characteristics of their organization, services they offered, experience providing similar services, curricula used, their process and procedures for identifying domestic violence, staff training; and any evaluations the grantees were conducting on their own. In order to identify respondents for our survey, we obtained lists of grantees and contact information from HHS’s Administration for Children and Families and their Office of Grants Management. We compared the two lists to compile the most accurate list of grant recipients and contact information. In some cases, we contacted the organization directly to determine the appropriate contact person and obtain updated information. Of the 216 grantees contacted, 211 provided information, for a response rate of 98 percent. The survey data was collected from February 2008 to April 2008.

Because this was not a sample survey, it has no sampling errors. However, the practical difficulties of conducting any survey may introduce errors, commonly referred to as nonsampling errors. For example, difficulties in interpreting a particular question, sources of information available to respondents, or entering data into a database or analyzing them can introduce unwanted variability into the survey results. We took steps in developing the questionnaire, collecting the data, and analyzing them to minimize such nonsampling error. For example, prior to launching our survey, we worked with social science survey specialists to develop the questionnaire and minimize error. We tested the content and format of the questionnaire with multiple grantees prior to administering the survey to address issues such as differences in question interpretation, and differences in data tracking. We conducted 10 survey pretests. As a result of our pretests, we changed survey questions as appropriate and tested those changes with grantees that participated in our original pretests. Further, the final pretests were performed using the Web-based survey tool, which checked for accuracy and usability. To ensure grantees responded to the survey, we sent e-mail reminders and conducted follow- up telephone calls with nonrespondents. Since this was a Web-based survey, respondents entered their answers directly into the electronic questionnaire, eliminating the need to key data into a database, minimizing error. We used content coding, computer edits, and independent analysts to assess the reliability of the information collected.

Site Visits to 14 Grantees

To gather information to respond to all of these questions, we visited 14 grantees—9 Healthy Marriage grantees and 5 Responsible Fatherhood grantees—in Washington, Oklahoma, New Mexico, Indiana, Oregon, and the District of Columbia. We selected grantees to achieve variation in geographic location, type of grant awarded, award amount, services, organization type, program curriculum, and the programs’ target populations. During each site visit we asked the grantees about the grant application process and their programs, including accessibility of funds, services provided, guidance and communication with HHS, and challenges the grantees experienced. During seven of these site visits, we observed the implementation of marriage and fatherhood services. Further, we also observed HHS staff in the process of conducting two grantee site visits. In analyzing our site visit interviews we arrayed and analyzed narrative responses thematically. The site visits were conducted from December 2007 through April 2008.

File Review

Further, to learn about the criteria used to award grants and HHS’s monitoring activities, we conducted a review of 50 grantee case files out of the total 229 grants awarded in September 2006. We conducted a simple random sample of 40 Healthy Marriage and Responsible Fatherhood grantee case files—28 Healthy Marriage grantees and 12 Responsible Fatherhood grantees. We also deliberately selected and reviewed an additional 10 grantee case files; the team deliberately reviewed case files for 1 technical assistance grantee, 6 grantees that assist other organizations with developing fatherhood programs, and 3 grantees we visited. During the case file review, we examined documents contained in the grantee’s case file including, the grantee’s original and continuation application, semiannual progress and financial reports, grantee selection panel score summary sheets, correspondences between the grantee and agency officials, and site visit reports. We reviewed the documents to assess HHS’s compliance with its grants policy manual and to understand how HHS monitors use of funds. We also reviewed Single Audit Reports for the selected sample of grantees. To facilitate the case file review, we developed a data collection instrument to record specific information for each case file reviewed. We used content coding to analyze the qualitative information from our data collection instrument. We conducted our review on-site at HHS’s Administration for Children and Families.

Review of Internal HHS Documents and Interviews with HHS Officials

We also reviewed the HHS grant selection criteria included in the grant announcements and HHS’s internal guidance on grant selection processes which we compared to the selection of Healthy Marriage and Responsible Fatherhood grant recipients. In addition to these reviews, we interviewed HHS and the contractor responsible for hiring reviewers and organizing the review panels.

To determine how HHS measures program impact, we collected survey instruments, design papers, and program guidelines for each of the four impact evaluations underway in order to assess their methodological soundness. In addition, we interviewed HHS staff responsible for overseeing the contractors responsible for the impact evaluations. To gauge how HHS is monitoring the progress of grantees, we interviewed HHS staff regarding its process for monitoring grantees, including guidance used and staff training provided to determine how HHS monitors and assesses program implementation and use of funds.

Interviews with Experts

To identify critical components that should be included in services provided by grantees, we interviewed multiple experts in the areas of marriage, fatherhood, and domestic violence. We also interviewed grantees and contractors that were not direct providers of healthy marriage and responsible fatherhood services but received funding under the Healthy Marriage and Responsible Fatherhood Initiative to provide technical assistance to demonstration grantees, conduct research, and help other organizations develop fatherhood programs.

We conducted this performance audit from July 2007 to September 2008, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Grantee Selection Criteria

Healthy Marriage Resource Center (1 grant awarded) awarded) grants awarded) grants awarded)

Appendix III: States and Territories with Grantees That Provide Direct Services to Participants as of February 2008

Appendix IV: Curricula Being Used by Healthy Marriage and Responsible Fatherhood Grantees and Frequency of Use

Prevention and Relationship Enhancement Program (PREP)

Appendix V: Comments from the Department of Health and Human Services

Appendix VI: GAO Contact and Staff Acknowledgments

Acknowledgments

Sherri Doughty (Assistant Director) and Ramona L. Burton (Analyst-in- Charge) managed all aspects of the assignment. Michelle Bracy, Melissa Jaynes, and Chhandasi Pandya made significant contributions to this report, in all aspects of the work. In addition, Cathy Hurley, Kevin Jackson, Stuart Kaufman, and Luann Moy provided technical support in design and methodology, survey research, and statistical analysis; Daniel Schwimer provided legal support; and Jessica Orr assisted in the message and report development.