Small Business Administration: Opportunities Exist to Improve Oversight of Women's Business Centers and Coordination among SBA's Business Assistance Programs
Released: 2007-11-16T00:00:00

Letter

Background

The WBC program is administered through OWBO. The program was established by the Women’s Business Ownership Act of 1988 to provide long-term training, counseling, networking, and mentoring to women who own businesses or are potential entrepreneurs after Congress found that existing business assistance programs for small business owners were not addressing women’s needs. The program’s goal is to add more well-trained women entrepreneurs to the U.S. business community and to specifically target services to women who are socially and economically disadvantaged. In fiscal year 2007, SBA funded 99 WBCs throughout the United States and its territories (fig. 1).

Private nonprofit organizations are eligible to apply for funds to set up WBCs, and successful applicants are initially awarded cooperative agreements for a maximum of 5 years. WBCs must raise matching funds from nonfederal sources such as state and local public funds, private individuals, corporations and foundations, and program income derived from WBC services. In the first 2 years of the 5-year award, each WBC is required to match SBA award funding at 1 nonfederal dollar for each 2 federal dollars. In the last 3 years, the match is 1 nonfederal dollar for each federal dollar. Award amounts may vary depending upon a WBC’s location, staff size, project objectives, performance, and agency priorities. However, awards cannot exceed $150,000 each fiscal year per recipient.

WBC funding is performance-based, and each additional 12-month budget period beyond the initial award may be exercised at SBA’s discretion. Among the factors involved in deciding whether to exercise an option for continued funding are the availability of funds, the extent to which past WBC funds have been spent, and satisfactory performance against SBA- established performance measures, including the number of clients served and jobs created. SBA requires WBCs to provide this performance data in quarterly reports.

Under the sustainability pilot program, WBCs that had been receiving funding for 5 years could receive sustainability awards for an additional 5 years. Criteria for receiving awards under the pilot program were similar to those for receiving the initial awards. WBCs were assessed on their record of performance and had to provide nonfederal matching funds equal to 1 dollar for each federal dollar. Unlike the WBC regular award, WBC sustainability award amounts could not exceed $125,000 each fiscal year per recipient. As noted earlier, Congress recently replaced these sustainability awards with 3-year renewable awards. SBA has not yet begun making these new awards, which are a maximum of $150,000 each year per recipient.

In addition to the WBC program, SBA’s SBDC and SCORE programs also provide training and counseling services to small business clients. The SBDC program was created by Congress in 1980. SBDC services include, but are not limited to, assisting prospective and existing small businesses with financial, marketing, production, organization, engineering, and technical problems and feasibility studies. Each state and U.S. territory has a lead organization that sponsors and manages the SBDC program. The lead organization coordinates program services offered to small businesses through a network of centers and satellite locations in each state that are located at colleges, universities, vocational schools, chambers of commerce, and economic development corporations. In fiscal year 2007, the SBDC program received $87 million to make awards to 63 lead SBDCs throughout the United States.

The SCORE program was founded in 1964 as a nonprofit organization. Under the Small Business Act, as amended, SCORE is sponsored by and may receive appropriations through SBA. The SCORE program is designed to provide free expert advice to prospective and existing small businesses in all aspects of business formation, advancement, and problem solving. SCORE counselors are volunteers who assist clients through a Web site, SCORE chapter offices, SBA district offices, and other establishments. In fiscal year 2007, the SCORE program received $5 million to support its activities and currently has 389 chapters throughout the United States.

SBA’s Office of Small Business Development Centers and Office of Business and Community Initiatives are components of OED, along with OWBO, and oversee the SBDC and SCORE programs, respectively. SBA’s Division of Procurement and Grants Management (DPGM) monitors financial activities and transactions and maintains award files for most of SBA’s award programs. The Office of SBDCs has its own grants specialists that conduct similar activities. With respect to the WBC program, DPGM is involved in, among other aspects, reviewing and making decisions on new WBC applications, providing final approval for all contracts, analyzing proposed budgets and negotiating budgets with OWBO, issuing modifications to terms and conditions of awards, reviewing matching funds documentation, and reviewing WBC financial reports and payment requests to authorize payment.

Recent Legislation Addressed Concerns about the WBC Program’s Funding

Before Congress passed recent legislation addressing concerns about long- term funding for WBCs, the WBC program’s funding structure had been in flux since its inception in 1988. In establishing the WBC program in 1988, Congress authorized SBA to help private nonprofit organizations conduct projects that benefit small business concerns owned and controlled by women. The 1988 act allowed for SBA to fund demonstration projects that terminated in 1991. However, in 1991, Congress authorized SBA to make awards for 3-year projects, and in 1997, Congress authorized SBA to make awards to WBCs for 5-year projects. In its 1999 reauthorization of the WBC program, as noted earlier, Congress added 5-year sustainability funding for WBCs that successfully completed 5-year projects to provide additional time for the centers to become self-sustaining (fig. 2). WBCs continue to receive SBA funds for the 5-year period as long as their performance is satisfactory although under the performance-based system, the award amount can vary from year to year.

WBCs that we spoke with identified two related factors that had largely been responsible for their funding uncertainties. First, because until recently the WBC program offered limited-term funding—in contrast to the SBDC and SCORE programs, which receive continuous funding— WBCs graduated from SBA support after 5 or 10 years. Second, Congress did not make the additional 5-year term for sustainability funding permanent. Instead, Congress extended the pilot program with each SBA reauthorization, raising concerns among the WBCs about its commitment to the program. Several WBCs that we spoke with expressed concern about the funding term limits and pointed out that the SBDC and SCORE programs did not have the same limits, even though SBA also administers those programs. Some WBCs in both the regular and sustainability programs also said that they were concerned about their ability to continue operations after losing SBA support. Because WBCs sometimes established their operations with SBA funds and depended on SBA funds to leverage other support, many were concerned about their ability to continue operations after 5 to 10 years of receiving SBA funding. One center that was receiving sustainability funds said that long-term funding would allow WBCs to continue operating without concerns of an end date after taking years to develop a valuable program. The center director added that a short-term program was less practical for the service that the WBC program provides, because it takes time to have client successes. Another center that graduated from SBA’s program in 2007 told us that although SBA funding had decreased each fiscal year, the WBC’s membership in SBA’s program and the funds it received were beneficial to the center’s ongoing success. One center president said that seamless funding for the program would greatly benefit centers that were meeting the needs of their communities, and the director of another center that was in the process of applying for sustainability funding told us that she was anxious to see the recent legislative changes that would make SBA funding for WBCs permanent. A district office official that we spoke with echoed the WBCs’ concern about sustainability, noting that the long-term viability of the WBC he oversaw might be threatened after the center graduated from SBA’s program in 2007.

The WBC program’s funding structure also created uncertainty that limited SBA’s ability to manage the program effectively. OMB’s 2007 PART report found that frequent changes by Congress in the WBC program’s funding structure, delays in extending sustainability funding, and uncertainty about the future had created challenges for the program. OMB’s report also noted that SBA had taken steps to foster more consistent management of the WBC program, but added that long-term planning was problematic because of the program’s funding structure. When we spoke with officials at OMB, they emphasized that SBA appeared to be making a significant effort to assist WBCs, given the program’s limitations. The officials also noted that the funding challenges that WBCs faced after graduating from the sustainability pilot could be related to the fact that these organizations operated resource-intensive programs and collected nominal revenues in program fees, largely because of their focus on economically disadvantaged clients, causing them to rely heavily on external support.

SBA will fund WBCs through the project term, subject to availability of funds, and our review indicates that WBCs that perform satisfactorily will continue to receive funds until they complete the program. SBA officials provided us with a list of eight centers that had terminated prior to completing the program and noted that the program had funded a total 150 WBCs since its inception. However, SBA officials in headquarters and the district offices were aware of the challenges WBCs faced in planning annual budgets without knowing how much they would receive or whether sustainability funds would continue to be available. According to SBA, two of the eight centers that left the program did so as a result of challenges securing matching funds, and one WBC not included in SBA’s list left the program during our review, in part due to funding challenges. In discussing the WBC program’s limited-term funding, some SBA district office officials emphasized that the agency had invested in creating successful WBCs and should be working to make those that performed well permanent SBA partners.

As we have seen, recent legislation for the WBC program replaces the sustainability pilot program with 3-year renewable awards, providing an opportunity for SBA to continue funding WBCs. Current program participants and those that have successfully graduated will be eligible to apply for continuous funding through these 3-year awards (table 1). SBA officials told us that by the end of fiscal year 2007, 21 WBCs that have graduated since the beginning of the program would be eligible to apply for the renewable awards. The award process will remain competitive, and the maximum amount for renewable awards will be $150,000 each year per recipient, as in the first 5 years of the WBC program. Also, the number of organizations competing could increase, but SBA’s annual budget for the WBC program may not increase beyond the approximate $12 million provided in the past 5 years. However, increased award competition provides an opportunity for SBA to continue funding high-performing centers. Prior to the new program changes, SBA officials emphasized that the WBC program is the agency’s only performance-based program and said that they believed this fact provided an incentive for WBCs to continuously improve. Because the WBC program is a competitive discretionary award program, WBCs in the program compete annually for the maximum award amount but continue to receive SBA funds for the length of the project as long as their performance is satisfactory. SBA has criteria for ranking new award applicants and performance-based criteria for placing existing program participants into three funding categories for annual awards. In September 2007, SBA made WBC awards for fiscal year 2007 to fund activities in fiscal year 2008, and SBA officials told us that they plan to begin providing the 3-year renewable awards in fiscal year 2008 as soon as practicable after appropriations are received.

As a result of the new legislation, which allows graduated WBCs to reenter the pool of applicants for continuous funding and changes the existing 5- year sustainability project terms going forward, SBA has begun revising its existing WBC award process. SBA officials said that they would have to create a new program announcement and update other documents to reflect the new program structure, and that they also anticipated revising the qualifying criteria and adding new considerations because they expected the competition for awards to increase with the availability of continuous funding.

SBA Has Oversight Procedures in Place, but Imbalances in Its Staff Resources and Ineffective Communication with WBCs Have Hindered Their Effectiveness

SBA has developed oversight procedures for the WBC program, but imbalances in the agency’s staff resources for WBC oversight and ineffective communication with WBCs reduce the effectiveness of these procedures. SBA’s oversight of WBCs includes ongoing assessments for performance-based funding, as required by the act authorizing the program; and SBA has requirements for WBCs to report quarterly on their program activities, performance, and finances. Although SBA had these oversight procedures in place, its staff resources for the WBC program have been limited, with the agency relying heavily on district office staff who may have too many responsibilities or lack relevant experience and training. Also, ineffective communication with WBCs has led to confusion about how to meet program requirements and on how their performance is being assessed.

SBA’s Oversight of WBCs Includes Ongoing Performance Assessments and Reporting Requirements

We found that SBA had developed written procedures for assessing the performance and financial management activities of WBCs and had taken steps to measure the WBC program’s effectiveness. Since 1997, as a condition of continued funding, SBA has been required to assess WBCs’ performance at least annually through programmatic and financial examinations, and SBA District Office Technical Representatives (DOTR) conduct these examinations semiannually, typically on site at the WBC location. SBA’s policies and procedures for the WBC program require the district office to make a recommendation on continued SBA funding for the WBC in the final or second examination report each year. As an added measure, SBA also requires WBCs to have an independent certified public accountant (CPA) certify the condition of their financial management system each year as part of the final programmatic and financial examination. We reviewed fiscal year 2006 final examination reports for 7 of the 10 WBCs that we visited and verified that the examination included a checklist of questions on the WBC’s personnel and facilities, financial management—including details of the funding match, data collection, program activities, and Web site support and other Internet activity. None of the examination reports that we reviewed included a recommendation from the district office that SBA discontinue funding to the WBC.

In addition to conducting semiannual examinations, SBA requires that WBCs submit quarterly reports on their program activities, performance, and financial status and transactions. Quarterly program activity reports include data on counseling, training, and information transfers; and SBA requires WBCs to report these data directly through its Entrepreneurial Development Management Information System (EDMIS) database. Most of the WBCs that we spoke with said that they tracked and maintained program data in a separate internal database and later uploaded the data to EDMIS for SBA reporting. The information that WBCs are required to provide in quarterly performance reports includes the WBCs’ actual accomplishments, compared with their performance goals for the reporting period; actual budget expenditures, compared with an estimated budget; cost of client fees; success stories; and names of WBC personnel and board members. Fourth quarter performance reports must also include a summary of the year’s activities and economic impact data that the WBCs collect from their clients, such as number of business start-ups, number of jobs created, and gross receipts. SBA reports some of these data in its annual performance reports to Congress through several output and outcome measures that are meant to reflect the WBC program’s performance and effectiveness (table 2). Quarterly financial reports detail the WBCs’ financial status and program expenses that qualify for SBA payment under the terms of the award. Fourth quarter financial reports may include adjustments to previous financial reports for the program year. Quarterly reporting is directly tied to the WBCs’ ability to access their award funds. OWBO and DPGM review WBC quarterly reports and separate award payment requests, and DPGM has the authority to authorize WBC requests for advance or reimbursement payments.

As noted, SBA reports to Congress annually on the performance of the WBC program. In addition to collecting output and outcome data from WBCs, and as part of a broader impact assessment of its business assistance programs, in 2004 SBA initiated a 3-year longitudinal study of the WBC program that surveys the program’s clients. In our review of the WBC portion of reports for the first 2 years of the study, we found that although the study had a sound design, low response rates from WBC clients in the second report may limit SBA’s ability to generalize the results to all WBCs. Appendix II includes additional information on our review of SBA’s study and other studies assessing the economic impact of WBCs and discusses the difficulty of obtaining high response rates from private citizens for voluntary surveys.

SBA Has Limited Staff Resources for the WBC Program and Relies Heavily on District Office Staff Who May Have Too Many Responsibilities or Lack Relevant Experience and Training

Within OWBO, program managers monitor a caseload of WBCs that are grouped by geographic region and perform a variety of functions such as communicating with the centers and DOTRs, reviewing WBC documents and maintaining a project file for each center, and coordinating with DPGM on funding matters. However, SBA relies heavily on its district offices, and specifically DOTRs, to carry out many WBC program responsibilities, although OED and OWBO do not have direct supervision of district office staff. Rather, SBA’s Office of Field Operations oversees the district offices and district directors assign responsibilities to individual staff. In 2001, we reported that DOTRs had been given an increased role in assessing WBCs’ performance to ensure that the programs were fiscally sound and functioning smoothly. To this end, we reported that DOTRs were receiving intensive training each year at the postaward conference at SBA headquarters on how to monitor the WBCs’ programmatic and financial activities. As noted earlier, DOTRs are expected to conduct the WBCs’ programmatic and financial examinations semiannually, but they also have other WBC program duties and other full- time agency responsibilities. District directors assign the role of DOTR as a collateral duty to district office staff, and DOTRs whom we met with held separate positions as business development specialists and assistant district directors. SBA has a list of 23 responsibilities for DOTRs, some of which involve oversight, including (1) reviewing WBCs’ requests for project revisions, (2) determining the extent to which WBCs are meeting the match requirement, (3) reviewing the scope and quality of services provided to clients, (4) reviewing all WBC signage and media, and (5) helping to resolve problems. According to the list of responsibilities provided to us, DOTRs are also expected to act as advocates for the WBCs within their district. Some of the responsibilities related to this role include (1) ensuring that the district office displays and distributes WBC brochures; (2) collecting success stories from WBCs to be used for publicizing the program; and (3) including WBCs in district office conferences, workshops, and other events for women business owners. SBA officials told us that ideally DOTRs should focus on the oversight responsibilities and act as a liaison between WBCs and the district office. In the past, district offices also had a Women’s Business Ownership Representative (WBOR) who would act as an advocate for all activities involving women’s business issues. However, SBA officials and some district offices that we spoke with said that this role was often performed by the same person who was the DOTR. OED and OWBO officials said that since they do not control the assignment of staff responsibilities, they could not influence whether a district office employee acted both as an overseer and advocate for WBCs.

The DOTR’s total responsibilities for the WBC program appear to be substantial, particularly since these responsibilities are part of a collateral role. Given SBA’s downsizing in recent years, some DOTRs may have more responsibilities than they had in the past, making it more challenging to perform their WBC program duties effectively. Others new to the role may lack the necessary experience and training or carry out DOTR responsibilities by default. For example, an assistant district director, who was familiar with the WBC in his district, told us that he had performed the role of DOTR for less than a year. He also said that he had previously supervised the DOTR, WBOR, and two other positions. The DOTR had retired in fiscal year 2005, and another staff member who had filled the position temporarily was no longer with SBA. The WBOR had also left the agency, and neither position had been filled. Although most WBCs we interviewed spoke positively of their relationship with their DOTR, several told us that reductions in district office staff had led to changes, including assigning DOTR responsibilities to a different district office staff member. DOTRs still attend required training for the WBC program annually at SBA headquarters, and SBA provides them with a handbook to assist them in performing their duties. However, three of the six DOTRs that we spoke with said that SBA’s training for DOTRs in WBC oversight had not always been adequate. One DOTR said that there had been recent improvements but that past training assumed that new DOTRs had prior knowledge of the WBC program. The other two DOTRs made similar statements, with one pointing out that a lack of guidance had led to challenges in monitoring the WBC in her district at the time that she first assumed the role of DOTR. In one location, the DOTR and other district office staff told us specifically that they did not feel that DOTRs were adequately trained to conduct the financial component of WBC programmatic and financial examinations, adding that SBA headquarters had previously coordinated financial examinations for WBCs.

A 2003 SBA Office of Inspector General (OIG) audit of a Texas WBC found that the center had misused award funds and that SBA had not adequately monitored its financial and accounting operations. The OIG report on the audit specifically noted that the SBA reviewer had concluded that the center had a good financial and client tracking record system, in contrast with the audit’s finding that the center’s financial reporting lacked the standards reasonably expected of such an entity. The report also noted that the district office personnel assigned to perform oversight of the WBC did not have the financial background or proper training to perform financial reviews. When we followed up with OWBO officials, they said that SBA added the 2004 requirement that a CPA review WBCs’ financial records annually both because the agency recognized that some DOTRs lacked this expertise and because there had been isolated incidents of mismanagement of WBC award funds. The CPA reviewing a WBC’s records must complete and sign a statement in the final examination report stating whether the records were found to be acceptable in accordance with federal standards. OWBO officials also told us that they were coordinating with SBA’s Office of SBDCs to use SBDC financial examiners for these on-site financial reviews of WBCs, but added that recently there had not been enough staff to do all of the reviews. The officials also said that OED was reviewing how future financial audits for all of SBA’s business assistance programs would be conducted.

When we reviewed examination reports for 7 of the 10 WBCs that we visited, we found some inconsistencies that may suggest the need for more practical and ongoing DOTR training. First, in one report, the DOTR noted that the funding match requirement did not apply to a WBC because the center did not charge fees for SBA-sponsored programs and therefore did not generate funds from such programs. As noted previously, SBA’s funding match requirement applies to all WBCs in its program, with the ratio changing from 1 nonfederal dollar for each 2 federal dollars in years one and two to 1 nonfederal dollar for each federal dollar in year three and thereafter. We followed up with the WBC, and the center director verified that the WBC did charge fees for WBC program offerings and was meeting its match requirement. Second, we found that most of the final examination reports did not include a CPA’s statement and that two reports included a note stating that the certification would be forthcoming because the CPA was unavailable on the review (examination) date.

We found that SBA had taken some steps to adapt WBC program oversight procedures to its limited staff resources and to increase efficiency in some areas. For example, until January 2006 DOTRs conducted programmatic and financial examinations quarterly, and SBA switched to semiannual examinations to conserve its staff resources. SBA officials told us that staff resources for WBC program oversight had been strained for some time, and that OWBO recently received approval to fill two vacant positions and was currently determining the roles and responsibilities for these new staff. OWBO currently has five program managers that monitor a caseload of between 15 and 30 centers each. In March 2007, SBA also revised its reporting procedures for WBCs to streamline communication and to reduce review and processing times. For example, WBCs previously submitted original payment requests to the DOTR for review and recommendation, the DOTR forwarded the paperwork to OWBO for review and recommendation, and OWBO then forwarded the paperwork to DPGM for approval. As a result of complaints from WBCs and DOTRs regarding delayed award payments and misplaced WBC paperwork, SBA revised this procedure, and WBCs now submit original payment requests directly to OWBO. OWBO reviews the paperwork and makes a recommendation for payment, forwarding the paperwork to DPGM for authorization and notifying the DOTR and WBC of the recommendation. Both WBCs and DOTRs that we spoke with following SBA’s revision of its payment request procedure said that the new procedure had significantly improved communication.

The new procedures also improved payment turnaround times. Many of the WBCs that we spoke with mentioned that they had experienced challenges with receiving payments in a timely manner. As noted, SBA was aware of this issue. During the course of our review, the SBA OIG conducted a study looking at award disbursements to WBCs for fiscal years 2004 through 2007, surveying 21 of the 99 centers in SBA’s program in fiscal year 2007. The OIG’s preliminary report, which was based on responses received from 18 of the centers, found that in fiscal years 2005 and 2006, the majority of SBA’s payments to WBCs were not made in a timely manner. The study did not determine the percentage of payment delays that were caused by SBA’s untimely processing or the percentage that were caused by errors the WBCs may have made in submitting their paperwork. However, the OIG is making recommendations for SBA to improve its reimbursement process. In a September 2007 congressional testimony addressing the challenges facing the WBC program, the Associate Administrator for OED pointed out that while the size of the WBC network had grown from an initial 13 centers in 1989, SBA’s resources assigned to OWBO and DPGM had declined due to reductions in SBA’s overall budget. He also noted that as a result, the WBC program had outgrown its original set of policies and procedures, and OWBO faced challenges in managing and supporting the program. Continued imbalances in SBA’s staff resources for the WBC program, including the agency’s significant reliance on DOTRs, could reduce assurances that its oversight of WBCs is effective and that WBCs are meeting the program’s requirements.

Ineffective Communication from SBA Led to Confusion about WBC Program Requirements and Performance Reviews

The WBCs that we spoke with also raised issues related to SBA’s communication on program procedure and their performance. One study we reviewed reported that 54 percent of 52 WBCs surveyed said that SBA could improve its communication with them. Timely and thorough communication of operational procedures is critical to ensuring that the agency and the WBCs are able to perform their responsibilities effectively. Our Standards for Internal Control in the Federal Government state that for an entity to run and control its operations, it must have relevant, reliable, and timely communications relating to internal as well as external events. For external communication, agency management should ensure that there are adequate means of communicating with and obtaining information from external stakeholders that may have a significant impact on the agency achieving its goals.

OWBO conducts monthly conference calls with the WBCs and DOTRs, and SBA officials said that the calls were intended to maintain contact with the centers and to provide program updates and information on best practices. OWBO program managers facilitate the conference calls each month; and SBA officials also said that they send an agenda and handouts to DOTRs and WBCs electronically, prior to making the calls, and record the calls so that the information is available later. We reviewed handouts from conference calls that OWBO conducted between February and July 2007 and noted that program updates and best practice presentations were included on some of the calls. However, when we spoke with WBCs about the calls, some told us that the calls were not meeting their communication needs. Three of the WBCs that we spoke with said that best practice presentations that allowed them to share information with other centers were helpful. Others said that the calls were less effective because administrative items were typically covered instead of new information. Although WBCs have an opportunity to ask questions during the calls, the WBCs that we spoke with had mixed opinions about whether monthly conference calls provided a good forum for asking questions. One experienced WBC said that questions unrelated to the call agenda sometimes caused the discussion to be sidetracked and suggested that OWBO officials address such questions off-line. The WBC director also said that the varying experiences of the WBCs participating resulted in the calls being more effective for some centers than for others and suggested that OWBO consider restructuring the calls by WBC experience (years in the program) to provide a more productive learning experience. SBA officials told us that in January 2007, as an opportunity to provide necessary instruction to newer DOTRs, OWBO reinstated separate conference calls for DOTRs, although they can still participate in the WBC conference calls.

OWBO also uses e-mail to communicate policy changes to the WBCs and DOTRs and to make interim information requests of the WBCs, but some WBCs told us that they had difficulty clarifying changes to requirements and that SBA’s communications were often insufficient. Several WBCs said that SBA had not responded in a timely manner when they submitted payment requests and other administrative paperwork and that such delays resulted in financial burden and led to confusion about whether they had followed appropriate procedure and met program requirements. For example, one WBC director said that the center’s request for an advance payment was denied because she incorrectly submitted the request to the DOTR during SBA’s procedural changeover and had not yet been notified of the revised procedure, which required the request to go directly to OWBO. According to the WBC director, the DOTR was out of the office during the week that she submitted the request, and DPGM denied the request several weeks after receiving it because it reached DPGM after the deadline. The WBC director said that she received no notification of receipt from DPGM until the request was denied.

While SBA removed the DOTRs from the payment request procedure to eliminate potential bottlenecks, this action has not completely resolved WBCs’ concern about communications with SBA headquarters offices. Another WBC director who identified communication issues with SBA when we initially spoke with her, later told us that the recent procedure revisions had eliminated some of the confusion caused by multiple layers of approval for payment requests but noted that the WBC still had difficulty with DPGM’s denying requests without any communication about items it may identify once the paperwork reached that office. The Associate Administrator for OED highlighted the revised payment request procedure as an example of recent efforts to address inefficiencies, saying that the processing of payment requests had also been centralized to one point of contact in OWBO. He added that OWBO had initiated a prescreening process to identify missing documentation prior to reviewing the payment request and said that OWBO’s new policy of notifying the WBC when a payment request had been forwarded to DPGM would increase transparency. In agreement with the OED official’s statement, the WBC director said that OWBO appeared to have a clearer understanding of DPGM’s requirements in conducting its initial review of payment requests and that OWBO had not denied any recent requests. However, the WBC director said that DPGM had still denied requests for minor items that the WBC became aware of during a self-initiated follow-up with DPGM. For example, the WBC director said that in one instance DPGM would not accept copies of forms for which the WBC previously submitted originals that were either lost or misplaced by OWBO. According to the WBC director, OWBO intervened to resolve this particular issue.

It appears that limited communication within SBA has played a role in some WBC communication issues, and the SBA OIG’s preliminary report found that until recently the agency lacked an integrated tracking mechanism to identify when a payment request was received, where it was in the review process, and whether a disbursement had been made within the required time frame. One DOTR also told us that there were opportunities for SBA headquarters to improve its communication with DOTRs on policy and procedural changes to assist DOTRs in their role, and a WBC director said that she would like to see improved communication between SBA headquarters and its district offices, noting that some changes to paperwork requirements and program policy had not been communicated to the district offices.

In addition to communication issues related to payment requests and other administrative procedures, several WBCs told us that SBA had provided inconsistent communication on setting annual performance goals. SBA has established procedures for OWBO to negotiate annual performance goals with WBCs as they work toward accomplishing their 5-year project goals. Prior to fiscal year 2008, SBA’s requirement was that WBCs establish a goal of increasing several performance measures by 10 percent each year over the 5-year period of the award. One WBC said that for fiscal year 2007, the district office had requested that the WBC serve over 250 additional clients after OWBO and the WBC had agreed to a 10 percent increase over the goal for fiscal year 2006. The WBC director expressed concern that they had followed program guidelines in setting the initial goal; had not received an official explanation for the revised goal; and had received a smaller award than for the previous year, although they would have to serve many more clients. She said that when the WBC inquired about the change at the district office, they were told that the new goal was an OWBO goal. Another WBC said that the district office had communicated revised goals for fiscal year 2006 but was unable to explain the basis for the new goals. The director of this WBC said that the new goals were subsequently retracted without any official communication from SBA. A third WBC specifically noted that several years ago, SBA had issued a midyear requirement that WBCs package a certain number of loans. Although the WBC had been packaging loans, this requirement was not a WBC program measure, and the WBC did not include loan packaging in its annual goals. The WBC director said that the added requirement forced WBCs to comply with a goal they had not been working toward previously, and the director added that the requirement would have been especially difficult for smaller centers that may have added staff and other resources to package loans, particularly since this was not a permanent program requirement. When we followed up with OWBO officials, they told us that the district offices had sometimes communicated different goals to the WBCs to assist them with meeting district office goals, but that the district offices’ goals were set by SBA’s Office of Field Operations, which is a separate office in SBA outside of OED and OWBO. One district office staff member did tell us that the district office would like the WBCs to be more involved in helping the district office to meet its goals as part of a joint effort in meeting the needs of their local communities. OWBO officials did not address a solution for the miscommunication of goals, but in agreement with what some WBCs told us, OWBO officials said that they recognized that it was unrealistic for WBCs to continue to increase their goals each year while receiving smaller awards. The officials said that OWBO is revising its formula for WBCs to set annual goals, and removed the 10 percent increase requirement in its fiscal year 2007 program announcement for centers that will be funded in 2008. They also said that OWBO would begin to consider prior year funding amounts in setting achievable goals.

Some WBCs also told us that they were not sure how well they were performing because SBA did not provide them with feedback on semiannual programmatic and financial examinations or the reports that they submitted quarterly. One WBC told us that SBA did not provide details on how programmatic and financial examination results were used to place the center in a particular funding category, and another WBC said that SBA should provide appropriate feedback to let the WBCs know what they were doing well or could do better. The second WBC pointed out that it could not tell whether anyone was actually reviewing the reports that it submitted to SBA. SBA officials told us that they were aware of the WBCs’ concern regarding a lack of performance feedback and would take steps to make the WBC program’s performance-based funding process more transparent. SBA’s ineffective communication with the WBCs and between its offices that oversee the WBC program has led to confusion among WBCs, limiting their understanding of the program’s requirements and potentially reducing their ability to effectively carry out these requirements.

WBCs Make Some Efforts to Coordinate with SBDCs and SCORE, but SBA Provides Limited Guidance to Support These Efforts

The WBCs that we spoke with focused on a different type of client than the SBDCs and SCORE chapters in their areas, and several WBCs actively coordinated with the other programs to avoid duplicating services. Consistent with SBA requirements and statutory authority, WBCs tailor services to meet the needs of socially and economically disadvantaged women and tend to serve clients with businesses that have fewer employees and lower revenues than clients of SBDCs and SCORE. Though WBCs serve different types of clients, most WBCs told us that they refer clients to and coordinate services with SBDCs and SCORE when appropriate to leverage resources and avoid duplication. Also, some of the coordination efforts were facilitated by the SBA district office, but we found that SBA provided limited guidance to WBCs on how coordination should occur. In addition, coordinating services can be difficult because WBCs, SBDCs, and SCORE have similar performance measures, which could lead to competition among the service providers in some locations. We also found that on some occasions, SBA encouraged WBCs to provide services that were similar to services already provided by SBDCs in their district. These issues and the lack of sufficient guidance could create barriers to coordination and increase the risk of duplication in some locations.

The WBC, SBDC, and SCORE Programs Provide Training and Counseling to Small Business Clients, but Have Different Target Groups

Like SBDCs and SCORE chapters, WBCs provide both counseling and training services to small business clients. Unlike SBDC and SCORE, however, WBCs tended to target services to women, socially and economically disadvantaged clients, and clients with smaller businesses. Our review of the statutory authorities governing the WBC, SBDC, and SCORE programs found that each of the programs is required to provide training and counseling, but the WBC program’s statutory authority requires SBA to evaluate WBCs on, among other things, their ability to target services to socially and economically disadvantaged clients. Consistent with the WBC program’s statutory authority and SBA requirements, WBCs targeted services to socially and economically disadvantaged clients. A study of WBCs conducted by the Center for Women’s Leadership at Babson College also confirmed that WBCs responding to its survey predominantly served socially and economically disadvantaged clients. According to the Babson College study, 67 percent of WBC clients came from households with incomes that were less than $50,000, and 55 percent of WBC clients had a high school diploma or less education.

Three WBCs that we spoke with were able to provide support to socially and economically disadvantaged clients through financial literacy, savings, and credit repair programs. For example, a WBC in California had a program that provided financial literacy and asset building services for its economically disadvantaged small business clients. Through this program, clients were able to attend financial literacy courses while gradually increasing their savings through an individual development account and savings club program. The individual development account and savings club program allowed low-income clients to receive matching funds to save toward the purchase of a home or to start a small business.

Consistent with the program’s statutory authority, we also found that WBCs tended to focus their programs on female clients. A study contracted by SBA on the impact of the WBC program reported that women made up 77 percent of WBC clients. The Babson College study also reported that the WBCs responding to its survey tailored their programming to meet the needs of women clients. Consistent with the findings of the SBA and Babson College studies, several WBCs that we contacted provided services specific to the needs of women clients. For example, a few WBCs had women’s networking or mentoring groups so that experienced women entrepreneurs could share advice with those who were new women business owners.

WBCs also tended to offer services that helped clients start and expand existing microenterprises or very small businesses. For example, a WBC in Chicago established a program to help its clients start home-based child care centers, and a WBC in Baltimore helped low-income clients with existing informal home-based businesses expand and increase their income in order to assist them with becoming economically self-sufficient. SBA’s impact study of WBCs also showed that they tended to serve clients with businesses that had fewer employees and lower revenues than clients of SBDCs and SCORE. According to the study, WBC clients had businesses with an average of 2.5 employees and an average revenue of $63,694. In contrast, SCORE worked with businesses with an average of 3.2 employees and $112,182 in average revenue, and SBDC worked with businesses with an average of 6.3 employees and $272,552 in average revenue.

We found that some WBCs offered services for clients with limited business experience. WBC directors interviewed for the Babson College study also reported that WBC clients had distinct needs that often reflected a lack of experience in the business world. Several WBCs that we contacted provided intensive long-term training and counseling to help clients through each phase of small business development from start-up through expansion. A WBC in California provided a 3-year long “virtual business incubator” program that targeted first generation immigrant entrepreneurs, helping them to develop their businesses and to set long- term asset-building goals. While in the virtual incubator program, WBC clients, through coaching and training over a 3-year period, produced a business plan, established a business accounting system and legal structure, and developed a marketing plan to start and establish their businesses. A WBC in Massachusetts also offered a 13-week and 20-week multiphase training and counseling program for its clients that was designed to help new small businesses through each phase of the business development process.

Coordination among WBCs, SBDCs, and SCORE in Some Locations Was Extensive, but SBA Provides Limited Guidance to Support These Efforts

While SBA requires WBCs to coordinate with SBDCs and SCORE chapters, SBA provides limited guidance or information on how these business assistance programs should coordinate. Increasingly, the government relies on new networks and partnerships to achieve critical results and develop public policy, often including multiple federal agencies, domestic and international non- or quasi-government organizations, for-profit and nonprofit contractors, and state and local governments. Notwithstanding the increased linkages in our system, each level of government often makes decisions on these interrelated programs independently, with little interaction or intergovernmental dialogue. According to the Grant Accountability Project, a working group chaired by the U.S. Comptroller General, coordination between federally supported programs that provide similar services, such as the WBC, SBDC, and SCORE programs, is important to avoid service duplication and to efficiently leverage federal funds.

Through the WBC notice of award, SBA policy requires that WBCs work collaboratively with SBDCs and SCORE chapters, with assistance from SBA district offices, to coordinate efforts in order to expand services and avoid duplication. When WBCs are located in communities with these resource partners, the WBCs are to coordinate with them in offering training and other forms of assistance to their clients. SBA headquarters officials also confirmed that they expected district offices to ensure that duplication between the programs did not occur.

Though SBA policy requires WBCs to coordinate, SBA does not provide detailed guidance to WBCs on how coordination should occur in order to efficiently leverage SBA funding. Through our review, we found that the only guidance SBA provided to WBCs was in the notice of award, which asks WBCs to coordinate with SBA resource partners and other WBCs, where appropriate, under cosponsorship arrangements or memorandums of understanding. However, neither the notice of award nor any other document prescribes any specific practices or methods for these efforts. When we asked SBA officials about the lack of guidance, they said that they expect WBCs to initiate coordination on their own without specific guidance from SBA. SBA officials also said that they had not issued specific guidance because they did not want to be overly prescriptive or dictate how coordination should occur, given that local conditions varied and that some forms of coordination might be effective in some locations but not in others.

Without specific SBA guidance, some WBCs used a variety of approaches to initiate coordination with other business assistance providers. Most WBCs said that they referred clients to SBDCs and SCORE chapters in their areas when appropriate and coordinated services with these other business assistance providers to leverage resources and avoid duplication. Some WBCs provided services to both start-up and experienced clients, but others referred more experienced or established small businesses to SBDCs. Some WBCs tended to refer clients seeking short-term counseling or specific industry expertise to SCORE. As an organization primarily staffed by volunteer small business counselors instead of full-time employees, SCORE services tended to be short-term and focused. For example, a small business client seeking restaurant industry expertise may be referred to a SCORE counselor that formerly owned his own restaurant.

In several locations, WBCs were colocated or shared space with SBDCs and SCORE chapters and were often able to benefit from reduced overhead costs that came from shared facilities. Five colocated WBCs and SBDCs we contacted shared administrative support and leveraged counseling staff in order to better serve clients. For example, in California, a WBC that was colocated with an SBDC often referred clients to SBDC counselors if WBC counselors were not available in order to maximize resources and provide better service.

Seven WBCs told us that the district office sometimes facilitated coordination between WBC, SBDC, and SCORE. Two SBA district offices that we contacted coordinated resource partner meetings at which representatives from the WBC, SBDC, and SCORE programs and other small business assistance providers met to discuss service coordination and to organize small business events. A few SBA district offices were involved in promoting WBC, SBDC, and SCORE activities but were not often directly involved in facilitating communication among the programs.

Some WBCs told us that coordination was sometimes independently initiated by WBC, SBDC, and SCORE representatives without assistance from the SBA district office. For example, under a memorandum of understanding, WBC, SBDC, and SCORE representatives in South Carolina organized informal groups with other area small business assistance providers to plan events, coordinate services, or facilitate training. In Wisconsin, a WBC coordinated with SBDC, SCORE, and other small business assistance providers to develop a detailed triage system for small business clients in their area. In order to better coordinate services, the WBC and its resource partners developed a flow chart to help service providers divide their resources and determine where to refer clients. Under this system, clients with existing businesses were referred to the SBDC, and clients not yet in business were generally referred to the WBC. Figure 3 illustrates some of the approaches that WBCs took to coordinate with SBDC and SCORE.

Though several WBCs provided examples of successful coordination efforts, a few WBCs that we contacted were unable to provide sufficient information to demonstrate that they were coordinating with SBDC and SCORE in order to decrease duplication of services. In two instances, WBCs that we spoke with had made efforts to decrease service duplication without coordinating with SBDCs and SCORE chapters in their area. For example, one WBC had limited contact with SBDC and SCORE chapters and attempted to eliminate duplication in services by reviewing some of the course and service information on SBDC and SCORE chapter Web sites.

Also, WBCs raised concerns about how to effectively coordinate by colocating with an SBDC or SCORE chapter. Several WBCs told us that they had considered coordinating with SBDC and SCORE by colocating or sharing space in order to reduce costs and leverage staff, but feared that doing so would inhibit their ability to maintain their identity and reach their target client group of low-income women. Until recent policy changes, WBCs and SBDCs were both measured on the number of clients that participate in small business training and counseling services, and one WBC told us that colocation would cause WBCs to compete for clients. SBA officials told us that the potential for competition between WBCs and SBDCs should have been reduced since SBDCs were no longer required to set a goal for the number of clients they serve. Until recently, if a WBC and an SBDC sponsored a joint training event, only one organization could count an individual client and the total number of training hours. WBCs and SCORE still have similar measures, and some measures could hinder collaborative efforts.

Some WBCs experienced challenges in their attempts to coordinate services with SBDC and SCORE. Some WBCs told us that coordinating services could be difficult. In some instances, SBA encouraged WBCs to provide services similar to those that SBDCs were already providing to small businesses. For example, in the WBC notice of award, SBA emphasizes that WBCs provide ongoing assistance to existing or established small businesses. However, several WBCs told us that they considered SBDCs and other service providers to be better equipped to serve existing and experienced small businesses. In another example, SBA’s WBC notice of award asks that each WBC make an effort to increase its focus on providing procurement assistance to small businesses; however this initiative could be interpreted as overlapping with the existing goals of Procurement Technical Assistance Centers, a program funded by the Department of Defense that provides procurement assistance to small business owners through SBDCs and other institutions. One WBC told us that the district office had encouraged the center to develop a government procurement curriculum, although the WBC was already referring clients to an SBDC in the area that provided this service to small business clients.

As we have seen, some WBCs were effectively coordinating with SBDCs and SCORE, but others faced challenges that SBA’s limited guidance has not addressed. The examples of successful and effective coordination that were shared with us demonstrate that a variety of approaches exists, and that some WBCs have overcome some of the challenges expressed by others. This type of information would be useful guidance to all WBCs. Without it, WBCs, SBDCs, and SCORE may be duplicating efforts and missing opportunities to use federal funds more efficiently.

Conclusions

The WBC program has undergone significant change since its inception in 1989. WBCs were initially envisioned as entities that would receive federal funding for only 5 years. However, concerns about whether the federal investment was sufficient to create sustainable WBCs led Congress to create a pilot program to provide sustainability awards for an additional 5 years. The creation of the sustainability awards, which were meant to provide some additional support to WBCs, also created some uncertainty about the amount of funding older WBCs would obtain year to year and whether sustainability awards would continue in the future. This year, Congress created a more permanent funding stream for WBCs that continue to meet the program’s requirements by establishing 3-year renewable awards. This change should address the concerns raised by existing WBCs. However, the program will remain competitive, and it is unclear how the new awards will impact SBA’s ability to fund new centers in the future. SBA is planning to begin providing the new 3-year renewable awards in 2008.

SBA’s downsizing and the subsequent impact on staffing in district offices, as well as the growth in the WBC program in terms of number of WBCs participating, have had an impact on how SBA oversees WBCs. The significant reliance on DOTRs in SBA’s district offices is particularly problematic because we found instances in which some DOTRs may not be able to carry out those responsibilities effectively. Some DOTRs had other district office responsibilities that could limit their ability to oversee WBCs, and others lacked the necessary skills and expertise. Because SBA relies on DOTRs, it is important that the agency ensure that such staff have the right mix of responsibilities and adequate guidance and training to carry out those responsibilities. Otherwise, DOTRs may not succeed in ensuring that WBCs are meeting all program requirements and that federal funds are not being misused or wasted.

Communication between SBA and WBCs could also be improved. Communication is a key internal control that ensures SBA’s policies and procedures are understood. The fact that several WBCs said that they did not obtain sufficient information on what it takes to be a successful WBC, even though OWBO has monthly conference calls with them, suggests the need to explore additional methods for providing information that will help WBCs to be successful. Improved communication would reduce the confusion expressed by many WBCs and increase the likelihood that the centers meet program requirements and perform well.

SBA can facilitate efficient and effective use of its resources by encouraging coordination among the WBC, SBDC and SCORE programs when coordination makes sense for the geographic areas they are serving. Though we found that the WBC program distinguishes itself from other business assistance programs by providing services to economically and socially disadvantaged populations, all of SBA’s business assistance providers—WBCs, SBDCs, and SCORE—provide training and counseling services to potential or existing entrepreneurs. As a result, the opportunity for duplication exists. SBA is aware that duplication could occur and has taken steps to encourage coordination but there is no explicit guidance on how to successfully coordinate services. WBCs are expected to demonstrate that they are coordinating with SBDCs and SCORE, and several of the WBCs gave us examples of how they coordinated, but the degree of coordination varied. Some WBCs had concerns about how they should coordinate while also ensuring that they meet their own program requirements. The instances of active coordination among SBA’s programs and other local business assistance programs provide a range of methods other geographic areas could also consider using. These examples demonstrated how coordination can leverage resources and help programs minimize or avoid duplication, but they are not necessarily familiar to all WBCs because SBA has not provided guidance based on these promising practices and examples of effective coordination.

Recommendations for Executive Action

To ensure that oversight of the WBC program is efficient and effective we recommend that the Administrator take the following two actions: evaluate and modify, as appropriate, the responsibilities assigned to DOTRs to ensure that DOTRs can conduct appropriate and effective monitoring of the centers, and establish a communication strategy to ensure that WBCs have access to up-to-date information on program requirements and help the centers better understand how they are performing .

To improve coordination and facilitate the efficient use of federally funded resources, we recommend that the Administrator direct the Associate Administrator of the Office of Entrepreneurial Development (OED) to take the following action: develop guidance or information for SBA’s district offices and WBCs, SBDCs, and SCORE that will facilitate successful coordination of services. This guidance or information could be developed by identifying promising practices currently in place in some geographic areas or by developing case studies or examples of successful coordination models. The guidance should also assist district offices, WBCs, SBDCs and SCORE in providing sound advice on how to coordinate services when doing so could conflict with meeting individual program requirements or initiatives.

Agency Comments

We provided SBA with a draft of this report for review and comment. SBA provided no comments on the draft report or its recommendations.

We will send copies of this report to the chair of the Committee on Small Business, House of Representatives, the Administrator of the Small Business Administration, and other interested parties. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-8678. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff acknowledgments are listed in appendix III.

Appendix I: Scope and Methodology

In this report, we address (1) the uncertainties associated with the funding process for Women’s Business Centers (WBC); (2) the Small Business Administration’s (SBA) oversight of the WBC program, including policies and procedures for monitoring compliance with program requirements and assessing program effectiveness; and (3) the services that WBCs provide to small businesses and actions that SBA and WBCs have taken to avoid duplicating the services offered by the Small Business Development Center (SBDC) and SCORE (formerly Service Corps of Retired Executives) programs.

To address all three objectives, we reviewed the legislative history of the WBC program, our previous reports, SBA’s policies and procedures for administering the program, and studies of the program conducted by SBA, SBA’s Office of Inspector General, and external organizations. We conducted site visits in 6 states and the District of Columbia and interviewed officials from 10 WBCs, 7 SBDCs, and 6 SBA district offices. We selected these locations to represent geographic diversity and to enable us to rely on staff from our field offices. We also conducted 17 telephone interviews with WBCs randomly selected from the universe of 99 WBCs that received SBA awards in fiscal year 2007, using criteria to ensure that we obtained a mix of newer and more established WBCs that would allow us to compare a range of experiences with the program. Table 3 lists the WBCs and SBDCs we contacted. We interviewed officials from SBA’s Office of Entrepreneurial Development (OED) and Office of Women’s Business Ownership (OWBO) who are responsible for overseeing the program.

To address uncertainties associated with the funding process, we interviewed officials in SBA’s Division of Procurement and Grants Management (DPGM) regarding the WBC award process and new legislation changing the WBC program’s funding structure. We interviewed officials at the Office of Management and Budget (OMB) regarding specific findings related to the WBC funding process in its Program Assessment Rating Tool (PART). Additionally, as part of our interviews, we asked the 27 WBCs to provide their perspectives on the WBC funding process.

To assess SBA’s oversight of the program, we reviewed documentation that SBA uses to oversee WBCs and the award applications and programmatic and financial examination reports for 7 of the 10 WBCs that we visited. We interviewed SBA district office staff on their role in overseeing the WBCs and on the guidance that SBA provides to them. We also interviewed officials in DPGM regarding DPGM’s role in WBC funding and oversight. We asked the 27 WBCs about their relationship with SBA.

To identify the services WBCs provide and actions WBCs and SBA take to avoid duplication with other SBA programs, we reviewed and compared the statutory authority for the WBC, SBDC, and SCORE programs. Additionally, we reviewed two reports from SBA’s contracted study and studies by three external entities on the impact of WBCs. We describe each study and provide an assessment of each study’s design in appendix II. We asked the 27 WBCs, 7 SBDCs, and SCORE’s national office about the services they provide and coordination among the programs. Additionally, we reviewed the Web sites of 24 of the WBCs and 6 of the SBDCs that we contacted, and SCORE to identify the services that they offer and determine whether the Web sites provided any information on how the three programs coordinate in their geographic areas (local markets).

We conducted our work in California, Georgia, Illinois, Maryland, Massachusetts, Virginia, and Washington, D.C. between August 2006 and November 2007 in accordance with generally accepted government auditing standards.

Appendix II: Studies Evaluating the Impact of WBCs

We reviewed four studies that have evaluated various aspects of the impact of WBCs. One study was sponsored by SBA. SBA contracted with Concentrance Consulting Group—a management consulting firm offering services to government and private sector clients—to conduct a longitudinal impact study of its business assistance programs, including the WBC program. We reviewed the first two reports of the SBA study and only reviewed the sections addressing the WBC program. The three other studies focused on the impact of WBCs and were sponsored by private organizations. The Association of Women’s Business Centers (AWBC)—a nonprofit organization representing WBCs—sponsored the study conducted by the Center for Women’s Leadership at Babson College. The National Women’s Business Council (NWBC)—a federal advisory council created by Congress—sponsored a study conducted by Quality Research Associates, a research firm. The fourth study was conducted by the Center for Women’s Business Research—a nonprofit research organization—and was sponsored by NWBC, AT&T, and American Express. We identified these three studies through Internet literature searches during July and August 2006 as being industry-conducted studies using the following search terms: “Women’s Business Centers” AND “Study” OR “Studies”; “WBC” AND “Study” OR “Studies.” Each study was reviewed by two staff members. Using a template, the first reviewer took notes on author’s affiliation, objectives, methodology, limitations, and other information. The second reviewer then reviewed these notes after reading the study. Where there was lack of agreement, the two reviewers discussed their points of view and reached agreement.

Based on our assessment of the studies’ design and methodology, we determined that the studies provide useful information on how some WBCs have impacted their clients, but these studies are limited because of low survey response rates and other study limitations. It is important to note that there are a number of considerations in attempting to evaluate the impact of service delivery programs like the WBC program. Voluntary surveys of private citizens tend to yield much lower response rates than surveys of organizations or nonvoluntary surveys, like the Census. Response rate is a key statistic toward understanding whether study results are representative of the population that has been sampled. As a consequence, any study of WBC clients may be limited in describing the universe of clients due to low response rates. The type of error associated with low response rates is called survey nonresponse error. There are some procedures that help mitigate against survey nonresponse error, and in our review of the study conducted for SBA and the other studies, we have noted when the study used these procedures. Also, when studies are based on small numbers, such as the studies using WBCs as the unit of analysis, it can be difficult to detect patterns in the data.

Overall, we believe that any inferences from these studies are largely limited to the centers and clients actually providing data for the studies, except perhaps in the first report for the SBA study, which included procedures to increase the confidence that the results are more representative of the population of clients served. In general, we found that all four of the studies appear to have reasonable and well thought through research designs. However, in some studies key information was not reported that would have enabled us to more completely evaluate them. Table 4 provides information on each of the studies and our assessment of the studies’ design.

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

The following individuals made key contributions to this report: Kay Kuhlman, Assistant Director; Heather Atkins; Bernice Benta; Carolyn Boyce; Michelle Bracy; Tania Calhoun; Emily Chalmers; and Rudy Chatlos.