Small Business Administration: Office of Advocacy Needs to Improve Controls over Research, Regulatory, and Workforce Planning Activities
Published: 2014-07-22T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Congress created Advocacy within SBA in 1976 as an independent voice for small businesses. Questions have recently been raised about Advocacy's efforts to represent small businesses in regulatory activities and some of its research on small business issues. In light of these questions, GAO was asked to review Advocacy's activities.
This report examines Advocacy's (1) research, (2) regulatory activities, and (3) workforce planning efforts. GAO analyzed Advocacy's research, comment letters, and other regulatory information for fiscal years 2008-2013; assessed Advocacy's policies and procedures against federal standards for internal control and information quality; and interviewed agency officials, and small business and industry representatives. GAO also analyzed the applicability of the Federal Advisory Committee Act to Advocacy roundtables.
section_title: What GAO Found
paragraphs: The Office of Advocacy (Advocacy) within the Small Business Administration (SBA) fulfills its mission by researching small business issues and providing input into federal rulemaking and related regulatory activities. However, GAO identified key areas in Advocacy's system of internal control that could be improved.
Research. GAO found that Advocacy did not ensure that its staff monitored the quality of the information the office disseminated, as required. GAO reviewed 20 selected research products and found that in 16 cases a required quality review had not been documented. Advocacy recently established a review policy for its research, but it does not include procedures for selecting the reviewers or documenting that a review occurred and how reviewer comments are addressed. GAO also found that Advocacy staff had not followed federal information quality guidelines to retain data and could not substantiate the quality of information in two cost-estimation reports—a research product it has contracted for every 5 years. Without better controls over its quality review process and efforts to substantiate the information it disseminates, Advocacy cannot ensure the validity of one of its core activities—research in support of small businesses.
Regulatory activities. Advocacy recently updated procedures for its regulatory activities, but these could be strengthened. GAO found the extent to which individual staff maintained records varied, in part, because the procedures lacked policies for documentation. For instance, the procedures state that when staff decide to intervene in the rulemaking process, they must follow up as appropriate with the interested groups to ensure that Advocacy has sufficient information and data to support its case. However, there is no policy that these interactions be documented. Federal internal control standards state that documentation and records should be maintained. If key procedures are not being documented, managers do not have an institutional record that agency goals and objectives in this area are being met. GAO also found that the Federal Advisory Committee Act's transparency and other requirements do not apply to Advocacy's meetings with stakeholders to get input on regulations (roundtables).
Workforce planning. Advocacy's workforce efforts include training and mentoring for new staff, but do not include succession planning, which is recommended by the Office of Personnel Management. According to federal internal control standards, effective management of a workforce is essential to achieving program results. Officials told GAO that Advocacy was a small office and that additional staff were hired on an as-needed basis. However, some key staff have been with Advocacy for many years and their experience will be difficult to replace. If Advocacy does not incorporate succession planning strategies into its workforce planning efforts, it is at risk of not having the skills and expertise to meet its mission when key staff leave or retire.
section_title: What GAO Recommends
paragraphs: GAO makes several recommendations to improve the Office of Advocacy's controls over the quality of its research, the documentation of its regulatory activities, and workforce planning. In commenting on a draft of this report, Advocacy agreed with our recommendations and noted some steps it will take to address them.

section_title: Letter
paragraphs: Congress created the Office of Advocacy (Advocacy) within the Small  Business Administration (SBA) in 1976 as an independent voice for small  businesses within the federal government. The Chief Counsel for  Advocacy, appointed by the President and confirmed by the U.S. Senate,  directs the office and is tasked with representing the views and interests  of small businesses to other federal agencies, including during  rulemaking and related regulatory activities. Advocacy’s mission is to  encourage the development and growth of America’s small businesses by producing research to keep policymakers and other stakeholders  informed about the impact of federal regulatory burdens on small  businesses, document the role of small businesses in the economy,  and explore and explain issues of concern to the small business  community;  intervening early in federal agencies’ regulatory development process  on proposals that affect small businesses and providing training in  complying with the Regulatory Flexibility Act (RFA) to agency  policymakers and regulatory development officials; and  fostering two-way communication between federal agencies and the  small business community.
Questions have recently been raised about Advocacy’s efforts to  represent small businesses. For example, some stakeholders questioned  how Advocacy staff gather and use information to develop comment  letters on proposed regulations and whether that information actually  conveys the views of small businesses. Also, some parties have  expressed concerns that Advocacy’s information-gathering events  (“roundtables”) are not fully transparent and open to the public, and  others have criticized the methodology of some of the research  conducted by contractors hired by Advocacy.
You asked us to review the Office of Advocacy. This report examines  Advocacy’s (1) research activities; (2) regulatory activities, including the  applicability of the Federal Advisory Committee Act’s (FACA)  transparency and other requirements to Advocacy’s roundtables; and (3)  workforce planning efforts.
To understand Advocacy’s mission, operations, and participation in the  rulemaking process, we interviewed Advocacy officials, and reviewed  Advocacy documents, including its policies and procedures, budget and  strategic planning documents, annual reports to Congress on the RFA,  and other publications. To evaluate Advocacy’s research activities, we  selected a nonprobability sample of 20 of its products and assessed them  against Advocacy’s review process for its research (“peer review”), the  Office of Management and Budget’s (OMB) Peer Review guidance, and  OMB and SBA’s Information Quality Guidelines. Our sample was  designed to include a variety of topics and report types and focused on  more recent publications in order to maximize the possibility that the  research staff involved would be available for interviews. In addition, we  evaluated three of Advocacy’s most recent external studies on the  economic costs of regulation and assessed the studies against OMB and  SBA Information Quality Guidelines. We also reviewed peer reviewers’  comments and other external reviews on the studies to understand  questions that had been raised about the studies. Lastly, we interviewed  the Advocacy officials responsible for this research to obtain information  on the steps taken to substantiate the quality of these studies.
To evaluate Advocacy’s regulatory activities, we analyzed comment  letters and information-gathering events (roundtables) from fiscal years  2009 through 2013 as well as relevant policies and procedures. We  interviewed Advocacy staff who were responsible for these activities and  reviewed supporting documentation to understand their origins and  purposes. In addition, we attended three roundtables. To analyze whether  FACA (5 U.S.C. App. II) applies to Advocacy’s roundtables, we reviewed  relevant statutes, case law, regulations, and guidance. We also  considered Advocacy’s views on the issue. Lastly, we interviewed officials  from the entities that interact with Advocacy in rulemaking—the  Environmental Protection Agency (EPA), Occupational Safety and Health  Administration (OSHA), Bureau of Consumer Financial Protection, also  known as the Consumer Financial Protection Bureau (CFPB), and OMB’s  Office of Information and Regulatory Affairs (OIRA). Also, we interviewed  officials from six industry associations that represent small and large  businesses; one large corporation; and two nonprofit organizations whose  missions relate to policies or rules under consideration, to understand  their perspectives on Advocacy’s roundtables and other work. We made  our selections to include representatives from a variety of sources, and  while the results from our interviews cannot be projected to all entities  that interact with Advocacy, the information we gathered provides insights  into how the selected groups view Advocacy and its work in representing  small businesses to federal policymakers.
To assess Advocacy’s workforce planning efforts, we interviewed senior  management to discuss any workforce planning efforts that were in place.  We evaluated those efforts and any related documents against applicable  federal standards for internal control and the Office of Personnel  Management’s Human Capital Assessment and Accountability  Framework (HCAAF).
We conducted this performance audit from August 2013 to July 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Under legislation that established Advocacy in its current form in 1976,  Advocacy’ duties are to  serve as a focal point for small businesses’ concerns about the  federal government’s policies and activities; advise small businesses on how to interact with the federal develop proposals for federal agencies on behalf of small businesses; represent the views and interests of small businesses before federal  agencies; enlist the cooperation and assistance of public and private agencies,  businesses, and other organizations in disseminating information  about the federal government’s programs and services that benefit  small businesses.
Since its establishment, a series of laws and executive orders has  increased Advocacy’s roles and responsibilities. First, in 1980 the White  House Conference on Small Business made recommendations that led  directly to the passage of the RFA, which requires government agencies  to consider the effects of their regulatory actions on small entities and,  where possible, mitigate them. Under the RFA, agencies provide a small  business impact analysis, known as an initial regulatory flexibility  analysis, with every proposed rule published for notice and comment and  a final regulatory flexibility analysis with every final rule. The Chief  Counsel for Advocacy was charged with monitoring federal agencies’  compliance with the act and with submitting an annual report to  Congress.
Second, in 1996 the Small Business Regulatory Enforcement Fairness  Act (SBREFA) provided for the judicial review of agency compliance with  key sections of the RFA. It also established a requirement that EPA and  OSHA convene panels whenever these agencies are developing a rule  for which an initial regulatory flexibility analysis would be required  (SBREFA panels). These panels consist of the agency, OIRA, and  Advocacy. The 2010 Dodd-Frank Act added the newly created CFPB to  the agencies required to convene SBREFA panels. The SBREFA panels  meet with representatives of the affected small businesses to review the  agencies’ draft proposed rules, identify alternative approaches to the  rules, and provide insight on the anticipated impact of the rules on small  entities. The panels issue a report, including any recommendations for  minimizing the economic impact of the rule on small entities.
Third, Advocacy’s responsibilities were further expanded by Executive  Order 13272, which was issued in 2002. The order required each agency  to establish procedures and policies to promote compliance with the RFA  and to publish a response in the Federal Register to any written comment  received from Advocacy on rules published. This requirement was  codified by the Small Business Jobs Act of 2010. Executive Order  13272 also directs Advocacy to provide training to federal agencies on  how to comply with the RFA.
Until 2010, Advocacy’s budget was part of SBA’s. As of fiscal year 2010,  however, Advocacy was given statutory line-item funding in a Treasury  account separate from other SBA funding, with Congress setting the  amount available for Advocacy’s direct costs. In fiscal year 2014,  Advocacy’s enacted budget was $8.75 million. Its fiscal year 2015 budget  request was $8.46 million. Of that amount, $7.75 million (92 percent) is to  be used to fund compensation and benefits for Advocacy’s professional  staff, with the balance of Advocacy’s budget split almost equally between  external research and all other direct expenses. Advocacy is organized  in five offices, as shown in figure 1.
subsections: section_title: Research Activities
paragraphs: Advocacy’s Office of Economic Research produces both internal and  external research, which is publicly disseminated, on a variety of small- business issues. More specialized research—requiring proprietary data or  econometric analysis—typically is conducted by contractors (external).  SBA handles the contracting process for Advocacy. The contracts  generally last for 1 year. Advocacy economists act as the official  contracting officers’ representatives (contracting officer) overseeing and  coordinating the work of the contractors. The contracting officers maintain  a contract file for each external research product. Each year the Office of  Economic Research solicits research topics from Advocacy staff and  small business stakeholders, such as associations comprised of small  businesses. In addition, Congress requests studies, either formally (by  putting the requirement into a law), or informally (through discussions with  Advocacy staff). A final list of potential research is presented to the Chief  Counsel before the beginning of the fiscal year, and the counsel chooses  the topics for the year. In fiscal year 2013, Advocacy produced 22  research products on topics that included access to capital, small  business exporters, entrepreneurship, and minority- and women-owned  businesses. Other Advocacy research addresses the concerns  highlighted in Advocacy’s authorizing statute, such as examining the role  of small business in the American economy, assessing the effectiveness  of existing federal subsidies and assistance programs for small  businesses, and evaluating efforts to assist small veteran-owned small  business concerns.
Advocacy follows a peer review process that was revised and formalized  in 2013 and applies to all internal and external research products.  Advocacy staff, including the Director of the Office of Economic  Research, conduct peer reviews for internal products. For external  research products, the peer review is initiated when Advocacy receives  the first draft of the product, typically 6 to 8 months after the contract has  been awarded, and generally includes external reviewers. In addition to  the peer review process, all of Advocacy’s research products are required  to pass Advocacy’s internal clearance process, which involves review by  the Director of Economic Research, editors in Advocacy’s Office of  Information, and individuals in Advocacy’s Office of Chief Counsel (Senior  Advisor, Deputy Chief Counsel, and Chief Counsel).
section_title: Regulatory Activities
paragraphs: Advocacy’s Office of Interagency Affairs oversees the office’s regulatory  activities, which aim to convey the views of small businesses on the  impact of federal regulations and related costs. These activities generally  fall into three categories—developing and issuing comment letters,  convening information roundtables, and providing RFA training.  Attorneys in this office (“regulatory attorneys”) are expected to become  experts in the policy areas they oversee and to establish and maintain  broad and effective networks of small business experts (e.g., trade  associations) in their policy area. The regulatory attorneys are  encouraged to go to trade association and other industry meetings in  order to maintain and expand those networks. In addition to maintaining  working relationships with industry members and experts, the staff are to  establish and maintain relationships with the regulatory staff within each  agency who write the rules.
One of the primary ways Advocacy provides input to agencies that are  issuing rules and regulations of concern to small businesses is through  public comment letters. Our review of comment letters from fiscal years  2009 through 2013 found that they covered a wide range of rulemakings  on issues such as food labeling, designations for critical habitat, and  emission standards. Advocacy made a number of recommendations in its  comment letters that included creating an exemption for small businesses  or strengthening economic analyses required by the RFA. Advocacy also  issued “nonrule” letters that involved the agencies’ other activities, such  as agencies’ scientific research. These letters constitute a small  proportion of Advocacy’s comment letters. Table 1 below shows the  number of comment letters on rulemakings by fiscal year.
Regulatory attorneys also convene information-gathering roundtables to  discuss the regulatory concerns of small businesses. Roundtables are  convened on a regular basis in two policy areas—environment and labor  safety—while events covering other areas are convened on an ad hoc  basis, depending on which regulatory or rulemaking issues might be  forthcoming. According to the regulatory attorneys, the most typical  reason for convening a roundtable was an upcoming rule or legislation  that would impact small businesses. The regulatory attorneys use the  information gathered from the roundtables, together with other  information, to inform Advocacy’s positions on the issues involved, and to  give Advocacy direction on proposed rules’ economic impacts and  possible regulatory alternatives. The attorneys also told us roundtable  discussions help them set priorities, and broaden their knowledge base.  The roundtables sometimes resulted in a comment letter, although not  always. Table 2 shows the number of roundtables by policy area for fiscal  years 2009 through 2013.
As discussed previously, Executive Order 13272 requires Advocacy to  provide training to the agencies on how to comply with the RFA.  According to data provided by Advocacy officials, in 2013, Advocacy staff  provided training on the RFA to 159 officials at nine agencies and to 22  congressional staff. In addition to the formal training sessions, regulatory  attorneys told us they were encouraged to interact regularly with the  relevant rulemaking officials at the agencies as rules were developed in  order to communicate the concerns of the small business advocates.
section_title: Advocacy’s Internal Controls Do Not Ensure the Quality of Its Research and That Required Information Quality Practices Are Followed
paragraphs: Producing research products, both internally and externally, on issues of  importance to small businesses is one of Advocacy’s primary  responsibilities. However, we found that Advocacy’s quality review  process lacked some key controls to substantiate the quality of the  research and did not take steps to ensure that staff were adhering to  existing controls.
subsections: section_title: Advocacy’s Peer Review Process Lacks Guidance for Identifying Reviewers
paragraphs: According to Advocacy officials, peer review is the main quality control  over the research it disseminates. Advocacy’s current Chief Counsel  recently directed the office to strengthen its peer review process with the  intent to make it more rigorous and consistent. As a result, during the  course of our review, Advocacy finalized a written peer review process.  We found that the written guidance discussed the various levels of review  for internal and external products as well as a process for initiating peer  review. However, it did not specify how the economists who managed the  research products were to identify peer reviewers. Instead, Advocacy  officials told us that they relied on their own expertise and professional  contacts to identify appropriate peer reviewers and provide  recommendations through the Director of Research and to the Chief  Counsel for Advocacy. The officials told us that within a specific subject  matter there is often a small group of available peer reviewers, in part  because their expertise is specialized and there are not many other  alternatives.
OMB’s peer review guidance calls on agencies to select peer reviewers  with the appropriate knowledge and expertise and to take into account  their independence and lack of conflicts of interest. Advocacy managers  told us that, in practice, the economists recommend peer reviewers based  on knowledge and experience in both subject matter and databases—as  discussed in the OMB guidelines. However, they did not provide specific  written guidance to the economists on how to identify peer reviewers.  Federal internal control standards state that internal control activities help  ensure that management’s directives are carried out and in implementing  those standards management is responsible for developing the detailed  policies, procedures and practices for their agency’s operations. With  additional guidance, Advocacy would be in a better position to help  ensure that the economists fully understand how best to identify qualified  peer reviewers and carry out the Chief Counsel’s directive to improve the  peer review process.
section_title: Advocacy Has Not Consistently Documented Peer Reviews
paragraphs: Advocacy does not have consistent documentation showing whether a  peer review occurred for all of its research products. Our review of 20  recent research products—10 internal and 10 external—revealed that 16  did not have documented peer reviews in the research files. According  to interviews with Advocacy economists who managed the research, all of  the 20 products underwent some form of peer review. The economists  said that the type of review was commensurate with the methodological  complexity of each product, among other factors. However, the Advocacy  officials were unable to produce any documentation that peer reviews  occurred for these 16 products. According to Advocacy’s peer review  guidance, the economists who manage the research should document all  correspondence pertaining to the peer review and maintain this  documentation in the research file. In addition, federal internal control  standards require all transactions and significant events be documented  and that the documentation be readily available.
Advocacy officials do not have procedures to review the external research  files to ensure that the peer reviews were documented. Furthermore, they  noted that for some of the less in-depth internal research products— typically 2 to 5 pages—such documentation would be administratively  burdensome. However, we note that the documentation could be  likewise concise, such as a checklist, or a form that reviewers sign,  similar to the one currently used by Advocacy for its internal clearance  process. Otherwise, absent written documentation, Advocacy managers  are limited in their ability to conduct oversight and ensure that this key  quality control activity is happening. For example, for one of the internal  research products we selected for review, the study author no longer  worked at Advocacy, and therefore no one could tell us with any certainty  whether the required peer review had occurred, or who had participated  in it. Without adequate documentation—a key internal control—of its peer  reviews, Advocacy does not have an institutional record of its activities  and cannot demonstrate that it is following its own peer review process.
In addition, Advocacy has not consistently documented how peer  reviewers’ comments were addressed by the authors of its external  research products. Of the 10 external research files we reviewed, 4 had  documentation that a peer review occurred, and 1 file included evidence  that the peer reviewer comments were incorporated into the final report.  Advocacy officials told us that the economists who managed external  research consolidated the peer reviewers’ comments and forwarded to  the author those that needed to be addressed, including methodological  and data issues and other comments, but not those that might change the  scope of the contracted research. They also told us that they did not  typically maintain documentation showing which comments had been  addressed and why, but included the final report in the research file.  However, Advocacy’s peer review process states that the economist  managing the research will analyze and incorporate, as needed, peer  reviewers’ suggestions and maintain all related documentation in the  research file. However, since the economists are not keeping records and  documenting that comments have been considered and addressed,  management does not have an institutional record to provide reasonable  assurance that its quality control process is being followed.
section_title: Advocacy Has Not Followed Some Required Elements of Information Quality Guidelines
paragraphs: While Advocacy has quality review policies for its peer review process, it  does not have policies and procedures that reflect the federal information  quality guidelines on retaining data for influential studies or taking other  steps to substantiate the quality of information in such studies when they  have not retained the data. Advocacy officials told us that they did not  retain the original data or underlying computer codes for three external  studies on the costs of regulation as required by the information quality  guidelines. We focused on external studies on estimating the costs of  regulation because it is a key research area for Advocacy, according to its  originating statute and the mission statement of its Office of Economic  Research. The OMB Information Quality Guidelines require that all  agencies producing and disseminating “influential statistical information”  help ensure a high degree of transparency about data and methods to  facilitate the reproducibility of such information by qualified third parties.  The SBA guidelines implement this standard for transparency by requiring  that the underlying data be stored and made available for public review  for as long as the agency-disseminated information based on the data are  valid. The guidelines also state that all formulas, calculations, matrixes,  and assumptions used in processing the data should be available.  Because Advocacy classified two of the regulatory cost studies as  “influential” according to the OMB guidelines, those data should have  been maintained.
Advocacy officials said that they did not maintain the data or models for  influential external research because there might be a cost associated  with obtaining such data, which would raise the costs of the studies,  possibly making them prohibitively expensive. However, in the case of  two of the studies, the original data were from publicly available sources  and involved a relatively small dataset, suggesting the cost would not  have been prohibitive. The OMB guidelines state that sufficient  transparency—achieved in part by storing the relevant data—results in  analyses that can be substantially reproduced. Not retaining the  underlying information for these influential research papers makes it  much more difficult to assess the quality of that work, including its  objectivity—a key goal of the information quality guidelines.
We also found that Advocacy staff had not taken additional steps, in the  absence of retaining the underlying data, to substantiate the quality of the  regulatory cost estimates in two of the studies that it sponsored and  disseminated. The OMB guidelines state that when data and methods  are not retained and made available to the public because of other  compelling interests such as privacy, trade secrets, intellectual property,  and other confidentiality protections, the agency shall apply rigorous  checks to the analytical results and document what checks were  undertaken. Because Advocacy had not retained data on the two cost  estimation studies that had been criticized, we interviewed senior  Advocacy officials, including the Director of its Office of Economic  Research, about the information and methodologies used in the studies.  We asked them a set of questions related to criticisms of the  methodologies, data, and models used in the studies that were identified  in our evaluation and the work of other researchers. Advocacy staff  declined to answer many of our questions and instead directed us to the  authors, stating that they, not Advocacy economists, were the experts on  the issues covered in the studies. However, the authors would not  speak with us, stating that they were no longer contractually obligated to  respond to our requests for information.
During our discussions with Advocacy officials, they stated that the  purpose of the studies was not to estimate the overall costs of  regulations, but rather to estimate the disproportionate share borne by  small businesses. They also noted that, as with all contract research, the  external research reports contain a disclaimer indicating that the views  presented did not necessarily represent those of Advocacy. In addition,  Advocacy added language to the 2010 report’s cover page about the  uncertainty of the authors’ estimates of the costs of regulation.  Advocacy officials noted that the majority of the research it conducts is  not classified “influential” according to the OMB and SBA guidelines, and  that they have no plans to engage in such work in the near future.  However, given Advocacy’s mission, it may do influential research in the  future as it has in the past even if on a limited basis. We acknowledge  that these reports may not necessarily be representative of all Advocacy’s  research efforts, but not substantiating the quality of the information in  even one study could call into question the credibility of Advocacy’s  research program. Thus, establishing policies and procedures that reflect  the federal information quality guidelines on retaining data for influential  studies or when such data are not retained because of certain compelling  interests, and requiring additional steps to substantiate the quality of the  information, would put Advocacy in a better position to provide  reasonable assurance about the quality of its research program.
OMB Information Quality Guidelines require that agencies develop  policies to ensure that managers be able to substantiate the quality of  information they disseminate. The guidelines also discuss narrow  circumstances under which an agency does not have to substantiate the  quality of the information that resulted from a research project of one of its  contractors or grantees. In those circumstances, the researcher is to  make clear with an appropriate disclaimer that the views expressed in the  research are his or her own and do not necessarily reflect those of the  agency. However, OMB cautions that “if an agency, as an institution,  disseminates information prepared by an outside party in a manner that  reasonably suggests that the agency agrees with the information, this  appearance of having the information represent agency views makes  agency dissemination of the information subject to these guidelines.”
As we noted previously, Advocacy placed a disclaimer on the two studies,  but its actions indicate its agreement with the information in the studies.  First, Advocacy made the two studies available on its website, where they  continue to be available. Second, Advocacy’s disclaimer that appeared on  the 2010 study noted that it contained information and analysis that was  reviewed and edited by Advocacy. Advocacy’s description of its review  role suggests that the agency contributed to the content, if not the  conclusions of the study. Finally, several Advocacy comment letters have  cited the 2005 study’s regulatory cost estimates in support of their  arguments. Because Advocacy’s actions raise, at the least, an  appearance of agreement with the information contained in the studies,  we believe that the office was required to substantiate the quality of the  estimates of the economic costs contained in the studies.
section_title: Documentation of Key Regulatory Activities is Inconsistent and Roundtables Are Not Publicized According to Advocacy’s Policies
paragraphs: Advocacy has practices and procedures (“policies”) for its regulatory  activities—comment letters and roundtables—however, documentation of  these key regulatory activities is inconsistent. Furthermore, while we  determined that transparency and other requirements in FACA do not  apply to Advocacy’s roundtables, Advocacy is not following its internal  policies meant to ensure its roundtables are as open to the public as they  could be. As a result Advocacy cannot demonstrate that it is always fully  meeting its mission to foster two-way communication between small  businesses and federal policymakers.
subsections: section_title: Advocacy’s Documentation for Key Regulatory Activities Is Inconsistent
paragraphs: Our review of Advocacy’s key regulatory activities—developing and  issuing comment letters and convening information roundtables—found  that Advocacy staff are inconsistently maintaining documentation of key  decisions and events. In April 2014, Advocacy provided us with a  “practices and procedures guide” for the Office of Interagency Affairs,  dated March 2014. The guide covers staff activities related to initiating  comment letters and convening roundtables, among other things.  Advocacy management told us that they update the guidance periodically,  including amending it during the course of our review to make it clear that  the decision to issue a public letter or hold a roundtable rests with the  Chief Counsel for Advocacy. However, our review found that the updated  guide continued to lack policies for documenting key decisions and  activities. For example, the guide stated that when regulatory attorneys  decide to intervene in the rulemaking process, they must follow up, as  appropriate, with the interested associations to ensure that Advocacy has  sufficient information and data to make its case for intervening. But there  is no policy to document these interactions. The following are other  instances where we found a lack of policies to document key activities.
Small Business Input into Comment Letters. Advocacy does not have  any policies requiring that the regulatory attorneys retain  documentation showing which entities provided input into comment  letters, and we found that the attorneys do not consistently do so. Our  analysis of Advocacy’s comment letters found that about 57 percent of  the letters referenced small business input or concerns. But when we  interviewed seven regulatory attorneys about how they developed  comment letters generally, some told us that they maintained a record  of the entities providing input for specific letters, but others said that  they did not. Further, when we asked Advocacy staff for  documentation on the sources of the small business input referred to  in a nonrepresentative sample of 11 comment letters, they were  unable to provide specific emails or notes of conversations.
Reasons for Convening Roundtables. Advocacy’s practices and  procedures guide states that the Chief Counsel must approve the  proposed agenda, speakers, and discussion topics for all proposed  roundtables before participants are invited. But the guide contains no  policy that this step be documented. We interviewed seven regulatory  attorneys responsible for the roundtables and their management and  were told that the attorneys set agendas and selected speakers based  on their own assessment of the issues and, in some cases, with  suggestions from small businesses or interested industry parties.  Advocacy officials emphasized that they only held roundtables when  there is sufficient interest or need on the part of small businesses.
Roundtable Discussions and Participants. Information gathered from  the roundtables is used to inform Advocacy’s positions on issues  related to small businesses and in comment letters, but Advocacy’s  guidance contains no policies to document roundtable discussions.  Our content analysis of Advocacy’s comment letters showed 19  percent of them referred to roundtable discussions. However, staff did  not routinely take and maintain notes of the discussions, according to  the interviews we conducted with the regulatory attorneys. In addition,  not all staff take attendance at the events. In most cases, Advocacy  staff keep an “RSVP list” of those who have indicated that they will  attend. However, some of the staff noted that the RSVP lists may not  include all participants, including those on the phone. As a result it is  difficult to determine the extent to which small businesses and related  entities were represented at these events.
Advocacy officials told us that they did not have guidance on maintaining  documentation on the sources of input to comment letters or on  roundtable discussions because they did not feel that setting such  standards was required to fulfill their duties under Advocacy’s statute.  Furthermore, they noted that, in the case of roundtables, keeping records  in a manner that identified specific speakers would inhibit discussion and  limit their ability to gain valuable input. They also cited logistical  challenges in taking accurate attendance at larger events. We  acknowledge that specific parties might not want to be publicly named,  but federal standards for internal control call for agencies to document  significant events. Taking steps to balance the need for privacy so  individuals can speak freely, with a commitment to maintain a basic level  of documentation of these events—that is, documenting which entities  provided input into its comment letters and roundtables—could help  Advocacy demonstrate that it is meeting its mission to represent the  interests of small businesses.
section_title: Advocacy’s Policy for Publicizing Information on Roundtable Events Is Not Followed
paragraphs: Key documents for Advocacy’s roundtables—agendas and presentation  materials—are not made available to the public at-large after the fact. The  regulatory attorneys cultivate email lists of relevant stakeholders who are  invited to roundtable events, and these lists are continually updated. It is  Advocacy’s policy to add any interested parties to the invitation list, if  asked, and several of the attorneys we interviewed said they did so. They  also told us they made agendas and presentations available to any  interested parties who requested them after the roundtable. However, the  agendas and presentations are not posted to the website or made  publically available in any systematic way and if small businesses and  other interested parties do not know about the roundtables, they cannot  request information from the events.
Advocacy’s policies and procedures state that agendas and presentations  should be posted on Advocacy’s website. Advocacy officials stated they  had been unable to post roundtable materials to the website, which SBA  maintains, because of difficulties in meeting certain readability and  accessibility requirements in the Americans with Disabilities Act.  However, a variety of other Advocacy reports and publications are posted  on its website that also must meet these requirements, and Advocacy  officials did not explain why information on the roundtables could not be  likewise included. Making the roundtable materials available on its  website would strengthen Advocacy’s efforts to inform small businesses  and other interested parties about its efforts to represent their interests to  federal decision makers.
section_title: Federal Advisory Committee Act Requirements Do Not Apply to Advocacy’s Roundtables
paragraphs: In addition to evaluating whether Advocacy publicizes and conducts its  roundtables in accordance with its internal policies and procedures, we  also analyzed whether Advocacy’s roundtable groups constitute “advisory  committees” subject to the public notice and other transparency  requirements of FACA. We found that the roundtables are not advisory  committees, and thus Advocacy is not required by law to follow these  rules. We first determined that Advocacy is an “agency” covered by  FACA. It is an “authority of the Government of the United States, whether  or not it is within or subject to review by another agency.” In addition,  Advocacy possesses the type of “substantial independent authority”  required by the courts. While located within SBA, Advocacy is  independent of SBA, and it has distinct statutory authorities and  responsibilities, a separate statutory charter, and an appropriation  account that is separate from the rest of SBA. Furthermore, its duties and  authorities under statute and Executive Order are substantial and its  recommendations made in furtherance of small entities’ concerns must,  by law, be given considerable weight by other agencies.
While we found that Advocacy is a FACA “agency,” we concluded that its  roundtable groups are not “advisory committees” as defined by the statute  and interpreted by the courts and implementing regulations. A covered  advisory committee is a panel, task force, or similar group created or  used by an agency for the purpose of providing advice or  recommendations on particular matters. Participants’ input must be  sought as a group, not as a collection of individuals. The formality and  fixed structure of the group also is an important factor in determining  coverage under FACA. Advocacy does not seek roundtable participants’  input as a group, however; rather, attendees provide their individual  perspectives on the agency rule or policy under discussion, do not  significantly interact with one another, and no attempt is made to reach a  consensus. Advocacy’s roundtables also do not have the requisite  organized or fixed structure; rather, as noted, Advocacy’s policy is to  extend roundtable invitations to anyone who expresses a desire to attend.  Finally, according to Advocacy, it is the agency itself, not the roundtable  participants, that develops policy advice and recommendations, albeit  based in part on the data and information provided by roundtable  participants (as well as obtained elsewhere). Our detailed legal analysis  of the applicability of FACA to Advocacy’s roundtables is contained in  appendix III.
section_title: Advocacy’s Workforce Planning Efforts Lack Succession Planning
paragraphs: Workforce planning presents some challenges for Advocacy, in part  because the office is small and has a large number of positions, which,  according to Advocacy officials, typically are replaced with each new  administration. While the office currently does some workforce planning,  its efforts do not include the long-term strategic plans that would help  ensure that it maintains the expertise and skills needed to fulfill its  mission. While Advocacy now has responsibility for developing its own  budget, goals, and performance measures, our review of its strategic plan  and goals did not find goals or objectives that discussed workforce issues  such as staff development and succession planning. Advocacy officials  told us that they used their organizational chart as the basis for their  workforce planning efforts and discussed workforce issues with staff and  managers at meetings as needed (see fig. 2). Advocacy officials said that  because Advocacy was a small office with only 46 staff, the  organizational chart and meetings to discuss staffing issues met their  workforce planning needs. Advocacy officials told us that to develop staff  they provided training to regulatory attorney and economist staff, had  established a mentoring program for newer staff, and were implementing  a knowledge-sharing database to help staff develop expertise in  specialized policy areas.
When we asked Advocacy officials about succession planning and their  plans for addressing the departure of senior or experienced staff, the  officials told us that most of the turnover in staff occurred during changes  in administration. For example, according to Advocacy officials, the  change of administration in 2009 resulted in the turnover of at least 15 of  20 senior officials. Among these were the 10 regional advocates, who are  located in each of SBA’s regional offices across the country, and make up  the Office of Regional Affairs. Turnover tends to be low among the  economists and regulatory attorneys, some of whom have been with the  office for years. Advocacy officials said that because rulemaking often  took so long, they generally had enough time to hire or realign any staff  as needed.
According to federal internal control standards, effective management of  a workforce is essential to achieving program results. Our body of work  on workforce planning has demonstrated the importance of such planning  and the need to develop long-term strategies—such as training and  succession planning—for acquiring, developing, and retaining staff to  achieve programmatic goals. OPM’s Human Capital Assessment and  Accountability Framework states that agencies and offices with workforce  planning are better able to manage their staff by, for example, ensuring  that systematic processes are in place for identifying and addressing any  gaps between current and future workforce needs. Further, OPM  recommends that succession planning is needed to ensure continuity in  leadership positions. In addition, workforce planning can help  management determine the type of training and other strategies that are  needed to address factors such as projected retirements and succession  planning. In past work on succession planning, we have found that in  addition to focusing on replacing individuals, succession planning  strategies can also strengthen current and future organizational  capacity.
Given the length of time some attorneys and economists have been with  Advocacy, the loss of their expertise through retirement, among other  things, could leave significant gaps in needed skills and knowledge,  according to Advocacy officials. Yet Advocacy has not incorporated  succession planning into its workforce planning efforts, such as its  training and mentoring initiatives. Further, Advocacy officials told us that  they discussed future staffing needs and various options for addressing  them, including training, but these efforts are not documented in a  manner that would ensure consistent implementation. However, ensuring  that activities such as staff training are consistently implemented is  particularly important when senior management can change significantly  every 4 or 8 years. Without incorporating long-term succession planning  into its workforce planning efforts, Advocacy is not in the best position to  ensure that it has qualified staff to fill leadership and other key positions  and a skilled workforce able to meet the demands of its mission on behalf  of small businesses.
section_title: Conclusions
paragraphs: Effective internal controls are critical for the Office of Advocacy if it is to  achieve program outcomes and minimize operational problems.  Recognizing this, Advocacy recently has taken action to improve some of  its guidance and controls. However, Advocacy’s research, regulatory, and  workforce planning functions could be improved by strengthening its  internal controls as follows:
Research activities. In its research operations, Advocacy has taken  some initial steps toward establishing stronger control policies by, for  example, formalizing its peer review process for internal and external  research products. However, the guidance provided did not include  information on how to select appropriate peer reviewers—the experts  whom Advocacy relies on to assess the quality of the research that it  disseminates. The guidance also lacks policies for documenting that a  peer review occurred and how reviewer comments were addressed  for its external research products. Such guidance would help ensure  that Advocacy research staff fully understand how best to identify the  most qualified peer reviewers, and how to document and incorporate  reviewer comments. In addition, Advocacy did not follow federal  guidelines for information quality for influential studies that set out  requirements for retaining data and taking additional steps to  substantiate the quality of information it disseminates when it has  certain compelling reasons to not retain the data. Improving policies  and procedures for its research activities would help support  Advocacy’s mission to provide quality research on small business  issues that decision makers and the public can rely on.
Regulatory activities. Advocacy’s lack of documentation and  transparency of the regulatory activities we reviewed made it difficult  to validate Advocacy’s efforts to represent small businesses.
Specifically, weaknesses exist in Advocacy’s documentation of both  the sources of the small business input in comment letters and the  views of small businesses discussed and conveyed at its roundtables.  As a result, the extent to which the small businesses’ views on  regulations were being obtained and communicated is unknown.  Improving its guidance to staff on its regulatory activities and  emphasizing the importance of documentation would enable  Advocacy to more effectively demonstrate to decision makers that it  was obtaining and communicating the interests of small businesses.  In addition, Advocacy is not following its internal policy to post  materials from roundtables on its website. As a result, it is missing an  opportunity to reach out more broadly to small businesses and other  interested parties and increase the transparency of its activities.
Workforce planning activities. While Advocacy’s workforce planning  efforts help in managing its current staff, these efforts do not include  any strategies to plan for succession, even though several staff have  been with the office for many years and will eventually need to be  replaced. Without this important element of workforce planning,  Advocacy could be in a vulnerable position when critical staff leave  the agency or staff face new demands. Although Advocacy is a  relatively small office, having a skilled workforce is critical to meeting  its mission. Succession planning would help ensure that Advocacy  was better prepared to maintain qualified staff to conduct its research  and regulatory activities in support of small businesses.
section_title: Recommendations for Executive Action
paragraphs: To improve Advocacy’s system of internal control, and help to provide  reasonable assurance that the office is meeting its mission, we  recommend that the Chief Counsel of Advocacy take the following five  actions:
Strengthen the accountability of its research activities by taking the  following two actions:
Enhance its peer review policies and procedures by including  written guidance on selecting peer reviewers and documenting all  steps of the peer review process—whether a peer review occurred  and how and to what extent peer reviewer comments were  addressed.
Develop policies and procedures that reflect the federal  information quality guidelines on retaining data for influential  studies, and when not retaining data, taking additional steps to  substantiate the quality of information disseminated.
Strengthen the accountability of its regulatory activities by developing  policies and procedures to ensure that key elements of that work— such as sources of input for comment letters and roundtable  discussions—are consistently documented.
Coordinate with SBA officials who oversee website administration to  comply with Advocacy’s roundtable policy to make information on the  events—agendas, presentation materials—publicly available on its  website so that its regulatory activities are more transparent to the  public.  Improve its workforce planning efforts to be better prepared to meet  its future workforce needs by incorporating succession planning.
section_title: Agency Comments
paragraphs: We provided a draft of this report to Office of Advocacy for its review and  comment. In its written comments (reproduced in app. II), Advocacy  agreed with our recommendations. Advocacy stated that for its research  activities, its current effort to further formalize procedures for the peer  review process will include steps for additional documentation and that it  also plans to develop written guidelines for determining which research  products are considered influential, which will clarify when Advocacy  needs to take additional information quality steps. Advocacy also said that  as it develops how it will disseminate information about its regulatory  activities, it will incorporate approaches that are responsive to our  recommendations. Finally, Advocacy agreed that workforce planning is  important for ensuring that the office maintains the skills and resources  needed to fulfill its mission and noted that the office is developing a  Leadership Succession Plan in response to our recommendation.
We are sending this report to the Office of Advocacy and interested  congressional committees. The report also will be available at no charge  on the GAO website at http://www.gao.gov. If you or your staff have any  questions about this report, please contact Cindy Brown Barnes at (202)  512-8678 or brownbarnesc@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report examines Advocacy’s (1) research activities; (2) regulatory  activities, including the applicability of the Federal Advisory Committee  Act (FACA) to Advocacy’s roundtables; and (3) workforce planning  efforts. The scope of our work covered key activities conducted from  fiscal year 2008 to fiscal year 2013, although in some cases, we  broadened that scope to include fiscal years 2005 through 2008, as noted  below.
To understand Advocacy’s mission, operations, and participation in the  rulemaking process, we reviewed relevant laws and regulations,  Advocacy documents, including its annual reports to Congress on the  Regulatory Flexibility Act (RFA), budget and strategic planning  documents, and other publications.
To assess Advocacy’s research activities, we determined: how Advocacy  staff chose its research topics; how they conducted the research; and  what controls they had in place to ensure quality products. We analyzed  Advocacy’s research products produced in the 5 most recent fiscal  years—2008 through 2012—and interviewed relevant Advocacy officials  on the processes by which they assess the quality of its research  products. To evaluate Advocacy’s research activities, we analyzed a  nonprobability sample of research, which included 10 research products  authored by Advocacy staff (internal), and 10 products produced by its  contractors (external). To select a sample of internal research products  that best represented the variety of internal research, we reviewed the 58  research products issued from fiscal years 2008 through 2012, and  categorized each as either “routine” (meaning published either quarterly  or semi-annually, and relying on the same underlying data set), or as  “nonroutine.” Within the routine category, we selected the mostly recently  published product from each of the following routine publications: (1)  Small Business Profiles of the States and Territories, (2) Small Business  Quarterly Bulletin, (3) Quarterly Lending Bulletin, (4) Small Business  Lending, and (5) The Small Business Economy. For the nonroutine  products, we created five categories, such as “analysis” and “fact sheets.”  We selected, from each of those five categories, the most recently  published report. Then, for each of the 10 selected internal research  products, we used a data collection instrument and interviewed the  Advocacy staff who wrote the products in order to see to what extent the  products adhered to Information Quality Guidelines issued by the Office  of Management and Budget (OMB) and the Small Business  Administration (SBA) and Advocacy’s internal quality review process  (“peer review”). To assess the quality of the peer review used for  Advocacy’s external research, we reviewed the contract files for the 10  most recently published external research products, as of year-end 2013,  and compared how the work was produced, including Advocacy’s peer  review process, against the OMB Peer Review Guidance and OMB and  SBA information quality guidelines, and applicable federal internal control  standards. The results from our reviews cannot be projected to all  Advocacy studies, but they provide an indication of how Advocacy staff  conducted or oversaw the research and what controls were in place to  ensure quality products.
We also assessed Advocacy’s contract studies that focused on the  economic costs of regulation. We focused on external studies in these  areas because they are key research areas for Advocacy, according to its  originating statute and the mission statement of its Office of Economic  Research. For the economic costs studies, we assessed them against  OMB and SBA Information Quality Guidelines, and examined Advocacy’s  compliance with related data retention policies therein. We also reviewed  peer reviewers’ comments and other external reviews on the studies as  part of our assessment. Lastly, we interviewed Advocacy officials to  obtain information on steps they took to substantiate the quality of  information for influential studies when data are not retained.
To evaluate Advocacy’s regulatory activities, we assessed: how and why  Advocacy decides to issue comment letters and convene roundtables; the  policies and practices in place that pertain to those activities; how  Advocacy staff solicit input from small businesses and other parties. We  interviewed relevant staff, as described below; reviewed relevant policies  and procedures, and analyzed comment letters produced and  roundtables convened from fiscal years 2009 through 2013. Specifically,  we performed a content analysis on the 181 comment letters issued  during that period, using NVivo software, to analyzed and categorize the  content of the letters and the nature and source of the small business  input provided. We also interviewed Advocacy staff who are responsible  for rulemaking and related activities, weighting our choices of  interviewees to reflect the distribution of comment letters by policy area,  and reviewed supporting documentation for comment letters to  understand how Advocacy staff develop and issue comment letters, and  compared this information to federal internal control standards.
Similarly, we reviewed available information on the 142 roundtables  convened during the same 5-year period and interviewed the responsible  Advocacy staff, weighting our choices of interviewees to reflect the  distribution of roundtables by policy area. We reviewed Advocacy’s  policies for roundtables to understand their origin, purpose, and  documentation requirements, and compared this information to federal  internal control standards. In order to understand the perspective of those  who attended the roundtables, we attended three of the events. In  addition, we selected and interviewed a nonprobability sample of past  participants from email lists provided by Advocacy’s regulatory attorneys.  Specifically, we talked to the following representatives: five from industry  associations that represent small and large businesses; one from a large  corporation; and one nonprofit organization whose mission relates to  policies or rules under consideration. We made our selections to include  representatives from a variety of sources, and while the results from our  interviews cannot be projected to all entities that interact with Advocacy,  the information we gathered does provide insights into how the selected  groups view Advocacy and its work in representing small businesses to  federal policymakers.
We also reviewed Advocacy’s training to agency staff in how to comply  with the RFA and interactions with agency rulemaking officials. In  addition, we interviewed officials from the entities that interact with  Advocacy in rulemaking—the Environmental Protection Agency (EPA),  Occupational Safety and Health Administration (OSHA), Bureau of  Consumer Financial Protection, also known as the Consumer Financial  Protection Bureau (CFPB), and Office of Management and Budget’s  (OMB) Office of Information and Regulatory Affairs (OIRA).
To analyze whether FACA (5 U.S.C. App. II) applies to Advocacy’s  roundtables, we reviewed relevant statutes, case law, regulations, and  guidance. In addition, we reviewed and considered Advocacy’s written  views on the issue. See appendix III for more information on the legal  analysis we conducted.
Finally, we reviewed Advocacy’s workforce efforts. To understand  Advocacy’s workforce planning, we reviewed Advocacy’s strategic goals  and other planning and budget documents and interviewed senior  management to determine what, if any, policies and procedures were in  place related to workforce and succession planning. We assessed any  such policies and procedures against applicable federal standards for  internal control and the Office of Personnel Management’s (OPM) Human  Capital Assessment and Accountability Framework (HCAAF). We also  reviewed GAO reports on workforce and succession planning to gain  insights about key practices and how agencies have used them.
We conducted this performance audit from August 2013 to July 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Office of Advocacy, Small Business Administration
section_title: Appendix III: Legal Analysis of the Applicability of the Federal Advisory Committee Act to the Office of Advocacy’s Roundtables
subsections: section_title: Conclusions
paragraphs: Advocacy is an “agency” under FACA first because it meets the statutory  definition of this term in FACA and the Administrative Procedure Act  incorporated by reference: it is an “authority of the Government of the  United States, whether or not it is within or subject to review by another  agency” and not subject to statutory exemption. In addition, Advocacy has  the type of “substantial independent authority” and other indicia of  “agency” status that courts interpreting the APA definition have required.  However, Advocacy’s roundtable groups are not FACA “advisory  committees” because they do not have an organized or fixed structure;  there is no attempt to reach consensus about an agency regulation or  policy but instead viewpoints are sought from individual participants; and  it is only data and information, according to Advocacy, rather than advice  or policy recommendations, that participants provide.
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Cindy Brown Barnes (202-512-8678), brownbarnesc@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Kay Kuhlman and Triana McNeil  (Assistant Directors), Kristeen McLain (Analyst-in-Charge), Emily  Chalmers, William Chatlos, John McGrail, Lauren Nunnally, Roberto  Pinero, Jena Sinkfield, Jack Wang, and Carrie Watkins made key  contributions to this report.