Agricultural Promotion Programs: USDA Could Build on Existing Efforts to Further Strengthen Its Oversight
Released: 2017-11-21T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: “Got milk?” and “Pork: The Other White Meat” are examples of advertising campaigns undertaken by 2 of the 22 federal agricultural research and promotion programs, commonly known as commodity check-off programs. These programs, funded by a fraction of the sale of each unit of a commodity, are led by boards consisting of industry members appointed by the Secretary of Agriculture. The programs conduct research and promotion activities to strengthen a commodity's position in the market. In 2016, check-off funds totaled over $885 million. By law, funds cannot be used for lobbying or disparaging other commodities, among other things. AMS has primary responsibility for overseeing the check-off programs.

GAO was asked to review AMS's oversight of the check-off programs. This report examines (1) the extent to which AMS has addressed previously identified weaknesses in its oversight and (2) how the effectiveness of the programs has been evaluated and what the results have indicated. GAO selected a sample of 8 such programs—selected, in part, based on total funds collected—and reviewed laws, regulations, and agency guidance. GAO interviewed agency officials, check-off board executives, and economists.

section_title: What GAO Found
paragraphs: The U.S. Department of Agriculture's (USDA) Agricultural Marketing Service (AMS) has improved its oversight of check-off programs since USDA's Office of Inspector General (OIG) made recommendations in a 2012 report. In response to two OIG recommendations, AMS developed and implemented standard operating procedures, which outline specific oversight responsibilities of AMS, and began to conduct internal reviews of its oversight functions. However, GAO found that AMS does not consistently review subcontracts—a legal agreement between a contractor and third party—or ensure that certain documents are shared with stakeholders on program websites.

Subcontracts. Under AMS's 2015 guidelines for check-off programs, which cover broad oversight activities, staff are to review a sample of subcontracts during agency reviews of program operations. However, AMS did not revise its standard operating procedures to match its guidelines with this responsibility, and GAO found that AMS reviewed subcontracts for only one check-off program in its sample of eight. Without revising the standard operating procedures to include a review of subcontracts, AMS's ability to prevent misuse of funds is impaired.

Transparency. According to leading business principles, transparency is central to stakeholders' access to regular, reliable, and comparable information. However, GAO found that four of the eight check-off programs reviewed posted all key documents, such as budget summaries and evaluations of effectiveness, to program websites. GAO found that AMS's guidelines state that budget summaries should be posted on program websites, while the other key documents are to be available on the website or otherwise made available to stakeholders. Agency officials said that boards would supply documentation if contacted by a stakeholder. Industry representatives GAO interviewed said that transparency of how funds are used and the effectiveness of programs are important to their members. Without including in its guidelines and standard operating procedures that all key documents should be posted on a check-off program's website, AMS may miss an opportunity to ensure that stakeholders have access to information on program operations and effectiveness.

Independent economic evaluations of the effectiveness of check-off programs, required by law to be conducted every 5 years, have generally shown positive financial benefits. For the eight evaluations GAO reviewed, benefits ranged from an average of $2.14 to $17.40 for every dollar invested in the programs. However, the evaluations varied in the methods used and had certain methodological limitations. For example, some evaluations did not account for the effects of promotion from competing commodities, which could overstate the programs' benefits. AMS's standard operating procedures state that the agency should review the evaluations to ensure that there is a credible methodology, among other things; however, AMS did not consistently document reviews of the evaluations or have criteria by which to review the evaluations. Without developing criteria to assess the methodology and results of evaluations, the agency's assessments of independent economic evaluations may be inconsistent across check-off programs and misleading to stakeholders.

section_title: What GAO Recommends
paragraphs: GAO is making five recommendations, including that USDA revise its standard operating procedures to include the review of subcontracts, include key documents on check-off program websites, and develop criteria to assess evaluations. USDA generally agreed with GAO's recommendations.

section_title: Letter
paragraphs: “Got milk?” “Pork: The Other White Meat”. “The Incredible, Edible Egg”.  These are three examples of advertising campaigns for agricultural  commodities undertaken through research and promotion programs,  commonly known as commodity check-off programs. At present, there are  22 commodity check-off programs authorized by the Secretary of  Agriculture through orders issued after public notice and comment. The  Secretary issues these orders under the authority provided in legislation  addressing specific agricultural commodities or under general authority  provided in the Commodity Promotion, Research and Information Act of  1996. As stated in the general act, these programs were formed to  strengthen a commodity’s position in the marketplace through the  development and implementation of promotion, research, and information  programs. The U.S. Department of Agriculture’s (USDA) Agricultural  Marketing Service (AMS) has primary oversight responsibility for ensuring  that check-off boards comply with legislative and regulatory requirements.

The term check-off refers to the way the research and promotion  programs are funded—an assessment is paid by producers, handlers,  processors, importers, or others in the marketing chain for each unit of a  commodity sold, produced, or imported. The programs are principally  funded by this assessment and do not receive federal appropriations. In  2016, program assessments totaled over $885 million for these federally  authorized check-off programs.

Each check-off program is operated by a board, such as the American  Egg Board or the Cattlemen’s Beef Board, whose members, for the most  part, are appointed by the Secretary of Agriculture from nominations  submitted by industry. Check-off board staff are primarily responsible for  the operations of the board and interacts daily with AMS representatives.

The check-off boards conduct activities to increase overall demand for the  commodities; expand markets in the United States and abroad; and  expand the commodities’ uses through research, promotion, and  consumer and industry information.

Among other things, check-off boards are explicitly prohibited from (1)  engaging in any action that could be a conflict of interest; (2) using  assessed funds to influence any legislation or governmental action or  policy; and (3) promoting any advertising that may be false, misleading, or  disparaging to another agricultural commodity. AMS has primary  oversight responsibility for ensuring that the boards follow these  prohibitions and abstain from these activities. In 2012, USDA’s Office of  Inspector General (OIG) found that AMS needed to improve certain areas  of its oversight of check-off board operations.

You asked us to review AMS’s oversight of check-off programs. This  report examines (1) the extent to which AMS has taken steps to address  previously identified weaknesses in its oversight of check-off programs  and (2) how, if at all, the effectiveness of the check-off programs has  been evaluated and what the results of those evaluations have indicated.

To perform this work, we reviewed statutes and regulations related to  check-off programs. In addition, we reviewed AMS guidance for check-off  programs, including the programs’ guidelines and standard operating  procedures. We also reviewed OIG reports on check-off program  oversight. For each of the objectives, we selected a nonprobability  sample of 8 of the 22 check-off programs. We selected these programs  to get a range of information based on the following criteria: (1) AMS  commodity area, (2) amount of total assessments collected, and (3)  whether each program was created under its own stand-alone legislation  or the Commodity Promotion, Research and Information Act of 1996. We  interviewed senior agency officials from each of the four AMS commodity  areas in which the 8 programs reside as well as the marketing specialists  who oversee the 8 programs. We also interviewed executives from the  check-off boards of the 8 programs in our sample. In addition, we  interviewed representatives from five industry organizations to discuss  information that they receive related to check-off programs and their  organizations’ views on any evaluations of check-off program  effectiveness. We selected organizations with members who pay check- off program assessments and who have an understanding of the check- off programs based on information found on the organization websites.

To determine AMS’s oversight activities, we reviewed relevant laws,  regulations, and agency guidance. Among the statutes we reviewed were  the Commodity Promotion, Research and Information Act of 1996 and the  various laws establishing other check-off programs along with the  regulations pertaining to the sample of 8 check-off programs. We  compiled a list of AMS’s oversight activities, and obtained documentation  from AMS for the 8 programs in our sample for fiscal years 2014 and  2015 to demonstrate whether those oversight activities were completed.  Using this documentation, a first GAO reviewer determined whether the  oversight activities were completed. A second reviewer then made his or  her own determination as to whether the activities were completed. The  first and second reviewers subsequently met to reconcile any differences.  Once the reviews were complete, we shared our findings with marketing  specialists from each of the 8 check-off programs in our sample. We held  meetings with the marketing specialists and the AMS directors in each of  the four commodity areas in which the 8 programs reside to discuss the  various oversight activities and how AMS conducts those activities.

To address the effectiveness of check-off program evaluations and their  results, a GAO economist assessed the most recent independent  economic evaluations of effectiveness for the 8 check-off programs in our  sample using GAO’s guidance for assessing economic analyses. This  included an assessment of the (1) objective and scope, (2) alternatives,  (3) analyses of effects, (4) sensitivity analyses, and (5) documentation. A  second GAO economist then reviewed this assessment. This assessment  was used to determine the various methodologies used in the check-off  program evaluations, the soundness of those methods, and any  limitations. In addition to reviewing the evaluations, we also spoke with  the economists who conducted independent evaluations of effectiveness  for our sample of 8 check-off programs to learn about their methods as  well as any challenges they may have faced in conducting the  evaluations. We also interviewed three additional economists who did not  conduct evaluations of the 8 check-off programs we assessed to get the  views of external experts. We selected these economists because they  had previously published journal articles on evaluating check-off  programs. Finally, we reviewed literature on conducting evaluations of the  effectiveness of check-off programs.

We conducted this performance audit from August 2016 to October 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Check-off programs are designed to expand the market for a given  agricultural commodity, such as eggs, pork, or highbush blueberries,  through generic promotion, research, and consumer and industry  information. A check-off program is meant to expand the demand for a  commodity rather than for any particular brand or producer. Although  some state, regional, and local check-off programs that have existed for  over 70 years may be voluntary, federal programs are mandatory. Many  commodity groups prefer mandatory programs to address the free rider  problem—that is, producers, handlers, processors, importers, or others in  the marketing chain who do not pay into a check-off program but benefit  economically from voluntary programs that others have funded.

After Congress authorized the Cotton Research and Promotion Act of  1966, the first federally mandated agricultural check-off program and  board—for cotton—was created. Over the next three decades, Congress  authorized the creation of an additional 11 commodity programs and their  respective boards. The 12 programs and boards created under the  authority of individual stand-alone legislation adhere to the specific  requirements as set forth in their respective authorizing legislation. The  passage of the Commodity Promotion, Research and Information Act of  1996 (generic legislation) gave USDA the authority to establish additional  commodity check-off programs and boards. Since then, 10 additional  boards were created based on this generic legislation. Those boards are  subject to the requirements set forth in the generic legislation. (See table  1 for the year established and authorizing legislation for all 22 check-off  programs.)

To create a check-off program, industry groups first identify the need for  such a program and then negotiate among themselves to agree on a  basic program framework. The framework includes the rate of  assessment and the various program activities to be undertaken, such as  promotion, advertising, research, and providing information to consumers  and industry. Additionally, each industry proposes regulations to USDA  for the structure of the board that will carry out these activities. Because  each industry has unique characteristics, a different board structure is  appropriate for each check-off program. The boards vary in size,  geographic representation, and types of individuals who are board  members—that is, producers, processors, handlers, importers, public  representatives, or others in the marketing chain. USDA, in consultation  with the industry, then develops regulations to define how the program  will be operated, how the funds will be collected, and how compliance  with the authorizing legislation will be maintained, among other things.  The check-off programs must be approved by a majority of producers— and in some cases processors, importers, and handler—subject to the  assessments. To gain approval, a referendum must be held either before  check-off program operations begin within some specified time after  assessments are first collected, depending on the authorizing legislation.

To fund a check-off program, producers, handlers, processors, importers,  or others in the marketing chain are assessed for each unit of the  commodity sold, produced, or imported. For example, for each 30-dozen  cases of eggs sold, a producer is assessed $0.10. These funds go to the  American Egg Board. The boards are to use assessments for the  research, promotion, and consumer and industry information activities as  well as for reimbursing AMS for its oversight costs. In 2016, total  assessments collected for the 22 check-off programs ranged from $0.6  million for the popcorn check-off program to $332.1 million for the dairy  check-off program (see table 2).
subsections: section_title: AMS’s Oversight and Past Recommendations
paragraphs: To facilitate oversight, AMS breaks the 22 check-off programs into four of  the agency’s commodity areas: (1) Cotton and Tobacco—the cotton  check-off program; (2) Dairy—the dairy and fluid milk check-off programs;  (3) Livestock, Poultry, and Seed—the beef, egg, lamb, pork, sorghum,  and soybean check-off programs; and (4) Specialty Crops—the  Christmas tree, Hass avocado, highbush blueberry, honey, mango,  mushroom, paper and packaging, peanut, popcorn, potato, processed  raspberries, softwood lumber, and watermelon programs. AMS has a  functional committee for the check-off programs, which comprises a chair  and the deputy administrators from the four AMS commodity areas and  meets quarterly. The functional committee reports to the AMS Associate  Administrator and was established to increase coordination and promote  best practices and consistency across the 22 check-off programs.  Additionally, the four commodity area directors and other senior agency  officials meet weekly to discuss any issues that have arisen and to  discuss any necessary policy changes.

AMS marketing specialists are responsible for the day-to-day oversight of  the check-off boards and for ensuring that board decisions and operations  are carried out in accordance with applicable legislation and regulations.  Each check-off program has a designated AMS marketing specialist  serving as the primary overseer of all check-off program activities. (Fig. 1  shows AMS’s oversight structure for the check-off programs.) As part of  their oversight duties, marketing specialists review and approve board  budgets, contracts, promotional activities, board policies, and bylaws,  among other activities. Every 3 years, marketing specialists also are to  conduct management reviews that assess each of the 22 check-off  boards’ internal controls intended to determine whether there is  reasonable assurance that the boards are in compliance with statutes,  regulations, and the board’s and AMS’s policies and procedures. AMS  management reviews are to include reviews of check registers, contract  and subcontract samples, assessments collected, and travel  reimbursements, among other items.

AMS’s Guidelines for AMS Oversight of Commodity Research and  Promotion Programs, most recently updated in September 2015, is  designed to facilitate the application of legislative and regulatory  provisions of the check-off programs and promote consistency in AMS’s  oversight of the 22 check-off programs. These guidelines, which pertain  to AMS as well as board members and board staff, are not intended to  cover the daily responsibilities of board operations or AMS’s oversight.  Instead, the guidelines provide broad information on AMS’s expectations  for how boards should operate and how AMS will oversee the programs  in activities such as budget approval, contracts, financial accountability,  referendum, and investments, among other items.

In March 2012, USDA OIG released a report on AMS’s oversight of  check-off programs. The work was initiated by the OIG after a 2010  investigative report, conducted at the request of the AMS Administrator,  identified the possibility of weak oversight controls over the check-off  boards. The 2012 report included two recommendations for AMS to  develop and implement (1) standard operating procedures that provide  detailed instructions for performing oversight activities to address all  areas listed in the agency’s guidelines and (2) guidance for conducting  periodic internal reviews of program area operations to ensure the  enforcement of AMS’s guidelines. AMS agreed with the two  recommendations and planned to implement them with a variety of  actions, as discussed below.

section_title: AMS Has Improved Its Oversight of Check-off Programs, but Some Oversight Activities Are Not Consistent across Programs
paragraphs: AMS has responded to recommendations for improving oversight made in  the OIG’s 2012 report, particularly by developing and implementing  standard operating procedures and conducting internal reviews of AMS  check-off program oversight. However, AMS does not provide consistent  oversight across check-off programs in some areas; specifically, it does  not routinely review check-off program subcontracts during its  management reviews, conduct follow-up on management review  recommendations, ensure that financial assurances are included in  annual audits, or ensure that check-off boards share information with  assessment payers on program websites. In conducting their oversight of  the check-off programs, senior agency officials and marketing specialists  said they face challenges because of increased use of social media, the  absence of an information system for tracking approvals, and complex  Freedom of Information Act (FOIA) requests for some programs, which  may delay the completion of some oversight priorities.
subsections: section_title: AMS Has Made Improvements in Response to Recommendations Made by USDA’s OIG
paragraphs: The OIG’s 2012 report included two recommendations that AMS has  since implemented: to develop and implement (1) standard operating  procedures and (2) guidance for conducting periodic internal reviews of  its oversight activities. In August 2013, AMS developed and  implemented its standard operating procedures, which provide marketing  specialists with more detailed guidance on the various oversight activities  that are outlined in the agency’s program guidelines. The standard  operating procedures cover a range of oversight activities, including  budget review, contract review, advertising and promotional materials  review, and financial and internal control oversight. Included in the more  detailed guidance are various checklists that marketing specialists can  use to itemize the requirements that boards must meet in a variety of  areas. For example, the budget review checklist includes a list designed  to ensure that budgets conform to law and contain, among other items,  accurate sums and categories, as well as clearly listed administrative  expenses. According to senior agency officials and marketing specialists,  the standard operating procedures have assisted AMS in providing  consistency across the 22 commodity check-off programs, have helped  ensure that oversight responsibilities are carried out, and have provided  documentation of specific duties for new marketing specialists.

In response to the OIG’s second recommendation, AMS has developed  and implemented guidance for conducting internal reviews of its oversight  of check-off programs. Internal reviews are conducted by AMS’s  Management and Analysis Program group to evaluate whether the AMS  commodity areas that oversee check-off programs employ controls that  provide reasonable assurance that the check-off programs are meeting  legislative and regulatory requirements. According to an AMS directive,  internal reviews of each of the four AMS commodity areas are to be  conducted on a rotating basis. An AMS internal review of the Cotton  commodity area was completed in November 2014, an internal review of  the Specialty Crops commodity area was completed in September 2015,  and an internal review of the Dairy commodity area was competed in May  2017. According to officials in the Management and Analysis Program,  the Livestock, Poultry, and Seed commodity area internal review began in  May 2017. The Cotton internal review found the program to provide  reasonable assurance that the boards were complying with legislative  requirements and that the oversight controls were adequate and  functioning as intended. The Specialty Crop internal review found that the  commodity area was fulfilling its oversight responsibilities but also found  opportunities to strengthen control practices, including ensuring  consistent and timely application in its use of checklists and tracking  management reviews to ensure that they are completed and issued in a  timely manner. As a result, the Specialty Crop commodity area  implemented changes to its use of checklists and agreed to complete  management reviews in a timely manner. The Dairy internal review found  opportunities to strengthen oversight, primarily with regard to  management reviews and recordkeeping. As a result, according to senior  agency officials, the Dairy commodity area has implemented changes to  its management review and recordkeeping processes.

section_title: AMS Does Not Provide Consistent Oversight across Check-off Programs in Some Areas
paragraphs: We identified four areas in which AMS does not provide consistency in its  oversight across its check-off programs: (1) review of subcontracts, (2)  follow-up on recommendations made to check-off boards, (3) ensuring  that independent financial audits contain statements of assurance, and  (4) ensuring that information is available on program websites for  assessment payers (i.e., transparency).

Subcontracts. The 2012 OIG report found that AMS did not recognize in  its guidelines for check-off programs that its oversight role extended to  monitoring subcontracts. Following the release of the OIG report, AMS  updated the guidelines to respond to the OIG finding. Under the 2015  AMS guidelines, marketing specialists are to review a sample of  subcontractor expenses during their management reviews. However, we  found that AMS did not similarly update its standard operating procedures  for the check-off programs and that these reviews are not being done  consistently across programs.

We found that the marketing specialist for one of the eight programs we  reviewed chose a sample of subcontracts for the management review and  documented this selection in the management review report. Marketing  specialists for three of the programs said they reviewed subcontracts only  if the sample of primary contracts that were part of the management  review included subcontracts. Marketing specialists for the other four  programs said they did not review subcontracts. Two marketing  specialists we interviewed said they do not select a sample of  subcontracts because check-off boards are responsible for overseeing  and monitoring subcontracts. Senior agency officials and marketing  specialists also noted that they review and approve all promotional  materials regardless of whether any material is from a contract or  subcontract. Senior agency officials also said that the contracting process  differs among the various check-off boards and may cause confusion  about what is considered a subcontract for purposes of a management  review. For example, the cotton board contracts with Cotton Inc. to carry  out the program’s research and promotion activities; Cotton Inc. may, in  turn, contract with entities to carry out those research and promotion  activities—considered a cotton board subcontract. This is in contrast to  processes of other boards, such as the honey board, which can directly  contract with entities to carry out research and promotion activities; those  contractors may, in turn, subcontract duties. In addition, the potential  exists for subcontract costs to total hundreds of thousands of dollars. A  2010 OIG investigative review found that a subcontractor of one check-off  board used subcontracts to pay employees unauthorized bonuses of  about $302,000. Without revising its standard operating procedures for  check-off programs to recognize that each management review is to  include a sample of subcontracts for review, AMS’s ability to prevent  misuse of subcontract funds is impaired.

Recommendation follow-up. Under AMS’s guidelines and standard  operating procedures, marketing specialists are to ensure that corrective  actions are taken by the boards in a timely manner if a matter is  recommended in the management review, conducted every 3 years. For  example, the standard operating procedures state that the board has 30  calendar days from the receipt of the management review report to  respond to the findings by formal letter and that follow-up should include  appropriate documentation of the corrective actions taken. The 2012 OIG  report found that there was little consistency among AMS commodity  areas regarding the reporting of management review results and follow- up procedures. Four of the check-off programs we reviewed obtained  written confirmation from boards about how they intended to address  issues identified during management reviews consistent with the standard  operating procedures; three others did not obtain written confirmation, but  said they obtain any check-off board plans for remediation via less formal  means, such as via e-mails or during board meetings. The management  review for the eighth program did not contain any recommendations.

According to marketing specialists we interviewed, the follow-up process  to ensure that boards have taken corrective actions is also informal—a  specialist learns how management review recommendations have been  implemented by attending board and committee meetings. Senior  agency officials verified that AMS has no mechanism for tracking follow- up with check-off boards to ensure that they have taken corrective  actions. Under federal internal control standards, management should  remediate identified internal control deficiencies on a timely basis and,  with oversight from the oversight body, monitor the status of remediation  efforts so that they are completed on a timely basis. Without  establishing a mechanism for documenting and tracking follow-up with  check-off boards on the implementation of management review  recommendations, AMS has no assurance that it is consistently  monitoring the status of corrective actions. Senior agency officials said  that having a formal method to track and follow up on management  review recommendations would allow them to identify trends, best  practices, and similar emerging issues among the check-off programs.

Independent financial audits. Each year, each check-off board is  required by law to hire an independent audit firm to conduct an audit of  the board’s financial statements in accordance with generally accepted  government auditing standards. This audit helps to ensure compliance  with legislative, regulatory, and policy directives. AMS guidelines direct marketing specialists to review the annual financial audits to determine  whether the auditor identified any misuse of board funds and if the audit  adequately addressed whether (1) funds were discovered to be used for  influencing government policy or action, (2) the board adhered to the  AMS investment policy, (3) internal controls over funds met auditing  standards, (4) funds were used only for projects and other expenses  authorized in a budget approved by USDA, and (5) funds were used in  accordance with AMS guidelines. The standard operating procedures  state that AMS is to ensure that audits contain these five statements of  assurance, and they state that the audit firm is to express an opinion on  the financial statements of the board and include a report on internal  controls and compliance with applicable laws and regulations.

The 2012 OIG report found that none of the independent audit reports  included the five statements of assurance for the 18 check-off boards  reviewed. In our sample, audit reports for four of the eight programs  included the five statements of assurance. For two of the programs in our  sample, the engagement letters, which document the agreed-upon terms  of the audit, contained all five assurances, but the audit reports did not  contain the five assurances. For the remaining two programs in our  sample, neither the engagement letters nor the audit reports contained all  five assurances, but senior agency officials said that the AMS marketing  specialists for those two programs ensured that these assurances were  adequately addressed during pre- and post-audit meetings. According to  marketing specialists we interviewed for those two programs, audits  following government auditing standards incorporate the requirements  and are fulfilled by a general statement that boards were in compliance  with laws and regulations. However, the 2012 OIG report found that an  independent auditor did not include the specific assurances because the  auditor was not asked to perform such work and only minimal  adjustments would be needed to provide for those assurances. Without  ensuring that its annual independent financial audits include the five  statements of assurance as outlined in the standard operating  procedures, AMS will have less certainty that check-off funds are not  subject to waste, fraud, or mismanagement.

Transparency. According to the Business Roundtable and the  Organisation for Economic Cooperation and Development’s principles of  corporate governance, a strong disclosure regime that promotes  transparency is central to stakeholders being able to access regular,  reliable, and comparable information. As check-off programs use  assessment money collected from stakeholders of the commodity being  promoted, AMS’s guidelines state that both transparency and oversight of  the check-off funds are critical. Moreover, AMS’s guidelines state that  annual budget summaries should be posted on the check-off board’s  website and that three additional documents are either to be on the  website or otherwise made available: (1) the bylaws and policy  statements, (2) annual reports, and (3) the independent economic  evaluation of effectiveness. Four of the eight check-off programs in our  sample posted all four documents on the programs’ websites. All eight  check-off programs posted their annual reports online. Four of the check- off programs, however, did not post to their websites at least one of the  remaining documents—the budget summary, bylaws, or independent  economic evaluation. Marketing specialists we interviewed said that  boards would supply information not included on the websites if an  assessment payer requested such information, which is consistent with  AMS guidelines. Board executives we interviewed from those programs  that do not post all four documents on their websites also said that they  would supply the information to assessment payers if contacted. Senior  AMS officials also said that there are stakeholders who may not have  computers or access to the Internet and may therefore request  information via postal mail.

Industry organization representatives we interviewed said that  transparency of how funds are used and the effectiveness of the  programs are important to their members. One industry organization  representative we interviewed said that, although some stakeholders may  not use the Internet, posting information on how assessments are being  used, such as the information provided in annual reports, is useful for  stakeholders and builds trust among check-off boards and stakeholders.  Posting information on the boards’ websites could convey information to  stakeholders who have access to the Internet at a low cost. Without  including in the guidelines and standard operating procedures that all four  key check-off board documents (i.e., bylaws and policy statements,  annual reports, and independent evaluations of economic effectiveness)  should be posted on a check-off program’s website, AMS may be missing  an opportunity to ensure that some assessment payers have access to  information on program operations and effectiveness.

section_title: AMS Officials Identified Challenges in Their Efforts to Oversee Check-off Programs
paragraphs: AMS officials identified ongoing challenges in check-off program  oversight. In particular, AMS marketing specialists and senior agency  officials identified three challenges: (1) the increase in some check-off  boards’ use of social media, (2) the absence of an information system to  track approvals, and (3) complex and time-consuming FOIA requests for  some programs. Because of competing priorities, some oversight duties  may be delayed as a result.

Increase in boards’ social media efforts. According to marketing  specialists, four of the eight check-off programs have seen a significant  increase in the boards’ use of social media, which has been a challenge  in terms of both workload and the need for additional AMS guidance  because the specialists must approve the social media content. Marketing  specialists for the other four programs said that the check-off programs  they oversee have not yet increased their social media presence enough  to make it a challenge for workload. Senior agency officials and marketing  specialists agreed that oversight of the check-off programs requires a  significant amount of time and effort that has been made more  complicated since some check-off programs began using social media.  For example, a marketing specialist for one check-off program approved  over 3,000 items, including social media for promotional and research  materials, in a 6-month period. According to this marketing specialist,  depending on the complexity of the item needing approval, there could  have been dozens of communications between the specialist and the  check-off board staff. In addition, marketing specialists and senior agency  officials said that because social media is constantly evolving, AMS has  needed to reevaluate its guidance to boards for social media. The senior  agency officials acknowledged that the duties of marketing specialists are  demanding and that they are working to find ways to provide support to  marketing specialists. Senior agency officials said that this is challenging  because the boards must reimburse AMS for oversight costs, so any  additional personnel would be paid for through check-off assessments.  Also, AMS established a social media committee made up of marketing  specialists who have drafted social media guidance for the boards to  follow.

Technology. Tracking the numerous promotional and research approvals  can be a challenge for some AMS marketing specialists because of the  absence of an information system to track approvals. According to two  marketing specialists, during busy times, they may be handling more than  20 requests for approvals a day. While marketing specialists for two of the  check-off programs we reviewed said that the use of approval tracking  software, paid for by the respective check-off boards, has made their  oversight function more efficient, other marketing specialists said that  they must rely on e-mail messages to organize the status of approvals.  Marketing specialists who have tracking software said that they can  quickly see the status of any approval at any given time; further, check-off  board staff can also use the software to prioritize approvals. One  marketing specialist said that, although she had developed a system for  organizing e-mails, a tracking system used by both AMS and the board  would ensure that oversight activities would not be delayed and could  expedite the approval process. Senior agency officials said that it would  be helpful if each marketing specialist had this software but that the  check-off boards would need to pay for this expense.

FOIA requests. Responding to complex FOIA requests about check-off  programs has been a challenge, according to senior agency officials,  marketing specialists, and board executives of four of the eight programs  we reviewed. Some requests do not take many resources to fulfill, but  others take significant time and money. For example, to respond to a  FOIA request, board staff and marketing specialists must identify  pertinent documents; review them to ensure that there is no proprietary or  sensitive information; and, as needed, involve the board’s legal counsel  or third-party businesses. According to senior agency officials, in one  case, a FOIA request resulted in the check-off board and AMS providing  approximately 10,000 documents to the requester. AMS estimates that in  fiscal year 2016, for the Livestock, Poultry, and Seed commodity area  programs, it cost the agency about $182,000 and more than 2,700 hours  to fulfill FOIA requests. For the same period for the Dairy commodity  area programs, AMS estimates that it cost over $365,000 and about  6,600 hours to fulfill FOIA requests. Because AMS is reimbursed for its  oversight costs, the funds to cover FOIA-related costs come directly from  check-off assessments. These cost estimates do not include check-off  board staff resources utilized to fulfill FOIA requests. Senior agency  officials said that there are legal constraints on the types of individuals  and organizations that they can request cover fees associated with  document retrieval under FOIA.

section_title: Check-off Evaluations Generally Indicate Positive Returns but Vary in How They Are Conducted and Reviewed
paragraphs: Independent economic evaluations of the effectiveness of check-off  programs, conducted at least every 5 years, have generally shown a  positive benefit to those who pay assessments. The evaluations we  reviewed varied both in the methods used to conduct the analysis and  how information was reported and revealed certain methodological  limitations. According to senior agency officials as well as the economists  who conducted the evaluations, the variations are in part due to the  differences in check-off board resources. We found that AMS does not  consistently document its review of independent economic evaluations  and has no criteria established for determining what makes for a credible  methodology and results.
subsections: section_title: Evaluations of Check-off Programs Were Conducted Every 5 Years and Show a Range of Positive Benefits for Assessment Payers
paragraphs: The Federal Agriculture Improvement and Reform Act of 1996 requires  check-off boards to (1) fund independent economic evaluations of the  effectiveness of their promotion activities every 5 years, (2) submit the  evaluation to USDA, and (3) make the results available to the public.  Check-off boards, through a request for proposals process, contract for  independent economic evaluation to determine the effectiveness of  promotion activities. The law does not specify how an independent  economic evaluation should be completed, and AMS does not offer any  guidance on the methodologies to use, the types of information to include,  or how the results of such an evaluation are to be presented. AMS  guidelines, which are available to the check-off boards, state that  evaluations: (1) have a credible methodology, (2) articulate shareholder  returns, and (3) present the results in a non-technical manner.

The eight independent economic evaluations of check-off programs we  reviewed focused on benefit-cost ratios (BCR) and returns on investment  (ROI). While BCRs and ROIs are slightly different, they both measure the  financial gain or loss generated from the costs of implementing a  program. In both cases, economists use economic, industry-specific  models to determine the benefits or economic gains from the check-off  programs by isolating the impacts of program promotion dollars from  other variables, such as competing products or changes in consumer  income. For example, some models include the effects of changes in the  prices of substitute food products, which may affect the demand for  commodities. The model used in the evaluation for the beef check-off  program, for instance, includes prices for both chicken and pork, as an  increase in the price of chicken or pork could lead to an increase in the  consumer demand for beef, regardless of check-off program activities.  Other variables that may affect demand include changes in (1) consumer  buying habits, (2) consumer income, and (3) government policy. These  variables can either increase or decrease the demand for commodities  despite the activities of check-off programs. Evaluation models may also  include variables that affect the supply of a commodity, such as increased  prices that send signals to farmers to increase production. Although it is  difficult to capture, some commodity evaluation models also model  increases in yields and acreage to determine how much the agricultural  research portion of a check-off program affects the supply of the  commodity. Increased supply as a result of agricultural research  expenditures can also increase producer benefits and economic gains,  but according to the sorghum and cotton evaluations, many of these  gains cannot be immediately or directly measured.

For the eight check-off programs we reviewed, the BCRs and ROIs  ranged from 2.14 to 17.40. In other words, for every dollar invested in the  check-off programs, the programs returned from $2.14 to $17.40 in  revenue to assessment payers (see table 3). However, it is important to  note that the results of the independent economic evaluations should not  be compared across check-off programs because of differing  methodologies, differing data, and differing demands for the products,  according to economists we interviewed. Economists we interviewed and  literature we reviewed suggested that although the results of an  independent economic evaluation may appear large, the amount invested  in promotion activities is small compared to the total value of industry  sales. Therefore, the overall impact of promotion activities on the market  may be small.

Program referenda largely show that most assessment payers approve of  check-off programs, but not all types of assessment payers may feel that  they share equally in the benefits that are found through the independent  economic evaluations, according to economists we interviewed. The  studies we reviewed report either average or marginal measures of  effectiveness, such as a BCR. Some economists we interviewed, both  those who have conducted the evaluations we reviewed and those who  did not, said that these types of studies do not address the level of ROIs  across the distribution of check-off program payers or how much more  larger-sized assessment payers receive in returns from their investment  in the check-off program as compared to smaller-sized ones. This view  was confirmed by representatives we interviewed from some of the  industry organizations, who indicated that their members would prefer to  better understand what they receive for their investment at the farm level.  In addition, one economist we interviewed said that assessment payers  may be skeptical of the results of independent economic evaluations of  program effectiveness because while the costs are tangible, the benefits  of the programs are not. That is, the producers cannot see what portion of  their revenues is directly attributable to check-off program activities.

section_title: Evaluations Vary in How They Are Conducted and Reveal Certain Methodological Limitations
paragraphs: The independent economic evaluations we reviewed were conducted  using different methodologies and reported different information.  According to senior agency officials, evaluations likely vary because  legislation does not include any details on how evaluations should be  completed and the amount of resources that each check-off board has  available to devote to evaluations varies. Nearly all of the economists we  interviewed said that it would be useful to have minimum standards for  information that should be included in the evaluations. Some independent  economic evaluations used different types of models and data to estimate  the benefits and costs to assessment payers. For example, for the egg  and honey check-off commodity models, the evaluation used two  separate types of methodologies to estimate increases in demand  because of the programs’ promotional activities. Some other  independent economic evaluations in our sample, such as for cotton and  fluid milk, used multi-market models that incorporated components for  substitute products, the foreign sector, and the government sector. Some  independent economic evaluations, such as the beef evaluation,  measured a marginal BCR and others measured an average benefit-cost  ratio. The cotton and sorghum evaluations performed an analysis of how  increases in yields, acreage, and production because of the research  portion of the check-off programs affected the supply of the commodities.  The independent economic evaluations also examined different time  periods in their analyses, depending on the available data (see table 3).  For example, the egg evaluation covered the period of 2007 through  2010, and the fluid milk evaluation covered 1995 through 2012.

In addition to having different methodologies to calculate benefits and  costs, information and analyses included in the independent economic  evaluation reports also varied among the eight programs we reviewed.

For example, the beef check-off evaluation includes a section on the  optimal allocation of funds to domestic activities of the program, which is  not included in any other report. Seven of the eight evaluation reports had  a conclusions section. One of the evaluation reports included a  recommendations section, while others did not. Although the law does not  specify information required to be included in the independent economic  evaluations, representatives from one industry organization we  interviewed said that having the information in a consistent format could  help ensure that stakeholders could compare information from one  evaluation to the next for a given check-off program.

The independent economic evaluations provided useful information to key  stakeholders and the general public, but we found that they also included  a number of caveats and limitations. Some of these limitations resulted  from the nature of a commodity or program itself and others from the  modeling procedures used. According to economists we interviewed and  senior agency officials, the law is not prescriptive about how evaluations  are to be conducted, and the boards differ in the amount of resources  available to devote to the evaluations. If, for example, a board has limited  resources available for an evaluation, there may not be funds available to  purchase a certain set of data. For the sample of eight evaluations we  reviewed, these limitations included the following:

Data limitations: A number of the independent economic evaluations  had data limitations. For example, one independent economic  evaluation (highbush blueberry check-off program) lacked either  wholesale or retail price data for its demand model, and another  (sorghum check-off program) lacked program data as it had only been  in existence for 5 years when the evaluation was performed. All of the  economists we interviewed who had completed the eight evaluations  we reviewed said that data are a challenge when conducting the  evaluations either because such data do not exist or the check-off  boards do not have the resources to buy the data.

Not discounting the BCR to present value: The cotton check-off  evaluation was the only one in our sample with a methodology that  discounted the BCR to present value to account for the time value of  money. Discounting a program’s benefits and costs to present value  transforms gains and losses occurring in different time periods to a  common unit of measurement.

Not accounting for spillover effects: Some independent  economic evaluations did not include the spillover effects—the cross- commodity impact of promotion—on related markets, though some,  such as the cotton evaluation, did account for spillover effects on  competing commodities. If spillover effects pertain to a commodity,  failure to account for these effects could overstate the benefits of a  program and cause an upward bias in computing the BCR.

Not adjusting models for structural changes: Some independent  economic evaluations did not adjust models for structural changes in  the industry over time. While some independent economic evaluations  we reviewed, such as those for the honey and beef check-off  programs, did use data or methods that accounted for changes in  market structure over time, others did not. For example, for the pork  check-off program, some hog farms have specialized in a single  phase of production, and have encountered substantial gains in  productivity because of technology over the past several decades, but  the independent economic evaluation did not reflect this. Failure to  correct for such structural change, if applicable to a commodity, can  lead to incorrect modeling and misleading policy implications.

AMS’s standard operating procedures acknowledge that each check-off  program varies in size and scope; therefore, the amount of resources  each program can devote to an independent economic evaluation varies.  Smaller programs may have independent economic evaluations that  reflect the realities of program scope, financial capability, and data  availability. Our discussions with the economists who conducted the  evaluations that we reviewed confirmed that this is the case. They said  that the smaller programs are able to devote fewer resources to  independent economic evaluations; therefore, the economist conducting  an evaluation may not be able to complete all of the analysis that could  be completed for a larger program that is able to pay for more complex  analysis. According to senior agency officials, in some instances, a  broader evaluation is not necessary because of the emphasis and goals  of the program. In addition, the resources a board is able to devote may  vary from evaluation to evaluation. For example, one economist said that  he worked with a board that wanted a more comprehensive evaluation  than was previously done. The new evaluation model included additional  data over a longer period of time, which ultimately led to an increased  ROI.

section_title: AMS Does Not Consistently Document Reviews of Check-off Evaluations
paragraphs: In addition to ensuring that independent economic evaluations are  conducted every 5 years and encouraging boards to make them available  to assessment payers, AMS’s standard operating procedures state that  marketing specialists should ensure that independent economic  evaluations (1) have a credible methodology and results, (2) articulate  shareholder benefits, and (3) present results in a non-technical manner.  To verify that these three directives are met, the standard operating  procedures state that marketing specialists may consult with agency  economists. They are directed to document verification in writing. Outside  of any agency review, there is no requirement that independent economic  evaluations be peer reviewed. A National Academies report states that  peer review is characterized, in part, as being a documented, critical  review of assumptions, calculations, and methodology, performed by a  person with technical expertise in the subject matter to be reviewed who  is independent and external of the work being reviewed. The report  further states that the peer, to the extent possible, should have sufficient  freedom from funding considerations to ensure that the work is impartially  reviewed. According to senior agency officials, AMS economists meet this  definition; and their review of the independent economic evaluations can  be considered peer review. Officials said that the economists on staff  critically review the evaluations; they all have PhDs in economics and are  independent as they do not work directly with the check-off programs  except for reviewing the evaluations.

Three of the four AMS commodity areas—Cotton and Tobacco, Dairy,  and Specialty Crops—utilized an AMS economist to review the  independent economic evaluations and document that review. Senior  agency officials said that the Livestock, Poultry, and Seed commodity  area has an AMS economist review the independent economic  evaluations but does not document that review. According to senior  agency officials, the Livestock, Poultry, and Seed commodity area has  relied on informal reviews of the evaluations by an economist, which are  orally presented to the director of the commodity area. Further, the  economists who completed the eight independent economic evaluations  we reviewed indicated that although their preference is to have the  evaluations peer reviewed, this is not always possible because of time  constraints and other priorities. One economist said that the board he  worked with included a contractual requirement that the independent  economic evaluation be peer reviewed.

Because the Livestock, Poultry, and Seed commodity area does not  document its reviews of independent economic evaluations, only four of  the eight check-off programs in our sample had documented reviews of  the evaluations. All four of the documented reviews ensured that the  independent economic evaluations had a credible methodology and  results and articulated shareholder benefits, as stated in the standard  operating procedures. However, only two of these four check-off  programs included in their documented review whether results were  presented in a non-technical manner, as also stated in the standard  operating procedures. Further, the internal reviews did not use standard  criteria to determine whether the independent economic evaluations had  a credible methodology or results, which is important because, as noted  earlier, the evaluations we reviewed varied in their methodology and we  found that they had certain limitations.

Although check-off programs are not subject to the guidelines in the  Office of Management and Budget’s Circular A-94, the circular provides  general guidance for conducting analyses to help federal agencies  efficiently allocate resources through well-informed decision making. For  example, Office of Management and Budget Circular A-94 establishes  key elements of an economic analysis, including (1) a statement of the  objective and scope of the analysis, (2) an identification of alternatives,  (3) an analysis of the economic effects, (4) a sensitivity analysis, and (5)  adequate documentation and transparency. Conducting and documenting  reviews of independent economic evaluations using criteria can be useful.  For example, in 2014, a senior agency official found several  inconsistencies in a check-off program independent economic evaluation.  The senior agency official assigned an AMS economist and marketing  specialist to work with the evaluator to revise econometric models to more  accurately capture the activities of the check-off program. According to  the official, if the independent economic evaluation had not been  reviewed, benefits of the program would have been understated and  would have misled those paying into the check-off program. Without  developing criteria by which AMS can assess the methodology and  results of independent evaluations and document those assessments to  ensure that the standard operating procedures are met, the agency’s  assessments of independent economic evaluations may be inconsistent  across check-off programs and misleading to agency officials, check-off  boards, and assessment payers.

section_title: Conclusions
paragraphs: AMS oversees commodity check-off programs that conduct research and  promotion activities to strengthen 22 commodities’ position in the  marketplace. The agency has taken steps to improve oversight activities  based on recommendations in USDA OIG’s 2012 report, but it continues  to face challenges in other oversight activities. For example, AMS has not  consistently reviewed subcontracts during its management reviews.  Without revising its standard operating procedures for check-off programs  to recognize that management reviews should include a sample of  subcontracts for review, AMS’s ability to prevent misuse of subcontract  funds is impaired.

In addition, AMS has not consistently followed up on recommendations  made to check-off boards, although its guidelines and standard operating  procedures state that marketing specialists are to ensure that corrective  actions are taken by the boards in a timely manner if a matter is  recommended in a management review. Without establishing a  mechanism for documenting and tracking follow-up with checkoff boards  on the implementation of management review recommendations, AMS  has no assurance that it is consistently monitoring the status of corrective  actions. Moreover, AMS has not ensured that independent financial  audits contain statements of assurance as called for in the agency’s  program guidelines or standard operating procedures. Without ensuring  that its annual independent financial audits include the five statements of  assurance outlined in the standard operating procedures, AMS will have  less certainty that check-off funds are not subject to waste, fraud, or  mismanagement.

Further, although principles of corporate governance state the importance  of transparency for stakeholders, AMS has not ensured that certain  information, such as budget summaries and program evaluations, are  presented on check-off program websites and has not included in its  guidelines or standard operating procedures that certain information  should be included on program websites, although the agency’s program  guidelines recognize that transparency of check-off funds is critical.  Without including in the guidelines and standard operating procedures  that key check-off board documents are to be posted on the check-off  program’s website, AMS may miss the opportunity to ensure that some  assessment payers have access to information on program operations  and effectiveness.

Finally, check-off boards are meeting legislative deadlines by completing  independent economic evaluations of effectiveness every 5 years;  however, the evaluations vary and have certain methodological  limitations. Without developing criteria by which AMS can assess whether  evaluations have a credible methodology and results and documenting  those assessments, the assessments may be inconsistent across check- off programs and misleading to agency officials, check-off boards, and  assessment payers.

section_title: Recommendations for Executive Action
paragraphs: We are making the following five recommendations to the Administrator of  the Agricultural Marketing Service:  The Administrator of AMS should revise the standard operating  procedures for AMS’s check-off programs to state that management  reviews include a sample of subcontracts for review. (Recommendation  1)

The Administrator of AMS should establish a mechanism for documenting  and tracking follow-up with check-off boards on the implementation of  management review recommendations. (Recommendation 2)

The Administrator of AMS should ensure that annual independent audits  include the five statements of assurance as outlined in the standard  operating procedures. (Recommendation 3)

The Administrator of AMS should include in the guidelines and standard  operating procedures that key check-off board documents, such as  bylaws and policy statements, annual reports, and independent  evaluations of economic effectiveness are posted on the check-off  programs’ websites. (Recommendation 4)

The Administrator of AMS should develop criteria by which to assess the  methodology and results of independent evaluations and document those  reviews to ensure that the standard operating procedures are met.  (Recommendation 5)

section_title: Agency Comments
paragraphs: We provided a draft of this report for review and comment to USDA. An  auditor with AMS’s Management and Analysis Program responded via e- mail on October 24, 2017, that the agency generally agreed with our  findings and recommendations.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Agriculture, and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or morriss@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix I.

section_title: Appendix I: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, key contributors to this report  included Thomas M. Cook (Assistant Director), Rose Almoguera, Kevin S.  Bray, Barbara El Osta, Cindy Gilbert, Holly Halifax, Khali Hampton, Dan  Royer, Holly Sasso, Sheryl Stein, and Kiki Theodoropoulos.

Fast facts: Remember "Got Milk?"

Those ads are sponsored by 1 of 22 industry-led research and promotion programs funded by producers of commodities like dairy or pork and such ads are overseen by USDA. These "check-off" programs have restrictions on their activities, such as being prohibited from promoting false advertising.

We reviewed USDA's process for overseeing program activities and financial controls. We made 5 recommendations, including that it consistently review subcontracts to prevent misuse of funds, and ensure documents are on program websites to promote transparency and ensure stakeholders have access to information on program operations.

Question-summary annotations: questions: question: How has AMS changed its oversight protocols?
answer: The U.S. Department of Agriculture's (USDA) Agricultural Marketing Service (AMS) has improved its oversight of check-off programs since USDA's Office of Inspector General (OIG) made recommendations in a 2012 report.
child_questions: question: What improvements did AMS undertake?
answer: In response to two OIG recommendations, AMS developed and implemented standard operating procedures, which outline specific oversight responsibilities of AMS, and began to conduct internal reviews of its oversight functions.

question: To what extent does AMS fully regulate relevant documents?
answer: However, GAO found that AMS does not consistently review subcontracts—a legal agreement between a contractor and third party—or ensure that certain documents are shared with stakeholders on program websites.

questions: question: What do AMS's guidelines for check-off programs stipulate?
answer: Under AMS's 2015 guidelines for check-off programs, which cover broad oversight activities, staff are to review a sample of subcontracts during agency reviews of program operations.
child_questions: question: To what extent did AMS keep its practices current?
answer: However, AMS did not revise its standard operating procedures to match its guidelines with this responsibility, and GAO found that AMS reviewed subcontracts for only one check-off program in its sample of eight.
child_questions: question: What effect does this have on AMS's capacity for broad oversight?
answer: Without revising the standard operating procedures to include a review of subcontracts, AMS's ability to prevent misuse of funds is impaired.

questions: question: Why is transparency important for business practice?
answer: According to leading business principles, transparency is central to stakeholders' access to regular, reliable, and comparable information.

question: What did GAO find about check-off programs?
answer: However, GAO found that four of the eight check-off programs reviewed posted all key documents, such as budget summaries and evaluations of effectiveness, to program websites.
child_questions: question: How does this compare to AMS's stated protocol?
answer: GAO found that AMS's guidelines state that budget summaries should be posted on program websites, while the other key documents are to be available on the website or otherwise made available to stakeholders.

question: What did stakeholers report about this situation?
answer: Agency officials said that boards would supply documentation if contacted by a stakeholder. Industry representatives GAO interviewed said that transparency of how funds are used and the effectiveness of programs are important to their members.

question: How might this pose a problem for AMS?
answer: Without including in its guidelines and standard operating procedures that all key documents should be posted on a check-off program's website, AMS may miss an opportunity to ensure that stakeholders have access to information on program operations and effectiveness.

questions: question: What are the effects of independent economic evaluations of the effectiveness of check-off programs?
answer: Independent economic evaluations of the effectiveness of check-off programs, required by law to be conducted every 5 years, have generally shown positive financial benefits.
child_questions: question: What benefits did GAO discover?
answer: For the eight evaluations GAO reviewed, benefits ranged from an average of $2.14 to $17.40 for every dollar invested in the programs.

question: What limitations did GAO discover?
answer: However, the evaluations varied in the methods used and had certain methodological limitations. For example, some evaluations did not account for the effects of promotion from competing commodities, which could overstate the programs' benefits.

question: To what extent did AMS review its own procedures?
answer: AMS's standard operating procedures state that the agency should review the evaluations to ensure that there is a credible methodology, among other things; however, AMS did not consistently document reviews of the evaluations or have criteria by which to review the evaluations.
child_questions: question: What risk doees this lack of oversight pose?
answer: Without developing criteria to assess the methodology and results of evaluations, the agency's assessments of independent economic evaluations may be inconsistent across check-off programs and misleading to stakeholders.

questions: question: What are some examples of advertising campaigns undertaken by federal agricultural research and promotion programs?
answer: “Got milk?” and “Pork: The Other White Meat” are examples of advertising campaigns undertaken by 2 of the 22 federal agricultural research and promotion programs, commonly known as commodity check-off programs.
child_questions: question: Who oversees these programs?
answer: MS has primary responsibility for overseeing the check-off programs. These programs, funded by a fraction of the sale of each unit of a commodity, are led by boards consisting of industry members appointed by the Secretary of Agriculture.

question: What do these programs consist of?
answer: The programs conduct research and promotion activities to strengthen a commodity's position in the market.

question: What limitations exist on these programs?
answer: By law, funds cannot be used for lobbying or disparaging other commodities, among other things.

questions: question: What was GAO asked to review?
answer: GAO was asked to review AMS's oversight of the check-off programs.
child_questions: question: How did GAO review this?
answer: GAO selected a sample of 8 such programs—selected, in part, based on total funds collected—and reviewed laws, regulations, and agency guidance. GAO interviewed agency officials, check-off board executives, and economists.

question: On what did GAO select its sample?
answer: GAO selected a sample of 8 such programs—selected, in part, based on total funds collected—and reviewed laws, regulations, and agency guidance.

question: What does the final report address?
answer: This report examines (1) the extent to which AMS has addressed previously identified weaknesses in its oversight and (2) how the effectiveness of the programs has been evaluated and what the results have indicated.