U.S. International Broadcasting: Management of Middle East Broadcasting Services Could Be Improved
Published: 2006-08-04T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Broadcasting Board of Governors' (BBG) broadcasting services, Radio Sawa, and the Alhurra satellite television networks--collectively known as the Middle East Broadcasting Networks, Inc. (MBN)--currently aim to reach Arabic speakers in 19 countries and areas throughout the Middle East. Annual spending for current activities amounts to about $78 million. GAO reviewed MBN's (1) strategic planning to address competition in the Middle Eastern media market, (2) implementation of internal control, (3) procedures MBN has developed to ensure compliance with its journalistic standards, and (4) performance indicators and whether targets have been met.
section_title: What GAO Found
paragraphs: MBN faces a number of competitive challenges in carrying out its mission of broadcasting in the Middle Eastern media market and has taken some steps to address them. However, MBN lacks a comprehensive, long-term strategic plan. As MBN emerges from its start-up mode and faces future challenges, a long-term strategic plan will be important. While MBN has developed financial and administrative controls to manage and safeguard its financial resources, it could take additional steps to strengthen its system of internal control. For example, the MBN has not (1) convened a meeting of its internal control board to formally develop its controls and coordinate audits, (2) completed an internal control plan, (3) completed a risk assessment to address potential risks to its operation, or (4) developed a comprehensive training program for its staff. MBN has procedures in place to help ensure its programming meets its journalistic standards. However, MBN lacks regular editorial training and has not fully implemented a comprehensive, regular program review process to determine whether its programming complies with those standards or with MBN's mission. While the BBG calls for its broadcasters to undergo an annual program review, Radio Sawa has only held one such review, and Alhurra has not completed one to date. The BBG has developed several performance indicators and targets for MBN's Radio Sawa and Alhurra services, including measures of audience size and program credibility. However, it is not clear whether the Radio Sawa and Alhurra performance targets have been met because of weaknesses in MBN's survey methodology and documentation. The BBG did not take certain steps that could have explained and increased the reliability of its estimates, such as by fully documenting its research and estimation methods, measuring the level of uncertainty surrounding its estimates, disclosing significant limitations, and consistently implementing policies and procedures for verifying data.

section_title: Letter
paragraphs: The President’s 2006 National Security Strategy stipulates that public  diplomacy, which includes U.S. international broadcasting, is an important  instrument for engaging and informing foreign audiences, creating and  maintaining a dialogue with Muslim leaders, and confronting  misinformation about the United States. Under the direction of the  Broadcasting Board of Governors (BBG), an independent agency, the  mission of all U.S. international broadcasting services is to promote  freedom and democracy and enhance understanding through objective  journalism. The BBG’s priority broadcasting services, Radio Sawa and the  Alhurra satellite television networks—both operated since 2005 by the  nonprofit grantee Middle East Broadcasting Networks, Inc. (MBN)— currently aim to reach Arabic speakers in 19 countries and areas  throughout the Middle East. Spending for MBN’s broadcasting services has  increased in recent years and currently amounts to about $78 million  annually. In fiscal year 2005, MBN received additional funding to support a  new Alhurra satellite television initiative to reach Arabic speakers in  Europe. MBN has recently requested funding to enhance its radio and  television programming content.
This report assesses (1) the extent to which MBN has used strategic  planning to address its competitive challenges in the Middle Eastern media  market, (2) the progress MBN has made in developing financial and  administrative controls to ensure that the organization operates efficiently  and complies with its grant agreement, (3) the procedures MBN has  developed to ensure compliance with its journalistic standards, and (4) the  extent to which BBG has developed performance indicators for MBN’s  efforts and met its primary performance targets.
To accomplish our objectives, we reviewed documentation and spoke with  officials from the BBG and its broadcasting entities, including the grantees  (MBN, Radio Free Europe/Radio Liberty (RFE/RL), Radio Free Asia  (RFA)), the Voice of America (VOA), and the International Broadcasting  Bureau. We also spoke with several outside experts, including  representatives from the Department of State, National Public Radio, the  InterMedia research organization, Zogby International, and foreign  international broadcasters such as the British Broadcasting Corporation  (BBC) and Deutsche Welle. In addition, we visited MBN’s offices in  Amman, Jordan, and Dubai, United Arab Emirates (UAE). We held  telephone interviews with MBN’s current affairs contractor in Beirut,  Lebanon, and corresponded with the head of the MBN office in Baghdad,  Iraq. We conducted our evaluation from August 2005 through May 2006 in  accordance with generally accepted government auditing standards.  (Appendix I provides a more detailed description of our scope and  methodology.)
subsections: section_title: Background
paragraphs: Middle East Broadcasting Networks, Inc. (MBN), which includes the  services Radio Sawa and Alhurra, is an independent nonprofit grantee  overseen by the Broadcasting Board of Governors (BBG).  The BBG is an  independent, federal agency responsible for overseeing all U.S.  government-sponsored, nonmilitary, international broadcasting programs.  The BBG also manages the operations of the International Broadcasting  Bureau, the Voice of America (VOA), and the Office of Cuba Broadcasting,  which are all federal entities. It also provides funding and oversight to three  independent grantees, MBN, Radio Free Europe/Radio Liberty (RFE/RL),  and Radio Free Asia (RFA) (see figure 1).
subsections: section_title: Development of MBN
paragraphs: In March 2002, due to concerns about the effectiveness of its outreach to  Arabic speakers in the Middle East, the BBG replaced VOA’s Arabic radio  service with Radio Sawa, a 24-hour, 7-day a week, Arabic language radio  station. In April 2003, Congress, at the request of the administration,  provided $26 million in the fiscal year 2003 Emergency Wartime  Supplemental Appropriations Act to establish a nonprofit corporation, the  Middle East Television Network (MTN), as a grantee of the BBG to launch  and operate Alhurra TV. Ten months later, on February 14, 2004, Alhurra,  MTN’s Arabic-language satellite television station, was launched and  initially started broadcasting 14 hours a day, expanding to 24 hours a day 2  months later. In November 2003, the fiscal year 2004 Emergency  Supplemental Act for Defense and for Reconstruction of Iraq and  Afghanistan included $40 million to establish a second 24-hour channel,  Alhurra-Iraq. On April 27, 2004, Alhurra-Iraq was launched. In 2005, MTN  was renamed the Middle East Broadcasting Networks, Inc. (MBN), and  Radio Sawa transferred to MBN. See figure 2 for a timeline of these events  and figure 3 for details on the current organization of MBN.
section_title: MBN Funding
paragraphs: Congress appropriated more than $274 million to fund Radio Sawa,  Alhurra, and Alhurra-Iraq from fiscal year 2002 through fiscal year 2006.
Each year MBN’s funding level has increased to support additional 24-hour  television streams and Radio Sawa’s 24-hour radio programming. MBN’s  grant in fiscal year 2006 is $78.7 million to support Alhurra, Alhurra-Iraq,  and Radio Sawa, as well as the launch of Alhurra-Europe. See figure 4 for a  breakdown of funds by fiscal year.
section_title: MBN’s Mission and Scope of Operations
paragraphs: The Alhurra networks and Radio Sawa are BBG’s priority broadcasting  services designed to support the BBG’s antiterror broadcasting initiatives  in the Middle East and counter media campaigns used by terrorists by  providing accurate reporting and analysis of the news and by explaining  U.S. policies. Although MBN and its Alhurra broadcasting services postdate  the BBG’s current 2002-2007 strategic plan, Radio Sawa, in particular, was  singled out as an opportunity in the plan to target Arabic-speaking youth  and provide them with news that is objective, comprehensive, fresh, and  relevant and to provide a forum for reasoned discussion of “hot button”  issues and U.S. policies. MBN’s current mission statement is to broadcast  factual, timely, and relevant news and information about the Middle East,  the United States, and the world to people of all ages in order to advance  the long-term U.S. interests of promoting freedom and democracy and  enhancing understanding in the Middle East. Radio Sawa and Alhurra aim  to be among the sources that audiences turn to in the Middle East for news  and information, to increase the standards of other broadcasters in the  region, and to offer distinctive and provocative programming unavailable  on other stations.
MBN’s target audience includes 19 Arabic-speaking countries and  territories in North Africa, the Near East, and the Gulf region, which are  home to approximately 250 million people. In addition to its headquarters  in Springfield, Virginia, MBN has several overseas offices, including a  production center in Dubai that broadcasts Radio Sawa’s live newscasts  during 8 hours each day and produces some opinion features for Radio  Sawa and current affairs programming for Alhurra (see figure 5).
section_title: Radio Sawa’s Operations
paragraphs: Radio Sawa’s broadcasts are designed to reach a target audience of 15- to  29-year olds in the Middle East with Western and Arab popular music, news  broadcasts, and specialized programming. Radio Sawa broadcasts 24 hours  of programming every day through a combination of FM, medium wave  (AM), digital audio satellite, and Internet transmission resources. See  figure 6 for a map of Radio Sawa’s regional reach.
Radio Sawa has developed seven distinct programming streams, including  (1) Iraq, (2) Jordan and the West Bank, (3) the Gulf, (4) Egypt and the  Levant, (5) Morocco, (6) Sudan and Yemen, and (7) Lebanon. All of the  streams generally feature the same major newscasts, current affairs, and  policy features; however, the Iraq program differs slightly, and the streams  all offer differentiated music programs. Radio Sawa’s streams broadcast  between 31 and 35 hours of news each week. (See appendix II for more on  Radio Sawa’s programming.)
section_title: Alhurra’s Operations
paragraphs: The Alhurra satellite television station is designed to reach a broad  audience in the Middle East by providing news, current affairs, and  entertainment programming 24 hours a day, 7 days a week. Alhurra-Iraq is  designed to provide Iraqi citizens with daily newscasts and talk shows that  specifically address issues in Iraq. Both Alhurra networks broadcast  between 36 and 43 hours of news and news updates a week. Alhurra  broadcasts on the Arabsat and Nilesat satellites, which currently allow it to  cover the entire Middle East. Alhurra-Iraq also broadcasts through these  satellites and a combination of terrestrial transmitters made available in  Iraq. Alhurra also expects to start broadcasting to Europe on August 1,  2006. (See appendix II for more information on Alhurra and Alhurra-Iraq  programming.)
section_title: Lack of Long-term, Comprehensive Planning Could Limit MBN’s Ability to Respond to Market Challenges
paragraphs: MBN faces a variety of challenges to broadcasting in the Middle East,  including operating in a competitive satellite television broadcast market,  operational and programming competitive disparities, and lack of coverage  for Radio Sawa in certain FM markets.  MBN has conducted some planning  efforts and, by using market research and internal assessments of its  competitors, has undertaken or proposed some initiatives to address many  of these challenges, such as increasing its hours of news coverage and  current affairs programming for Alhurra and increasing the amount of local  content Radio Sawa broadcasts. However, MBN has not developed a long- term strategic plan that fully addresses its operational and competitive  challenges.
subsections: section_title: Radio Sawa and Alhurra Face Significant Competitive Challenges
paragraphs: MBN faces several significant competitive challenges. These include the  competitive Middle East satellite television market; operational and  programming competitive disparities, such as Alhurra’s lack of news  bureaus compared with its competitors; and lack of coverage in certain FM  radio markets.
MBN operates in the competitive Middle Eastern satellite television market  that has over 140 channels. Pan-Arab satellite television stations–in  particular, the news stations Al Jazeera and Al Arabiya--are currently the  primary competitors to Alhurra. According to the BBG’s research firm  Intermedia, Al Jazeera is currently the top international broadcaster as a  source of news and information for audiences in many countries  throughout the Middle East. Moreover, Alhurra will face new competition  from the BBC’s entry to the Middle Eastern satellite television market in  2007. BBC officials have indicated that the new station’s overall approach  in the region will be multimedia in focus, taking advantage of the BBC’s  more than 60 years experience of broadcasting on the radio to the region as  well as its award-winning Arabic-language news Web site. For Radio Sawa,  the primary competitive challenge comes from existing local radio stations  in its broadcast range and the BBC World Service in Arabic, as well as from  the generally increasing competitiveness of the Middle East radio market.
Alhurra also faces operational and programmatic competitive disparities,  since both Al Jazeera and Al Arabiya are estimated to receive significant,  although unknown, levels of funding from their respective supporting  Qatari and Saudi financiers--allowing them to develop large networks of  correspondents and bureaus throughout the Middle East and other parts of  the globe. Al Jazeera, in particular, has bureaus in over 30 locations across  6 continents, which enable it to respond to breaking news events on a  timely basis. Alhurra and Radio Sawa, by comparison, only have overseas  bureaus in Baghdad, Dubai, and Amman. In addition, the BBC has a large  network of correspondents and bureaus around the globe and, unlike  Alhurra, has a vast in-house library of desirable BBC-produced content,  including documentaries and current affairs programming, which can be  readily translated into Arabic. The BBC also has favorable licensing and co- production arrangements with many companies.
One of Radio Sawa’s other primary challenges is its lack of broadcast  coverage in certain countries in the Middle East region. For example, Radio  Sawa does not have any broadcasting coverage in Tunisia, Libya, and  Algeria. Moreover, it has faced difficulties expanding its transmission to  include FM coverage in some countries, such as Egypt, Saudi Arabia, Syria,  Yemen, and Oman. MBN is attempting to negotiate transmission  agreements with several of these countries, but still faces significant  challenges to finalizing agreements.
section_title: MBN Has Taken Some Steps to Address Its Competitive Challenges
paragraphs: Since its inception, MBN has conducted some planning exercises to  address its competitive challenges. These have included developing a “2006  Goals and Strategies” overview document to guide operations for the  current fiscal year, establishing a 2006 annual performance plan as part of  the Office of Management and Budget’s Program Assessment Rating Tool  process, and participating in the development of the BBG’s new long-term  strategic plan covering fiscal years 2008-2012. In addition, MBN conducts  ongoing assessments of its competitors and uses various types of market  research to gain information about its audience and media usage patterns  in the Middle East. It has used this information to make adjustments to its  programming within its current budget, and also to develop proposals for  obtaining additional funding for new efforts. BBG and MBN officials have  explained that they use audience surveys, audience monitoring panels,  focus groups, in-depth interviews, and Arab television and music station  monitoring to inform MBN’s current efforts and planning, whenever  possible.
As a result of market research performed in August 2004, MBN officials  identified television viewing patterns and made changes to Alhurra  program schedules, such as by offering programming appealing to  women (e.g., current affairs and health and fitness programming) during  the daytime.
Through a review of the current competition in the market, MBN  officials decided that it was important to increase the number of debate  programs they broadcast on Alhurra. As a result, MBN created a series  of “town hall meetings” that allowed journalists and experts to discuss  issues of regional interest with interaction from a live audience.
As a result of audience monitoring panels, MBN officials made changes  to Radio Sawa’s program schedule by adding new features on subjects  such as social and cultural issues. MBN officials also learned of the  importance of efforts to localize those features, and made changes to  tailor programs to the interests of audiences of the various Radio Sawa  streams.
MBN has also developed program enhancement proposals for Alhurra and  Radio Sawa as part of the BBG’s language review and budget request  processes. Several proposals were included in the President’s fiscal year  2007 budget request. For example, the budget request includes a proposal  to increase Alhurra’s newsroom hours to increase on-the-spot and breaking  news coverage. In addition, in fiscal year 2005, the administration  requested additional funds for providing satellite Alhurra broadcasts to  Europe. According to planning documents, many of the Alhurra proposals  were designed to reinforce one another with the goal of improving  Alhurra’s credibility, as well as building audience size and increasing  viewing time of those who already tune in. MBN also developed one  proposal for enhancing Radio Sawa’s operations that calls for increasing  the amount of localized news content offered on five regional streams,  which officials say would allow the station to more effectively compete  with local stations in its broadcast range. MBN’s president said that, given  the increasing level of competitiveness in radio broadcasting in the Middle  East and expressed audience interest in news about their home country,  creating more localized content on Radio Sawa streams is important.
section_title: MBN Has Conducted Planning Exercises but Lacks a Long-term Strategic Plan
paragraphs: Strategic planning is a good management practice for all organizations.  Although MBN has conducted some planning exercises, it lacks a long-term  strategic plan and a strategic approach that outlines (1) a shared vision of  operations for Alhurra and Radio Sawa, (2) detailed implementation  strategies to achieve measurable outcomes for its goals, and (3) the  competitive challenges it faces and how it plans to address its key  challenges to broadcasting in the Middle East.
Strategic planning, including the development of a strategic plan, is a good  management practice for all organizations. Additionally, risk assessment is  an integral part of strategic planning. According to GAO guidance,  organizations should make management decisions in the context of a  strategic plan, with clearly articulated goals and objectives that identify  resource issues and internal and external threats, or challenges, that could  impede the organization from efficiently and effectively accomplishing its  objectives. Additionally, Office of Management and Budget (OMB)  guidance suggests that strategic plans contain, among other things, a  statement of the organization’s long-term goals and objectives; define  approaches or strategies to achieve goals and objectives; and identify the  various resources needed and the key factors, risks, or challenges that  could significantly affect the achievement of the strategic goals.
MBN has yet to create its own long-term strategic plan. MBN’s president  stated that funding uncertainties and other more pressing organizational  needs--such as the development of financial and administrative policies and  procedures--have delayed the development of MBN’s strategic plan and  related planning policies. In addition, he commented that MBN did not  emphasize planning in its early stages because it was focusing on making  its networks broadcast-ready. BBG officials said another reason for the  delays in planning is that the BBG and MBN are still learning about the  market, especially for Alhurra, and are taking a close look at the results of  audience surveys, focus groups, and in-depth interviews to determine the  best direction for these initiatives. MBN has stated that, to date, it has  primarily used the BBG strategic plan for organizational guidance.  Nevertheless, BBG officials said the BBG also has the expectation that all  broadcasting entities will develop their own strategic plans, particularly to  guide funding decisions.
In the absence of a strategic plan of its own, MBN lacks a comprehensive,  strategic approach that fully outlines (1) a shared vision of operations for  Alhurra and Radio Sawa, (2) detailed implementation strategies to achieve  measurable outcomes related to its goals, and (3) the competitive  challenges it faces and how it plans to address them.
First, MBN does not have a comprehensive strategic vision for the  integration of Radio Sawa and Alhurra operations in the organization. For  example, although MBN’s most recent annual performance plan contains a  goal to “integrate news operations for more effective television and radio  news-gathering,” none of MBN’s current plans outline specific, shared  objectives for Radio Sawa and Alhurra. MBN officials told us that several  steps toward integration of Alhurra and Sawa have occurred to date, such  as sharing financial and administrative support staff. However, Radio Sawa  and overseas bureau staff we talked with said that cooperation between  Alhurra and Radio Sawa is limited, the identities of the stations are  separate, and the two stations work largely independently of one another.  Radio Sawa staff noted several areas for further increasing cooperation,  including more sharing of interviews, sound bites, field correspondents,  Web site stories, and copy editors. One staff member in the Baghdad  bureau said that Radio Sawa and Alhurra operations in Iraq are completely  independent, including separate offices, and only the financial activities of  both offices are supervised by the same person.  In addition to gaining  more efficiency in operations, a vision for further integration of Radio  Sawa and Alhurra may help MBN more effectively identify opportunities to  address its challenges from increasing competition in the Middle East.
Second, MBN has not yet developed detailed implementation and resource  strategies needed for successful implementation of its goals. For example,  with regard to MBN’s initiative to localize content on Radio Sawa, we were  not able to identify a plan directing what types of local news and features  will be considered on the various streams, to what degree existing program  schedules might be affected, and how required resources might be divided  among the various streams. Additionally, although MBN’s most recent  annual performance plan states a goal of “expanding overseas production  of news coverage for radio and television,” neither that document nor any  other plan MBN identified provides details or direction for the overseas  production of news for Alhurra in existing overseas offices. MBN officials  have stated that uncertainties in the future commitment of resources to  Alhurra have affected MBN’s ability to, for example, plan for and use  existing overseas offices for Alhurra news. Further, BBG officials have said  MBN is still learning about the Middle Eastern media market. However,  given MBN’s internal enhancement requests to the BBG to increase the  number of news bureaus in the region, among others, it should clarify, for  example, what implementation steps are necessary to maximize the use of  existing overseas offices.
Third, MBN has not yet comprehensively outlined its challenges or  developed a strategic approach for how it plans to address its key  challenges to broadcasting in the Middle East. While MBN is planning to  expand its broadcast operations into Europe, it has not clearly identified  how its broadcasts will meet competitive challenges in the Middle East. As  an example, MBN has not indicated how it will address the implications of  the upcoming BBC Arabic-language television initiative. The BBC could  gain a significant audience that potentially would interfere with Alhurra’s  market share, credibility, and use as a source of alternative information.
section_title: MBN Has Made Progress in Developing Internal Control, but Additional Steps Could Improve Its Control Structure
paragraphs: MBN was initially limited in developing its internal control structure  because it was focused on quickly starting up its broadcasting operations.  In response to an external review of its financial operations by Grant  Thornton LLP in May 2004, MBN strengthened several of its controls, after  which it received an unqualified opinion on its Fiscal Year 2005 Single  Audit. However, MBN has not fully implemented several of the Grant  Thornton review’s key recommendations related to its control  environment, including (1) establishing an internal control board to  formally develop its controls and coordinate audits, (2) preparing an  internal control plan, (3) conducting a risk assessment to address potential  risks to its operation, and (4) developing a training program for its staff.
subsections: section_title: MBN Faced Initial Challenges in Developing Its Internal Control Structure
paragraphs: Internal control refers to the policies and procedures that help ensure the  proper management and application of an organization’s assets.  Clear,  strong controls can provide some assurance that management problems  are unlikely to occur or will be addressed if they do occur. MBN’s internal  control is governed by several OMB circulars cited in its grant agreement.
The Comptroller General’s Standards for Internal Control in the Federal  Government also provides guidance that is available to MBN.
MBN faced some initial challenges in establishing its internal control  structure. According to MBN documents and officials, MBN management  initially focused on establishing broadcasting operations rather than the  development of internal control policies and procedures, because the  organization had only several months to plan the launch of its 24-hour a day  Alhurra television network. As a result, MBN’s internal control lagged  behind.  The MBN chief financial officer (CFO) told us that problems  encountered by MBN in hiring and retaining staff added to the delay in  developing internal controls.
Due to concern over the slow development of MBN’s internal control  structure, the BBG commissioned a review by Grant Thornton LLP  accountants and management consultants, which was completed in spring  2004, to assess MBN’s internal control and make recommendations for  improvement. The report by Grant Thornton LLP cited numerous findings,  such as inadequate financial policies and procedures, understaffing, and  inadequate training, that impeded MBN’s ability to successfully mitigate  risks. The report, however, also noted that MBN’s controls were  improving.
section_title: MBN Has Made Progress in Developing Its System of Internal Control, but Some Control Elements Could Be Improved
paragraphs: Grant Thornton’s May 2004 report on MBN’s system of internal control  made recommendations that covered staffing, MBN’s financial system,  training, administrative policies and procedures, developing a decision  support structure, improving logs and records, and MBN’s control  environment. MBN accepted the review’s recommendations and agreed to  implement them, according to MBN officials, and the BBG concurred with  these recommendations. Our analysis shows that MBN subsequently  responded to many of the recommendations, including hiring additional  financial department staff, developing financial and administrative policies  and procedures, and completing an annual single audit (see table 1).  For  example, although MBN provided us with copies of its Fiscal Year 2003 and  Fiscal Year 2004 Single Audits well past the deadlines for completing those  audits, MBN’s Fiscal Year 2005 Single Audit was completed on time,  provided an unqualified opinion, and showed marked improvement over  previous years.
Some control elements could be improved in order to better implement  best management practices based on OMB circulars and GAO internal  control standards. For example, although MBN is establishing an internal  control board, the board has not met to establish protocols and outline its  responsibilities. The organization also has not developed an internal  control plan. Furthermore, MBN has not established a comprehensive  process to analyze risks the organization faces from internal and external  sources. Finally, MBN has provided some training on internal control but  has not yet developed a regular structured training program for its staff.
subsections: section_title: MBN’s Internal Control Board Has Not Met to Establish Protocols to Develop and Monitor Its Internal Control Structure
paragraphs: The Grant Thornton review recommended that MBN establish an internal  control board of key managers and officers to determine the internal  control risks facing MBN, work towards decreasing these risks, and  oversee MBN’s efforts to employ strong controls. Moreover, according to  GAO guidance, organizations should have an audit committee or senior  management council–similar to an internal control board–that reviews the  internal audit work and coordinates closely with external auditors.
According to the MBN president, MBN is establishing an internal control  board consisting of three members, including, as recommended, the MBN  president, general counsel, and CFO. However, the board has not yet  formally met to establish protocols and outline responsibilities. In addition,  according to MBN’s general counsel, MBN has not appointed a member  from the BBG, as was recommended, to serve on the board. MBN’s  executive committee, which examines issues affecting MBN and reports  back to the BBG, has provided some support on management and  administrative issues, such as approving the construction and expansion of  MBN’s new facility and providing guidance on hiring high-level staff.   However, the committee has not fulfilled the role of an internal control  board as previously described. Instead, MBN’s CFO has largely taken on the  sole responsibility of establishing and overseeing MBN’s controls,  reviewing audits, and coordinating with external auditors. The MBN  president told us that they have not convened an internal control board  because the organization is too new and, therefore, is focused on  developing policies and procedures rather than mechanisms to review  them. MBN is planning to develop an internal audit function, implemented  by an external firm, to provide assurance to MBN management that the  organization is operating appropriately.
section_title: MBN Has Not Completed an Internal Control Plan
paragraphs: According to the Grant Thornton review, MBN should develop an internal  control plan to ensure that effective controls are established and monitored  regularly. Such a plan should identify the roles and responsibilities of all  individuals whose work affects internal control, lay out specific control  areas, cover risk assessment and mitigation planning, and include  monitoring and remediation procedures. MBN officials told us in January  2006 that they were in the process of developing such a plan and have  developed internal control guidelines, but as of the end of May 2006, they  had not provided us with a finalized plan.
section_title: MBN Has Not Established a Comprehensive Process to Analyze Risk
paragraphs: The Grant Thornton review called on MBN to conduct a broad risk  assessment, led by its internal control board, to evaluate and mitigate  potential obstacles to efficiently and effectively achieving its operational  objectives. According to Grant Thornton LLP, failure to conduct an MBN- wide risk assessment could result in the loss of resources and could  decrease confidence of the grantor and of Congress, which could  ultimately compromise MBN’s achievement of its mission. According to  GAO guidance, risks should be identified during both short- and long-term  forecasting and as part of strategic planning. Moreover, after conducting a  risk assessment, organizations need to develop internal control activities to  manage or mitigate the risks that have been identified. For example, Radio  Free Asia identified avian influenza and signal jamming from China as two  threats to their operations. As part of their risk assessment, they  considered how to address and overcome these issues, such as by  broadcasting from alternate locations.
In February 2005, MBN prepared an initial risk assessment that identified a  list of actions taken to address the issues raised in the Grant Thornton LLP  report. However, the document did not identify the organization’s  objectives or the risks it faces, nor did it analyze the possible effects of the  risks or propose a strategy to mitigate them, as recommended by GAO  guidance. MBN officials told us that they assess risk on an ongoing,  biweekly basis. However, in taking a short-term approach to analyzing risk,  MBN lacks a comprehensive basis from which to establish a strong internal  control structure. Some risks identified by MBN include threats to the  security of its staff and bureaus in the field, particularly in Iraq, and the risk  of a terrorist attack on its facilities in the United States. There are some  risks that MBN has not identified or addressed. For example,  PriceWaterhouseCoopers, MBN’s external auditor, noted that MBN has not  adequately addressed its risks related to information security. Doing so  would reduce the risk of security incidents and unauthorized system  activity, according to the auditor. PriceWaterhouseCoopers also found that  MBN’s lack of a business continuity plan or an adequate disaster recovery  plan could result in slower recovery in the case of such an event, as well as  significant loss of revenue, inability to meet customer needs and third party  obligations, and potential noncompliance with legal requirements. The  internal audit function planned by MBN may at some point take on the  function of assessing risk, but this body is not yet operational.
section_title: MBN Has Not Developed a Comprehensive Training Program on Internal Controls
paragraphs: MBN has provided some training on internal controls but has not yet  developed a regular structured training program for its staff of about 240,  as recommended by the Grant Thornton LLP review and by GAO leading  practices. MBN’s CFO and controller attended a seminar on grants  management in October 2004 and subsequently shared the information  with the 16 in-house financial staff. MBN also provided internal control  compliance training to its managers in December 2004, and the  organization regularly provides training to staff at its business manager  meetings.  However, other U.S. broadcasting entities, such as Radio Free  Europe/Radio Liberty (RFE/RL) and  Radio Free Asia (RFA), have more  organized, ongoing training programs on internal control. The MBN CFO  concurred that there is a great need for internal training. According to him,  the underdeveloped training situation is due to a lack of resources,  including a lack of funds specifically designated for training and limited  time to plan or implement training. This lack of structured, recurrent  internal control training can cause problems if staff are unfamiliar with an  organization’s business processes and controls, and can lead to the  inefficient or improper use of resources.
section_title: MBN Has Developed Editorial Procedures to Support Its Journalistic Standards, but Could Improve Monitoring and Some Quality Control Measures
paragraphs: MBN has established journalistic standards, as well as procedures to help  ensure that the organization’s broadcasts comply with these standards.  However, it has not fully developed some quality control measures, such as  the use of listener and viewer feedback. Additionally, the BBG has not held  regular comprehensive program reviews for MBN, thereby making it  difficult for MBN to assure its audience, Congress, the administration, and  the BBG that its controls are working and that it is broadcasting quality  programming.
subsections: section_title: MBN Has Developed Journalistic Standards and Implemented a Wide Variety of Pre- and Postbroadcast Editorial Procedures
paragraphs: The International Broadcasting Act of 1994 calls for U.S. international  broadcasting to be conducted in accordance with the highest professional  standards of broadcast journalism, including the production of news that is  consistently reliable, authoritative, accurate, and objective. The act also  calls for U.S. international broadcasting to present a balanced and  comprehensive projection of U.S. thoughts and institutions, as well as clear  and effective presentation of U.S. government policies and responsible  discussion of those policies. MBN’s mission statement, which partly draws  upon the principles and standards contained in the U.S. International  Broadcasting Act of 1994, calls for MBN to broadcast factual, timely, and  relevant news and information that promotes freedom and democracy.
MBN has developed journalistic standards, including a code of ethics, as  part of its effort to ensure that its news broadcasts are consistently  accurate, authoritative, objective, balanced, and comprehensive. MBN’s  general counsel said that the code was also established to ensure that MBN  fully complies with its mission and the U.S. International Broadcasting Act  of 1994. MBN’s journalistic standards were drafted by MBN management  using input from professional journalistic organizations and another  grantee. According to BBG officials, MBN’s standards appear to be as good  as those of other U.S. international broadcasters. In our analysis, we found  that the standards cover areas similar to the codes of other broadcasters-- such as RFE/RL, RFA, Voice of America (VOA), and National Public Radio-- focusing on accuracy, impartiality, establishing context, clearly  distinguishing analysis from reporting, using a tone of moderation and  respect, avoiding advocacy, and promoting ethical conduct. The standards  also include guidelines for conducting interviews, as well as editing and  production requirements. For example, according to MBN officials, MBN  strives to present opposing views accurately and achieve a balance among  the guests on its current affairs shows. According to MBN officials, when  broadcasting about the war in Iraq, they try to ensure that the program  incorporates both pro-war and antiwar views.
To help ensure that staff comply with MBN’s journalistic standards, Radio  Sawa and Alhurra have established a number of pre- and postbroadcast  procedures, which are roughly similar to those of other U.S. international  broadcasting entities. Examples of MBN editorial procedures include daily  editorial meetings, the use of two or more sources to support a news item,  checks by editors and producers to determine whether news stories are  properly written and accurate, a headquarters-level review of all materials  produced in MBN’s overseas offices before broadcast, and postbroadcast  discussions. In addition, we observed that MBN employs an experienced  journalist from the Arab world to review all of Alhurra’s and Alhurra-Iraq’s  weekday newscasts for technical and stylistic errors, a control not  implemented by other U.S. broadcasting entities. The journalist watches  the newscasts just before they are aired, then provides MBN management  with an evaluation of the newscast’s quality. MBN provided us with records  of these evaluations, which assess whether the newscasts are presentable,  balanced, and free of technical errors. In some cases, technical mistakes  can be caught before the piece is aired. However, while MBN management  follows up on critiques of individual journalists, they do not systematically  review and assess the journalist’s evaluations.
MBN officials told us that the organization places a high value on  journalistic controls, particularly due to the volatility of certain areas in the  Middle East and the impact news reports can have. In addition, MBN’s  controls can serve as an assurance to its audiences and others that they are  broadcasting quality programming. Since Radio Sawa and Alhurra were  established, MBN has not had to retract a single story or apologize for any  error, according to MBN officials.
section_title: MBN Could More Fully Develop Some Quality Control Measures and Lacks Regular Editorial Training
paragraphs: There are several areas in which MBN could more fully develop some  quality control measures for its programming. These areas include using  listener and viewer feedback to improve program quality, making better use  of weekly compilations, and ensuring its style guide is distributed to all  staff.  Unlike other U.S. international broadcasters, MBN typically only  partially utilizes the following measures:  Although MBN collects feedback from its listeners and viewers, it does  not rely extensively on this feedback as a program quality control.
Although MBN produces weekly compilations that are distributed to  interested parties summarizing what was broadcast on Radio Sawa and  Alhurra that week, it does not use these compilations to do any formal,  long-term analysis of errors and programming.
MBN has developed a style guide to provide critical guidance on the use  of sensitive terms and to help staff avoid grammatical mistakes.  However, Radio Sawa staff did not receive the guide until early 2006,  and MBN has not distributed it to all of its overseas offices, inhibiting its  use by some staff.
In addition, MBN has not provided regular training to its journalists and  producers. MBN told us that it does not provide regular training to help its  editorial staff maintain and increase their professional competence. MBN’s  level of training for its editorial staff is also not on a par with the training  offered by other grantees. For example, RFE/RL and RFA both have  extensive training programs for their employees at their headquarters and  in their bureaus in different countries, according to RFE/RL and RFA  officials. While MBN provides some initial technical training to its  journalists, an informal mentoring program, and sporadic training  overseas, according to MBN officials, the organization does not have an  ongoing training program to educate journalists throughout the  organization about editorial and ethical issues they might encounter on the  job. According to the MBN CFO, the organization does not have a well- established training program because the network is relatively new and  lacks training resources. MBN officials have also noted that their most  important controls are the editors themselves, and that MBN tries to hire  experienced staff. However, regardless of experience, staff can and do  make mistakes, and MBN’s lack of regular training increases the risk that  correspondents and editors will make mistakes.
section_title: MBN Lacks Comprehensive, Regular Program Reviews to Determine Compliance with Mission and Journalistic Standards
paragraphs: The BBG’s main mechanism to determine whether its broadcasting services  comply with its mission and journalistic standards is a regular program  review, which is designed to improve programming and ensure quality  control. However, only one review has been conducted for Radio Sawa, and  none have been conducted for Alhurra. The Radio Sawa review was more  limited in scope than program reviews conducted by other entities. In  addition, Radio Sawa’s program quality score is inconsistent with other  BBG entities, and without a program review to develop a program quality  score for Alhurra, the BBG will not be able to measure the contribution of  these efforts to the goals of the organization, or be able to ensure that the  quality of Alhurra’s broadcasts conforms to applicable standards. Finally,  many MBN staff were not extensively involved in the Radio Sawa review.
subsections: section_title: Annual Program Reviews Have Not Been Conducted for MBN’s Services
paragraphs: According to written guidance from the International Broadcasting Bureau,  which coordinates and supports all VOA program reviews, program review  is an annual process by which an institution judges itself and solicits the  judgment of others to make improvements and fulfill its mission with  regard to U.S. national interests. The process enables the broadcasting  entity to better connect its mission to the market where it is broadcasting  and assess whether its editorial procedures are functioning effectively,  while also allowing the BBG to fulfill its requirements from OMB that it  conduct regular program evaluations to capture a program’s impact over  time. In addition, the Senate Committee on Foreign Relations Report on the  Fiscal Year 2003 Foreign Relations Authorization Act called for significant  resources to be dedicated to postbroadcast analysis of Radio Sawa  programming to ensure that broadcasts are consistent with U.S. interests  and values and with the standards in the U.S. International Broadcasting  Act. Program review typically includes a study of the target area’s media  environment, analyses by internal and external reviewers, background  quantitative research, reports on marketing and transmission, and target  area profiles compiled by the broadcaster itself. Reviewers rate programs based on criteria for content--such as accuracy, timeliness, objectivity,  relevance, and quality of analysis and interviews--and for presentation,  such as pace and liveliness, presentation style, sound quality, and host  interaction. These inputs are then discussed at a meeting that includes the  program review coordinators and the management and staff of the entity  being reviewed. Following the main program review meeting, key  participants develop an action plan, and 3 months later the group meets  again to determine to what extent the action plan has been carried out.
Although the BBG calls for program reviews to be conducted annually,  MBN has not complied with this guidance. Neither the BBG nor MBN has a  regular mechanism in place to systematically review MBN programs. In  December 2004, the BBG convened a program review meeting for Radio  Sawa, which had begun operating in March 2002. However, there has been  no Radio Sawa review since then, while Alhurra, which has been operating  for more than 2 years, has had no program review at all. The BBG is  planning to initiate a program review of Alhurra by the end of this year, but  has not set a firm date.
section_title: Radio Sawa’s Program Review Was Limited by Its Emphasis on Audience Input
paragraphs: BBG officials have stated that program quality should be sampled and  assessed by both internal and external evaluators. This is designed to  produce a balanced and robust review. In 2000, in response to our report  that found a lack of consistency in how program quality scores were  developed, the BBG stated that it intended to harmonize and standardize program reviews across broadcasting entities. BBG guidance now calls  for all U.S. international broadcasting entities to be evaluated using the  same standards, definitions, and scoring methods.
However, when conducting the Radio Sawa program review, the BBG relied  only on audience monitoring panels to assess program quality and did not  utilize internal analysts or external control listeners, as is common practice  among other U.S. international broadcasters. Using only audience  monitoring panels gives the audience more weight in the review results  and, in turn, more potential to influence the strategic direction of the  organization. In addition, since the audience tends to be unfamiliar with a  broadcasting service’s journalistic standards and editorial procedures,  having input only from monitoring panels makes it more difficult for a  service to provide reasonable assurance that its editorial procedures are  working and that it is broadcasting quality programming. Also, external  control listeners are specifically tasked with examining the programming in  light of the service’s mission, something audience monitoring panels are  not asked to do. As a result, Radio Sawa’s program review strongly  emphasized the audience’s perspective and therefore provided a less  thorough evaluation of Radio Sawa’s mission and standards. (For more  information on Radio Sawa’s program review, see appendix III.)
section_title: Radio Sawa’s Program Quality Score Is Inconsistent with Other BBG Entities
paragraphs: Furthermore, MBN’s program score for Radio Sawa is not comparable with  other BBG broadcaster scores for program quality--the percentage of a  station’s language services judged on both content and presentation criteria  as being of good-or-better quality. The BBG says that to measure Radio  Sawa’s program quality, it has developed standardized criteria applicable to  different media and methods of delivery, while minimizing subjective  judgments on content and presentation. Although the criteria used to  measure Radio Sawa’s program quality are similar to those used by other  broadcasting entities, the BBG did not use as many inputs when calculating  the program quality scores for Radio Sawa, leading to a less robust result  (see table 2).
Without a consistent process for broadcaster program reviews, the BBG is  limited in its ability to assess and compare broadcaster performance. In  addition, without a program review to develop a program quality score for  Alhurra, the BBG will not be able to measure the contribution of these  efforts to the goals of the organization, or be able to ensure that the quality  of Alhurra’s broadcasts conforms to applicable standards. Currently, the  BBG is considering how it will conduct the future program reviews for  Radio Sawa, but officials could not yet provide specifics about the  approach BBG will use.
section_title: MBN Staff Had Limited Awareness of the Radio Sawa Program Review and Follow-up
paragraphs: MBN staff in the United States and overseas do not appear to have much  knowledge of the Radio Sawa review or its follow-up. Program review can  be a learning experience for staff, who are usually encouraged to sit in on  their language service’s program review meeting, and the process can also  bring people together who normally do not interact, which can help  generate ideas and improvements. However, according to MBN  management, most Radio Sawa staff did not attend the Radio Sawa  program review. In addition, there was also not much awareness about the  program review in MBN’s overseas offices.
While Radio Sawa staff lacked awareness about their review, some staff  who had worked at VOA in the past spoke positively of the program review  process in general. Other broadcasters routinely involve staff in their  program reviews. For example, RFA requires all staff from the service that  is being reviewed to be present at the program review meeting. According  to an RFA official, program review is one of the few times that all of the key  players for a service are in one place, and it therefore presents a good  opportunity for communication.  In addition, a BBC official noted that  program review can be a good way to expose staff to an organization’s  values. Increasing staff involvement in program review could therefore  increase chances for communication throughout all levels of the  organization, as well as provide a forum for discussing potential  programming improvements.
section_title: BBG Has Developed Several Performance Indicators, Although Weaknesses in Audience Surveys Call into Question Whether MBN Audience Size and Credibility Targets Have Been Met
paragraphs: The BBG has established several standard performance indicators and  targets for MBN programs, including measures of audience size and  credibility; however, it has not implemented some performance indicators  fully, including a program quality measure. Additionally, we were unable to  determine the accuracy of MBN’s audience size and program credibility  estimates due to weaknesses in MBN’s methodology and documentation.  Therefore, it is not clear whether the Radio Sawa and Alhurra performance  targets have been met.
subsections: section_title: MBN Performance Indicators and Targets Have Been Established, but Some Are Not Fully Developed
paragraphs: The Government Performance and Results Act of 1993 (GPRA) requires  that all government agencies establish performance indicators, or  measures, that provide a meaningful reading of how well the organization  is progressing towards its goals. The BBG has developed a standard set of  performance indicators for its broadcasting entities, which it says are a  best effort to measure its level of effectiveness now and where its  performance is targeted to be in the future. The BBG established common  indicators for its entities to allow it to better assess overall progress for the  organization. Although the BBG has made progress in establishing these  standard performance indicators, as well as targets for MBN’s programs, it has not formally established or implemented all of them for Radio Sawa  and Alhurra.
According to BBG, the three most important standard indicators for its  entities, referred to as primary performance indicators, are (1) audience  size, or the overall weekly audience of a station; (2) credibility, which  represents the percentage of viewers in a target area that consider the  station’s news programs somewhat or very reliable; and (3) program  quality, or the percentage of a station’s programs judged on standard  criteria as being of good-or-better quality. These three indicators are tied to  the current BBG mission and strategic plan for U.S. international  broadcasting. In addition to its standard primary performance indicators,  the BBG has a number of “secondary” measures that provide management  with additional information for gauging cost- effectiveness, marketing and  promotion activities, and transmission efforts, including the number of  transmitters and affiliates, cost per listener, signal strength (radio only),  and awareness.
The BBG has established and implemented two primary indicators– audience size and credibility--for both Radio Sawa and Alhurra. The BBG  has also established performance targets for Radio Sawa and Alhurra for  these two indicators. For example, MBN’s reported fiscal year 2005  audience size performance target for Radio Sawa was 18 million listeners  and for Alhurra was 12.8 million viewers. However, the BBG has not yet  established Alhurra’s program quality indicator, or consistently  implemented this measure for Radio Sawa. Without a measure of program  quality for Alhurra and Radio Sawa, the BBG will not be able to  consistently assess MBN’s performance against that of other grantees, or  fully assess MBN’s contribution to the overall goals of the BBG  organization. The BBG has established all of its standard secondary  performance indicators for MBN’s services except for audience  awareness. See table 3 for a list of the performance indicators  implemented for Alhurra and Radio Sawa.
section_title: BBG Should Increase Steps to Ensure Reliability and Transparency of Its Performance Estimates
paragraphs: We and others have noted that agencies’ performance indicators and data  should provide a reliable means to assess progress. However, we were  unable to determine the accuracy of MBN’s reported audience size and  program credibility estimates due to weaknesses in MBN’s methodology  and documentation. Therefore, it is not clear whether the Radio Sawa and  Alhurra performance targets have been met. While BBG has taken several  important steps to ensure the validity and reliability of its performance  measurement approach, it has primarily used a methodology that cannot be  reliably projected to the broader population. Although it is difficult to  conduct probability sampling in many locations in the Middle East, the  BBG has not taken steps to explain and increase the reliability of MBN’s  performance information, such as by maintaining more detailed  documentation to support its estimates, reporting significant data  limitations, limiting the scope of its projections to areas actually covered  by its surveys, and developing BBG policies and procedures for verifying  performance data.
subsections: section_title: Measurement and Research Methods Should Allow for Confidence in Reliability of Performance Information
paragraphs: We have previously reported that performance indicators should provide a  reliable way to assess progress. In particular, agencies should implement  quality control procedures to mitigate errors that can occur at various  points in the collection, maintenance, processing, and reporting of  performance data and can impact its reliability. In addition, agencies  should select sampling methods that ensure representative samples, where  possible. For example, probability surveys are designed to ensure each  person in the population has a measurable chance of being selected for the  survey, enabling the results to be reliably projected to the larger population  with known levels of precision. Additionally, agency performance reporting  should provide sources, disclose limitations, and discuss the implications  of them. Explaining the limitations of performance information--as well as  actions taken to compensate for low-quality data–can provide context for  understanding and assessing agencies’ performance and the costs and  challenges that agencies face in gathering, processing, and analyzing  needed data.
The U.S. International Broadcasting Act of 1994 requires the BBG’s use of  audience research to guide its decisions about its language services. For  the past 5 years, the BBG has contracted with Intermedia to serve as the  primary research contractor for the BBG and its broadcasting entities,  including MBN.  Intermedia works with subcontractors, partners, or both  to gather audience research survey data from citizens in various locations  overseas. This survey information is used by the BBG to develop estimates  for MBN and other entities’ audience size and credibility performance  indictors. For example, for fiscal year 2005, the BBG estimated that Radio  Sawa and Alhurra had each achieved an audience size of about 21.6 million  people. Table 4 shows the breakdown of individual survey data reported by  the BBG.
Although the BBG has taken several important steps to enhance the validity  and reliability of its audience survey designs, several factors call into  question the accuracy of the data used for BBG’s audience size and  credibility performance indicators for Radio Sawa and Alhurra. First, we  observed that, in 12 out of 14 cases, the BBG used nonprobability surveys,  which cannot be reliably projected to the broader population, to develop its  regional estimates for audience size and credibility. While it is difficult to  conduct probability surveys in hostile environments, such as those in Iraq  and Saudi Arabia, the BBG did not take certain steps that could have  increased the accuracy of its estimates and explained their limitations,  thereby increasing confidence in the data. Therefore, we are unable to  determine whether MBN actually met its performance targets for fiscal  year 2005.
BBG has taken several important steps to enhance the validity and  reliability of its audience survey designs. For example, the BBG’s  questionnaires are reviewed by multiple parties, and its contractors  extensively pretest the questionnaires in the field, conduct pilot studies,  and use throwaway surveys for training new subcontractors. In addition,  the BBG’s contractors exercise quality control when collecting data and  receiving it from the field, including by conducting preliminary electronic  testing of the data, among other things.
However, in 12 of 14 cases, we found the country-level estimates used for  generating Radio Sawa and Alhurra’s fiscal year 2005 performance  indicators of audience size and credibility were not based on probability  survey results. Many of the surveys conducted used judgment sampling, a  form of nonprobability sampling, instead. BBG officials told us that its use  of nonprobability surveys for certain countries is due to either the cultural,  political, or security situation in those countries, which limits the selection  of individuals or the geographical areas that can be surveyed. In addition,  the International Broadcasting Bureau director of research stated that in  many developing countries, existing map and population data is not  adequate to support pure probability-based sampling. The Conference of  International Broadcasters’ Audience Research Services, called CIBAR,  whose standards are specified as a source of guidance for BBG research  contractors, requires that audience measurement use samples based on the  principles of random probability and that other sampling methods should  only be used in cases where, for reasons of practicality or cost, proper  random samples cannot be used. (See appendix IV for more on the CIBAR  standards.)
We recognize that many agencies face challenges in collecting credible  performance data and that, due to security risks and political  considerations in many Middle Eastern countries, it is not always possible  to expect BBG to use random samples. However, the BBG did not take  certain steps that could have explained and increased the reliability of its  estimates, such as fully documenting its research methods, measuring the  level of uncertainty surrounding its estimates, disclosing significant  limitations, limiting the scope of its projections to areas actually covered  by its surveys, and developing and implementing procedures for verifying  data.
First, the BBG and its research contractors were unable to provide us with  certain documentation commonly required by international broadcasting  research standards. CIBAR requires that, in all measurement research, the  sampling methods used and other technical aspects of the survey be both  fully and accurately described in the project documentation and open to  independent scrutiny. We asked the BBG and its contractors to provide us  with detailed documentation–including clear information on sampling  plans and related assumptions, response rates, and adjustments applied to  the data to reflect the effects of the survey design–for all 14 of its Middle  Eastern country surveys used to develop its fiscal year 2005 performance  indicator estimates. For two cases, the BBG was unable to provide us with  any survey documentation, and for all but one case, the BBG and its  contractors were unable to provide us all the detailed information we requested. BBG officials acknowledged the lack of complete  documentation in contractor technical reports, and said it was due in part  to their failure to follow up with the contractors to obtain the details,  aswell as to the contractors’ general practice of not generating such  detailed documentation.
Second, the BBG has not sufficiently measured the level of uncertainty  surrounding MBN’s performance estimates. CIBAR requires that in all  measurement research, technical aspects of the survey, including margins  of error and confidence levels where appropriate, be both fully and  accurately described in the project documentation. The BBG has not been  able to measure sampling errors for its surveys, in part because it has not  required contractors to document the information that is needed to  calculate the sampling errors accurately. Moreover, officials said that it is  not customary for their contractors to maintain this information.  Therefore, the BBG only has a rough idea of what the margins of error  might be for its surveys, further limiting confidence in the reliability of its  current performance information. One research official told us that he  believed that the overall margins of error for Radio Sawa and Alhurra’s  audience size and credibility estimates are large, but said that currently the  agency cannot accurately calculate them.
Third, the BBG lacks transparency in reporting data sources and significant  limitations affecting MBN’s audience size and credibility performance  information. In reporting performance data, agencies should provide data  sources, disclose limitations, and discuss the implications of them. CIBAR  standards also recommend that proper care be exercised when reporting  estimates, to ensure that the type of audience covered by the estimate is  clearly stated and that, at all stages in the calculation and extrapolation  process, sources, assumptions, and methods be fully documented and  available for independent scrutiny. However, the BBG has not sufficiently  explained the specific methods used for generating estimates for its  performance indicators, such as the number and names of the countries  surveyed, methods of sampling used, sources of the population data, and  basic procedures used to create the estimates. Moreover, the BBG has  reported only two limitations for MBN performance indicators to date: (1)  that credibility ratings are highly dependent on volatile political factors;  and (2) that, depending on political, social, and media conditions,  measurement of audience size may either be easily attained or impossible.  However, the BBG has not explained many significant limitations, and their  resulting implications, on the data reliability of the performance  information for MBN. The largest identifiable limitations not reported are  that many of the BBG surveys are not based on probability sampling, cover  only part of the country, have very low response rates, or have high  substitution rates; therefore, those results cannot be reliably projected to  be representative of the larger population. In the case of the survey in  Morocco, we calculated that the survey only covered 35 percent of the  general population and had a 48 percent substitution rate, but the results  were projected to represent a broader population. In addition, we  calculated low response rates for a number of MBN’s surveys; in the case of  the survey of Egypt, the response rate was about 19 percent. BBG research  staff explained that their stakeholders to date, including the BBG, have not  required such a level of detail in reporting, and specifically have not  required margins of error. However, a discussion on data limitations in  performance reporting can help decision makers determine their level of  confidence in the agency’s ability to report on its performance goals and  indicators and identify actions needed to improve its ability to measure  performance.
Fourth, we found the BBG has not taken sufficient steps to avoid  projections to areas outside the population surveyed. For example, the  elimination of projections to the rural population of Morocco, when only  urban areas were surveyed, or to those governorates of Jordan not  contained in the survey, would have resulted in estimates of audience size  and credibility with fewer limitations for those countries. In addition, BBG  project documentation does not clearly describe what steps have been  taken to restrict the scope of MBN’s surveys to provinces or areas that have  sufficient map or population data. Probability sampling projected to those  areas only would result in more reliable estimates of audience size and  credibility.
Fifth, we found the lack of verification procedures inhibits assessments of  BBG’s data quality. Data reliability is increased by the use of verification  and validation procedures, such as checking performance data for  significant errors by formal evaluation or audit. However, BBG has not fully  implemented such procedures or formally documented policies and  procedures governing its research. BBG officials said they have  implemented some forms of verification, such as a research director  review, at various stages of performance data analysis. However, it is not  clear that the BBG always thoroughly verifies performance indicator  estimates and calculations used for reporting. For example, we identified  some errors in internal calculation spreadsheets for performance  estimates, in official external reporting, and in an informal presentation on  MBN performance. BBG officials have acknowledged that they need to  develop policies and procedures and implement more rigorous verification  of performance data. They said that, as a result of our review, they have  taken some steps to expand their verification procedures and will  document those and other research procedures in the BBG’s research  contract and manual of operations.
section_title: Conclusions
paragraphs: MBN was established to broadcast accurate and relevant news and  information to the Middle East in order to advance freedom, democracy,  and long-term U.S. interests in the region. MBN’s programs are challenged  by existing television competitors, such as Al Jazeera and Al Arabiya, and  planned initiatives such as the BBC’s planned Arabic-language television  network; by numerous local radio stations across the region; and by limited  opportunities for increasing coverage of radio transmission. It has  attempted to address these challenges through some planning efforts, such  as by developing proposals to increase its news time for Alhurra and  expand its number of news bureaus in the Middle East. However, MBN has  not developed a strategic plan or taken a detailed strategic approach to  addressing certain issues, including identifying opportunities for additional  gains that could be made from further integrating Radio Sawa and Alhurra  or fully utilizing MBN’s overseas offices. Developing a strategic plan that  establishes specific objectives, provides an overall shared vision or  framework for decision-making, and comprehensively addresses  competitive challenges could enable MBN to identify efficiencies or  opportunities to address its competitors more effectively and increase  congressional confidence in its operations.
Although MBN has developed a number of financial and administrative  controls, it could take additional steps to ensure its system of internal  control is fully implemented. MBN’s planned internal control board needs  to establish protocols to oversee and monitor its internal control structure  and to ensure timely completion of MBN’s financial audits. Additionally,  MBN should develop an internal control plan, conduct a comprehensive  risk assessment, and develop an organized training program for its staff.
Further, MBN has developed journalistic standards for its broadcast  operations and has put in place a number of editorial procedures. However,  the network has not conducted annual program reviews called for in BBG  guidance. These reviews are a key mechanism for improving programming  and ensuring quality control.
Finally, the BBG has established performance indicators and targets for  MBN services related to measures of audience size and program credibility.  For example, in fiscal year 2005, the BBG reported that Radio Sawa and  Alhurra had each achieved radio and television audiences of about 21.6  million. However, limitations in the survey methods and documentation  used for developing these estimates raise questions about the accuracy of  MBN’s performance estimates. In addition, the BBG has not put in place  policies and procedures for verifying the accuracy of its performance  information. These weaknesses in methodology and documentation inhibit  an accurate assessment of whether the Radio Sawa and Alhurra  performance targets have been met. The accuracy of the MBN estimates  could be improved by more rigorous reliance on probability sampling,  wherever possible, and avoidance of projections to areas not contained  within the survey. At the same time, insistence on proper fieldwork  documentation from contractors, full disclosure of survey methods and  limitations, and greater transparency in the development of aggregate  audience estimates would further enhance confidence in MBN  performance estimates.
section_title: Recommendations for Executive Action
paragraphs: To improve efforts to monitor performance and the efficiency and  effectiveness of the broadcasting activities of the Middle East Broadcasting  Networks, Inc., we recommend that the chairman of the BBG do the  following:  Require that MBN develop a long-term strategic plan, which  incorporates a shared vision for Alhurra and Radio Sawa operations and  details specific measurable objectives and implementation strategies for  the successful implementation of the goals in the plan.
Require that MBN implement the remaining recommendations from the  Grant Thornton LLP report and require that its internal control board  meet on a regular basis to coordinate MBN’s single audits and oversee  MBN’s ongoing efforts to use sound internal control procedures.
Develop a process for analyzing risk as part of strategic planning that  identifies approaches to mitigate the potential obstacles to efficiently  and effectively achieving MBN’s operational objectives.
Require MBN to develop a comprehensive training program covering  both internal controls and editorial procedures to meet the continuing  needs of all employees.
Initiate a schedule of annual program reviews for Radio Sawa and  Alhurra to regularly ensure that the quality of Alhurra’s broadcasts  conforms to applicable standards.
Implement program quality performance indicators for MBN’s broadcast  services, consistent with other BBG entities, to assess and compare  their performance and measure the contribution of these efforts to the  goals of the overall organization.
Require research contractors to improve the methods used in audience  research to allow for probability sampling and document the sample  selection so that survey sampling errors can be calculated, where  possible.
Identify and report significant methodological limitations and the  implications of them for performance indicators, including, where  applicable, sampling errors, margins of error, or confidence intervals.
Develop, document, and report policies and procedures for verification  and analysis of performance indicator estimates.
section_title: Agency Comments and Our Evaluation
paragraphs: The Broadcasting Board of Governors (BBG) provided written comments  on a draft of this report. The BBG’s comments, along with our response to  specific points, are reprinted in appendix V. The BBG also provided  technical comments, which we incorporated where appropriate.
In general, the BBG concurred with all of our recommendations and said it  looked forward to implementing them. The BBG said that MBN has made  significant progress in the 2 years of its operations in establishing a sound  journalistic organization with financial and administrative controls.  However, the BBG raised a number of concerns about the report’s  criticisms of the audience research conducted by the BBG and its  contractors. Specifically, the BBG said that we did not fully understand the  difficulty in surveying audiences in Middle Eastern countries and that the  research practices used by the BBG and its contractors follow industry  standards for commercial and media research.
Our report examines the reliability of the BBG’s fiscal year 2005  performance information in order to determine whether or not MBN’s  performance targets have been met. We acknowledge, in our report, that  BBG has taken positive steps to enhance the validity and reliability of its  audience survey designs. We also acknowledge that there are challenges to  conducting audience research in the Middle East, and that there are  tradeoffs between cost and data reliability when conducting research. In  conducting our data reliability assessment, as referenced in the draft, we  largely used the international audience research guidelines published by  the Conference of International Broadcasters’ Audience Research Services  (CIBAR), which are specified as a source of guidance for BBG research  contractors. For example, CIBAR standards recommend that the proper  care be exercised when reporting estimates to ensure that the type of  audience covered by the estimate is clearly stated, and that, at all stages in  the calculation and extrapolation process, sources, assumptions, and  methods are fully documented and available for independent scrutiny. We  were unable to determine the accuracy of MBN’s fiscal year 2005 audience  size and program credibility estimates due to weaknesses in MBN’s  methodology and documentation. As noted in the report, in several  instances the BBG and its contractors departed from CIBAR research  standards. In particular, the BBG did not take certain steps that could have  explained and increased the reliability of its estimates, such as by fully  documenting its research methods, measuring the level of uncertainty  surrounding its estimates, disclosing significant limitations, limiting the  scope of its projections to areas actually covered by its surveys, and  developing and consistently implementing policies and procedures for  verifying data. For these reasons, it is not clear whether the Radio Sawa  and Alhurra performance targets have been met.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution of it until 30 days from  the date of this letter. We will then send copies to others who are interested  and make copies available to others who request them.
If you or your staff have any questions about this report, please contact me  on (202) 512-4128. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  Key contributors are listed in appendix VI.
section_title: Scope and Methodology
paragraphs: To accomplish our objectives, we reviewed documentation and spoke with  officials from the Department of State; the Broadcasting Board of  Governors (BBG) and its grantees; including the Middle East Broadcasting  Networks; Inc. (MBN); Radio Free Europe/Radio Liberty (RFE/RL); Radio  Free Asia (RFA); the Voice of America (VOA); and the International  Broadcasting Bureau. We also spoke with several outside experts,  including representatives from National Public Radio, the InterMedia  research organization, Zogby International, and foreign international  broadcasters such as the British Broadcasting Corporation (BBC) and  Deutsche Welle.
In addition to our audit work in the Washington, D.C., area, we visited  MBN’s offices in Amman, Jordan, and Dubai, United Arab Emirates. We  held telephone interviews with MBN’s current affairs contractor in Beirut,  Lebanon, and sent questions to the MBN office in Baghdad, Iraq.
Because our work was focused on reviewing the BBG’s MBN initiatives-- Alhurra and Radio Sawa–we did not include RFE/RL Arabic-language  media activities in Iraq in the scope of our work.
subsections: section_title: Review of MBN Internal Controls
paragraphs: To address our objective of assessing MBN’s internal controls, we used  criteria contained in MBN’s grant agreement, OMB Circulars No. A-110 and  A-133, GAO published standards on internal controls, and a report on  MBN’s internal controls completed by Grant Thornton LLP accountants  and management consultants. We also met with other broadcasters,  including RFE/RL, RFA, and National Public Radio, to obtain an  understanding of their internal controls and to make comparisons with  MBN. We held discussions with MBN officials and reviewed documentation  provided by them to determine whether the organization was complying  with appropriate criteria. We spoke with PriceWaterhouseCoopers officials  responsible for the 2003, 2004, and 2005 single audits to obtain their  perspectives on the status of problems cited in the Grant Thornton LLP  report and to learn about any new issues arising from their audit work. We  did not test MBN’s controls, since doing so would have been beyond the  scope of our review.
section_title: Review of MBN’s Editorial Procedures
paragraphs: To address our objective of assessing the procedures MBN has developed  to ensure it complies with its journalistic standards, we spoke with other  broadcasters, including RFE/RL, RFA, VOA, and National Public Radio, to  compare their editorial standards and procedures with those of MBN. We  also met with the BBG general counsel, as well as MBN producers, editors,  and journalists. To better understand the program review process, we  attended several VOA program reviews. We also held detailed discussions  and obtained documentation on program review from the BBG officials  who conducted Radio Sawa’s review, the International Broadcasting  Bureau office of performance review, and the research directors of RFA  and RFE/RL. We did not test MBN’s editorial procedures.
section_title: Review of MBN Audience Survey Methods and Performance Indicator Estimates
paragraphs: To address our objective of determining the extent to which the BBG met  or exceeded its fiscal year 2005 performance MBN targets, we reviewed the  reliability of BBG audience research that was used to generate the  estimates of credibility and audience size contained in the BBG’s Fiscal  Year 2007 Budget Request, which was the BBG’s most up-to-date source of  officially-reported GPRA performance information for MBN for fiscal year  2005. We did not review the inputs used to generate performance  information for fiscal years other than 2005, nor did we review the  reliability of the data used to develop other MBN performance indictors  reported in the 2007 Budget Request.
To conduct our assessment of the reliability of BBG estimates of credibility  and audience size for fiscal year 2005, we reviewed available  documentation provided as of May 2005 against applicable standards and  common professional research practices, including Conference of  International Broadcasters’ Audience Research Services (CIBAR)  guidelines and the professional standards of the American Association for  Public Opinion Research. We reviewed relevant surveys, including  technical reports and printouts of the survey results, as well as  spreadsheets used to calculate the performance indicator estimates. We  also conducted a series of interviews and corresponded extensively with  the BBG’s coordinator for performance planning and research, the head of  the International Broadcasting Bureau office of research, and officials from  Intermedia to discuss the survey methods and how performance indicator  estimates were generated. In addition, we requested information from the  BBG’s research subcontractors, including AC Nielsen and D3 Systems, to  obtain details on the survey sampling plans and related assumptions,  response rates, and adjustments applied to the data to reflect the effects of  the survey design. As a result of our review, we determined that MBN’s  reported audience size and credibility estimates are not statistically reliable  and are rather rough estimates of performance.
section_title: Radio Sawa and Alhurra Programming
paragraphs: Radio Sawa currently has seven programming streams, with the Iraq  programming stream airing more news and features than the other streams.  We reviewed one week of programming and found that the Iraq stream  aired about 50 hours of news and features a week, while five other streams  aired about 40 hours a week (see table 5).
Alhurra’s programming focuses on news and information, including hourly  news updates, daily hour-long newscasts, and current affairs talk shows  (see table 6).   In addition, Alhurra broadcasts current affairs shows on  subjects including health and fitness, entertainment, sports, and science  and technology. Alhurra also airs documentaries on a diverse range of  topics designed to appeal to a broad audience. In a sample week, Alhurra’s  acquired programming, which mainly includes documentaries, accounted  for the largest number of hours aired, about 35 percent of the total weekly  programming hours--or approximately 60 hours in a week. News and news  updates amounted to 22 percent of weekly programming hours.
In contrast, Alhurra-Iraq’s talk shows accounted for the largest number of  hours aired--- about 46 hours, or 27 percent of total programming time (see  table 7).
section_title: Radio Sawa Program Review
paragraphs: The BBG initiated the Radio Sawa review because MBN did not have the in- house capacity to do so, according to a BBG official. Moreover, both the  BBG and MBN wanted to ensure that the review was conducted by an  entity other than MBN to make it more independent.
To plan and conduct this review, BBG officials reported that they held  preliminary discussions with the International Broadcasting Bureau’s office  of performance review, hired an outside expert to serve as a consultant to  the project, and contracted with the InterMedia research organization to  convene panels of regular listeners in Iraq, Morocco, Egypt, and Jordan.  According to representatives of InterMedia, they recruited a large number  of panelists because the BBG wanted more input than usual. Panelists  listened to and evaluated 6 days worth of programming, responded to a  questionnaire that evaluated all of Radio Sawa’s broadcast criteria,  answered open-ended questions, provided detailed feedback, and made  recommendations about the programming. Overall, the results of Radio  Sawa’s monitoring panels were generally positive, with an overall program  quality score of 2.9 out of 4, which is considered “good” by the BBG. To  follow up on the program review, MBN is attempting to implement action  points drafted by the BBG after the review, such as reducing abrupt  transitions between music and news and localizing Radio Sawa’s streams.
The BBG told us it placed great emphasis on the audience’s response,  noting that if there had been any problems, they would have heard about  them from the listeners. The BBG’s research contractor concurred that it is  important to obtain the audience’s perspective, since the audience can  judge programming in the context of the alternatives available to them in  the local media market. According to a BBG official, it was decided that  information obtained from the audience monitoring panel would capture  the issues that would normally be covered by the internal analysts in the  content and production analyses.
However, even a detailed, well-thought-out questionnaire does not replace  the different perspectives internal and external control reviewers bring to  the review, according to a grantee official. According to an International  Broadcasting Bureau official, even though the three components of a  program review–audience panels and internal and external reviewers–use  nearly the same criteria, they provide different insights into the program’s  quality, direction, and context. According to several media experts, while  audience panelists can evaluate news and information from their own  perspective, they are less qualified to evaluate a service’s mission or judge  the extent to which a show complies with journalistic standards. In  particular, the absence of an internal review meant that Radio Sawa’s  programming was not evaluated by people familiar with BBG and MBN  standards and controls.  Moreover, the external control listener’s  evaluation is intended to give the service a sense of what the panel’s  responses would be like if the environment were freer and more open,  information that a monitoring panel cannot provide.
The BBG told us that it did not believe an internal evaluation was necessary  because the English translations of each of the panel reports enabled them  to examine and assess MBN’s programming. The BBG also told us that it  did not use external control listeners in the review for several reasons: (1)  Radio Sawa’s focus on youth and popular music would make it difficult for  a typical older control listener to evaluate Sawa’s programs; (2) the BBG  lacked resources to use control listeners for the multiple countries in Radio  Sawa’s review; (3) it would have been hard to find impartial listeners; and  (4) the four in-country panels ensured enough diversity of opinion so that  there was less need for a control reviewer. However, according to an  International Broadcasting Bureau official, the VOA is able to successfully  evaluate youth shows using its regular internal reviewers. Moreover, it may  be possible to use younger control listeners, or to use the older ones  selectively. For example, RFE/RL uses a control listener to evaluate the  news of Radio Farda. In addition, the challenge of finding impartial control  listeners is not unique to the Middle East, but is potentially present to some  degree for every language service.
section_title: CIBAR Standards
paragraphs: The Conference of International Broadcasters’ Audience Research Services  (CIBAR) guidelines were developed by an international group of  broadcasters – including the BBG’s International Broadcasting Bureau -- to  encourage, among other things, the appropriate use of audience and  market data for decision-making within international broadcasting  organizations, the establishment and maintenance of standards in  international audience research, and the efficient and responsible use and  application within member organizations of all forms of audience  feedback. By providing a context for international audience research and a  clear statement of the minimum standards required, the guidelines attempt  to address the challenges faced by international broadcasters, including the  tension between the needs of proper audience measurement and survey  practice and the conditions and expertise in many of the countries where  the research is carried out.
The third edition of the guidelines, published in 2001, covers the nature of  international audience research measures; survey design, sampling, and  fieldwork; and data management and reporting. Specific guidelines  relevant to this report include the following: 4.1:  Samples and method:  The basis for audience measurement should be samples based on the  principles of random probability.
The principles of random selection should be applied at all stages of the  sampling process, from initial sampling point to selection of individuals.
Quota samples should be used only in cases where, for reasons of  practicality, cost, or both, proper random samples cannot be used. When  quota samples are used, this should be clearly labeled in the reporting  and documentation of the data.
It should be a requirement of all measurement research that the  sampling methods used and other technical aspects of the survey,  including margins of error and confidence levels where appropriate, be  both fully and accurately described in the project documentation, and  open to independent scrutiny.  4.2:  Survey coverage:  Where certain groups are disproportionately sampled and weighting  techniques are applied at the analysis stage to correct for this, project  documentation should give full details of the weights applied.  5.2:  Data reporting:  The research agency should provide the following information to a   Background information – client for whom the study was conducted;  purpose of the study; names of subcontractors and consultants  performing any substantial part of the work.
Intended and achieved sample -- universe covered; statistics used  (e.g. census data; size, nature and geographical distribution of the  sample; sampling method and weighting methods used; response  rates and possible bias due to non-response).
Data collection – method of collection; field staff; briefing and field  quality control; method of recruiting respondents; and fieldwork  dates.
Presentation of results – relevant factual findings obtained; bases of  percentages (weighted and unweighted); margins of error; and  questionnaire and other documents used.
Proper care be exercised when reporting such estimates, to ensure that  the type of audience covered by the estimate is clearly stated.
At all stages in the calculation and extrapolation process, it is vital that  sources, assumptions, and methods be fully documented and available  for independent scrutiny.
All reporting of worldwide and regional audiences should be  accompanied by a technical appendix giving details of the sources,  assumptions, and measures used.
section_title: Comments from the Broadcasting Board of Governors
paragraphs: The following are GAO’s comments on the Broadcasting Board of  Governor’s (BBG) letter dated July 20, 2006.
subsections: section_title: GAO Comments
paragraphs: 1. GAO acknowledges in the report that MBN has participated in long- range planning in coordination with the BBG’s update of its 2008-2012  strategic plan, and that MBN has developed a 2006 annual performance  plan as part of the OMB Program Assessment Rating Tool process.  However, contrary to the BBG’s comments, the fiscal year 2006  performance plan document for MBN that we obtained did not contain  all of the elements of a stand-alone strategic plan. In addition, we were  told by the former executive director of the BBG, who is currently the  president of MBN, that the Radio Sawa strategic plan drafted in 2002 by  the BBG–which the BBG stated in its comments contributed to its  considerable long-term planning for MBN--was never adopted by the  BBG. We were also told by MBN’s former president that he had never  received a copy of the draft Radio Sawa strategic plan. 2. The BBG stated that MBN’s Single Audit evaluates the effectiveness of  its internal control program. However, while MBN’s Single Audits  address the fair presentation of the organization’s financial statements,  its compliance with certain laws and regulations, and the organization’s  internal control as it relates to financial reporting, MBN’s 2005 Single  Audit as prepared by PriceWaterhouseCoopers does not–contrary to  what the BBG stated in its comments--offer an opinion on the overall  effectiveness of MBN’s internal control. Thus, the unqualified opinion  that MBN received on its fiscal year 2005 Single Audit does not provide  a broad assessment or opinion of MBN’s internal control system. For  the purposes of our report, it was therefore necessary to consider other  assessments of MBN’s financial and administrative controls, including  the Grant Thornton report.  3. MBN’s grant agreement states that MBN should make every reasonable  effort to achieve the purpose of the grant in accordance with OMB  Circulars A-110, A-122, and A-133. Our report did not attempt to make  an assessment of MBN’s compliance with its grant agreement, but  rather focused on the progress MBN has made in developing its system  of internal control and the ways in which MBN can continue to  strengthen this system. To that end, we drew upon the work of Grant  Thornton LLP, which assessed the status of MBN’s controls against  relevant OMB Circulars and GAO guidance on internal control.   4. While the Grant Thornton recommendations are not obligatory, they  are based on best management practices from OMB and GAO.  According to officials, both the BBG and MBN accepted the results of  the report, which provides detailed insight into MBN’s financial  operations, when it was completed in May 2004. Adopting the report’s  remaining recommendations would help MBN in its efforts to build a  fully mature internal financial management operation. 5. The BBG stated that it believes its standards yield data of sufficient  reliability to allow it to estimate MBN’s audiences and credibility.  However, the BBG offers no evidence for this. BBG’s estimates for MBN  are based on judgment and not statistics, and its current methods do  not and cannot estimate the error in its estimates. As noted in the  report, in several instances the BBG and its contractors departed from  CIBAR research standards. For example, we analyzed 12 of the 14  country surveys conducted by BBG contractors for 2005. In the BBG’s  fiscal year 2007 budget request, the BBG reported the results of each  country survey and an overall estimate for audience size of 21.5 for  Alhurra and 21.6 million for Radio Sawa. However, our analysis of the  12 surveys identified a number of methodological weaknesses,  including BBG’s failure to fully document research methods, measure  the level of uncertainty surrounding its estimates, disclose significant  limitations, limit the scope of its projections to areas actually covered  by its surveys, and develop and consistently implement policies and  procedures for verifying data. These limitations were not reported in  BBG’s audience survey estimates and prevent us from concluding that  the estimates are accurate and reliable. 6. We acknowledge in the report that there are challenges to conducting  audience research in the Middle East, and that there are tradeoffs  between cost and data reliability when conducting research. 7. As we noted in the report, it is difficult to conduct probability sampling  in many locations in the Middle East. But, it is not impossible.  Organizations, including the United States Census Bureau, have  collaborated on probability surveys in the Middle East, including in  Saudi Arabia and Jordan. The major problem of the BBG estimates is  their lack of transparency, and the lack of an explanation for the  methodology behind BBG’s estimates.
8.
In conducting our data reliability assessment, as referenced in the  report, we largely used the international audience research guidelines  published by CIBAR. The BBG participated in drafting these standards,  which are specified as a source of guidance for BBG research  contractors. CIBAR standards recommend that proper care be  exercised when reporting estimates to ensure that the type of audience  covered by the estimate is clearly stated, and that, at all stages in the  calculation and extrapolation process, sources, assumptions, and  methods are fully documented and available for independent scrutiny.  As noted in the report, in several instances the BBG and its contractors  departed from CIBAR research standards, leading to weaknesses in  MBN’s methodology and documentation. As a result, we were unable to  determine the accuracy of MBN’s fiscal year 2005 audience size and  program credibility estimates.  9. The BBG stated that it believes the level of methodological detail it has  provided in public documents is comparable with that typically used by  other research organizations for studies in the Islamic world; therefore,  it believes it is consistent with industry practice. However, the Reports  Consolidation Act of 2000 requires federal agencies, such as the BBG,  to assess the completeness and reliability of the performance data in  their performance reports and to discuss any material inadequacies in  the completeness and reliability of their performance data, as well as  actions to address the inadequacies. In performance reporting such as  the BBG fiscal year 2007 budget request, the BBG reported the results  of its research for MBN as 21.5 million viewers for Alhurra and 21.6  million for Radio Sawa. However, the BBG did not sufficiently explain  the specific methods used for generating estimates for its performance  indicators, such as the number and names of the countries surveyed  (including the sizes of its samples), methods of sampling used, sources  of the population data, and basic procedures used to create the  estimates. Moreover, the BBG reported only two limitations for MBN  performance indicators to date: (1) that credibility ratings are highly  dependent on volatile political factors; and (2) that, depending on  political, social, and media conditions, measurement of audience size  may either be easily attained or impossible. Significant limitations not  reported are that many of the BBG surveys are not based on probability  sampling, cover only part of the country, have very low response rates,  or have high substitution rates. Therefore, the BBG does not provide a  full level of confidence in the credibility of its performance data for  MBN. In contrast, the Department of Transportation, has a separate  compendiums available online that provides source and accuracy  statements, which provide detail on the methods used to collect  performance data, sources of variation and bias in the data, and  methods used to verify and validate the data. 10. The BBG cannot calculate the sampling error for the BBG estimates of  audience size and credibility because the probability of selection is not  known. Although increased sample size will generally decrease  sampling error, it is impossible to accurately estimate the sampling  error of the BBG surveys because they are nonprobability surveys. As  the BBG has stated, its use of a simple random sampling formula when  calculating sampling errors for its surveys underestimates the sampling  error. This formula is not appropriate for the sample designs used by  MBN.  11. Although our recommendation asks the BBG to require research  contractors to improve the methods used in audience research to allow  for probability sampling and document the sample selection so that  survey sampling errors can be calculated, where possible, it is the  responsibility of the BBG to explain and justify the need to conduct  nonprobability samples.
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: Diana Glod, Melissa Pickworth, Eve Weisberg, Dorian Herring, and Joe  Carney made key contributions to this report. Chanetta Reed, Jay Smale,  Karen O’Conor, and Jackie Nowicki provided technical assistance.