Homeland Security: Agriculture Inspection Program Has Made Some Improvements, but Management Challenges Persist
Published: 2012-09-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: According to DHS, invasive species cause an estimated $136 billion in lost agricultural revenue annually, and since September 11, 2001, concerns have persisted about the vulnerability of agriculture to deliberate introduction of foreign pests and disease. DHS and USDA manage the AQI program, which places agriculture inspectors at U.S. ports of entry to inspect imported agriculture products and intercept foreign pests. GAO reported in 2006 on management challenges in the program and made seven recommendations to improve it. GAO was asked to examine the extent to which (1) DHS and USDA implemented GAO's recommendations; (2) data on arrivals, inspections, and interceptions are used for managing the program; and (3) the views of AQI agriculture specialists on their work environment have changed since 2006. GAO surveyed a representative sample of agriculture specialists and supervisors; reviewed key documents and inspection procedures; visited five selected ports of entry based on size and entry pathways, such as air or sea; interviewed DHS and USDA officials; and reviewed AQI data. The survey instrument and most results can be viewed at GAO-12-884SP .
section_title: What GAO Found
paragraphs: The Department of Homeland Security (DHS) and the U.S. Department of Agriculture (USDA) have taken steps to implement all seven of the recommendations GAO made in 2006 to improve the Agriculture Quarantine Inspection (AQI) program, but they face challenges in fully implementing four of them. Specifically, DHS and USDA have implemented GAO's recommendations to improve information sharing, review DHS's financial management system for the AQI program, and remove barriers to timely and accurate transfers of AQI user fees--collected for AQI services provided in connection with the arrival of international air passengers and conveyances at U.S. ports. However, DHS and USDA face challenges in fully implementing GAO's recommendations to adopt meaningful performance measures, establish a national risk-based staffing model, improve the agriculture canine program, and revise user fees to cover program costs. For example, in 2006, GAO recommended that DHS and USDA adopt meaningful performance measures for assessing the AQI program's effectiveness at intercepting foreign pests and disease. DHS and USDA have expanded the use of one type of performance measure but have not developed measures for all aspects of the AQI program that are important for its management. In addition, the AQI program does not have a strategic plan--a leading practice that would provide DHS and USDA with a framework for defining the mission of what the program seeks to accomplish, setting goals to achieve desired results, and identifying performance measures for gauging progress toward those goals. Furthermore, DHS has undertaken efforts to respond to GAO's recommendation to develop a national, risk-based staffing model but does not yet have one, and DHS anticipates that the model will recommend significant staffing increases. DHS officials told GAO they do not have the resources to increase staff, but the agency has not developed a plan that assesses the risk of potential fiscal constraints on its ability to implement the staffing model. Without a plan or strategy to address potential resource constraints on staffing by considering the fiscal resources that may realistically be available, DHS risks increasing the vulnerability of the agriculture sector to foreign pests and disease.
The AQI program uses data on arrivals, inspections, and interceptions at U.S. ports of entry to determine how well agriculture inspections identify prohibited materials and to review ports' performance, but data quality issues may prevent AQI program officials from making full use of the data. For example, the data may not be reliable. DHS and USDA recognize that data quality is an ongoing issue and in 2004 created an interagency group to address this issue. However, from 2010 to 2011, joint DHS-USDA reviews of 22 selected ports found discrepancies in the data at about half of the ports reviewed (10 out of 22). Data reliability has the potential to affect other key efforts that are currently under way, such as the program's staffing model. Without reliable data, AQI program officials do not have assurance that they have the information needed to manage the program.
GAO also presents analysis of survey data from 2006 and 2012 on agriculture specialists' and supervisors' views about their work environment.
section_title: What GAO Recommends
paragraphs: GAO recommends, among other things, that (1) DHS and USDA develop a joint strategic plan for the AQI program, (2) DHS develop a plan for implementing a staffing model, and (3) DHS and USDA take steps to improve the reliability of certain data. DHS and USDA agreed with the recommendations.

section_title: Letter
paragraphs: The United States is one of the world’s largest producers, consumers,  exporters, and importers of agricultural commodities. However, some of  these imported products may contain exotic pests and diseases.  According to the Department of Homeland Security (DHS), invasive  species cause an estimated $136 billion in lost agricultural revenue  annually. For example, the Asian Longhorned Beetle has destroyed more  than 30,000 trees in Illinois, New Jersey, and New York since it was  discovered in 1996 in imported cargo, and it remains a significant problem  today. Damages for this infestation have been estimated at more than  $269 million, and according to the U.S. Department of Agriculture  (USDA), if the infestation were to expand nationwide, it could cause more  than $41 billion in losses. Furthermore, the terrorist attacks of September  11, 2001, heightened concerns about agriculture’s vulnerability to  terrorism, including the deliberate introduction of livestock, poultry, and  crop diseases.
Under the Agricultural Quarantine Inspection (AQI) program, international  passengers and cargo at U.S. ports of entry are inspected to seize  prohibited material and intercept foreign agricultural pests. Historically,  USDA was responsible for the AQI program, but the Homeland Security  Act of 2002 split responsibility for the AQI program between DHS and  USDA’s Animal and Plant Health Inspection Service (APHIS). DHS  acquired the authority to inspect passenger declarations and cargo  manifests; international passengers; baggage; cargo; and conveyances  such as ships, aircraft, vehicles, buses, and rail cars; and to seize and  quarantine suspect articles. Under this authority, DHS’s Customs and  Border Protection’s (CBP) agriculture specialists perform these duties at  ports of entry located at airports, seaports, and land borders throughout  the United States. Inspection methods include the examination of  baggage by hand, X-ray, and canine inspection, and the examination of  documents accompanying incoming cargo to ensure compliance with  plant and animal health and trade agreements. An agriculture inspection  may result in an interception—the identification of items that may be  confiscated or transferred to an APHIS facility for subsequent evaluation  or treatment, depending on the product or material. APHIS remains  responsible for setting inspection policy, overseeing CBP agriculture  specialists’ training, and managing and collecting AQI user fees––fees  collected in connection with the arrival of international air passengers and  conveyances at ports in the customs territory of the United States.
For more than a decade, we have reported on challenges related to the  AQI program. In particular, in 2006, we issued two reports on  management challenges in the AQI program, including one in May 2006,  in which we identified shortcomings in the program’s staffing,  performance measures, inter-agency coordination, and inspection  performance. In our May 2006 report, we made seven recommendations  to DHS and USDA to improve the AQI program by (1) improving  information-sharing, (2) undertaking a full review of DHS’s financial  management system for the AQI program, (3) removing barriers to timely  and accurate AQI user fee transfers from USDA to DHS, (4) adopting  meaningful performance measures, (5) establishing a national risk-based  staffing model, (6) improving the agriculture canine program, and (7)  revising user fees to cover AQI program costs. DHS and USDA generally  agreed with these recommendations and in 2007 created a joint task  force to, among other things, identify and implement solutions. In  November 2006, we issued a report with additional analysis of survey  data from our May 2006 report regarding agriculture specialists’ views on  their work environment.
Concerns persist that the agriculture sector remains vulnerable to the  unintentional or deliberate introduction of pests and disease. In this  context, you asked us to examine the AQI program. Our objectives for  this report were to examine the extent to which (1) USDA and DHS have  implemented recommendations we made in 2006 to improve the AQI  program; (2) data on arrivals, inspections, and interceptions are used for  managing the program; and (3) the views of agriculture specialists at CBP  regarding their work environment have changed, if at all, since our 2006  reports.
To address all of these objectives, we selected a nonprobability sample of  five CBP ports of entry to visit in order to observe agriculture inspection  work activities and discuss challenges related to conducting agriculture  inspections and interceptions. These ports of entry included Austin,  Texas; Baltimore, Maryland; Dallas, Texas; Miami, Florida; and San  Ysidro, California. We selected these ports of entry based on size and  entry pathways, such as air, sea, and land. Because we used a  nonprobability sample, the information we obtained from these visits  cannot be generalized to other CBP ports of entry. The visits instead  provided us with more in-depth information on the perspectives of various  agriculture specialists in these ports of entry about the management of  the AQI program.
To examine the implementation of recommendations from our May 2006  report, we interviewed DHS and USDA officials and reviewed  documentation and criteria related to CBP’s performance measures,  information-sharing between CBP and APHIS, DHS’s financial  management system, CBP’s agriculture staffing model, CBP’s agriculture  canine program, AQI’s user fee transfers, and AQI user fee modifications  to cover AQI program costs. To examine the extent to which data on  agriculture arrivals, inspections, and interceptions are used to manage  the program, we reviewed DHS and USDA inspection and interception  data, related summary reports, and policies, and interviewed key program  officials at DHS and USDA headquarters, in the field, and at ports of  entry. To address our objective on agriculture specialists’ views, we  developed a questionnaire to survey CBP agriculture specialists, which  contained closed-ended and open-ended questions. Specifically, we  surveyed a stratified random probability sample of 556 agriculture  specialists and supervisors from the universe of 2,311 agriculture  specialists, including supervisors and agriculture canine handlers at CBP.  To facilitate comparison with the findings from our May 2006 and  November 2006 reports, we developed similar questions for this survey.  We also used the results of our survey to inform our analysis on the  extent to which DHS and USDA have implemented some of the  recommendations we made in 2006. We received an unweighted  response rate of 74 percent, and the results of our analysis of closed- ended survey questions are generalizable to all agriculture specialists and  supervisors at CBP. More details on our scope and methodology appear  in appendix I. To view the full results of our survey, please see an  electronic supplement to this report (GAO-12-884SP).
We conducted this performance audit from August 2011 to September  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: From 1913 until 2002, USDA was responsible for the inspection of plants  and animals at U.S. ports of entry. Following the September 11, 2001,  terrorist attacks, Congress passed the Homeland Security Act of 2002,  which combined the inspection activities of the Department of the  Treasury’s Customs Service, the Department of Justice’s Immigration and  Naturalization Service, and USDA’s APHIS into the newly created DHS.  The Secretaries of DHS and of USDA signed a memorandum of  agreement in February 2003, agreeing to work cooperatively to  implement the relevant provisions of the Homeland Security Act of 2002.  Under the memorandum of agreement, APHIS’s responsibilities include  managing user fees, overseeing agriculture specialists’ training, and  providing pest identification services; and CBP’s responsibilities include  conducting inspections and related activities. Consistent with the  Homeland Security Act of 2002 and the agreement, in 2003  approximately 1,500 agriculture specialists who had formerly worked for  APHIS became CBP employees.
In 2003, CBP established an initiative called “One Face at the Border,”  which unified its three missions—customs, immigration, and agricultural  inspection—by cross-training CBP customs and immigration officers and  agriculture specialists in all three areas. The inspection procedures vary  somewhat by the transportation pathway, such as airports, seaports, or  land border crossings, but, generally, CBP officers conduct primary  inspections, including interviewing passengers, and may refer passengers  to agriculture specialists who conduct more detailed secondary  inspections, including subsequent interviews of passengers or  examination of baggage. Figure 1 shows an example of primary and  secondary inspection procedures at an airport.
As of April 2012, approximately 2,360 CBP agriculture specialists,  including those who formerly worked with APHIS and those hired by CBP,  were assigned to about half, or 167 of the 329 U.S. ports of entry. Each  U.S. port of entry can include one or more pathway, such as airports,  seaports, or land border crossings. For example, the port of Baltimore,  Maryland, has an airport and seaport, whereas the port of Dallas, Texas,  has just an airport. CBP port directors are responsible for overseeing  operations at ports of entry and assigning agriculture specialists to  specific port facilities. The ports of entry are organized into 20 district field  offices across the United States, and these district offices are headed by  CBP field office directors who also serve as liaisons between CBP  headquarters and port management. In 2005, APHIS and CBP  established a formal assessment process at select ports—known as joint  quality assurance reviews—to ensure that ports of entry carry out  agricultural inspections in accordance with APHIS’s regulations, policies,  and procedures. APHIS and CBP spent about $33,000 conducting eight  such reviews in 2011, and, according to CBP officials, the reviews focus  on ports that are considered to be high risk for agriculture.
The Food, Agriculture, Conservation, and Trade Act of 1990, as  amended, authorizes the Secretary of Agriculture to set and collect user  fees for AQI services provided in connection with the arrival, at a port in  the customs territory of the United States, of commercial vessels,  commercial trucks, commercial railroad cars, commercial aircraft, and  international passengers. The Secretary subsequently delegated this  authority to APHIS. The fees are paid either directly by shipping  companies or indirectly by air passengers through fees on tickets. CBP  collects user fees for commercial vessels and trucks and deposits the  collections into APHIS’s user fees account. APHIS collects all other user  fees and periodically transfers a pre-agreed portion of the total collections  to CBP to support CBP’s agriculture-related operations. APHIS has  revised the AQI user fees several times since the act was passed. For  example, through the rulemaking process, APHIS increased the user fees  in November 1999 and, in January 2003, extended the adjusted fees  indefinitely. In August 2006, APHIS published a final rule affirming a  December 2004 interim rule that increased user fees again. We have  previously reported on how the fees are set, collected, and distributed,  and the benefits and challenges of this process to agencies and  stakeholders, including the implications of consolidating these fees under  the authority of DHS.
section_title: DHS and USDA Have Taken Steps to Implement GAO’s Seven Recommendations, but Have Not Fully Implemented Four of Them
paragraphs: DHS and USDA took steps to implement all seven recommendations we  made in our May 2006 report, but they faced challenges fully  implementing four of them.
subsections: section_title: DHS and USDA Have Implemented Three Recommendations to Enhance the Effectiveness of the AQI Program
paragraphs: DHS and USDA have implemented our recommendations to (1) improve  information-sharing, (2) undertake a full review of DHS’s financial  management system for the AQI program, and (3) remove barriers to  timely and accurate AQI user fee transfers from USDA to DHS.
Improve information-sharing. In our May 2006 report,that DHS and USDA ensure that they more effectively share urgent alerts  and other information essential to safeguarding U.S. agriculture and that  they transmit such information to agriculture specialists at the ports. At  that time, we reported that agriculture specialists were not consistently  receiving notifications of changes to inspection procedures or policies and  urgent alerts from APHIS—notices of emerging concerns about foreign  pests and diseases––in a timely manner. In response to our  recommendation, in 2007, DHS established the position of Deputy  Executive Director for Agriculture Operational Oversight to serve as a   we recommended  primary point of contact for coordination among CBP, APHIS, and AQI  stakeholders, such as state departments of agriculture, who would be  largely responsible for eradicating foreign pests. In addition, CBP  implemented a policy to disseminate urgent agriculture alerts to its field  offices within 24 hours of receipt, and the alerts are then disseminated to  agriculture specialists at the ports. These alerts, as well as any changes  in inspection procedures or policies, are also available on the AQI  program’s intranet site. Our survey results indicate that an estimated 83  percent of the agriculture specialists and supervisors at CBP found that  information regarding agriculture regulatory changes was delivered in a  timely manner either always or most of the time. Our survey results also  indicate that an estimated 4 percent of agriculture specialists and  supervisors at CBP believe that the timeliness of information provided to  agriculture specialists is a very major challenge, while an estimated 57  percent believe it is only a minor challenge or not a challenge.
Review DHS’s financial management system. In 2006, we recommended  that DHS undertake a full review of its financial management systems,  policies, and procedures for the AQI program to ensure financial  accountability for funds allocated for agricultural quarantine inspections.
GAO-06-644. spent on agriculture activities and the associated costs. Since we last  reported, CBP added agriculture-specific activity codes to its financial  management system, which, according to CBP officials, allows CBP to  provide APHIS with actual costs related to user fees. In addition, CBP  issued guidance to clarify how employees should account for activities  that are simultaneously related to immigration, customs, and agriculture  activities so that this information could be tracked for the purpose of  collecting AQI user fees. According to APHIS and CBP officials, the new  activity codes and guidance allow CBP to accurately report its costs by  user fee type to APHIS and ensure financial accountability for funds  allocated to AQI user fees.
Accurately transfer user fees. In 2006, we recommended that USDA take  steps to assess and remove barriers to the timely and accurate transfer of  AQI user fees to DHS. At that time, transfers of user fees from USDA’s  APHIS to DHS’s CBP were often delayed and their amounts were  sometimes less than CBP expected, which adversely affected agricultural  inspection activities. For example, in 2006, we reported that during fiscal  years 2004 and 2005, CBP frequently did not receive the transfers at the  time specified or for the agreed upon amount, causing some ports to  reduce spending for supplies or to delay hiring or purchasing equipment.  Since that time, APHIS and CBP have fully implemented a user fee  transfer agreement that allows for a bimonthly transfer schedule that  better accommodates the inflow of AQI user fee funds. This new user fee  agreement also created chief budget liaisons at APHIS and CBP and  provides for meetings at least 4 times a fiscal year to discuss AQI  funding. Our analysis of transfer records provided by APHIS shows that,  from fiscal year 2006 through 2011, 34 of the 36 scheduled user fee  transfers were made in accordance with the new, bimonthly transfer  schedule. During that time, six scheduled transfers were less than the  scheduled amount, and, for each, APHIS made up the difference either  by adding an unscheduled transfer or by adding the difference to the next  scheduled transfer. APHIS and CBP officials told us that transfers have  been more timely and accurate since the changes were implemented.
section_title: DHS and USDA Faced Challenges Implementing Four of Seven Recommendations
paragraphs: DHS and USDA faced challenges in fully implementing our  recommendations to (1) adopt meaningful performance measures, (2)  establish a national risk-based staffing model, (3) improve the agriculture  canine program, and (4) revise user fees to cover AQI program costs.
In 2006, we recommended that DHS and USDA adopt meaningful  performance measures for assessing the AQI program’s effectiveness at  intercepting foreign pests and disease on agricultural materials entering  the country by all pathways and posing a risk to U.S. agriculture. At that  time, CBP had not adopted performance measures that took into account  the agency’s expanded mission or considered all pathways by which  prohibited agriculture items or foreign pests may enter the country. In  response to our recommendation, in 2007, APHIS and CBP created a  joint task force that expanded existing performance measures to more  pathways, including passengers’ baggage, pedestrians, and vehicles, as  well as some cargo pathways. These performance measures provide  information about the amount of prohibited agriculture items expected to  be found in specific pathways during inspections, which can then be  compared with data on the actual amount found. According to APHIS and  CBP officials, this comparison can provide some information regarding  the effectiveness of agricultural inspection activities as well as the relative  risk that prohibited pests pose in a particular pathway. For example, in  fiscal year 2011, APHIS estimated that 3.5 percent of international airline  passengers were expected to be carrying prohibited agriculture items,  and agricultural inspections actually found prohibited items in 3.2 percent  of international airline passengers, suggesting that the agricultural  inspections may not have found all items for that year.
Even as APHIS and CBP expanded existing performance measures, they  did not develop measures for all aspects of the AQI program that are  important for its management. Thus, existing performance measures are  not sufficient to assess the program’s overall effectiveness. For example,  APHIS officials told us that the AQI program does not have performance  measures for gauging the timeliness of APHIS’s pest identification  services, which form the basis for deciding how to treat inspected  material. According to APHIS officials, they are working independently to  develop such measures. In addition, CBP officials told us that there are  no performance measures for gauging the extent to which the AQI  program targets inspections to commodities or pathways of higher risk.  Furthermore, there are no performance measures for assessing progress  related to the AQI program’s expanded mission to prevent agroterrorism,  such as preventing the intentional introduction of pests harmful to  agriculture by coordinating with intelligence entities. According to the  Director of CBP’s Agriculture/Bio Terror Countermeasures division, there  is an ongoing effort to develop a measure of CBP’s ability to prevent  intentional introductions of harmful pathogens. CBP officials also told us  that because the current performance measures are not designed to  indicate what type of inspection procedure would most likely identify a  prohibited item––for example, whether an item would be more likely to be  found using an x-ray or a canine team—they are limited in their  usefulness for effectively deploying their inspection resources. According  to APHIS and CBP officials, the two agencies have not created additional  performance measures for the AQI program because they have largely  focused their performance management efforts on expanding their  existing performance measures to additional pathways. In addition, they  told us that developing additional performance measures has been a  challenge because of the diverse missions of the two agencies. Our work  on results-oriented organizations states that performance goals and  measures that successfully address important and varied aspects of  program performance are key elements of results-oriented  organizations.
Pub. L., No. 103-62, 107 Stat. 285, amended by GPRA Modernization Act of 2010, Pub.  L. No. 111-352, 142 Stat. 3866. divisions, programs, or initiatives.established APHIS and CBP roles in the AQI program also emphasized  the need for coordination between the two agencies. In 2007, a task force  comprising of APHIS and CBP employees identified the need to create a  joint agency AQI strategic plan, but an official with the task force told us  that this effort was not successful because the two agencies took  fundamentally different approaches to strategic planning that could not be  resolved. Instead, according to APHIS and CBP officials, the two  agencies have incorporated elements of AQI activities into each agency’s  separate strategic planning efforts. For example, the strategic plan for  APHIS’s Plant Protection and Quarantine program has a program goal of  optimizing the effectiveness of pest exclusion and prevention activities. At  the same time, CBP’s Office of Field Operations has a separate draft  strategic plan that contains an objective for protecting against the  introduction of plant pests and foreign animal diseases. Each plan has  different performance measures. These separate strategic plans do not  reflect a coherent joint mission with program goals that can serve as the  foundation for developing performance measures for the AQI program as  a whole. Without a joint strategic plan defining the mission and goals of  the AQI program, APHIS and CBP do not have a unified framework for  developing meaningful performance measures.
GAO-06-644. develop a model, and the agency received the model in September 2010.  However, CBP determined that the model was inadequate because it did  not incorporate specialized staffing needs, such as ports where unique  commodities (e.g., cut flowers) are inspected. CBP is in the process of  soliciting for a new contract to revise the staffing model to account for  these specialized staffing needs, and they estimate that this new contract  will cost an additional $300,000. CBP officials told us their goal is for the  revised staffing model to be completed and approved by September  2014, but as of June 2012, officials said that the effort to finalize the  contract solicitation is 4 months behind schedule. In addition, CBP  officials were unable to provide contracting documents to substantiate  their timeline. CBP officials told us they are having difficulty writing the  new contract solicitation because DHS’s information requirements for  contracting documents have changed since they wrote the previous  contract, and they were unaware of the new requirements when they  submitted their draft documents. In addition, a CBP official involved in the  contracting process told us that the AQI program generally does not use  contractors; therefore, the program’s experience in managing the  contracting process is limited.
CBP officials told us they anticipate that the revised staffing model will  recommend significant increases in staff at many locations because the  staffing model they received in September 2010 recommended a 32- percent increase in the total number of agriculture specialists for the AQI  program. CBP officials also told us they currently do not have the  resources to increase staff above replacement levels. However, the  agency has not developed a plan or strategy that assesses the risk of  potential fiscal constraints on their ability to implement the staffing model.  Under standards for internal control in the federal government, agencies  are to assess the risks they face from external and internal sources and  to determine what actions should be taken to mitigate them. Because  the AQI program may not have the fiscal resources to add the number of  agriculture specialists that the model recommends, CBP faces the risk of  being unable to keep up with the demand for agricultural inspections. In  2006, we reported that the changing nature of international travel and  agricultural imports creates risks for introducing pests and disease.
These risks include the volume of passengers and cargo, the type of  agricultural products, countries of origin, and ports of entry where  passengers and cargo arrive in the United States. Without a plan or  strategy to optimize the allocation of staff to those ports of highest need  that considers the fiscal resources that may realistically be available, CBP  risks investing in a staffing model it cannot execute and increasing the  vulnerability of the agriculture sector to foreign pests and disease.  International trade could also be affected if agriculture specialists are not  available to inspect cargo in a timely manner.
In 2006, we recommended that DHS and USDA work together to improve  the effectiveness of the agriculture canine program by reviewing policies  and procedures regarding training and staffing of agriculture canines and  ensure that these policies and procedures are followed in the ports. At  that time, the agriculture canine program was understaffed and  proficiency scores of canine teams had declined. Agriculture canines are  a key tool for targeting passengers and cargo for inspection by detecting  the scent of specific prohibited agricultural items, such as citrus or beef.  CBP officials told us that the use of canines can increase the number of  agriculture interceptions. For example, at one port of entry that we visited,  after a canine handler was assigned to work a pathway where no canine  team had previously worked, interceptions increased by 800 percent. In  response to our recommendation, CBP has  Increased staffing levels. According to CBP officials, there are  currently 114 active agriculture canine teams compared with the  approximately 80 teams that were in place in 2006. We could not  determine if the staffing levels were adequate without a final staffing  model for the AQI program. According to CBP officials, the staffing  model being developed by CBP includes staffing recommendations  for agriculture canine teams. However, as discussed above, CBP  officials told us that they do not have a plan to implement the staffing  model results for deploying canine teams effectively at the ports  because of current resource constraints.
Increased canine training requirements. In 2006, we reported that 60  percent of the 43 agriculture canine teams tested failed the 2005  proficiency test—an annual performance evaluation all canine  enforcement teams must undergo to maintain certification of their  detection capability. We recommended that USDA and DHS review  canine training policies. In response, in 2007, trainers from USDA’s  National Detector Dog Training Center—the training center for  agriculture canine teams— evaluated the annual agriculture canine  certification program at selected ports. As a result of these  evaluations, officials from CBP and the training center decided to  extend the agriculture canine field training course by 1 week to add,  among other things, hands-on training for conducting certifications at  the ports. According to the national agriculture canine program  manager, all current canine teams have passed the annual proficiency  test.
Expanded management oversight. CBP hired a national agriculture  canine program manager with canine and agriculture experience to  help ensure that staffing and training policies are followed in the ports.  This program manager is responsible for canine handler recruitment  activities; monitoring the training and staffing status of canine  handlers; and coordinating the flow of information among canine  handlers, field offices, and training centers. For example, the  agriculture canine program manager noted that one of her primary  responsibilities is to coordinate with the National Detector Dog  Training Center to assist agriculture canine teams with training  procedures and policies.
Even with these efforts, the agriculture canine program still faces  challenges related to supervisory training and data reliability. First,  supervisors may not have canine training or experience. Based on the  results of our survey of agriculture specialists, we estimate that 35  percent of agriculture canine specialists at CBP believe that their  supervisor did not have adequate agriculture canine expertise to advise  them on any work-related concerns in the past year.findings from the joint CBP-APHIS quality assurance reviews in 2010 and  2011 shows that canine handlers at 4 of the 15 ports with canine handlers  that were reviewed reported to a supervisor who had not taken the formal  canine supervisory training course. According to USDA’s National  Detector Dog Manual, the supervisory training class covers topics related   Our analysis of  to using canine teams, proficiency training, veterinary requirements, and  work expectations. The agriculture canine program manager told us that  this formal training class is available, but not mandatory, for canine  supervisors. Port directors at individual ports of entry are responsible for  determining whether a canine supervisor should take the class and  whether the port has the resources to support sending the supervisor to  the class. Not having supervisors with experience in agriculture canines  can, according to the canine handlers we spoke with at ports of entry,  make it difficult for the canine handlers to get support when they  encounter difficulty or have questions. Also, in March 2004, we reported  that investing in and enhancing the value of employees through training  and development is crucial for the federal government to successfully  acquire, develop, and retain talent.
Second, we found that some data being used to support the agriculture  canine program may not be sufficiently reliable for assessment because  they are incomplete and inaccurate. In addition, some data being  collected may not be meaningful. The agriculture canine program relies  on multiple types of data to track the work activities of canine handlers,  such as whether the handler is active or on leave and what training  activities the team has completed. However, we found the following:
Some data are incomplete and inaccurate. For example, we analyzed  data from fiscal years 2010 and 2011 on canine teams’ weekly  training exercises. The program manager uses this information to  ensure that canine teams are meeting their training requirement and  maintaining a minimum required proficiency level to conduct  agricultural inspections. However, we estimate that the percent of  missing values in this data set ranges from 13 percent to 22 percent  from January 2010 through July 2011. We could not accurately  determine the precise number of missing values because, according  to the canine program manager, the field offices do not maintain  accurate records of the number of active canine teams. The canine  program manager told us that canine teams periodically become  inactive for a variety of reasons, such as medical leave or temporary  reassignment of canine handlers, but the field offices do not change  the team’s status to inactive in the human capital tracking system, and  as a result, the number of active canine handling teams in each  quarter may be overstated. The canine program manager also noted  that the proficiency data—which are collected quarterly by CBP field  offices and forwarded to the agriculture canine program manager in  CBP headquarters—have missing values because field offices do not  consistently review the data for errors or missing values. When there  are missing values, too much time may have elapsed for the canine  handler to remember what the correct data should be. According to  the agriculture canine program manager, to compensate for these  data problems, she maintains a separate record of the current status  of each canine team, which is generated from personal  communications between her, the field offices, and canine handlers.  The agriculture canine program manager also told us she is  developing a web-based data reporting system that is intended to  have a supervisory review process before the data are submitted;  however, she did not have a timeline for this project because it is  being developed informally. Without complete and accurate  information regarding its canine teams, the AQI program does not  have sufficient information to assess the status of the canine teams’  proficiency level to ensure agriculture canine inspections are being  conducted by qualified canine teams.
Some data being collected may not be meaningful. Specifically, the  agriculture canine program is gathering data on the amount of time  canine teams actively inspect and intercept pests; however, according  to CBP officials, these data are not useful for the AQI program. Before  the creation of DHS, the U.S. Customs Service collected these data to  measure the cost effectiveness of its drug-detecting canines, and  CBP continued to collect these data for agriculture canines. The use  of these data to determine the cost effectiveness of a canine program  depends heavily on determining the value of the seized contraband,  which is not easily determined for agriculture. For example, according  to CBP officials, a pound of seized cocaine has an associated street  value that can be compared with the cost of performing inspections.  For agriculture products, the monetary value of the seized item may  be minimal; instead, the value of the activity is in preventing the  introduction of a harmful agricultural pest or disease and avoiding the  potential economic harm, which, according to CBP, is difficult to  project or calculate.
Without sufficiently reliable data, the AQI program cannot assess the  effectiveness of the canine program. Furthermore, by collecting data that  are not meaningful or useful for assessing the effectiveness of its canine  handlers, the AQI program may be using resources to collect data it does  not need.
In 2006 we recommended that DHS and USDA work together to revise  the user fees to ensure that they cover the AQI program’s costs. At that  time we found that, although the Secretary of Agriculture had the  discretion to prescribe user fees to cover the costs of the AQI program,  program costs had exceeded user fee collections since the transfer of  AQI inspection activities to DHS’s CBP. In 2009, APHIS attempted to  increase the AQI user fees using an interim rulemaking process that  increased AQI user fees by approximately 10 percent. However, later that  year they withdrew the interim rule because, according to APHIS officials,  stakeholders objected to the lack of a formal comment period in the  interim rulemaking process. We have previously reported that APHIS’s  use of interim rules to adjust user fees limits stakeholder input. APHIS  officials are exploring other regulatory alternatives for adjusting the fees.  In October 2010, APHIS hired a contractor to conduct a comprehensive  fee review to determine the full cost of AQI services, identify potential  changes to the fee structure, and recommend new fees. APHIS officials  estimated that, should APHIS and DHS agree that user fees need to be  increased, APHIS will publish a proposed rule by the summer of 2013.  GAO is conducting a separate evaluation of the AQI user fees’ structure  and options for setting and distributing them.
section_title: Limitations in the Arrival, Inspection, and Interception Data May Hinder Their Use in Managing the Program
paragraphs: The AQI program uses data on arrivals, inspections, and interceptions to  determine how well agricultural inspections identify prohibited materials  and to review ports’ performance, but data quality issues may prevent  AQI program officials from making full use of the data. More specifically,  CBP and APHIS officials told us that they determine how well agricultural  inspections identify prohibited materials by comparing the actual counts of  arrivals, inspections, and interceptions against the existing performance  measure of the amount of prohibited agricultural items expected to be  found in a specific pathway. In addition, AQI officials use the data to  identify annual trends specific to a particular port and to determine  whether any changes in the number of inspections or interceptions at that  port can be explained by external factors––such as a change in trade or  travel patterns––or indicate a performance problem at the port that needs  to be corrected. However, we found limitations that may undermine the  data’s overall usefulness for managing the program. Specifically:
The data may not be reliable. According to APHIS officials, data  quality is an ongoing issue with the AQI data systems, including the  Work Accomplishment Data System (WADS)—the primary repository  for arrival, inspection, and interception data. As a result, in 2004,  APHIS and CBP created an interagency stakeholder group with a goal  of ensuring data quality and accuracy by performing quarterly visual  checks of WADS data and providing data quality training to agriculture  specialists at the ports upon request. Since 2006, 30 of the 167 ports  with agriculture specialists have received this training. At some ports,  agriculture specialists record arrivals, inspections, and interceptions  every day on paper or spreadsheets and enter this information into  WADS at the end of the month, which may affect the reliability of the  data by introducing transcription errors. In addition, from 2010 through  2011, the joint CBP-APHIS quality assurance reviews, which focused  on 22 ports, found instances of discrepancies between data recorded  in WADS and data recorded on daily logs at about half of the ports  reviewed (10 out of the 22 ports). For example, at one port that was  reviewed, the number of passengers referred to secondary inspection  was underreported by nearly 10 percent, and reviews of other ports  identified some data that were overreported or had not been recorded.  APHIS and CBP officials told us that supervisors review the data  monthly at all the ports, and that there is a quarterly review process by  the interagency stakeholder group; however both of these reviews rely  primarily on a visual comparison of current and historical numbers and  thus detect only unusually large changes at a port. According to  APHIS officials, the monthly supervisory review process does not  require supervisors to compare the numbers in WADS with the daily  log sheets or other databases because the policy and procedures do  not have specific requirements on how data are to be reviewed. We  have previously reported that agencies should ensure that data are  free of systematic error or bias and that what is intended to be  measured is actually measured. According to a senior APHIS  official, ensuring AQI data quality and accuracy is a continuing  challenge because of various factors, including changes in the APHIS  and CBP personnel involved in data collection.
The data cannot be easily analyzed. WADS captures data on the  number of arrivals, inspections, and interceptions independently from  one another, and these data cannot be linked without extensive  analysis. CBP officials told us that being able to connect a specific  arrival to an inspection and the resulting interceptions would be useful  information for assessing the effectiveness of targeting activities and  inspection procedures. However, the analysis necessary to do so is  time consuming and cumbersome, undermining the use of the data for  managing the program. In addition, the way in which some data are  recorded in WADS complicates analysis, according to APHIS and  CBP officials. For example, WADS captures the number of agricultural  inspections carried out at individual ports by agriculture specialists. It  does not count the number of agricultural inspections carried out by  CBP officers when agriculture specialists are not present. Therefore,  WADS is not a representation of the total number of agricultural  inspections carried out at ports and cannot be used to analyze the  number of agricultural inspections conducted in relation to total  number of arrivals. In another example, each inspection is counted  once in WADS, but the number of interceptions that are recorded from  an inspection varies depending on what is found. Specifically, if a  passenger were inspected and found to be carrying five different  types of fruit, the agriculture specialist would record that result as five  interceptions for one inspection. However, if the passenger had five  pieces of one type of fruit, it would be recorded as one interception.  As a result, it is difficult to analyze the number of agricultural  interceptions found in relation to the total number of inspections  conducted.
The data have inherent limitations beyond AQI program control. The  number of agricultural arrivals, inspections, and interceptions varies  from year to year and by pathway. According to APHIS and CBP  officials, the variation is caused by numerous factors, including  weather, economics, and politics that affect travel and trade, as well  as variances in inspection targeting procedures. For example, arrivals  of a commodity from a particular country may drop sharply because of  severe weather conditions, such as hurricanes or droughts, or  because trade sanctions and embargoes have been put in place for  political reasons. APHIS and CBP officials noted that they target  inspections on the basis of factors such as pathways, commodities  known to be a risk for agricultural pests, country of origin, and a  shipper’s history of violations. Together, these factors can result in  inspection and interception numbers that do not correlate with the  number of arrivals for a particular pathway. For example, APHIS  officials told us that one pathway, such as cargo trucks, may have a  high number of arrivals, but if the cargo these trucks are carrying is  considered a low risk for carrying prohibited pests, then it would be  expected that the data for this pathway would show fewer inspections  and interceptions.
Some of these data limitations have the potential to affect other key  efforts that are currently underway, such as the program’s staffing model  and the effort to analyze the structure and amounts of AQI user fees–– both critical to the AQI program. For example, contractor documents  related to the staffing model stated that WADS data are not reliable for  understanding the actual workload of agriculture specialists, and that the  contractor had to undertake significant effort to make the data usable.  Specifically, the contractor’s analysis estimated that about 2 percent of  pedestrians are inspected by agriculture specialists, but that the data in  WADS indicated that agriculture specialists inspected 33 percent of  pedestrians.concerns regarding potentially inconsistent or inaccurate data entry,  which could result in decreased confidence with data reporting and  analysis. Without reliable data on work activities, AQI program officials  cannot be assured that they have the information they need to manage  the program.
section_title: Survey Responses Show Some Changes in Agriculture Specialists’ Views
paragraphs: According to our analysis of agriculture specialists’ and supervisors’  responses to our 2012 survey and the results from our 2006 survey,  some aspects of agriculture specialists’ and supervisors’ views about their  work environment have changed since 2006. The responses to our open-  and closed-ended questions show that views about some areas have  improved, some concerns expressed in 2006 have persisted, and some  concerns have increased. (To view the full results of our survey, please  see the electronic supplement to this report, GAO-12-884SP.)
subsections: section_title: Survey Responses Show Improvement in Some Areas
paragraphs: According to our analysis of the responses to the 2006 and 2012 surveys,  agriculture specialists and supervisors believe some aspects of the AQI  program have improved since we last reported. Notably, in 2006, “nothing  is going well” was the second most common response to our open-ended  question, “What is going well with respect to your work as an agriculture  specialist?”; in 2012, it was one of the least common responses,  suggesting that, in general, respondents believe aspects of the AQI  program have improved. (See appendix II for a summary of responses to  this open-ended question for 2006 compared to 2012.) Based on our  analysis of survey results, agriculture specialists and supervisors  specifically believe that training and information-sharing have improved.
Training. More agriculture specialists and supervisors at CBP now believe  they are sufficiently trained to perform their agriculture inspection duties  than did in 2006, according to our analysis of responses to the closed- ended questions in our 2012 survey. Specifically, we estimate that 76  percent of agriculture specialists and supervisors at CBP employed  before the merger believe they are definitely sufficiently trained to perform  their agricultural inspection duties, and 51 percent employed after the  merger believe they are definitely sufficiently trained. This represents an  increase from 2006, when we estimated 58 and 36 percent,  respectively. (See table 1.)
Additionally, based on our analysis of the responses to our closed-ended  questions, we estimate 28 percent of agriculture specialists and  supervisors at CBP believe that the sufficiency of training is not a  challenge to conducting agriculture duties at their port, and another 29  percent believe it is only a minor challenge. We did not ask about  whether or not training was a challenge in our closed-ended questions in  2006, but at that time, training was the fourth most frequent response to  our open-ended question, “What would you like to see changed or  improved with respect to your work as an agriculture specialist?” Our  analysis of the responses to our 2012 survey shows that views about  training have generally improved, but out of 406 respondents, we  received 68 comments in response to an open-ended question listing  certain aspects of training they would like to see changed or improved.  For example, some respondents commented that they would like to see  more specific or refresher training on agriculture pests that they may  encounter on the job, and one respondent commented that he would like  the opportunity for more hands-on—rather than web-based—training.
Information-sharing. According to our analysis of 2006 and 2012 survey  responses, information-sharing has also improved. Based on the  responses to our closed-ended question on whether information is  delivered in a timely manner, we estimate that 36 percent of agriculture  specialists and supervisors at CBP believe that information on agriculture  regulatory changes is always delivered in a timely manner, and another  48 percent believe the information is delivered in a timely manner most of  the time. These views reflect an increase from 2006, when we  estimated 17 percent of agriculture specialists and supervisors at CBP  believed such information was always delivered in a timely manner, and  38 percent believed it was delivered in a timely manner most of the  time. Additionally, we estimate that, compared with 2006, statistically  more agriculture specialists and supervisors at CBP in 2012 believe they  always receive information on urgent alerts, pest alerts, and updated  pages for the agriculture regulatory manual in a timely manner. (See  table 2.) We estimate that 57 percent of agriculture specialists and  supervisors at CBP in 2012 believe the timeliness of information provided  to agriculture specialists is either not a challenge or is a minor challenge  to conducting agriculture duties at their port. We did not ask this  question in 2006.
According to our analysis of the responses to our 2012 survey, some of  the concerns agriculture specialists and supervisors at CBP reported in  2006 persist. In particular, concerns remain about CBP’s chain of  command, the agriculture mission, and working relationships among co- workers.
CBP’s chain of command. Our analysis indicates that respondents remain  concerned about CBP’s chain of command—including concerns about  poor communication related to internal policies, supervisors lacking  agriculture experience, and the need for an agriculture chain of command  that is separate from CBP. In 2006, CBP’s chain of command was the  third most frequent response to our open-ended question, “What would  you like to see changed or improved with respect to your work as an  agriculture specialist?” Similarly, in 2012, it was the category noted most  often as something to be changed or improved. Furthermore, based on  our analysis of closed-ended questions in 2012, we estimate that 30  percent of agriculture specialists and supervisors at CBP believe the CBP  chain of command is a very major challenge to conducting agriculture  duties at their port, and another 23 percent believe it is a major  challenge. (See appendix II for a summary of responses to the open- ended question, “What would you like to see changed or improved with  respect to your work as an agriculture specialist?” in 2006 compared to  2012.) We did not ask about CBP’s chain of command in our closed- ended questions in 2006.
Agriculture mission. As with the results of our 2006 survey, 2012 survey  respondents identified the priority given to the agriculture mission among  the top four aspects of the AQI program they would like to see changed or  improved. Further, we estimate that 39 percent of agriculture specialists  and supervisors at CBP believe the priority given to the agriculture  mission is a very major challenge to conducting agriculture duties at their  port, and another 23 percent believe it is a major challenge based on our   Out of 406  analysis of responses to closed-ended questions in 2012. survey respondents, 86 respondents identified the priority given to the  agriculture mission as something they would like to see changed or  improved in response to our open-ended question. Respondents noted  that competing priorities sometimes prevented them from performing their  missions and that agriculture specialists do not have enough time to  complete their tasks, such as looking for pests in agriculture materials  intercepted from passengers. Not having enough time to complete  agriculture-mission related tasks is a concern for about the same  percentage of agriculture specialists and supervisors at CBP in 2012 as it  was in 2006. (See table 3.)
The 95 percent confidence interval surrounding the estimate of 39 percent of agriculture  specialists and supervisors at CBP identifying the priority given to the agriculture mission  is a very major challenge ranges from 34.0 percent to 44.9 percent, and the 95 percent  confidence interval surrounding the estimate of 23 percent ranges from 18.5 percent to  28.0 percent.
Concern for the priority given to the agriculture mission persists, but 91  out of 406 respondents indicated in their response to the open-ended  question about what is going well that they believe some aspects of the  AQI mission are going well, such as inspections and interceptions.
Working relationships. Working relationships among co-workers is still a  concern for agriculture specialists and supervisors, according to our  analysis of the 2006 and 2012 survey data. Specifically, we estimate a  similar percentage of agriculture specialists and supervisors at CBP in  2006 and 2012 believe their work is definitely not respected by CBP  management—42 percent in 2006 and 43 percent in 2012—and by CBP  officers—35 percent in 2006 and 28 percent in 2012.Respondents identified working relationships in both 2006 and 2012 as  one of the most frequent responses to the open-ended question asking  respondents what they would like to see changed or improved. In  particular, some respondents stated that CBP officers and management  do not respect their work or mission, and others noted that agriculture  specialists are not treated as part of the team. However, in both 2006 and  2012, respondents also identified working relationships among co- workers as one of the most frequent responses to our open-ended  question of what is going well, suggesting that while some aspects of  working relationships continue to be a concern, other aspects of working  relationships are going well. In particular, some respondents noted that  their fellow agriculture specialists were dedicated, highly-qualified, or hard  working.   (See table 4).
section_title: Conclusions
paragraphs: The AQI program is a key component of U.S. efforts to protect agriculture  from the unintentional or deliberate introduction of pests and disease, and  effective management of the AQI program requires a coordinated effort  by DHS and USDA. Since we reported in 2006, both DHS and USDA  have made progress in implementing our recommendations and  overcoming some of the management challenges we identified, including  ensuring that user fees are transferred to CBP and that important  agriculture information is shared with agriculture specialists in a timely  manner.
However, the AQI program continues to wrestle with fundamental  problems that undermine the management of the program and risks  wasting resources in a fiscally constrained environment. Specifically, in  the absence of a strategic plan that lays out the program’s joint mission  and goals, APHIS and CBP do not have a framework for defining the  program’s mission, setting goals to achieve the desired results, and  identifying performance measures for gauging progress towards those  goals.
Furthermore, the agencies may not have sufficient information on which  to base key decisions to support the AQI program because the data the  two agencies are collecting and using for managing the program may not  be reliable. Without ensuring that the data on arrivals, inspections, and  interceptions across ports are entered accurately, the AQI program may  not have sufficiently reliable data for supporting critical efforts for  managing the program. When discrepancies exist between data in WADS  and data recorded on daily logs at about half of the ports reviewed by the  2010 to 2011 joint CBP-APHIS quality assurance reviews, it could be an  indicator of more widespread data quality issues related to failures with  the supervisory review policy and procedures in ensuring that data are  entered accurately. In addition, the data supporting the agriculture canine  program are hindered by the absence of a timely and consistent data  review process at CBP field offices, resulting in incomplete and  inaccurate data. Moreover, the agriculture canine program continues to  collect some data that may not be meaningful or relevant to the program,  further straining the program’s resources. Without sufficiently reliable  data, the AQI program cannot assess the effectiveness of the canine  program. Furthermore, by collecting data that are not meaningful or useful  for assessing the effectiveness of its canine handlers, the AQI program  may be using resources to collect data it does not need.
Finally, the agencies may not be using their resources effectively. Having  devoted 6 years and hundreds of thousands of dollars in contracts to  develop a staffing model, CBP still does not have a risk-based staffing  model that provides assurance that those ports of highest vulnerability for  the entry of pests and disease are adequately staffed. Furthermore, DHS  has not assessed the risk that insufficient resources may pose to  implementing the staffing model once it is completed. Without a plan or  strategy for how it will implement this staffing model that considers the  fiscal resources that may realistically be available, the agency risks  investing in a staffing model that it cannot execute. CBP also risks  increasing the vulnerability of the agriculture sector to foreign pests and  disease or disrupting international trade if agriculture specialists cannot  keep up with demand for agriculture inspections. In addition, well-trained  supervisory staff are a crucial resource for supporting the agriculture  canine teams, but the results of our survey and the joint CBP-APHIS  quality assurance reviews indicate that some supervisory staff may not  have canine training or experience because of barriers such as a lack of  resources at ports and the voluntary nature of supervisory canine training.  As a result, supervisors in the AQI program may not have the skills  necessary to manage issues unique to the canine program, such as the  health and training of the canine. By overcoming these challenges, the  United States would be in a better position to protect agriculture from the  economic harm posed by foreign pests and disease.
section_title: Recommendations for Executive Action
paragraphs: We are making the following six recommendations.
To help ensure the CBP and APHIS agricultural quarantine inspection  program protects U.S. agriculture from accidental or deliberate  introduction of foreign pests and disease, we recommend that the  Secretaries of Agriculture and of Homeland Security work together to take  the following three actions:
Develop a strategic plan for the AQI program that lays out its joint  mission and program goals.
Once the strategic plan is completed, as part of a coordinated  strategic planning effort, identify corresponding meaningful  performance measures for monitoring progress towards those goals.
Continue taking steps to improve the reliability of AQI data on arrivals,  inspections, and interceptions across ports, including reviewing the  supervisory review policy and procedures to ensure the data are  entered accurately.
In addition, we recommend the Secretary of the Department of Homeland  Security, in consultation with USDA where appropriate, take the following  three actions:
Take steps to ensure the agriculture canine program has reliable and  meaningful data, including instituting a timely and consistent review  process at CBP field offices, and evaluate the relevance of data  collected for the agriculture canine program.
Develop a plan or strategy for implementing the forthcoming AQI  staffing model that assesses the risk of potential fiscal constraints and  determines what actions should be taken to mitigate that risk by  considering the fiscal resources that may realistically be available to  ensure that agriculture staffing levels at each port are sufficient.
Identify any agriculture canine supervisors who do not have canine  training or experience and work with port directors to overcome any  barriers to providing formal training.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided USDA and DHS with a draft of this report for their review  and comment. In their written comments, both agencies agreed with our  recommendations, and DHS provided technical comments, which we  incorporated as appropriate. USDA’s comments appear in appendix III.  DHS’s comments appear in appendix IV.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  Secretaries of Agriculture and Homeland Security, the appropriate  congressional committees, and other interested parties. In addition, the  report is available at no charge on GAO’s website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-3841 or shamesl@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to examine the extent to which (1) U.S. Department  of Agriculture (USDA) and the Department of Homeland Security (DHS)  have implemented recommendations we made in 2006 to improve the  Agricultural Quarantine Inspection (AQI) program; (2) data on arrivals,  inspections, and interceptions are used for managing the program; and  (3) the views of agriculture specialists at Customs and Border Protection  (CBP) regarding their work environment have changed, if at all, since our  2006 reports.
To address all of these objectives, we visited or spoke with officials at a  nonprobability sample of five CBP ports of entry to observe agriculture  work activities and discuss challenges related to conducting agriculture  inspections and interceptions. These ports of entry were Austin, Texas;  Baltimore, Maryland; Dallas, Texas; Miami, Florida; and San Ysidro,  California. We selected these ports of entry based on size; the presence  of agriculture specialists, supervisors, and canine handlers; and entry  pathways, such as air, sea, and land. Because we used a nonprobability  sample, the information we obtained from these visits cannot be  generalized to other CBP ports of entry. The visits instead provided us  with more in-depth information on the perspectives of various agriculture  specialists in these ports of entry about the management of the AQI  program.
To examine the extent to which GAO’s May 2006 recommendations have  been implemented, we interviewed DHS and USDA officials and  reviewed documentation related to (1) improving information-sharing, (2)  undertaking a full review of DHS’s financial management system for the  AQI program, (3) removing barriers to timely and accurate AQI user fee  transfers from USDA to DHS, (4) adopting meaningful performance  measures, (5) establishing a national risk-based staffing model, (6)  improving the agriculture canine program, and (7) revising user fees to  cover AQI program costs. Specifically:
To determine the extent to which USDA’s Animal and Plant Health  Inspection Service (APHIS) and DHS’s CBP improved information- sharing, we obtained and reviewed CBP’s agriculture information  sharing policies and protocols regarding agriculture alerts and policy  updates, among other things, to evaluate changes since our 2006  report. We discussed these policies with agriculture specialists and  supervisors during our site visits. We also interviewed the Deputy  Executive Director for Agriculture Operational Oversight to discuss  changes in information-sharing policies and protocols since 2006. We  conducted a nationally representative survey of agriculture specialists  and supervisors at CBP, details of which are discussed below, and  analyzed the results to determine the extent to which agriculture  specialists and supervisors believe that information-sharing is a  challenge.
To determine the extent to which DHS reviewed its financial  management system, we reviewed guidance on procedures for  accounting for CBP work activities related to agriculture and  interviewed officials with CBP’s Budget Group and APHIS’s Financial  Management Division to discuss how the changes have affected the  financial accountability for funds allocated for agriculture quarantine  inspections.
To review how USDA and DHS have removed barriers to user fee  transfers, we obtained the updated Memorandum of Agreement  signed by the Secretaries of Agriculture and Homeland Security that  modified the user fee fund transfer schedule. We also received AQI  user fee fund transfer records from APHIS to compare the timing and  amount of transfers to the agreed upon amount. We interviewed  officials from CBP’s Budget Group and APHIS’s Financial  Management Division regarding the transfers to evaluate whether the  steps taken by USDA have resulted in timely and accurate transfers to  DHS.
To evaluate APHIS’s and CBP’s progress in developing meaningful  performance measures for the AQI program, we reviewed strategic  planning documents, such as the Plant Protection and Quarantine  (PPQ) strategic plan and selected sections of the Office of Field  Operations’ draft strategic plan, for how AQI activities are  incorporated into these plans. We interviewed AQI officials, including  PPQ officials with strategic planning responsibilities, CBP officials with  the Office of Field Operations Strategic Planning Division, and an  official with the APHIS-CBP joint agency task force to discuss the  existing AQI performance measures, including how they are used to  assess performance, their limitations for measuring key AQI activities,  and the extent to which APHIS and CBP are developing additional  performance measures for managing the AQI program. We also  discussed the extent to which the AQI program has developed a  strategic plan with a joint mission and program goals, and we  compared this to leading practices we have previously reported on for  federal strategic planning at lower levels within federal agencies, such  as planning for individual divisions, programs, or initiatives.
To evaluate the extent to which CBP has established a national risk- based staffing model, we reviewed CBP contracting documents  related to CBP’s efforts to develop a model and interviewed officials  from CBP’s Office of Field Operations and the Deputy Executive  Director for Agriculture Operational Oversight regarding their progress  towards, and their implementation plan and strategy for, a model. As  part of evaluating the potential impact of the AQI program not having  a risk-based staffing model, we reviewed standards for internal control  in the federal government.
To assess the effectiveness of the canine program, we analyzed and  reviewed agriculture canine units’ quarterly proficiency scores from  fiscal years 2006 through 2011, the agriculture canine training  manual, agriculture canine units’ leash time data from fiscal years  2006 through 2011, and CBP’s agriculture canine daily statistics  template. We discussed these with the CBP agriculture canine  program manager to determine the extent to which the data can be  used for assessing the agriculture canine program. To determine the  extent to which the proficiency data have missing values, we  compared the number of active agriculture canine teams for each  quarter provided by the agriculture canine program manager to the  number of canine teams that reported a proficiency score for that  quarter and discussed the limitations of this approach with the  agriculture canine program manager. We also reviewed the results of  APHIS-CBP joint quality assurance reviews from 2010 through 2011,  the most recent years for which complete data are available, for those  ports that had canine handlers to determine the number of reviews  that found canine handlers did not have supervisors with canine  training. We also analyzed the results of our survey, discussed below,  to determine the extent to which canine handlers believe their  supervisor has adequate experience. We interviewed the agriculture  canine program manager on training policy and procedures for canine  handlers and supervisors, steps CBP has taken to address staffing  issues raised in our 2006 report, and data quality issues with the  agriculture canine program.
To evaluate how DHS and USDA have revised user fees since 2006,  we reviewed Federal Register documents relating to APHIS’s  attempts to revise AQI user fees. We interviewed APHIS officials on  their plans for future revisions to the user fee structure. GAO is  conducting a separate evaluation of the AQI user fee structure and  options of setting and distributing them.
To examine the extent to which data on agriculture arrivals, inspections,  and interceptions are used for managing the AQI program, we reviewed  DHS and USDA inspection and interception data, related summary  reports, and policies. Specifically, we obtained and analyzed data from  USDA APHIS’s Work Accomplishment Data System for fiscal years 2001  through 2011 and reviewed data collection and entry procedures and  definitions of required data elements that may impact the use of the data  for managing the program. We reviewed DHS and USDA joint Quality  Assurance Reviews— joint agency reports on port compliance with  agricultural inspection policy— from 2010 through 2011, the most recent  years for which data were available. We reviewed documents related to  contractor-led efforts for key aspects of the AQI program to determine  how those efforts were impacted by data quality issues. Additionally, we  interviewed key program officials at USDA and DHS to discuss the  reliability of the data and whether there were limitations in the data that  may affect the extent to which these data on agricultural quarantine  inspections and interceptions are used to make AQI program decisions.  To assess the reliability of arrival, inspection and interception data, we  observed data collection and data entry procedures at selected ports of  entry; reviewed the finding of CBP-APHIS joint quality assurance reviews  from 2010 and 2011; and interviewed agriculture inspectors, supervisors  and program managers. We concluded that inspection and interception  data in APHIS’s Work Accomplishment Data System (WADS) may hinder  managers’ ability to definitively determine whether patterns observed in  inspection and interception data are attributable to real changes in the  number of inspections and interceptions, or to errors in the data collection  and entry processes. We found that potential weaknesses exist at several  places in the data collection process. The process for recording  inspections and interceptions varies significantly from port to port. At  some ports, inspectors record inspection information on a paper clipboard  and then proceed to enter that information into an electronic system  following the inspection. However, at other locations, paper records of  inspections and interceptions are compiled and entered into electronic  form as much as one month after the inspection took place. The multiple  steps involved in data entry increase the likelihood of errors; and at some  ports, the amount of time that elapses between inspection and  interception activities and data entry reduces the usefulness of inspector  recall to correct any ambiguities or errors found in paper records.
To determine the extent to which the views of agriculture specialists  regarding their work environment have changed, if at all, since GAO last  reported, we conducted a nationally representative survey of agriculture  specialists and supervisors at CBP regarding their work environment. The  survey consisted of closed-ended and open-ended questions.  Specifically, we drew a stratified random probability sample of 556  agriculture specialists and supervisors from the universe of 2,311  specialists in the DHS’s CBP who were engaged in agricultural inspection  duty as of October 2011 and still engaged in this duty as of February  2012. We excluded 10 respondents from our sample of 556 as out-of- scope because these individuals performed management duties outside  of CBP ports. All agriculture canine handlers were placed in one stratum;  other strata were defined by the number of specialists at the respective  ports. Each sample was subsequently weighted in the analysis to account  statistically for all specialists in the population. We received an  unweighted response rate of 74 percent. The survey results for the  closed-ended questions are generalizable to all agriculture specialists and  supervisors at CBP.
Because we followed a probability procedure based on random  selections, our sample is only one of a large number of samples we might  have drawn. Since each sample could have provided different estimates,  we express our confidence in the precision of our particular sample’s  results as 95 percent confidence intervals. This is the interval that would  contain the actual population values for 95 percent of the samples we  could have drawn. In developing the survey, we met with CBP and APHIS  officials to gain a thorough understanding of the AQI program. We also  shared a draft copy of the questionnaire with CBP officials who provided  us with comments, including technical corrections, to ensure that the  questions were clear and unambiguous, terminology was used correctly,  the questionnaire did not place an undue burden on agency officials, the  information could feasibly be obtained, and the survey was  comprehensive and unbiased. On the basis of the feedback we received,  we made changes to the content of the survey. We posted the  questionnaire on GAO’s survey website. When the survey was activated,  the agriculture specialists who had been selected to participate were  informed of its availability by an e-mail message that contained a unique  user name and password. This allowed respondents to log on and fill out  a questionnaire but did not allow respondents access to the  questionnaires of others. We sent reminder and follow-up e-mails to  agriculture specialists who had not completed their surveys in a specific  time period to increase our response rate. Since this was a Web-based  survey, respondents entered their answers directly into the electronic  questionnaire, eliminating the need to key data into a database, thus  minimizing error. The survey was available from February 6, 2012, until  March 9, 2012. Results of the survey are summarized in GAO-12-884SP.
To facilitate comparison with the findings from our prior report, we used a  questionnaire for this survey similar to the one we used in 2006. The  2006 questionnaire focused on the transition from agriculture specialists  and supervisors in USDA to CBP, and we recognize that there may be  demographic shifts in these agriculture specialists and supervisors that  we did not calculate. However, we do not believe this affected the results  of our analysis. The 2012 survey contained 25 closed-ended questions  that asked for opinions and assessments of (1) agricultural inspection  training, (2) agricultural inspection duties, (3) agricultural inspection  supplies and equipment, (4) communication and information-sharing  within CBP and between other agencies, and (5) challenges related to  conducting agricultural work activities. Some of those questions were  specifically for agriculture canine handlers. In addition, the survey  contained two open-ended questions asking for opinions and  assessments of what is going well and what could be improved in regards  to the work of agriculture specialists. We conducted a content analysis on  the responses to the open-ended questions using all of the same  categories that were used in 2006, adding new categories as appropriate.  We categorized the responses based on the descriptions of the  categories from 2006. The responses were placed into as many  categories as appropriate; however, each response was placed no more  than once in any category. We conducted an independent review of the  results and resolved any disagreement.
We conducted this performance audit from August 2011 to September  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Summary of Responses to Open- ended Survey Questions in 2006 and 2012
section_title: Appendix III: Comments from the U.S. Department of Agriculture
section_title: Appendix IV: Comments from the Department of Homeland Security
section_title: Appendix V: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Mary Denigan-Macauley  (Assistant Director), Adam Anguiano, Carl Barden, Michelle Cooper,  N’Kenge Gibson, Amanda Harris, Stuart Kaufman, Kirsten Lauber, Alison  O’Neill, and Kiki Theodoropoulos made key contributions to this report.  Important contributions were also made by Kevin Bray, Joyce Evans, and  Jena Sinkfield.