Drug Control: DEA Could Improve Its Heroin Signature and Domestic Monitor Programs' Geographic Source Data
Published: 2002-03-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Drug Enforcement Administration (DEA) runs two programs--the Heroin Signature Program and the Domestic Monitor Program--that provide information on trends in heroin trafficking. The only programs of their kind in this country, these two program conduct chemical analyses to pinpoint the geographic origin of heroin being sold on the streets. The Domestic Monitor Program determines (1) the source of heroin that has been bought undercover in 23 U.S. cities and (2) the purity and price of heroin at the retail level. The Heroin Signature Program provides law enforcement with information on the origins of heroin at the wholesale and retail level in some U.S. cities. Data from the two programs are included in intelligence and investigative reports provided to DEA and other federal law enforcement agencies, which use this information to adjust their drug enforcement efforts. The quantity of heroin seized by the Customs Service at ports-of-entry but not sent to DEA for testing may make a difference in the results reported by DEA. All seizures at ports-of-entry forwarded to DEA are tested for geographic source, according to DEA officials. However, Customs is not required to send all heroin seizures to DEA.
section_title: What GAO Found
paragraphs: GAO found that the usefulness of the data from and Heroin Signature Program and the Domestic Monitor Program is limited because they are based on nonrepresentative samples of their respective populations. GAO believes that DEA could produce stronger estimates if it modified its methodology.

section_title: Letter
paragraphs: The Drug Enforcement Administration (DEA) administers two programs, the Heroin Signature Program (HSP) and the Domestic Monitor Program (DMP), that serve as indicators for assessing trends in the geographic source of heroin supplied to the United States. These programs produce data on the geographic source of heroin through a series of chemical analyses and are the only programs of their kind in the United States. The HSP reports the geographic source of heroin seized at ports-of-entry (primarily by the U.S. Customs Service) and other locations, as well as wholesale level purity. The DMP reports the geographic source of heroin purchased undercover in 23 U.S. metropolitan areas, as well as retail level heroin purity and price. This report does not focus on purity and price.
This report responds to your request that we review how DEA captures and reports the data for the HSP and DMP and that we provide, if appropriate, possible alternative approaches for DEA to consider to improve the HSP and DMP data. Specifically, this report discusses the purpose of the HSP and DMP;   how federal law enforcement uses the data generated by the HSP and   whether the quantity of heroin seized at ports-of-entry by Customs, but not sent to DEA for testing, is sufficient to make a difference in the results reported by DEA; and   whether the HSP and DMP data could be improved.
To address these areas, we met with officials from DEA, Customs, and the Office of National Drug Control Policy (ONDCP) concerning the purpose and operations of the HSP and DMP and how federal law enforcement uses the data generated by the programs. We obtained and analyzed data from Customs to determine the quantity of heroin that is seized at ports-of- entry by Customs, but not sent to DEA for testing. To determine if improvements could be made in the HSP and DMP data, we met with DEA officials and reviewed the methodology used by DEA in its design and implementation of the programs.
subsections: section_title: Background
paragraphs: Each heroin-producing region has a unique production process, or signature, which generally can be determined through chemical analyses. In the HSP, seized or purchased substances are forwarded to one of DEA’s regional laboratories, which confirms whether the substance is heroin. If the substance is confirmed to be heroin, the laboratory is responsible for preparing a written report for judicial purposes and, in certain circumstances, providing a sample to DEA’s Special Testing and Research Laboratory (STRL) for signature analysis. STRL analyzes the heroin samples; in most instances, these analyses result in the identification of the heroin’s geographic source. In the DMP, purchased substances are sent directly to STRL for analysis.
The heroin that is selected for HSP testing is selected from either “cases” or “exhibits.” Figure 1 illustrates a possible case that includes multiple seizures and exhibits.
subsections: section_title: Seizure
paragraphs: As shown in figure 1, a single case can include more than one seizure of heroin. For example, a single case would include more than one seizure if agents seized heroin from the same person (the same case), but on different dates. In turn, a single seizure might include more than one exhibit, if agents find heroin associated with the same person on the same date, but at multiple locations. For example, a seizure would include three exhibits if agents, on the same date, seize heroin from multiple locations such as the person’s desk, closet, and coat pocket.
DEA initiated the HSP in 1977. Heroin for the HSP is obtained from eight sources and selected for signature analysis as shown in table 1.
DEA initiated the DMP in its New York Field Division in 1979. The program has expanded to include 23 metropolitan areas. The DMP was originally designed to enable DEA to monitor the price and purity of retail- level heroin sold in the United States; it now also provides for the purchase of heroin for signature analysis. In the DMP, DEA provides funding for quarterly purchases by DEA field divisions, that may utilize cooperating sources to make retail-level purchases of heroin. Ten purchases are to be made in 22 of the 23 metropolitan areas, each quarter; in New York City, 20 purchases are to be made in each quarter.
DEA guidelines provide that a certain number of DMP purchases be made each quarter, throughout the 3 months of each quarter, and in various locations in the metropolitan areas. According to the DEA guidelines, it would seldom be necessary to make more than one purchase in any one location, per day. The most important requirement is that exhibits should be purchased from locations within the metropolitan areas that are dissimilar enough to ensure that they come from different suppliers. Each purchase should weigh at least 1 gram net, including diluents and adulterants, to ensure that there is a sufficient amount of pure heroin available to perform a signature analysis.
According to DEA, the average time necessary to complete the signature analysis of a heroin sample is about 4 hours, with an associated cost of about $375 per sample. DEA indicated that the STRL performs signature analyses of 3,000 heroin samples annually, for a total annual STRL cost of about $1.13 million. DEA provides its field divisions with a total of $200,000 annually for the DMP purchases.
DEA prepares annual reports on the HSP and DMP data. The HSP reports display data on geographic source by net weight on a national basis. The DMP reports display data on geographic source by the number of exhibits by metropolitan area.
section_title: Scope and Methodology
paragraphs: To determine the purpose of the HSP and DMP and how they operate, we interviewed officials at DEA’s Office of Intelligence, Forensic Sciences, and the STRL. We reviewed relevant policies, reports, and other documentation.
To determine how federal law enforcement uses the data generated by the programs, we interviewed officials at DEA’s Office of Intelligence and Office of Forensic Sciences and ONDCP. We also reviewed relevant documentation of the use made of the data, including documentation of instances in which changes have been made in federal law enforcement efforts as a result of the data.
To determine if the heroin seized at ports-of-entry, but not tested by DEA, is of sufficient quantity to make a difference in the results reported by DEA, we interviewed officials at DEA’s Office of Intelligence, Office of Forensic Sciences, and the STRL, as well as Custom’s Smuggling Investigations Division and Office of Intelligence. We also obtained and analyzed data from Customs regarding seized heroin in fiscal years 1998- 2000. These data were derived from the Department of the Treasury’s Enforcement Communications System (TECS), Seized Assets and Case Tracking (SEACATS) subsystem. We did not verify the accuracy of these data. The data included the date and location of the seizure, the total weight of the seizure (less the weight of any packaging or container), and the disposition of the heroin. According to Customs, a seizure is recorded in the system when contraband is discovered and physical custody is taken. Under Customs’ policy, seizures that involve taking heroin from more than one place, for example, from an individual’s pocket and from his suitcase, are counted as one seizure but entered into the system as two “line items.” We were provided data by line items. Each line item from the same seizure equates to a DEA exhibit.
To determine if the sample-based HSP and DMP data could be improved, we interviewed officials at DEA’s Office of Intelligence and Office of Forensic Sciences. We reviewed sample design and sample selection methodology and the formulas and methodology used to develop data on the geographic source of heroin. We obtained and reviewed HSP and DMP data files for calendar years 1999 and 2000. We reviewed earlier reports, analyzed current methodology, and how DEA reports and caveats the figures.
In performing our work, we did not talk with officials from all federal law enforcement agencies that may make use of the programs’ data. We performed our work from May 2001 to February 2002, in accordance with generally accepted government auditing standards. We requested comments from DEA, Customs, and ONDCP. Comments from DEA are summarized at the end of this report and contained in appendix III.
section_title: Purpose of the HSP and DMP Is To Produce Data That Detect Trends in Heroin Sources
paragraphs: According to DEA officials, the HSP and DMP produce data for detecting trends in the geographic source of heroin supplied to the United States. Officials stressed that the purpose of these data is not to provide overall estimates of where all heroin supplied to the United States originates. The HSP data are intended to provide law enforcement with a “snapshot” of where heroin at the wholesale level originates. The DMP data are intended to provide law enforcement with a snapshot of where heroin at the retail level, in certain metropolitan areas, originates. Officials explained that they believe that, over time, the snapshots begin to tell a story about what is happening with drug trafficking patterns. When this happens, officials can make their decisions, in conjunction with other investigative and intelligence data.
Officials also stressed that direct comparisons should not be made between the geographic source data from the HSP and DMP. For example, the wholesale heroin seized in one market (HSP seizures) may not be intended for retail-level sale (DMP purchases) in the same market. In addition, comparisons should not be made between the HSP data and DMP data because the HSP data reflect law enforcement investigative priorities and techniques, in terms of where and how seizures are made, as well as the difficulties associated with the various concealment techniques used by smugglers. In addition, large quantity seizures of heroin from one geographic source area may boost that geographic source area’s representation in the HSP data. This may be especially applicable to heroin from Southeast and Southwest Asia that has been traditionally smuggled in large, multikilogram quantities.
The officials also noted that these same factors could influence year-to- year fluctuations in the proportion of heroin from each geographic source area. For example, law enforcement priorities and smuggler concealment techniques are reflected in the numerous small-quantity heroin seizures from Colombian air couriers.
section_title: Federal Law Enforcement Used the HSP and DMP Data for Intelligence Purposes and Management
paragraphs: According to DEA officials, the HSP and DMP data are used for intelligence purposes and as a management tool by federal law enforcement. Data drawn from a variety of sources, including the HSP and DMP, are used to develop a comprehensive picture of heroin trafficking in the United States.
For example, HSP and DMP data are frequently included in DEA intelligence and investigative reports to corroborate heroin trafficking trends in the United States and to inform DEA and other federal law enforcement agencies about heroin trafficking. In addition, ONDCP officials noted that the data are used as a confirmation of data from other sources, such as opium production and cultivation estimates provided by the Central Intelligence Agency.
ONDCP officials noted that it uses data from the HSP and DMP for drug- flow modeling and that the data, which are viewed as one of the better heroin market indicators, are key components of ONDCP’s data analysis efforts. ONDCP officials also said that data from the HSP and DMP are used for such purposes as testimony before the Congress and in ONDCP’s annual National Drug Control Strategy.
According to DEA, federal law enforcement also uses the HSP and DMP data as management tools to make adjustments in enforcement activities. Changes in HSP and DMP data could alert management to changing trafficking patterns. DEA cited the emergence of heroin from South America as an example of how the HSP and DMP geographic source data have been used in law enforcement intelligence and management. Southeast Asian heroin dominated the market on the East Coast until 1991; Southwest Asian heroin was also readily available. In 1991, a high- purity heroin entered the eastern U.S. market and was initially identified by DEA as high-purity but atypical Southwest Asian heroin. Intelligence reports indicated that heroin was entering the United States from South America. Also, reportedly Southwest Asian heroin producers had taught Colombians their methods of processing opium into heroin. DEA’s subsequent determination of a signature unique to South America confirmed this intelligence and, as a result, South America was identified as a new supplier of heroin into the United States.
Officials said that the data are also used to monitor the success of various initiatives. For example, a decrease in the amount of tested heroin that is found to have originated in a particular geographic source area can be an indicator that law enforcement initiatives against that particular area have been successful.
section_title: Quantity of Heroin Seized at Ports-of- Entry by Customs but not Sent to DEA for Testing May Be Sufficient To Make a Difference in Results Reported by DEA
paragraphs: According to DEA officials, all ports-of-entry seizures sent to DEA by Customs are tested by DEA for geographic source. However, for several reasons, Customs is not required to send all seized heroin to DEA. DEA and Customs officials noted that Customs is not required to send to DEA abandoned heroin or heroin that is turned over to state or local officials for prosecution. “Abandoned heroin” is heroin that cannot be connected to any individual or defendant. For example, an unmanifested kilogram of heroin found in an aircraft cargo hold is considered abandoned. Under its MOU with DEA, Customs is not required to send abandoned heroin to DEA. Instead, it is to be reported on a Customs Search, Arrest, Seizure report (CF-151) and turned over to the Customs seized property custodian for destruction. Also, under the MOU, Customs does not submit for testing heroin that does not meet local U.S. attorney prosecution guidelines.Instead, Customs officials explained that in most instances, this heroin is to be turned over to state or local officials for prosecution. According to Customs officials, there are also instances in which the weight of the seized heroin is so low that it is not turned over to state or local officials for prosecution. In these instances, it is to be destroyed.
Our analysis of Customs’ heroin seizure data revealed that, for fiscal years 1998-2000, 57 percent of the total weight of the heroin seized by Customs was not sent to DEA. Data on the number and weight of Customs heroin seizures, including the number and weight of heroin seizures not sent to DEA for testing, are displayed in table 2.
According to DEA officials, it is not crucial to test the relatively smaller seizures. Table 3 displays amounts seized by Customs over a 3-year period. Of the total number of line items that were not forwarded to DEA, about 72 percent exceeded 100 grams in weight; these line items accounted for over 99 percent of the total weight of all line items not forwarded to DEA for testing.
section_title: The HSP and DMP Data Could Be Improved
paragraphs: The HSP and DMP data on the geographic source of heroin could be improved. The HSP data have limitations; appendix I of this report describes the current HSP selection methodology, its limitations, and opportunities for improvements. The DMP data also have limitations; appendix II describes the current DMP selection methodology, its limitations, and opportunities for improvements.
subsections: section_title: HSP Limitations and Opportunities for Improvement
paragraphs: According to DEA, it does not intend that the HSP data be used either to produce estimates as to where all wholesale heroin supplied to the United States originates or as to where all wholesale heroin seized in the United States and forwarded to DEA for testing originates. Our analysis showed that the data, with some modifications to DEA’s methodology, could be used to produce estimates about the geographic source of all wholesale heroin seized in the United States and forwarded to DEA for testing. To make these estimates, the DEA data must be based on a probability sample. The HSP data, however, are not based on a probability sample because not all exhibits have a known chance of being selected for testing. Consequently, there is no way to tell how the HSP sample relates to the universe of all heroin seized in the United States and forwarded to DEA for testing. Our analysis revealed an additional problem. Even if the HSP data were based on a probability sample, DEA’s current methodology for reporting testing results does not include procedures to adjust for the probability of exhibits being selected for the test sample. Thus, DEA’s current methodology for reporting HSP testing results would not produce valid estimates even if a probability sample were used.
With these limitations in mind, opportunities exist for making improvements that would allow DEA to make valid estimates about the geographic source of all seized wholesale heroin that is sent to DEA for testing. These improvements could include modification of sampling procedures and record keeping to ensure that the HSP data are based on a probability sample and revision of its methodology for reporting testing results to include procedures to adjust for the probability of exhibits being selected for the test sample. (See app. I of this report for detailed information.)
section_title: DMP Limitations and Opportunities for Improvement
paragraphs: According to DEA, it does not intend that the DMP data be used to either produce estimates about retail heroin markets outside the 23 metropolitan areas covered by the DMP or about the geographic source of all retail level purchases within the 23 metropolitan areas. Our analysis found that the DMP data, with some modifications, could produce estimates about the geographic source of retail level purchases within the 23 metropolitan areas covered by the DMP, and the DMP estimates could possibly be combined across the 23 metropolitan areas. These estimates cannot be made now because of limitations in the DMP sampling and estimation procedures.
The DMP data are limited for two reasons. First, our analysis showed that the purchases made by DEA agents were not made in accordance with the DEA guidelines that indicate that a certain number of purchases should be made each quarter and that the purchases should be made throughout the 3 months of each quarter. DEA officials told us that they perform periodic reviews to determine compliance with the guidelines. However, our analysis found that the required number of purchases was not always made and that they tended to occur in certain periods of each quarter and on certain days of the week. Second, the DMP contains no information on the size of the market in each of the metropolitan areas. For example, City A could have 10 out of 10,000 purchases tested by the DMP, but in City B there may be relatively few heroin users and the quarterly DMP sample could be 10 out of 1,000 purchases. The size of the markets is not known. As a result, DMP data reflect only DMP purchases.
With these limitations in mind, there are opportunities for making improvements that would allow DEA to produce estimates about the geographic source of heroin purchased in the 23 metropolitan areas and to combine them across the metropolitan areas. These improvements could include taking action to ensure that DEA agents follow DEA guidelines when making the DMP purchases and utilizing alternative data sources for the total number of retail heroin purchases in an area, such as the number of hospital emergency room admissions related to heroin. (See app. II of this report for detailed information.)
section_title: Conclusions
paragraphs: The HSP and DMP data are used for important purposes by federal law enforcement. For instance, DEA uses the data as an indicator of the geographic source of heroin found in the United States, to measure the success of law enforcement initiatives, and to corroborate trends in heroin trafficking over time. ONDCP uses the data for drug-flow modeling, in testimony before the Congress, and in its annual National Drug Control Strategy. We recognize the challenges and difficulties of the HSP and DMP programs. However, current HSP and DMP data could be providing misleading information about the geographic source of heroin found in the United States because of sampling and statistical analysis problems.
Our analysis showed that problems with sampling and statistical analysis in the HSP might lead to misleading information about the geographic source of heroin in the wholesale market. HSP data are derived from a sample of seized heroin. However, our analysis showed that DEA did not obtain a large proportion of the heroin seized by Customs, the seized heroin that was analyzed was not obtained from a random probability sample, and that the reporting methodology did not include procedures to adjust for the probability of exhibits being selected for the sample. If a snapshot of wholesale heroin geographic source is based on HSP data, then this snapshot may be misleading because accurate information about seized heroin cannot be developed from the flawed sample.
Our analysis also showed that problems with sampling and statistical analysis in the DMP might lead to misleading information about the geographic source of heroin in the retail market also. DMP data are to be collected from random undercover purchases made in select metropolitan areas. However, our analysis of DMP data showed that these undercover purchases were not spread randomly over the year, as provided by DEA guidelines, but instead were concentrated in certain time periods of the quarter. Therefore, if the retail market characteristics vary over time or vary between midweek and weekend, the data could produce results that would be different from those that might have been obtained had these guidelines been followed. Furthermore, without knowledge of the size of the retail markets in the sampled metropolitan areas there are difficulties in combining DMP results across those metropolitan areas.
The quality/validity of data derived from these two programs could be improved by more careful sampling, adhering to existing DEA guidelines, and enhancing data analysis.
section_title: Recommendations
paragraphs: To help improve the HSP and DMP data on the geographic source of heroin, we recommend that the attorney general direct the administrator of DEA to   ensure that the HSP data are based on a probability sample so that all HSP exhibits have a known chance of selection, revise the HSP methodology for reporting testing results to include procedures to adjust for the probability of exhibits being selected for the test sample, take action to ensure that DMP purchases are made according to DEA guidelines, and study the utilization of alternative data sources for an estimate of the total number of retail heroin purchases in an area that could allow the DMP data to be combined across metropolitan areas.
To enhance the usefulness of HSP data, we recommend that the attorney general and the secretary of the treasury direct the administrator of DEA and commissioner of Customs, respectively, to enter into discussions to determine whether additional seized heroin should be forwarded to DEA by Customs.
section_title: Agency Comments
paragraphs: We requested comments on a draft of this report from DEA, Customs, and ONDCP.  In its comments, DEA indicated that it strongly disagreed with two of our recommendations and concurred with the remaining three. (See app. III.)
DEA disagreed with our recommendations that it (1) ensure the HSP data are based on a probability sample and (2) revise the HSP methodology for reporting testing results.  DEA said that the HSP should remain a program whose data are based solely on the results of signature analysis.  This seems to imply that the application of statistical analysis to signature testing results would yield unscientific data.  We disagree.  Without the use of data based on a probability sample, it is impossible to know how to interpret the HSP data.  In addition, DEA seems to imply that we want the signature of untested exhibits to be imputed based on the results of actual analyses of tested exhibits.  This is incorrect.  We recommended the use of a probability sample and standard weighting procedures that would allow the estimation of the geographic source of all seized heroin.  DEA also said that it should implement a stratified sample similar to the one we proposed to ensure that a significant portion of the total weight of heroin seized by DEA is sampled.  However, DEA did not want to use the stratified sample to produce estimates with the resulting data.  We believe that by implementing a stratified probability sample, DEA could produce estimates of the seized heroin, which would improve the overall data.
Furthermore, DEA suggested expanding the number of DEA and Customs exhibits submitted for analysis.  However, this method would increase reliability only if a probability sample were used.  Expanding the number of exhibits submitted without using a probability sample limits the interpretation of the data to the tested exhibits alone.  Finally, DEA said that the estimation model proposed in our report was simplistic. We provided this straightforward model only as an example and strongly endorse any attempt DEA might make to enhance the suggested model. DEA concurred with the remainder of our recommendations, that it take action to ensure that DMP purchases are made according to DMP guidelines; study the utilization of alternative data sources for an estimate of the total number of retail heroin purchases in an area; and that the attorney general and the secretary of the treasury direct DEA and Customs to enter into discussions to determine whether additional seized heroin should be forwarded to DEA by Customs.
DEA concurred with our recommendation that it study the utilization of alternative data sources for the DMP.  DEA also commented that data sources do not exist that measure either the number of retail heroin purchases or the prevalence of heroin abuse in a metropolitan area.  While we recognize that these data sources may not exist, we suggested that the DAWN data could provide a useful surrogate measure; without such a measure, the DMP data could be misleading.
DEA concurred with our recommendation that DEA and Customs discuss whether additional seized heroin should be forwarded to DEA. However, DEA incorrectly characterized our recommendation by saying we recommended they work to ensure that more seizures are sampled.  We did not make this recommendation.  We recommended only that DEA and Customs discuss whether additional heroin should be forwarded.  DEA officials also provided additional technical comments, which we have incorporated where appropriate.
Customs and ONDCP said that they had no comments on the draft report.
As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date.  At that time, we will send copies of this report to the co- chairman, Senate Caucus on International Narcotics Control; the attorney general; the secretary of the treasury; the administrator of DEA; the commissioner of Customs; and the director, Office of National Drug Control Policy. We will also make copies available to others upon request. This report will also be available on GAO’s home page at http://www.gao.gov.
If you or your staff have any questions about this report, please call me or Weldon McPhail at (202) 512-8777. Other key contributors to this report were Doris Page, Mark Ramage, Anthony Patterson, David Alexander, and Geoffrey Hamilton.
section_title: Appendix I: Heroin Signature Program
paragraphs: The Heroin Signature Program (HSP) sample was selected using sampling procedures for two categories of eligible exhibits. The first category included exhibits from seizures made by the Drug Enforcement Administration (DEA) and the Washington D.C., Metropolitan Police Department. This sample is referred to as the “DEA exhibit sample.” The second category was comprised of all other exhibits, including those seizures submitted to DEA by the Federal Bureau of Investigation (FBI), and those submitted to DEA by the U.S. Customs Service from passengers, luggage, or cargo on flights that originated outside of the United States. This sample is referred to as the “POE exhibit sample.” (See table 1 of this report for additional details on the source and selection of HSP heroin.)
subsections: section_title: HSP Methodology
paragraphs: DEA calculated the results of the HSP signature analyses in the following manner:   First, the total weight of heroin from sampled exhibits from each geographic source was computed.
Second, the total weight of heroin from sampled exhibits from all geographic sources was combined.
Last, the total weight of heroin from each geographic source was divided by the total weight of heroin for all areas. The result was expressed as a percentage.
We analyzed calendar year 1999 and 2000 data in our review of the HSP. Because 1999 was the most recent year for which finalized reports were available, some analyses are based solely on the 1999 HSP data. According to DEA, there were no methodological changes from 1999 to 2000.
HSP analyses results for calendar year 1999 are displayed in table 4.
section_title: HSP Limitations
paragraphs: Our analysis of the HSP methodology found that the HSP data, with modifications, could produce valid estimates about the geographic source of all wholesale heroin seized in the United States and forwarded to DEA for testing. These estimates could not currently be made because not all exhibits had a known chance of being selected for testing; therefore, the HSP data were not based on a probability sample. In addition, our analysis revealed that, even if the data were based on a probability sample, DEA’s current methodology that is used to report testing results would not produce valid estimates, because it does not include procedures to adjust for the probability of exhibits being selected for the test sample.
subsections: section_title: HSP Data Not Based on a Probability Sample
paragraphs: Our analysis showed that, for the following three reasons, not all exhibits had a known positive chance of being selected for testing. 1.  The procedure for selecting the POE exhibit sample resulted in the selection of heavier exhibits. DEA laboratory personnel were responsible for identifying POE sample exhibits for testing and, according to DEA, selected exhibits generally tended to be the heaviest exhibit from a case. For example, if a POE seizure resulted in 3 exhibits weighing 10 gram, 15 grams, and 50 grams, it was likely that the 50-gram exhibit was tested. While not conclusive, the DEA exhibit sample also contained heavier than expected exhibits, suggesting that the heavier exhibits may generally have been selected for testing. Although only about 13 percent of the DEA exhibits were sampled in 1999, these sampled exhibits accounted for about 45 percent of the total weight of seized wholesale heroin sent to DEA for testing.2.  Exhibits did not have a known chance of selection because it appears that DEA did not consistently follow its policy for selecting exhibits, for either the POE or the DEA exhibit samples. The policy requires that, except for special request exhibits, no more than one exhibit per case be chosen. However, based on our review of 1999 and 2000 HSP data, this policy was apparently not implemented consistently. There were multiple exhibits for individual cases in the HSP sample for both 1999 and 2000. In 1999, about 9.5 percent and 8.4 percent of the DEA and the POE sample exhibits, respectively, were from cases that had multiple exhibits in the sample. For 2000, 10.5 percent and 3.4 percent of the DEA and the POE sample exhibits, respectively, were from cases that had multiple exhibits in the sample. 3.  Exhibits for the DEA exhibit sample also did not have a known chance of selection because DEA’s sampling procedures are apparently not working as intended. Before the start of each 6-month sampling period, DEA produced a list of random numbers for the DEA exhibit sample. A separate list of random numbers was sent to each DEA regional laboratory. The random numbers were to contain approximately 20 percent of the exhibits expected in the sampling period. However, our analysis showed that the actual percentage of exhibits sampled was 13 percent for 1999 and 8 percent for 2000.
section_title: DEA’s Methodology Used to Report HSP Testing Results Did Not Include Procedures to Adjust for the Probability of Exhibits Being Selected for the Test Sample
paragraphs: If DEA developed a probability sample, it would still be unable to produce valid estimates because its current methodology for reporting HSP testing results does not include procedures to adjust for the probability of exhibits being selected for the test sample. In computing estimates using HSP data, DEA did not take into account that some exhibits had a low probability of selection and some had a high probability of selection. As a result, DEA produced only simple tabulations of sample data that could not be used to produce estimates about the geographic source of all seized wholesale heroin sent to DEA for testing.
section_title: HSP Opportunities for Improvements
paragraphs: Opportunities exist for making improvements that would allow DEA to estimate the geographic source of all seized wholesale heroin that is sent to DEA for testing. These improvements could include (1) modification of sampling procedures and record keeping to ensure that the HSP data are based on a probability sample and (2) revision of its methodology for reporting testing results to include procedures to adjust for the probability of exhibits being selected for the test sample. If these improvements were made, it would also be possible to produce confidence intervals for estimates.
subsections: section_title: Modification of Sampling Procedures to Ensure That HSP Data Are Based on a Probability Sample
paragraphs: HSP sampling procedures could be improved to ensure that all exhibits have a known chance of selection. For example, DEA could use a stratified sample in which exhibits are divided into different strata, or categories, and sample selections are made from each stratum. The heaviest exhibits and the exhibits DEA considers to have special intelligence importance could all be selected, and a portion of the remaining exhibits could also be chosen. Table 5 shows this alternative sample design.
This suggested design would result in a known chance of selection for all heroin exhibits. All exhibits in strata 1 and 2 would be selected. Exhibits in strata 3 and 4 would have a less than 100 percent chance of selection. For example, if k equals 5, then 5 percent of exhibits in stratum 3 would be selected. If h equals 20, then 20 percent of exhibits in stratum 4 would be selected.
DEA should also draw the sample and centrally maintain sufficient records in such a way that the resulting sample selection could be verified. For example, if exhibits are selected manually, with the use of a list of random numbers, the exhibits should first be numbered sequentially, before the random number list is used to identify sample exhibits;   after identifying the sample exhibits, DEA should ensure that sufficient information is maintained to identify each exhibit, indicate whether the exhibit was selected for the sample, and whether it was sent for signature analysis testing; and if an exhibit, that is ineligible for testing due to its low weight, were selected by the sampling procedure, it should be recorded as “insufficient for signature analysis.”
section_title: Revision of Current Methodology to Include Procedures to Adjust for the Probability of Exhibits Being Selected for the Test Sample
paragraphs: Several changes to DEA’s methodology are possible that would include procedures to adjust for the probability of exhibits being selected for the test sample. For example, DEA could use statistical weights for the sample design to produce estimates about the geographic source of all seized wholesale heroin sent to DEA for testing. These statistical weights would be determined by calculating a value that is the inverse of an exhibit’s chance of selection. For example, referring to the example following table 5, all exhibits in stratum 1 would be selected and 5 percent of exhibits in stratum 3 would be selected. Stratum 1 sample exhibits would get a weighting factor of 1/1=1 and stratum 3 sample exhibits would get a weighting factor of 1/.05 = 20.
DEA could also use supplemental data to improve the estimates. DEA could use data on the total weight of seized heroin sent to DEA for testing, by regional laboratory. This same quantity could be estimated from the HSP sample. The ratio of these two quantities could then be computed and used as a refinement to the statistical weights described above. For example, if one DEA regional laboratory received a total of 210 kg of heroin, and the sample estimate of heroin for that regional laboratory was 200 kg, then the final statistical weight for sample data from that regional laboratory would be the initial statistical weight multiplied by 210/200=1.05. Using the weighting factor for stratum 3 (from above), the final statistical weight for stratum 3 sample exhibits selected from that laboratory would be 20 x 1.05=21.
Estimates of the percentage of seized heroin by geographic source area would then be computed as follows:   Compute the total weighted sum of heroin in grams for each of six categories. The weighted sum would use the final statistical weighting factors described earlier. This would produce six sums, H1, …, H6;   Compute the total over all six categories, H = H1 + … + H6;   The ratio of each category to the total (converted to percentages) would then yield estimates of the percentages of seized wholesale heroin, sent to DEA for testing, by geographic region. For example, the percentage from geographic region i would be pi = 100(Hi / H).
section_title: Appendix II: Domestic Monitor Program
paragraphs: DEA produced tables using the Domestic Monitor Program (DMP) data. For example, one table displayed, by metropolitan area, the number of exhibits from each geographic source. Table 6 displays this information for calendar year 1999.
subsections: section_title: DMP Limitations
paragraphs: Our analysis of the DMP methodology found that the DMP data, with modifications, could (1) produce estimates about the geographic source of retail heroin purchases in the 23 metropolitan areas covered by the DMP and (2) possibly be combined across the 23 areas. These estimates could not currently be made because the DMP data have the following limitations. First, the purchases were not made by DEA agents in accordance with the DEA guidelines that provide that a certain number of purchases be made each quarter and that the purchases be made throughout each quarter. Second, DMP data analysis did not take into account the size of the heroin market in each of the metropolitan areas.
subsections: section_title: DMP Purchases Are Not Made in Accordance with DEA Guidelines
paragraphs: DEA agents did not make the DMP purchases in accordance with the DEA guidelines that provide that a certain number of purchases be made each quarter and that the purchases be made throughout each quarter.
A review of the 1999 DMP data showed that the DEA guidelines, that provide that 10 purchases be made in each quarter (20 in New York City) and that the DMP purchases be spread over the quarter, were not consistently met. For example, in the first quarter there were less than 5 purchases made in 3 of the metropolitan areas. Our analysis also showed that a disproportionate number of DMP purchases occurred in a certain month of a quarter, and on certain days of the week. If the purchases were spread randomly throughout each quarter, approximately a third of the purchases would be expected each month, and approximately one seventh of the purchases would be expected each day of the week.  However, based on our analysis of the DMP data, the purchase dates were not random by day of the week or by month of quarter.
As a result, it is unlikely that the DMP data for 1999 were derived from a sample in which the days in each quarter selected for purchases were chosen with a known, equal probability of selection. In addition, assuming that the characteristics of retail heroin might change over time, biases may have been introduced due to oversampling in certain time periods within the quarter. For example, if the geographic source of heroin supplied to one location changed between the beginning and the end of a quarter, making most of the purchases during one part of the quarter would not reflect the geographic source of the heroin over the entire quarter.
section_title: DEA Does Not Take into Consideration the Size of the Heroin Markets
paragraphs: The total number of retail sales in each metropolitan area in each quarter is not known. As a result, it is difficult to appropriately combine DMP data across metropolitan areas. For example, in one metropolitan area (City A) there may be 1,000 retail sales in a quarter, but in another area (City B) there may be 5,000 retail sales in a quarter. So, for City A, DMP would sample and analyze 10 purchases representing 1,000 sales, but for City B, the 10 purchases would represent 5,000 sales. Without knowing how many purchases are represented by the DMP purchases, the data should not be combined across metropolitan areas.
section_title: DMP Opportunities for Improvements
paragraphs: Opportunities exist for making improvements that would allow DEA to estimate the geographic source of heroin purchases in the 23 metropolitan areas and possibly to combine them across the areas. These improvements could include ensuring that DEA agents follow guidelines when making the purchases and using alternative data sources for the total number of retail heroin purchases in an area.
subsections: section_title: Ensure that DEA Agents Follow Guidelines When Making DMP Purchases
paragraphs: DEA could take actions to ensure compliance with its own guidelines. For example, a list of randomly chosen dates on which purchases are to be made each quarter could be sent to each metropolitan area. Additionally, actual purchase dates in each metropolitan area could be more closely monitored, to ensure that purchases are made according to scheduled dates.
section_title: DEA Should Study the Use of Alternative Data Sources for the Size of the Heroin Markets
paragraphs: To calculate the chance of selecting any one purchase, the total number of retail heroin sales per quarter for each metropolitan area is needed. It is unlikely that this number would ever be known. However, DEA could study using alternative data sources as a substitute for the total number of retail heroin sales. For example, the Drug Abuse Warning Network (DAWN) collects information on hospital emergency room admissions that are drug abuse related. If the number of heroin purchases in a metropolitan area is proportional to emergency room mentions for heroin- related admissions, then DAWN could provide information on the relative number of heroin purchases by metropolitan area.
section_title: Appendix III: Comments from the Drug Enforcement Administration