**Economic Evaluation of Radiopharmaceutical Research at NIST**

**I. Introduction**

Nuclear medicine is the branch of medicine that relies on radiation
emitting substances, called radioisotopes or radionuclides, for the
diagnosis or treatment of a disease or condition. The field of nuclear
medical and biological research can be traced to the 1895 discovery by
Wilhelm Conrad Roentgen of a ray that could penetrate the human body and
display bone structures.[^1] Soon thereafter, in 1896, Henri Becquerel
demonstrated that uranium in its normal state gave off X-rays. Following
these two path-breaking discoveries came the radioactivity discoveries
by Marie Curie in 1897. Then, in 1925, Herrman Blumgart and Otto Yens
used radioactive substances in human patients for the first time. In
1936, John H. Lawrence and Joseph Gilbert independently undertook the
first therapeutic clinical applications of a radioactive substance to
treat leukemia.

The use of radiochemicals or radiopharmaceuticals increased slowly
during the pre-war period and then more rapidly in the decades following
World War II. Finally, in the early 1970s, radiopharmaceutical
manufacturers realized the importance of calibrations for the
short-lived radionuclides being used in nuclear medicine. According to
Golas \[1996\], in the early 1970s Standard Reference Materials (SRMs)
were available in unsuitable physical forms and activity levels. Also,
there were no standard methods for measuring the activity of a
radionuclide, even ignoring the imprecise decay information that was
available. As a result, no method existed for a pharmacy to assure the
accuracy of the radiation in a prescribed dose.

In the early 1970s, there was not an established research program on
radiopharmaceutical measurement at NIST \[then the National Bureau of
Standards (NBS)\], or at any other federal facility. In 1972 the Atomic
Industrial Forum (AIF) responded to industry needs and appointed a
subcommittee of Manufacturers of Radioactive Reference Standards, with
the objective of "... obtaining a high degree of consistency and
reliability in commercially available radioactive reference standards
and their accompanying Certificates of Calibrations" \[Seidel and
Hutchinson, 1976 (p. 1)\].[^2] The NBS was represented on this
committee, as were all major commercial manufacturers of radiochemicals.

In 1973, following the formation of this subcommittee, a symposium was
held at NBS to address the concerns of radionuclide manufacturers
regarding the proposed standards. Speakers at the symposium emphasized
the need for such standards, citing:

1.  the lack of standards for about 75 percent of the more than 100
    radionuclides then produced by industry;

2.  the unusable physical form and activity levels of the few standards
    that were available; and,

3.  the failure to have industry-wide adopted decay-scheme data upon
    which to base derived standard instrument calibrations.

It was also noted at the symposium that the lack of standards was an
obstacle to both the assurance of accuracy in the administration of
radioisotopes, and to the approval of new drugs by the federal Food and
Drug Administration (FDA).

In response to industry-wide concerns expressed at this symposium, NBS
entered into a cooperative research agreement[^3] with the Atomic
Industrial Forum (AIF) \[now the Nuclear Energy Institute (NEI)\]. "...
whereby NBS will supervise and administer on behalf of AIF a
measurements technology quality assurance program which caters more
specifically to the needs of the radiopharmaceutical industry" \[Collé,
1976 (p. 71)\].[^4] The NBS also entered into an interagency agreement
with the FDA to "ensure the continuous availability of national
radioactivity standards at appropriate levels of activity for use by the
radiopharmaceutical industry, and thus to establish a degree of
uniformity in the measurements throughout the industry" \[Collé, 1976
(p. 71)\].

Today, radiopharmaceutical Standard Reference Materials are produced
within the Radioactivity Group of the Ionizing Radiation Division of the
Physics Laboratory at NIST through a CRADA with NEI, and sold to the
members of the NEI/NIST Radioactivity Measurement Assurance Program
(MAP) and to the public. The purpose of this report is to document and
quantify selected broad-based economic impacts associated with this
program's research activities.

The report is outlined as follows. In Section II, the research
activities of the Radioactivity Group are discussed in detail. In
Section III, a conceptual framework for evaluating the economic effects
of radiopharmaceutical research at NIST is described and evaluated. In
Section IV, the findings from the economic evaluation are interpreted.
Finally, concluding remarks are offered in Section V.

**II. Radiopharmaceutical Research at NIST**

**A. Incidence of Radiation Treatment**

In 1996, over 3,900 hospital-based nuclear medicine departments in the
United States performed over 10 million nuclear procedures; about 90
percent of these procedures were diagnostic and about 10 percent were
therapeutic.[^5] The number of such procedures has increased from 7.7
million just a decade ago. Experts expect these numbers to increase
between 5 percent and 10 percent per year over the next 5 years.[^6]
Patients undergoing a nuclear medical exam receive a prescribed dosage
of a radiopharmaceutical. These radiopharmaceuticals produce radioactive
emissions specifically formulated to be collected temporarily in the
organ being examined. Gamma cameras and PET scanners are used to detect
and transfer these emissions to film or to a computer image for
study.[^7]

Data from the Health Physics Society \[1996\] show that in 1987, 26
percent of the 7,690,000 diagnostic tests conducted in the United States
were bone studies, 18 percent were heart studies, and 17 percent were
lung studies.[^8]

If a patient receives too little radiation the procedure will often have
to be redone, particularly for a therapeutic procedure. If the patient
receives too much radiation, injury or death could result. In either
case, there is an economic cost to imprecise measurement. It is
therefore incumbent upon the nuclear pharmacist to deliver as close to
the prescribed dosage as measurably possible, that is to ensure that the
patient receives *x* amount of radiation, where *x* is measured in terms
of number of atoms of the radioactive substance decaying per unit of
time.

**B. Research Mission of the Radioactivity Group**

The Ionizing Radiation Division, one of six divisions in the Physics
Laboratory at NIST, provides national leadership in:[^9]

> "... fundamental research and measurement standards for all types of
> ionizing radiation, including X-rays, gamma rays, electrons, neutrons,
> energetic heavy charged particles, and radioactivity. The Division's
> extensive domestic and international collaborations in applied
> research help to make ionizing radiation effective in industrial and
> health applications and safe in the environment and the workplace.

The Radioactivity Group is one of three research groups within the
Division. Its mission is to develop and maintain U.S. radioactivity
standards and to confirm the standards internationally; to perform
research on radioactivity decay scheme characteristics; to develop and
apply radioactivity measurement techniques; and to disseminate results
using standard reference materials, calibrations and testing services,
and cooperative industrial measurement traceability programs.

This mission complements NIST's overall responsibilities. The NBS was
established in the Treasury Department under the Organic Act of March 3,
1901:

> *Be it enacted by the Senate and House of Representatives of the
> United States of America in Congress assembled,* That the Office of
> Standard Weights and Measures shall hereafter be known as the National
> Bureau of Standards ... That the functions of the bureau shall consist
> in ... the determination of physical constants and the properties of
> materials, when such data are of great importance to scientific or
> manufacturing interests and are not to be obtained of sufficient
> accuracy elsewhere.

The NBS was transferred to the Department of Commerce and Labor in 1903
and to the Department of Commerce in 1913.[^10] All responsibilities
were transferred to NIST when NBS was re-named under the Omnibus Trade
and Competitiveness Act of 1988. As described in the United States Code,
Title 15, Chapter 7:

> \[The\] National Institute of Standards and Technology \[shall\]
> enhance the competitiveness of American industry while maintaining its
> traditional function as lead national laboratory for providing the
> measurement, calibrations, and quality assurance techniques which
> underpin United States commerce, technological progress, improved
> product reliability and manufacturing processes, and public safety ...
> \[and it shall\] prepare, certify, and sell standard reference
> materials for use in ensuring the accuracy of chemical analyses and
> measurements of physical and other properties of materials.

**C. Research Activities of the Radioactivity Group**[^11]

To date, NIST has produced the 28 radiopharmaceutical standards listed
in Table 1.[^12] Also shown in Table 1 is the primary diagnostic use of
each of the radionuclides.

A radiopharmaceutical standard produced at NIST is typically accurate to
within ±1% at the one standard deviation level; that is all standards
produced at NIST are calibrated within ±1% of the actual decay rate of
the substance. Or stated alternatively, NIST's calibration error on
these SRMs is ±1%. These calibrated SRMs are provided to the
radiopharmaceutical manufacturing companies that belong to the NEI/NIST
MAP for an approximate fee of \$600.00 per vial.[^13]^,^[^14] The
current members of the assurance program are listed in Table 2.

Not all radiopharmaceutical manufacturers are members of the MAP.
Non-member companies can obtain standards from commercial laboratories
for a similar fee; however, the standards produced and sold by
commercial laboratories are less accurate than those produced at NIST.
On average, commercial laboratories are within ±5% of the actual decay
rate of the substance, but these laboratories are traceable to
NIST.[^15]

Ten SRMs are produced at NIST each year, one each month following a
schedule agreed upon by MAP members. The months of May and November are
so-called "open months." During these two months, members of the MAP can
submit their own samples to NIST for calibration as long as it contains
a radionuclide already standardized by NIST. This allows companies that
produce radioactive substances to have calibrated samples of
radionuclides that are important to them, but that are not on NIST's
current production schedule to obtain traceability.

Table 3 shows NIST research costs related to the production of the
radionuclide SRMs since 1990. While NIST initially became active in
providing these SRMs in 1974, the cost data relevant to this study, as
discussed below, begin with 1990. These cost data include all NIST
capital, labor, and materials expenses.

**Table 1. Radiopharmaceutical Standard Reference Materials (SRMs)
Produced at NIST**

  --------------------- ------------- ---------------------------------------
  **Radionuclide**      **SRM         **Diagnostic/Therapeutic Use or
                        Number**      Relevant Organ**

  Chromium-51           4400 N        Diagnosis of blood cell survival

  Iodine-131            4401 V        Thyroid and other organs

  Tin-113-Indium-113m   4402 C        \[Replaced by Indium-111\]

  Strontium-85          4403 B        Bone studies

  Thallium-201          4404 S        Cardiac studies

  Gold-198              4405 B        Synovectomy, treatment for rheumatoid
                                      arthritis

  Phosphorus-32         4406 N        In vitro traces/biotechnology

  Iodine-125            4407 T        In vitro clinical chemistry

  Cobalt-57             4408 F        Standard for calibration of imaging
                                      instruments

  Selenium-75           4409 D        Biochemical tracer

  Technetium-99m        4410 V        Imaging of multiple organs

  Iron-59               4411 B        Hemoglobin studies

  Molybdenum-99         4412 U        \[Parent of Technetium-99m\]

  Mercury-197           4413 A        In vitro tracer studies

  Iodine-123            4414 C        Thyroid and other organs

  Xenon-133             4415 T        Lung imaging studies

  Gallium-67            4416 Q        Gastrointestinal tract imaging

  Indium-111            4417 P        Radioimmunotherapy

  Mercury-203           4418 A        In vitro tracer studies

  Ytterbium-169         4419 C        Brachytherapy

  Lead-203              4420 B        In vitro tracer studies

  Gold-195              4421 A        Tracer for new therapeutic agents

  Chlorine-36           4422 A        Beta-particle standard

  Strontium-90          4423 A        Beta-particle standard

  Sulfur-35             4424 A        Cell kinetics and metabolism;
                                      biochemical tracer

  Samarium-153          4425 B        Bone palliation agent

  Strontium-89          4426 A        Bone palliation agent

  Yttrium-90            4427 A        Radiolabelled antibody therapy
  --------------------- ------------- ---------------------------------------

Source: Institute of Medicine \[1995\] and information provided by
experts within the Radioactivity Group at NIST.

> **Table 2**

**Current Participants in the NEI/NIST Radioactivity Measurement
Assurance Program**

+-----------------------------------------------------------------------+
| Bristol-Meyers Squibb Company                                         |
|                                                                       |
| DuPont Merck Pharmaceuticals Company                                  |
|                                                                       |
| Institute of Nuclear Energy Research (Taiwan)                         |
|                                                                       |
| Mallinckrodt Medical, Incorporated                                    |
|                                                                       |
| Medi+Physics, Incorporated                                            |
|                                                                       |
| Nordion International, Incorporated                                   |
|                                                                       |
| Packard Instruments, Incorporated                                     |
|                                                                       |
| Syncor International Corporation                                      |
|                                                                       |
| U.S. Food and Drug Administration                                     |
+-----------------------------------------------------------------------+

Source: Golas \[1996\] and the Radioactivity Group at NIST.

**Table 3. NIST Costs Related to the Production of Radiopharmaceutical
SRMs**

**(in \$1,000)**

  ------------------------------------ ----------------------------------
  **Year**                             **NIST Costs**

  1990                                 \$210

  1991                                 \$218

  1992                                 \$218

  1993                                 \$265

  1994                                 \$226

  1995                                 \$384

  1996                                 \$364
  ------------------------------------ ----------------------------------

> Source: The Radioactivity Group at NIST.
>
> **III. An Economic Analysis of Radiopharmaceutical Research at NIST**

**A. The Conceptual Framework**

The approach selected for evaluating the economic impacts associated
with the research activities of the Radioactivity Group at NIST has been
adopted from previous evaluations of other laboratory research programs
at NIST and from evaluations of other public investments in
infrastructure technology.[^16] The actual NIST costs related to the
production of radiopharmaceutical SRMs (Table 3) are compared to
estimates of the economic benefits received by SRM users derived from a
hypothetical counterfactual experiment. The experiment assumes that the
first-level economic benefits associated with the Radioactivity Group's
research can be approximated in terms of the additional costs that
radiopharmaceutical manufacturers and patients would incur in the
absence of NIST's research.

The hypothetical counterfactual experiment is used because this case
study lacks comparable baseline observations. In other words, it is not
the case that some members of MAP rely on NIST for only selected SRMs.
Were that the case, then a comparison of diagnostic/ therapeutic
efficiency could in principle have been conducted between calibrated and
non-calibrated substances.[^17]

It is important to keep in mind that this study focuses on only one part
of the production and distribution chain---the radiopharmaceutical
manufacturer. The radiopharmaceutical itself requires a radionuclide
that is produced either in a nuclear reactor or a particle accelerator.
Some manufacturers operate their own reactors or accelerators, but many
do not. Therefore, the radionuclides must be purchased from other
laboratories such as those operated by the U.S. Department of Energy or
by universities. Once the radiopharmaceuticals are prepared at the
manufacturer, they are shipped to the clinic or hospital. At this point,
it is the responsibility of the radiopharmacist to dispense the bulk
shipment into individual doses for the patients. Finally, the
radiopharmaceutical is administered to the patient by the nuclear
medicine technician. At each of these steps in the distribution chain,
it is important to maintain good measurement and quality assurance
practice because any errors introduced, especially early in the
distribution chain, are quickly multiplied before the drug is given to
the patient.

**B. Survey Data and Analysis**

Previous experience with the collection of information related to the
economic benefits associated with NIST's laboratory research suggests
that the most efficient, and presumably the most accurate means to
collect data is through semi-structured, interactive telephone
interviews. Accordingly, the Radioactivity Group identified a contact
person in each of the seven manufacturing companies in the MAP listed in
Table 2. In effect, this group defined the domestic industry of
radiopharmaceutical manufacturers for this study.

The radionuclide industry is of modest size. Based on data from the
Department of Commerce's Bureau of the Census in its 1992 *Census of
Manufacturers,* SIC industry 2835 is Diagnostic Substances. In 1992, the
value of product shipments from this four-digit industry was \$6,177.1
million. The five-digit SIC industry 28352, *In Vivo* Diagnostic
Substances, had a value of product shipments of \$1,051.8 million in
1992. The value of product shipments of *In Vivo* Radioactive Reagents
(diagnostic and therapeutic), SIC 2835220, was \$323.2 million in that
year, and there were seven companies with shipments over \$100,000.

Prior to the telephone interview, each identified participant was sent
an electronic mail message from the Radioactivity Group introducing the
study, soliciting participation in the study, and assuring
confidentiality of individual responses. Each identified participant was
asked to think about two general topics prior to the telephone
interview. The first topic related to what their company would do under
the hypothetical counterfactual situation where NIST ceased to produce
SRMs. The second topic related to how the industry would adjust to such
a situation and what the consequences would be.

Each identified manufacturing individual was interviewed. The following
conclusions are based on these interviews.

1.  Absent NIST's Radioactivity Group's involvement in SRMs, it would
    take between 5 and 10 years for some industry group or association
    to form and become accepted as a de facto standard setting body. The
    total (summed over the seven companies) expected transaction costs
    during this transition period would be at least \$1.3 million per
    year. This amount represents the additional labor costs expected to
    be required on the part of manufacturers to resolve measurement
    disputes between manufacturers and customers, and the associated
    additional measurement equipment needed by manufacturers. The
    expressed expectation was that these costs would increase between 4
    percent and 10 percent per year until a steady state situation was
    reached.

2.  During this 5 to 10 year transition period, the level of accuracy at
    the manufacturing stage would decrease from the current ±3% to
    between ±5% and ±10% owing to the lack of accurate reference
    materials and measurement methods. At the hospital, the accuracy of
    dosages would fall from the current ±10% to at least ±15%.

3.  Reliance on foreign national laboratories for SRMs has never been
    considered due to the measurement quality at NIST. Thus, no
    respondent could reasonably assess the likelihood that the industry
    would turn to a foreign national laboratory for SRMs in the
    hypothetical counterfactual situation.

These findings are summarized in Table 4.

**Table 4. Summary of Findings from Interviews with the MAP
Manufacturing Members**

  ------------------------------------------- ---------------------------
  **Hypothetical Counterfactual Situation**   **Range of Responses**

  Absent NIST's radiopharmaceutical research, Between five and ten years
  how long would it take an industry group or 
  association to form and become accepted as  
  a de facto standard setting body?           

  During this five to ten year transition     Accuracy would decrease
  period, what would happen to the accuracy   from ±3% to between ±5% and
  of radiopharmaceuticals at the              ±10%
  manufacturing stage?                        

  During this five to ten year transition     Dosage accuracy would
  period, what would happen to the accuracy   decrease from ±10% to at
  of the radiopharmaceuticals administered by least ±15%
  hospitals?                                  
  ------------------------------------------- ---------------------------

The findings from these interviews with radiopharmaceutical
manufacturers are, that should NIST's current efforts regarding SRMs be
absent, an industry association would over a 5 to 10 year period evolve
to replicate NIST's current role and level of accuracy in SRMs. During
this transition period manufacturers would experience sizable
transaction costs and patients would receive less accurate dosages of
radionuclides. The economic costs to manufacturers during this
transition period are estimated to be at least \$1.3 million per year,
to increase at a rate not less than 4 percent per year; and patient
dosages of radioactive substances will fall by at least 5 percentage
points, from ±10% to ±15%.[^18]

To quantify, in dollar terms, the economic cost of decreased accuracy,
the Radioactivity Group identified three recognized experts at renowned
U.S. medical centers as candidates for telephone interviews. Each was
interviewed, and as a group they expressed the following opinions:

1.  A decrease in the accuracy of a radioactive substance from ±10% to
    ±15% would require that, on average, 1 percent of all diagnostic
    procedures be re-done due to too low a dosage (e.g., imprecise
    imaging) at an estimated average cost of between \$500 and \$750 per
    diagnostic procedure.

2.  A decrease in the accuracy of a radioactive substance from ±10% to
    ±15% would require that, on average, 3 percent of all therapeutic
    procedures be re-done due to too low a dosage (e.g., ineffective
    treatment) at an estimated average cost of between \$1,500 and
    \$2,500 per therapeutic procedure.

3.  The health implications (e.g., organ damage) of too high a dosage
    could not be estimated owing to the fact that such consequences
    would not be known immediately. However, it is probable that
    substantial long-term economic costs would be associated with this
    type of misadministration.

Table 5 summarizes these economic benefits.

**Table 5. Summary of Economic Benefits from Radioactivity Research at
NIST**

  --------------------------------------------- -------------------------
  **Category of Economic Benefit**              **Economic Benefit**

  Reduced transaction costs between             Saved \$1.3 million
  radiopharmaceutical manufacturers and their   
  customers                                     

  Increased accuracy of diagnostic procedures   Saved re-doing 1% of all
                                                procedures at an average
                                                cost of \$500 to \$750
                                                per procedure

  Increased accuracy of therapeutic procedures  Saved re-doing 3% of all
                                                procedures at an average
                                                cost of \$1,500 to
                                                \$2,500 per procedure
  --------------------------------------------- -------------------------

**C. Calculation of Economic Evaluation Metrics**

Table 6 replicates in the second column the NIST research costs from
Table 3 from 1990 through 1996. The cost datum for 1997 is the 1997
budgeted cost amount; estimated costs (rounded) for 1998 through 2001
are based on a 5.5 percent annual increase, the actual increase from
1996 to 1997. The five year forecast period is based on the opinion of
radiopharmaceutical manufacturers that, absent NIST, the industry would
incur transactions costs and decreased accuracy for a period of five to
ten years. For the sake of conservativeness, the lower bound of the
forecasted range is used---five years.

The third column in the table contains manufacturers' estimates of the
total industry cost savings from NIST continuing its SRM activity as
estimated by the transaction costs (labor and equipment costs) that
manufacturers would have had to incur in the absence of NIST's
radiopharmaceutical research. The 1997 estimate of \$1.3 million is a
direct interview aggregate. The range of opinion on the growth rate of
this amount was 4 percent to 10 percent per year. Again for the sake of
conservativeness, the lower bound of the forecasted range is used---4
percent.

The fourth column contains the estimated value of the cost savings to
patients from avoiding having to have procedures re-done because of an
underdose of radiation. With respect to diagnostic procedures, the base
estimate of 1 percent of 9,000,000 diagnostic procedures per year was
used, valued at the lower-bound estimate of \$500 per procedure. For
therapeutic procedures, the base estimate of 3 percent of 1,000,000
therapeutic procedures per year was used, valued at the lower-bound
estimate of \$1,500 per procedure. Coincidentally, the estimate of
patient costs avoided given NIST's research involvement is \$45,000,000
for each procedure, or a total annual economic benefit estimate of
\$90,000,000. Based on expert opinion that the number of such procedures
will increase between 5 percent and 10 percent per year over the next
five years, a conservative lower-bound rate of increase of 5 percent per
year is imputed to the \$90,000,000 estimate. Absent opinion on the
annual rate of increase in procedure costs, it was assumed, again for
the sake of conservativeness, that per procedure costs will remain
constant over the next five years.

Finally, the fifth column in Table 6 contains total net benefits,
defined, by year, to be the sum of manufacturer and patient benefits
less NIST costs. Net benefits are negative in value in years 1990
through 1996 owing to the fact that economic benefits are not realized
until 1997.

**Table 6. Actual and Projected NIST Costs and Economic Benefits**

**(in \$000)**

  ----------- ------------- ---------------- -------------- ---------------
  **Year**    **NIST        **Manufacturer   **Patient      **Total Net
              Costs**       Benefits**       Benefits**     Benefits**

  1990        \$210         ---              ---            \- \$210

  1991        \$218         ---              ---            \- \$218

  1992        \$218         ---              ---            -\$218

  1993        \$265         ---              ---            \- \$265

  1994        \$226         ---              ---            \- \$226

  1995        \$384         ---              ---            \- \$384

  1996        \$364         ---              ---            \- \$364

  1997        \$384         \$1,300          \$90,000       \$90,916

  1998        \$405         \$1,352          \$94,500       \$95,447

  1999        \$427         \$1,406          \$99,225       \$100,204

  2000        \$450         \$1,462          \$104,186      \$105,198

  2001        \$475         \$1,521          \$109,395      \$110,441
  ----------- ------------- ---------------- -------------- ---------------

> **IV. Interpretation of Findings**
>
> The data in Table 6 are used for the calculation of three evaluation
> metrics; an internal rate of return, an adjusted internal rate of
> return, and a benefit-to-cost ratio. Each of these is an accepted
> metric for quantifying the net economic impact associated with a
> public research program like that within the Radioactivity Group.
>
> A critical issue related to the calculation of the evaluation metrics
> is the choice of a base period and the choice of a terminal period for
> the analysis. The choice of the terminal period was discussed
> previously with regard to a five year forecast. However, some judgment
> must be exercised in determining the base period. The relevant
> question is the following: the current level of research expertise
> within the Radioactivity Group represents the culmination of previous
> efforts dating back to what year? Certainly, knowledge depreciates
> over time. Based on discussion with the Radioactivity Group, current
> economic benefits are traceable to research that began four to six
> years ago. Again, for the sake of conservativeness, a six-year period
> is used, and thus the cost data in Table 3 and Table 6 begin with
> 1990.

**A. Internal Rate of Return**

By definition, the internal rate of return (IRR) is the value of the
discount rate, *i*, that equates the present value (PV) of a net benefit
stream to zero. Mathematically, the IRR is the rate of discount, *i*,
that satisfies the equation:

\(1\) PV = \[(B~0~-C~0~)/(1+*i*)^0^\] + \... +
\[(B~n~-C~n~)/(1+*i*)^n^\] = 0

where (B~t~-C~t~) represents net benefits in year t, and n is the number
of years under consideration.

Based on the net benefit data in the fifth column of Table 6, the
calculated value of i for which PV=0 is 1.38 (rounded), implying an
internal rate of return to NIST\'s investments in radiopharmaceutical
research of 138 percent.

The internal rate of return estimate of 138 percent means that 1.38
(rounded) is the value of the discount rate that equates the present
value (1990 as the base year) of net benefits to zero, or stated
alternatively the present value of total benefits to the present value
of costs.

Economists and policy makers generally use internal rate of return
measures, for on-going or completed public-sector research projects, to
estimate what is referred to in the economics literature as an
approximation of the social rate of return. If 138 percent internal rate
of return calculated above, is greater than the hurdle rate (generally
accepted rate of return) experienced by NIST, then the
radiopharmaceutical research program is surely worthwhile from a social
perspective. This is especially evident when one considers that this
internal rate of return is a conservative estimate, as discussed in
Section V.

**B. Adjusted Internal Rate of Return**

It is not uncommon to misinterpret an internal rate of return measure as
an annual yield similar to that earned on, say, a bank deposit. One
invests, say \$100, and then earns interest on that \$100 each year plus
interest on the interest. That is not the case in an R&D project, in
general, or in the case of NIST\'s radiopharmaceutical research, in
particular.

Under an alternative set of assumptions, an adjusted internal rate of
return (AIRR) can be calculated from the data in Table 6 to provide a
rate of return that is more analogous to an annual yield (as earned on a
bank deposit for example). It is important to re-emphasize that the AIRR
is not the same as the IRR and that the AIRR is not directly comparable
to the economics literature on social rates of return. Nevertheless, the
AIRR remains an accepted evaluation metric.

If all net NIST costs are referenced to 1990 using a discount rate of
8.78 percent, the 1990 present value of NIST costs is \$2,435,720. The
discount rate of 8.78 percent is the sum of the Office of Management and
Budget's \[1992\] recommended real rate of discount of 7.0 percent plus
and inflation factor of 1.78 percent.[^19] If all economic benefits (the
sum of manufacturer benefits and patient benefits) are referenced
forward to the year 2001 using the same rate, the 2001 present value of
industry benefits is \$596,138,530. The AIRR is the rate of return that
equated a single NIST investment of \$2,435,720 in 1990 to a single
benefit estimate of \$596,138,530 in 2001.

Thus, the annual compounded rate of return that corresponds to such an
investment in 1990 culminating in 2001 equals 65 percent (rounded) based
on the value of *x* that satisfied the following relationship:

\(2\) \$2,435,720 (1+*x*)^11^ = \$596,138,530

**C. Benefit-to-Cost Ratio**

The third evaluation metric is a benefit-to-cost ratio (B/C). This ratio
is, by definition the present value of all benefits to the present value
of all costs, where the point of reference for both benefits and costs
is the base time period, 1990. The relevant discount rate for the
calculation of present value is again 8.78 percent. For the data in
Table 6, the present value of all benefits equals \$236,215,070 and the
present value of all costs equals \$2,435,720. Thus, the ratio of
benefits-to-costs is:

3.  B/C = 97-to-1

It should be noted that the calculated IRR of 138 percent from equation
(1) implies a benefit-to-cost ratio of unity. Rewriting equation (1)
using summation notation:

\(4\) PV = \[Σ~t=0\ to\ n~ B~t~/(1+*[i]{.underline}*)^t^\] -
\[Σ~t=0\ to\ n~ C~t~/(1+*i*)^t^\]

where *i* is the IRR that equated PV=0. When PV=0, then it follows that:

\(5\) \[Σ~t=0\ to\ n~ B~t~/(1+*i*)^t^\] = \[Σ~t=0\ to\ n~
C~t~/(1+*i*)^t^\]

or that the present value of benefits equals the present value of costs,
or B/C=1. Thus, as is the case here, when B/C \> 1 it implies that the
IRR is greater than the social rate of discount used in the
benefit-to-cost calculation.

**V. Conclusions**

NIST responded to the radiopharmaceutical industry's needs for standards
as outlined by the Atomic Industrial Forum in 1973. Related research has
been ongoing at NBS/NIST since 1974. This research, in the opinion of
manufacturers and in the opinion of medical experts today has been
invaluable. Quantitatively, the activities of the now Radioactivity
Group are significantly benefiting society as measured by accepted
evaluation metrics.

Three standard evaluation metrics were quantified in this study: an
internal rate of return of 138 percent, an adjusted internal rate of
return of 65 percent, and a benefit-to-cost ratio of 97-to-1. Each of
these metrics was calculated under the most conservative set of
assumptions possible. In every instance when experts expressed an
opinion in terms of a range of values, the most conservative end point
of the range was used. Also, there are certainly costs associated with
both underdose and overdose misadministrations that go beyond the direct
cost of readministering a dosage (in the case of an underdose). The
benefits associated with these cost savings also have not been
considered in this analysis. Were they to be considered, the estimated
values of all three of the metrics above would increase. Still, when
compared to the social cost of resources used to generate the economic
benefits described herein---an 8.78 percent return on resource
costs---the radiopharmaceutical research at NIST is without question
valuable to society.

**References**

Collé, R., 1976. "AIF-NBS Radioactivity Measurements Assurance Program
for the Radiophar­ma­ceutical Industry," National Bureau of Standards
Special Publication No. 456, pp. 71-76.

Golas, Daniel B., 1996. "NIST Radiopharmaceutical Standard Reference
Materials and the NEI/NIST Radiophar­m­a­ceutical Measurement Assurance
Program" (mimeographed November 1996).

Golas, Daniel B., 1993. "USCEA/NIST Measurement Assurance Programs for
the Radiophar­ma­ceutical and Nuclear Power Industries," *Proceedings of
the Workshop on Measurement Quality Assurance for Ionizing Radiation*
(March 1993).

Health Physics Society, 1996. "Centennial Calendar."

Hutchinson, J. Robin, 1996. "Collaboration with and Support of
Radiopharmaceudical Manufacturers through the U.S. Council on Energy
Awareness" (mimeographed).

Institute of Medicine, 1995. Committee on Biomedical Isotopes, *Isotopes
for Medicine and the Life Sciences*, edited by S.J. Adelstein and F.J.
Manning, Washington, DC: National Academy Press.

Link, Albert N., 1996a. "Economic Impact Assessments: Guidelines for
Conducting and Interpreting Assessment Studies," NIST Planning Report
96-1.

Link, Albert N., 1996b. *Evaluating Public Sector Research and
Development*, New York: Praeger Publishers.

Link, Albert N. and John T. Scott, 1997. "Evaluating Technology-Based
Public Institutions: Lessons from the National Institute of Standards
and Technology," in *Policy Evaluation in Innovation and Technology,*
edited by G. Papaconstantinou, Paris: OECD (in press).

Office of Management and Budget, 1992. "Memorandum for Heads of
Executive Departments and Establishments, Circular No. A-94,"
Washington, DC, October 29, 1992.

Physics Laboratory Brochure, 1996.
http://physics.nist.gov/overview/div846bro.html.

Seidel, C.W. and J.M.R. Hutchinson, 1976. "American National Standards
Institute Quality Assurance Program in Radioactivity Measurements," NBS
Special Publication No. 456, pp. 1-6.

Society of Nuclear Medicine, 1996a. *Nuclear Medicine: 100 Years in the
Making, 1896-1966,* Reston, VA: Society of Nuclear Medicine.

Society of Nuclear Medicine, 1996b. "Nuclear Medicine, X Rays, CT and
MRI,"

<http://www.snm.org/nmx-ray.html>

[^1]: This information draws directly from Society of Nuclear Medicine
    \[1996a\].

[^2]: This committee became subcommittee N42.2 of the American National
    Standards Institute (ANSI).

[^3]: Such agreements are now known as Cooperative Research and
    Development Agreements or CRADAs.

[^4]: Today, this program is known as the NEI/NIST Radioactivity
    Measurements Assurance Program.

[^5]: Health Physics Society \[1996\]. See also, Institute of Medicine
    \[1995\].

[^6]: This forecast was provided by experts within the Radioactivity
    Group at NIST. See also Institute of Medicine \[1995\].

[^7]: Society of Nuclear Medicine \[1996b\].

[^8]: These are the most recent study-specific data available.

[^9]: Physics Laboratory \[1996\].

[^10]: Between 1903 and 1934 it was called the Bureau of Standards.

[^11]: This section draws directly from Golas \[1996\].

[^12]: Some of these standards are no longer produced or are produced
    only rarely for a user to verify response rates. The letter at the
    end of the SRM number indicates how many times the substance has
    been produced (e.g., A=1, B=2, and so on).

[^13]: In general, a vial is a standard NIST glass ampoule that contains
    5 milliliters of solution.

[^14]: These revenues are divided between NIST and NEI; \$108.00 to NIST
    to underwrite, in part, research and the production of the SRM, and
    \$492.00 to NEI.

[^15]: This information was provided by experts within the Radioactivity
    Group at NIST, and it was verified during the interview stage of the
    study.

[^16]: See Link \[1996a, 1996b\] and Link and Scott \[1997\].

[^17]: Data could not be obtained from manufacturers of
    radiopharmaceuticals that are not members of the MAP.

[^18]: Certainly, manufacturers and patients received benefits from
    NIST's research during the 1990 to 1996 period. However, as shown in
    Table 6, these benefits were not quantified owing to the
    counterfactual methodology adopted for estimating avoided costs if
    NIST were to cease its related research activity.

[^19]: The Office of Management and Budget \[1992\] recommends using a
    nominal discount rate equal to a 7 percent real discount rate plus
    an inflation factor. The inflation rate of 1.78 percent is the
    Federal Reserve Board's published implicit price deflator for the
    past four quarters.