![](media/image1.png){width="1.5083333333333333in" height="0.7in"}

## Food and Drug Administration

###### Center for Devices and Radiological Health

**Report on the Evaluation of Manufacturers'**

**Activities to Assess the Year 2000**

**Compliance Status of Their Medical Devices**

### 

### November 9, 1999

Prepared for the FDA by Battelle Memorial Institute

Under Contract \# 223-97-5511/TO16

FDA Project Officer:

E. Stewart Crumpler

Office of Compliance

Center for Devices and Radiological Health

**Evaluation of Manufacturers' Activities to Assess the**

**Year 2000 Compliance Status of Their Medical Devices**

Healthcare facilities need accurate, comprehensive and reliable
information on the Year 2000 compliance status of their medical devices
in order to know which devices can continue to be used safely, and which
need to be upgraded or taken out of service. For the past two years, the
Food and Drug Administration (FDA) has been working with the device
industry and the healthcare community to develop such information and
make it available.

###### What Has Been Learned?

The vast majority of medical devices are not affected by the Year 2000
problem because they do not use or depend on computers or embedded
microprocessors. Moreover, most devices that are computerized are not
dependent on any date-related function. Even for those devices that are
vulnerable to a Year 2000 problem, most do not depend on date-related
information for any of their critical functions. Record-keeping or date
display functions that are adversely affected by a date problem present
little chance of any significant risk to patients. However, there are a
few devices for which a Year 2000 problem could pose a risk, and these
deserve special attention.

###### Availability of Year 2000 Compliance Information

To help healthcare facilities screen their medical devices, it is
valuable to have reliable information in a single, searchable location.
Comprehensive information on the Year 2000 compliance status of
computerized medical devices has been made available through the efforts
of medical device manufacturers and the establishment of the Federal
Year 2000 Biomedical Equipment Clearinghouse. In addition, FDA has
published a list of computer-controlled, potentially high-risk devices
(PHRDs) that are critical to patient care and whose unexpected failure
or malfunction could present significant risks to patients. FDA has
suggested that healthcare facilities pay particular attention to the
Year 2000 compliance status of these types of devices.

###### Is Information in the Clearinghouse and on Manufacturers' Websites Reliable?

To provide additional confidence in the Clearinghouse, and an
independent verification of the reliability of the information provided
by manufacturers, the FDA contracted with Battelle Memorial Institute
for a study of manufacturers of PHRDs to assess their efforts to address
the Year 2000 issue. A randomly selected sample of 80 manufacturers were
visited on-site to examine their Year 2000 procedures and records. The
results of that study are described in the contractor's attached report.
The study examined the extent of activities undertaken by manufacturers
of PHRDs to assure that the Year 2000 status of these types of medical
devices was known and appropriately addressed. The contractor assessed
the processes and procedures used by the manufacturers to determine
whether their computerized medical devices had any Year 2000 compliance
problems. The contractor also assessed the procedures used to design,
develop and test upgrades or modifications to correct any Year 2000
non-compliant devices, and the actions taken by the manufacturers to
communicate relevant information to users of their products.

**What Do We Now Know About Computer-Controlled, Potentially High-Risk
Devices?**

Based on the results of the assessments, as described in the attached
report, FDA has a high level of confidence that manufacturers have taken
appropriate steps, in conformance with FDA regulations, to carefully
evaluate their computerized medical devices for vulnerabilities to Year
2000 problems. In addition, when devices were identified with Year
2000-related problems, manufacturers have taken the appropriate steps,
in conformance with FDA regulations, to advise users of potential
problems, to develop and validate upgrades or to describe devices as
obsolete and warn against their continued use.

The FDA has reviewed the results of each assessment and those areas of
moderate or high levels of concern identified and described by the
contractor. In FDA's judgment, the contractor assessments have revealed
no issues or conditions at the firms assessed that raise significant
concerns regarding the adequacy of manufacturers' programs to assure the
safe functioning of their medical devices. FDA has reviewed the areas of
concern identified during the very thorough and detailed review by the
examiners and the agency has concluded that none of these issues raise
fundamental questions about the safety of the medical devices. On the
basis of this statistical sample, one can conclude with a high level of
confidence that the remaining firms in the population of manufacturers
of computer-controlled, potentially high-risk devices have taken similar
and equally appropriate actions.

The results of this study provide the FDA and the public with confidence
that information in the Federal Year 2000 Biomedical Equipment
Clearinghouse is reliable. The vast majority of manufacturers of
computer-controlled medical devices have assessed their products for
Year 2000-related problems and, when upgrades were required, have used
appropriate processes and procedures to design and test these upgrades
in conformance with FDA regulations.

Attachment: PHRD Study Final Report

![](media/image2.png)

Food and Drug Administration

CDRH Year 2000 Medical Device Assessment

[Final Report]{.underline}

[Summary of Medical Device Assessments]{.underline}

Battelle/Unisys/LGS

Table of Contents

*Executive Summary [i](#executive-summary)*

1 Introduction [1](#introduction)

1.1 Overview [1](#overview)

1.2 Background [1](#background)

2 Preparation for Assessments [3](#preparation-for-assessments)

2.1 Assessment Plan Development [3](#assessment-plan-development)

2.2 Examiners' Training [4](#examiners-training)

2.3 Random Selection Methodology [4](#random-selection-methodology)

2.4 Identification of Participants [6](#identification-of-participants)

2.5 Preparation for Examinations [8](#preparation-for-examinations)

2.6 Problems Encountered [8](#problems-encountered)

3 Execution and Reporting of the Assessments
[10](#execution-and-reporting-of-the-assessments)

3.1 Execution of the Assessment [10](#execution-of-the-assessment)

3.2 Reporting of Assessment Results
[11](#reporting-of-assessment-results)

4 Summary of Findings [13](#summary-of-findings)

4.1 Executive Leadership and Control
[13](#executive-leadership-and-control)

4.1.1 Management Chain [13](#management-chain)

4.1.2 Planning [14](#planning)

4.1.3 Design Control and Configuration Management
[15](#design-control-and-configuration-management)

4.2 Risk Management [16](#risk-management)

4.3 Corrective and Preventative Actions
[18](#corrective-and-preventative-actions)

4.4 Test Planning and Procedures [20](#test-planning-and-procedures)

4.4.1 Vendor Related [20](#vendor-related)

4.4.2 Test Planning [21](#test-planning)

4.4.3 Execution of the Testing [23](#execution-of-the-testing)

4.4.4 Independent Verification and Validation
[24](#independent-verification-and-validation)

4.5 Communications with Consignee [24](#communications-with-consignee)

4.6 Implementation and Contingency Planning
[26](#implementation-and-contingency-planning)

4.7 Obsolete PHRD Devices [28](#obsolete-phrd-devices)

4.8 Federal Year 2000 Biomedical Equipment Clearinghouse
[29](#federal-year-2000-biomedical-equipment-clearinghouse)

4.9 Conclusions [29](#conclusions)

Attachments [30](#attachments)

Attachment 1 - Results of Telephone Calls
[31](#results-of-telephone-calls)

Attachment 2 - Summary of Assessment Area Results
[32](#summary-of-assessment-area-results)

# Executive Summary

The Food and Drug Administration (FDA) has chosen to address concerns
about the adequacy of the Medical Device Industry's manufacturers'
actions taken to avoid serious Year 2000 problems by independently
validating manufacturers' Year 2000 self-assessments. The FDA has
identified a list of 90 device types that could pose a risk to patients
if a date-related failure affects the function or operation of the
device, and has designated these as computer-controlled potentially
high-risk devices (PHRDs)

A study designated as a "Special Year 2000 Data Gathering Request" was
initiated by the FDA to examine the Year 2000 programs at a random
sample of manufacturers of PHRDs. The goals of the study were as
follows:

-   To provide a high level of assurance that manufacturers have
    properly assessed the Year 2000 status of their computer--controlled
    medical devices

-   To verify that the manufacturers have developed and properly
    validated appropriate upgrades to correct any Year 2000 problems for
    these devices

-   To confirm the information provided by manufacturers for the Federal
    Year 2000 Biomedical Equipment Clearinghouse by examining the
    supporting documentation of the manufacturers.

From the population of 803 PHRD manufacturers that market their devices
in the United States a group of companies was chosen based on random
selection. A screening process to ensure that computer-controlled PHRDs
were part of the companies' product lines followed that selection.
Participation was solicited on a voluntary basis.

Battelle contacted 325 manufacturing companies about the study. Eighty
agreed to participate. Of the remainder, 197 (60.6 percent) did not
manufacture a PHRD at that location; 29 (8.9 percent) declined to
participate in the study; and 5 (1.5 percent) were non-responsive
(callers were unable to contact them by telephone).

During the 80 on-site visits, the Examiners assessed six critical areas.
Those assessment areas were Executive Leadership and Control, Risk
Management, Corrective and Preventative Action, Test Planning and
Procedures, Communication with Consignee, and Implementation and
Contingency Planning. Only one company received a rating at the high
level of concern in any of the six assessment areas. The overwhelming
majority of ratings received were at the Low Concern level (91.3 percent
of all ratings given). Medium Concerns were given for 8.3 percent of all
ratings. High Concern ratings constituted only 0.4 percent of all
ratings for the entire study. The medium concerns were spread across all
the assessment areas. The high concerns were received by the same
company in the areas of (1) Executive Leadership and Control and (2)
Test Planning and Procedures.

Based on the results of the assessments of the 80 medical device
manufacturers' Year 2000 programs, Battelle finds that the selected
manufacturers have quality systems that are adequate to deal with the
Year 2000 issue for their devices. Those firms have systematically
applied quality processes and procedures to determine the Year 2000
compliance of their products. Moreover, where Year 2000 compliance
problems existed, the companies have taken the necessary corrective
actions to remedy the problems. Since these results were obtained from a
representative, statistical sample of manufacturers, Battelle believes
that further assessments would produce similar results and the
additional results obtained would not offer any substantial benefit to
the FDA.

# 1 Introduction

## 1.1 Overview

The US Food and Drug Administration (FDA) has identified a list of 90
device types that could pose a risk to patients if a date-related
failure affects the function or operation of the device, and has
designated these as computer-controlled potentially high-risk devices
(PHRDs). The FDA initiated a study designated as a "Special Year 2000
Data Gathering Request" for the examination of the Year 2000 programs at
a random sample of manufacturers of PHRDs. Battelle contracted with the
FDA to perform all tasks necessary to carry out assessments at 80
foreign and domestic manufacturers. This report presents the results of
that study.

## 1.2 Background

The Food and Drug Administration (FDA) chose to address concerns about
the adequacy of the Medical Device Industry's manufacturers' actions
taken to avoid serious Year 2000 problems by independently validating
manufacturers' Year 2000 self-assessments.

The general approach was as follows:

-   Battelle, and its subcontractors, Unisys Corporation and LGS Group,
    Inc., with the assistance of the FDA, identified the objectives for
    the assessments of participant manufacturers' Year 2000 programs.

-   An Examiner's Guideline was developed to ensure that the same
    questions were asked of all manufacturers. Six areas of concern were
    identified in the guideline.

-   Battelle selected a random sample from the population of
    manufacturers who have distributed computer-controlled, potentially
    high-risk devices and obtained agreement of the companies to
    participate in the study. Participation was solicited on a voluntary
    basis.

```{=html}
<!-- -->
```
-   LGS Group, Inc. provided experienced software quality engineers (or
    persons with similar qualifications) to examine the design records
    of each participating manufacturer, examine the adequacy of the
    manufacturers' procedures for Year 2000 assessments and, if
    applicable, Year 2000 corrective actions. The examinations also
    provided reasonable assurance that a Year 2000 assessment and, if
    applicable, remediation procedures had been consistently applied to
    all currently produced or previously manufactured PHRDs. This was
    done through an examination of a random sample of PHRDs. The
    examination was not intended to be exhaustive, but was intended to
    cover a representative sample of the manufacturers' PHRDs,
    sustaining confidence in the general accuracy of the manufacturers'
    claims in the Federal Year 2000 Biomedical Equipment Clearinghouse.

-   A report was prepared on each manufacturer and a level of concern
    (Low, Medium, or High) was assigned to each of the six assessment
    areas. These reports were submitted to the FDA.

These assessments were designated a Special Year 2000 Data Gathering
Request under the Year 2000 Information and Readiness Disclosure Act.
Under this law, the FDA will not use the results of this study for any
civil action, and the FDA will not publicly release specific study
results without the prior consent of the subject companies. Each
Examiner, and any other contractor or subcontractor personnel that
handled confidential trade secret information, signed non-disclosure
agreements that are on file at the FDA.

# 2 Preparation for Assessments

## 2.1 Assessment Plan Development 

The first task in the project was to develop an overall project plan,
including all tasks to be completed, the schedule for completion of the
tasks, and the staffing, resources, and training plans for the project.
The project plan had to identify all government-provided information,
materials, and personnel needed for successful completion of the
project. The project plan also had to include or reference a specific
assessment plan for how Y2K assessments would be conducted, and detailed
assessment procedures for Examiners in conducting their Y2K assessments
of potentially high risk medical device manufacturers (referenced
henceforth as the Examiner's Guideline).

The project plan, assessment plan, Examiner's Guideline, assessment
checklists, draft assessment report format, training plan and any
subsequent changes to these plans and procedures had to be reviewed and
approved by the FDA project officer prior to implementation. Given the
very short deadline for completion of this project, very close
coordination between the contractor's project manager and the FDA
project officer was required.

To meet the tight deadlines imposed by the project, immediately after
the award of the contract, the Battelle team conducted a two day
"Groupware" workshop with the staff of the three contractors on the task
along with the FDA project officer and other FDA personnel. The purpose
of the workshop was to create the methodology and tools needed for the
assessments. The objectives for the team to reach at the end of the two
days were as follows:

-   Define team roles and responsibilities.

-   Create the assessment processes.

-   Create the assessment tools.

-   Define the assessment products (the site assessments and final
    reports).

-   Determine the assessment plan.

-   Develop the training curriculum and plan.

"Groupware" was chosen for its ability to speed up the process of
obtaining consensus on managing each part of the project. Each
participant in the meeting had use of a personal computer to input and
respond simultaneously to structured activities designed to meet the
objectives. "Groupware" also allows each participant an equal amount of
time to discuss or provide input on the issues. The "Groupware" workshop
addressed all the deliverables: project plan, training plan and proposed
training site, assessment plan, specific assessment procedures,
assessment checklists, and draft assessment report format.

At the end of the two-day workshop, all participants had agreed to the
following:

a firm assessment plan, a well-developed list of questions for the
Examiners in draft form, decisions on report content and format,
Examiners' training course content, and team roles and responsibilities.

## 2.2 Examiners' Training

The Examiners participated in a training program conducted jointly by
the Battelle team and the FDA. This training was intended to ensure
consistency in assessment performance, data collection and reporting.
The FDA portion of the training was videotaped during the first training
session so that all Examiners would receive the same instructions. The
Battelle team provided the instructors and curriculum on Y2K technical
issues and on assessment and reporting procedures.

Training was held over a two-week period in three separate locations.
Initial train the trainer sessions were held at Battelle facilities in
Arlington, Virginia. The second training session was held at LGS Group,
Inc. facilities in Ottowa, Ontario, Canada, from July 26 -- 28, 1999.
The third training session was held at LGS Group, Inc. facilites in
Vancouver, British Columbia, Canada, from July 28 -- 30, 1999.

The FDA project officer attended the initial training session and was
available to answer questions from participants. FDA personnel covered
the following topics and assisted in conducting Examiner training
regarding:

-   Types of devices to be covered

-   Types of Y2K problems already seen or anticipated by FDA

-   FDA procedures, rules, special concerns

-   FDA quality system requirements

-   Types of manufacturer records available

-   Manufacturer sensitivities

-   Manufacturer internal assessment issues

-   Non-availability / access to independent validation & verification
    results

-   Gaining voluntary access to the company

-   Ethics, integrity and confidentiality issues

-   Non-disclosure agreement

-   FDA contacts if Examiner has problems

```{=html}
<!-- -->
```
-   Frequently asked questions

```{=html}
<!-- -->
```
-   FDA guidance documents and other references

```{=html}
<!-- -->
```
-   Any clearances needed from foreign governments prior to foreign
    assessments

## 2.3 Random Selection Methodology

The FDA requested that Battelle assess the Year 2000 programs at 80
manufacturing facilities. A population of 803 facilities was provided to
Battelle in a database for use in selecting the manufacturers to assess.
It should be noted that many manufacturers have multiple facilities.
Therefore, it was anticipated that in a random sampling, these
manufacturers could have more than one facility identified as a
candidate for an assessment. The results were as anticipated. Therefore,
though 80 visits were conducted, only 68 unique companies were involved
in the assessments.

Preliminary estimates of possible responses to a request to participate
were as follows:

-   10 percent of the manufacturers would not actually manufacture a
    computer-controlled PHRD. These manufacturers would be considered to
    "fall out" of the study because their devices presented no danger to
    the public due to Year 2000 problems.

-   10 percent of the manufacturers would decline to participate for
    various reasons.

-   80 percent of the manufacturers would agree to participate.

However, an analysis of the project indicated the possibility that a
single, random sample of manufacturers might fail to provide an adequate
number of manufacturers for the study and additional samples might be
required to obtain the necessary data. Therefore, the methodology used
to generate the sample from the population had to support sequential
extraction of multiple samples.

Two methods for extracting samples from a population were considered.
One method involved ordering the population, then selecting entries at
random. Samples thus generated would have to be statistically tested
against the population as a whole to ensure they were properly
representative. Generating additional samples would require that the
entire process be repeated. Further, the process by which random entries
were selected would have to be checked against previous iterations to
ensure that additional samples were random relative to one another as
well as to the population as a whole.

A second sampling method involved randomizing the entire population,
then selecting records at regular intervals as a sample. As samples are
taken, the remaining population retains its random order, which ensures
that all samples are random relative to one another. This method
requires careful attention to and proper testing of the randomizing
process, but once this is accomplished extraction of multiple samples
becomes trivial.

The second sample generation method was selected. Randomizing of the
FDA-supplied database was accomplished by appending a random number to
each record, then sorting against that number. The random nature of the
sequence was then checked both by subjective review and by statistical
testing of the sequence of numbers used.

Creation of the random numbers was accomplished using the RANUNI random
number generator (uniform distribution) by SAS Institute, Inc.
Experience with this software indicates that it is a reliable source for
sequences of statistically random numbers. A simple program loop
produced 1,000 random numbers within the range zero to one with seven
decimal places. Each number was then multiplied by 1,000,000 to adjust
the magnitude. Finally, 0.5 was added prior to conversion to integer
value to eliminate any remaining fractional parts without uniform
truncation (which might skew the results).

The list of numbers was tested for statistical validity and both the
standard deviation and the distribution appeared random. The list was
then tested again by dividing it in half and comparing the standard
deviations of the two parts. This test disclosed no significant
variation in statistical parameters between the first 500 numbers and
the second or between the parts and the whole.

The 803 PHRD facilities identified by the FDA were stored in a Microsoft
Access database. At the request of the FDA project officer, 102 of the
facilities were removed from the sample (their records were deleted from
the database) because the FDA had already planned visits to those
facilities. Random numbers were then appended to the remaining 701
records and the database was sorted according to the random number
assigned to each record. This effectively randomized the database.
Multiple samples could now be extracted by selecting records at regular
intervals. Sampled records were then marked as "used."

The first sample was extracted by the selection of every 6^th^ record.
This sample was provided to Battelle's Statistician for review. His
analysis showed a uniform distribution with no statistical anomalies,
confirming its random character. Additionally, FDA personnel with
knowledge of the manufacturers involved reviewed the list and
subjectively judged it to be random.

Ultimately nine samples totaling 325 manufacturing facilities were
extracted from the database. The number of facilities that "fall out,"
that is, that did not actually manufacture a "computer-controlled" PHRD
and thus posed no danger to the public, was much higher than estimated
(60.6 percent). This necessitated a much larger sample than initially
estimated in order to secure the 80 visits required for the study. The
actual responses to the request to participate in the assessment are
summarized in a table in Attachment 1. Manufacturers were categorized as
"Fall Out," "Visit Scheduled," "NO's," "No Response," or "Duplicate."
"No Response" was used to refer to manufacturers that could not be
contacted. "Duplicate" was used to refer to manufacturers with multiple
locations that had agreed to assessments at two or more locations, and
which had additional locations identified in the latter samples. The FDA
decided that additional assessments of previously assessed companies
would not contribute to the value of the project.

## 2.4 Identification of Participants

The assessment plan required first contact telephone calls by
contractors from the Battelle team to determine if companies would agree
to a voluntary site assessment. Using a sample of 103 firms drawn
according to the random selection methodology, the callers began working
the week of July 19. Site visits were to begin during the week of August
2, 1999, and to end during the last week of September 1999. All 103
companies on the first sample received a letter from the FDA
Commissioner explaining the nature of this unique study and asking for
their voluntary participation.

The first task of the caller was to research the companies to become
familiar with the FDA-identified products and to verify the address,
telephone number, and point of contact. The caller researched the FDA
Clearinghouse database for a website address and to verify the address
and telephone number of the companies. Most companies' websites provided
sufficient information for the caller to judge the kind of device and
the reported status of the Year 2000 compliance of the cited device on
the selected sample.

Callers used a prepared script to identify themselves, to explain the
purpose of the call to this particular company, and to ask a standard
set of questions in order to determine the companies' suitability for a
site visit. When the caller talked to the appropriate point of contact,
the main points to determine were if the companies had received the FDA
Commissioner's letter and if the product(s) identified by the FDA met
the definition of computer controlled potentially high risk medical
devices. If the companies had not received the letter or needed the
official definition of a PHRD, the caller had to fax the information
before the discussion could continue. Once the companies had the
required information, the caller asked for permission to make a site
visit to conduct an assessment.

Every company on the samples received calls, many of them receiving
multiple calls. All of the results of the calls were documented.
Companies were considered "outstanding" (meaning not finished with this
process) until a letter of invitation was received or for some reason
the firm was removed from the study. The calling document was used to
provide weekly updates and status to the FDA project officer with the
number of fallout firms, number of visits scheduled, visits completed,
firms saying "no," firms still undecided saying "maybe," and those firms
that were not reachable or that provided "no response."

When a company agreed to a visit, the caller asked for a letter of
invitation to be sent to the Battelle team. Usually, the company asked
to review the Examiner's Guidelines, which were sent upon request. When
the invitation letter was received by Battelle, the company was added to
a calendar of scheduled visits and it was noted that the letter of
invitation had been received. The information about the firm was
provided to LGS Group, Inc. for use in making the pre-examination call.

When the first sample of 103 firms yielded a 64.1 percent fallout rate,
and only 28 of the 80 required visits had been scheduled, another random
sample was pulled. Since none of the firms in this second sample had
been sent the FDA Commissioner's letter, the callers had to adjust the
calling procedure to explain the study, fax the FDA letter, and wait for
responses. In all, nine samples were pulled for a total of 325 companies
that were to be called. The final calls were made the week of September
20, 1999. Attachment 1 summarizes the results of the calls to all 325
companies.

For various reasons firms were determined to "fall out" of the realm of
possible site visits. Many of these reasons were not anticipated,
resulting in a reevaluation of the calling process and the use of the
random selection methodology to identify potential candidates.
Fortunately, the process was adaptable to these unforeseen
contingencies. Of the 325 firms called, 197, or 60.6 percent did not
meet the requirements for the site visit. Either the product listed on
the sample was [not]{.underline} computerized, did not meet the FDA
definition of a potentially high-risk medical device, or had been
transferred or sold to another manufacturer. All of these reasons were
acceptable reasons for not scheduling a visit.

Callers encountered difficulties in getting the firms onto the schedule.
The main problems include the following:

-   In many cases either the appropriate decision-maker had not been
    identified or was not available to make decisions. This resulted in
    callers being required to make many attempts to reach the
    appropriate personnel.

-   The period of performance for the study was an issue. Many European
    companies schedule vacations in August. Thus many key staff members
    were not available during the period of time allocated for this
    study.

-   Few Asian companies could not be contacted during normal Eastern
    Time Zone business hours. LGS staff in Vancouver, British Columbia
    attempted to reach the companies during their business hours.

-   Many foreign companies do not have voice mail, and a few did not
    have personnel who spoke English.

-   Companies were reluctant to commit time and resources for various
    reasons and questioned if the study was actually "voluntary."

Only 29 (8.9 percent) of the companies called actually declined to
participate in the study. The reasons given fall into distinct
categories that are summarized in the following table.

+--------------------------------------------------------+-------------+
| ###### Reason for Declining                            | Number of   |
|                                                        | Firms       |
+--------------------------------------------------------+-------------+
| Audited recently by FDA                                | 3           |
+--------------------------------------------------------+-------------+
| Not available during period of study                   | 3           |
+--------------------------------------------------------+-------------+
| Device manufactured in another location                | 2           |
+--------------------------------------------------------+-------------+
| Device not affected by the date function               | 1           |
+--------------------------------------------------------+-------------+
| Information already given to FDA                       | 1           |
+--------------------------------------------------------+-------------+
| No reason                                              | 1           |
+--------------------------------------------------------+-------------+
| No resources or too time-consuming                     | 11          |
+--------------------------------------------------------+-------------+
| Reorganization in progress                             | 2           |
+--------------------------------------------------------+-------------+
| Timing wrong - can only accommodate a visit after the  | 5           |
| end of the study                                       |             |
+--------------------------------------------------------+-------------+
| Total                                                  | 29          |
+--------------------------------------------------------+-------------+

Only 5 or 1.5 percent of companies were categorized as providing "no
response." Companies were categorized in this manner if they could not
be reached by telephone or facsimile transmission (numbers provided were
incorrect or out of service), if the telephone was not answered, or if
no response was received to messages left on an answering machine or to
facsimile transmissions.

## 2.5 Preparation for Examinations

Once a manufacturer voluntarily agreed to the examination, a pre-visit
contact was made with the manufacturer to confirm the types and models
of PHRDs that were actually manufactured at a specific site. The
manufacturer was notified (by documented telephone call) of the intent
to perform an examination, given an explanation of the approach and the
name of the Examiner who would perform the examination. A request was
made that the manufacturer have available the personnel responsible for
the Year 2000 project at the site and that adequate meeting and
communications facilities be provided for the Examiner. The Examiner's
Guideline that had been provided to the manufacturer was discussed to
ensure an understanding of the assessment process. Any logistical
problems that existed were discussed and resolved in the telephone call.

## 2.6 Problems Encountered

During the course of this study, Battelle experienced difficulty in
achieving the target of scheduling 80 assessments of manufacturing
facilities of PHRDs. Preliminary estimates from the FDA indicated that
80 percent of the companies would agree to participate, 10 percent of
the devices classified as PHRDs would not actually be PHRDs and would
fall out of the sample, and 10 percent of the companies would decline to
participate in this voluntary study. Actually 24.6 percent agreed to
participate, 60.6 percent fell out and 8.9 percent of the companies
declined to 39 participate. The remaining companies in the random sample
either could not be contacted or were subsidiaries of large corporations
that had already had three or more manufacturing facilities assessed. In
this latter situation a decision was made by the FDA to exclude
duplicates from the study since there would be no benefit gained from
additional assessments of large companies with standardized procedures.
A significant increase in the number of samples was necessary in order
to obtain the required level of participation. The time required for
performing research on each company, identifying and contacting the
companies' Year 2000 points of contact, determining if the companies
really manufactured computer-controlled PHRDs, and then scheduling
visits exceeded preliminary estimates. This pushed some site visits into
late September and early October, and the project was not completed in
September 1999, as originally planned.

# 3 Execution and Reporting of the Assessments

## 3.1 Execution of the Assessment

The assessments were conducted using an Examiner's Guideline developed
by Battelle, Unisys, LGS and the FDA. The Office of Management and
Budget (OMB) reviewed and approved this guideline prior to its use by
the Examiners and assigned a control number to the document. All
participant companies were given a copy of the guideline prior to the
start of the site visit.

The review was conducted in five steps:

-   Incoming Briefing: introduction and development of mutual
    understanding with the company representatives; description of the
    overall terms of reference, scope, assessment process, constraints,
    schedule, etc., of the examination;

-   Facilities Tour: development of appreciation of size and scope of
    the operation, the physical process including all phases of design,
    manufacturing and control;

-   Interviews and Information/Documentation Gathering: scheduled and/or
    extemporaneous meetings with key managers to glean the necessary
    Year 2000 compliance process information and documentation;

-   Exit Briefing: summary of information, facts and preliminary
    findings (without conclusions or recommendations) conducted with key
    company representatives. Objective is to assure agreement with
    company representatives that the information, facts and preliminary
    findings are accurately and completely portrayed;

```{=html}
<!-- -->
```
-   Analysis and Report Writing: analysis of the information obtained,
    development of the findings and conclusions and writing the final
    site report.

    The examinations were structured around six Assessment Areas that
    were identified during the Assessment Plan Development phase as
    important to the success of Year 2000 projects. The individual
    assessment reports discussed strengths and concerns in each area and
    summarized the findings by applying a high, medium or low concern
    rating to each assessment area. The concern ratings were based on
    the identification of the most severe of the following observed
    characteristics:

> **High** Late, no plan, or inadequate plan
>
> Inadequate or incomplete resources
>
> Incomplete or inaccurate deliverables
>
> Unable to validate results - due diligence inadequate
>
> **Medium** Somewhat late, or incomplete plan
>
> Insufficient or incomplete resources
>
> Deficiencies in deliverables
>
> Incomplete validation of results
>
> **Low** On schedule
>
> Adequate resources
>
> Completed

Through interviews of knowledgeable staff and review of available
documentation, the Examiners evaluated the adequacy of the
manufacturers' Year 2000 self-assessment process for PHRDs. To support
the concern ratings given, the Examiner's Guideline required
consideration of the following:

-   Evidence of a systematic approach to assessments of all devices
    including PHRDs

-   Manufacturers' logic process for Year 2000 assessments

-   The nature of Year 2000 functional problems in devices

-   Risk analysis documentation, both before and after any Year 2000
    correction

-   Design change documents

-   Changed parts

-   Validation / verification / testing protocols

-   Comprehensiveness of testing to determine the exact nature of the
    Year 2000 problem

-   Year 2000 test reports (not to include test results unless offered
    by the manufacturer)

-   Any Independent Validation and Verifications (IV&V) reports

```{=html}
<!-- -->
```
-   Confirmation that those IV&V procedures were followed and actions
    taken in response to IV&V findings

```{=html}
<!-- -->
```
-   Contingency planning for any problems that have not or cannot be
    corrected

-   Communications to customers (consignees) regarding the up-to-date
    status of Year 2000 readiness of manufactured PHRDs

If a company declared a device to be obsolete and did not provide a Year
2000 correction, the Examiners collected information regarding that
decision, including:

-   Specific details of the nature of a Year 2000 failure (if known)

-   Information regarding the date functionality of the device

-   The basis of the decision to declare the device obsolete

-   How that decision was communicated to customers (consignees)

-   Current status of removing all obsolete devices from use (if known)

-   Contingency planning if users refuse to discontinue use of the
    device

    Using the information gathered from the personnel interviews, review
    of participant companies' related documentation and project reports,
    and the comparison with the guidelines, the Examiners were able to
    assess the manufacturers' Year 2000 programs and report on those
    findings.

## 3.2 Reporting of Assessment Results

Direct feedback was provided to each manufacturer regarding findings of
the examination of their facility. The Examiners conducted a debriefing
with each company's management at the conclusion of the examination. A
draft report was submitted to the Battelle team for a quality review and
consistency check. Any discrepancies were analyzed and FDA direction was
requested for any issues that could not be resolved by the review team.
Subsequent to completion of each draft report an additional technical
review was conducted. Use of these independent reviews and a standard
reporting template helped to ensure that all Examiners were basing their
judgments on the same criteria. All site visit reports were forwarded to
the companies involved and to the FDA. Any responses received from the
companies were reviewed by the Examiners and forwarded to the FDA along
with the Examiner's agreement or disagreement to the responses. The
results of the individual assessments have been incorporated in this
summary report.

# 4 Summary of Findings

The following sections address the six assessment areas. A summary of
the assessment of all areas for all sites is provided as Attachment 2.

## 4.1 Executive Leadership and Control

Ratings

  --------------------------------------------- ------------- ------- --------- ---------
  **Assessment Area**                           **Concern**                     

                                                **HIGH**              **MED**   **LOW**

  **Executive Leadership and Control:**         **1**         **8**             **71**
  --------------------------------------------- ------------- ------- --------- ---------

Seventy-one of the eighty companies assessed (88.8 percent) received a
Low Concern rating for this assessment area. Of the 71 companies with
low concern ratings, 50 had no specific concerns identified for any of
the three rating criteria. The remaining 21 companies had minor concerns
identified, primarily due to informality in the record keeping processes
and failure to have formal definitions of Year 2000 compliance. In all
cases these minor concerns were alleviated by the overall familiarity
that the companies had with the compliance status of their products.

One company did receive a High Concern rating. That company did not
follow the processes specified in the various guidelines of the FDA, did
not have a structure with which to effectively manage their Year 2000
project, did not have a Year 2000 compliance definition, did not have a
repository for the documentation necessary to meet due diligence
requirements, did not have a formalized strategy or plan to assess or
define the scope of remediation for its device, and did not have any
provisions in place in the event of system failure.

The remaining eight companies received Medium Concern ratings. The main
reasons cited for this rating was the absence of formal project
organizations and funding, informal recording keeping, and poorly
defined processes for Year 2000 remediations.

### 4.1.1 Management Chain

**Strengths**

The Examiners found that 74 of the 80 companies (92.5 percent) had
established Year 2000 projects, generally with separate, clearly defined
budgets. Activities were typically undertaken through a matrix of shared
responsibilities between line and program managers, or through
cross-disciplinary project teams. There was regular contact between the
participants and reports were submitted to corporate executive
management. In many cases job performance ratings took into account
performance against Year 2000 objectives.

**Concerns**

High Concern -- None observed.

Medium Concern

The primary concern for three of the eight companies that received a
medium rating centered on the fact that they did not have a Year 2000
project in existence with separate, clearly defined budgets, resources,
and plans. However, in all three cases, the companies were very small
and upper management was very aware of all activities concerning Year
2000 compliance. One larger company had a very loose management
structure, resulting in somewhat unclear lines of responsibility for
remediation and implementation. One of the reasons for this structure
involved the recent integration of an acquired company into the
organization. In addition, two of the eight companies were very informal
in the manner in which reporting occurred. One provided reports on an
irregular schedule to the President of the company and the other had no
written records of results and no indication that the President was made
aware of the results. Still, despite the absence of records, the
Examiners were reassured by the companies' Year 2000 program
coordinators that in both cases the companies' upper management was very
aware of all Year 2000 activities and results. Finally, one company
lacked strong Senior Management oversight of its Year 2000 efforts, with
no evidence of management quality audits and infrequent status reports.
Also, the processes used to make key Year 2000 project decisions have
not been documented and existing documentation was not centrally filed.
This situation was mitigated by strong efforts by departmental managers
and the recent appointment of a strong Year 2000 Program Manager.

Low Concern

The Examiners identified minor issues in the management area for nine of
the companies reviewed. One of the nine did not have a separately
established Year 2000 project, instead using a matrix approach to
management, linking line management to a parallel Year 2000 Program with
executive accountability for performance management objectives. Other
low risk concerns pertained to informal documentation of authority and
reports on Year 2000 activities and results. Only one company was
identified as being unaware of potential issues with real-time operating
systems, compilers, and embedded systems. When advised of the issues,
management of that company promised to immediately initiate an
investigation of the issues and stated that remedial actions would be
taken as needed.

### Planning

**Strengths**

Generally, the Examiners found that planning activities were well
organized; detailed procedures for achieving compliance were in place;
documentation of compliance activities was available; and design reviews
and any design changes required were tracked, validated, and
communicated to users.

**Concerns**

High Concern

The one firm with a High Concern rating did not have a clear Year 2000
compliance definition, no schedules or budget information, and no
provisions for defining activities that would identify operations or
processes potentially affected by Year 2000 problems. Their plans did
not make provisions to raise the level of awareness in the organization
such that staff would know and understand the magnitude of the Year 2000
impact.

Medium Concern

For three companies receiving a medium concern rating in this category,
separate project plans and timelines had not been established. For these
companies compliance activities were conducted as part of regular duties
and tended to be relatively informal and on an irregular schedule.
Therefore, progress could not be tracked. However, both companies were
relatively small with a fairly flat management structure that resulted
in speedy communication of issues to top management. A fourth company to
receive a medium concern rating did not record its Year 2000 compliance
definition and did not keep records on devices with no date
functionality to document that testing was not required. In addition,
this one company did not have any records to document that devices with
a date function successfully passed Year 2000 compliance tests. The
fifth company lacked a formal plan featuring a Gantt chart to plot the
project time line and manage task interrelationships. The final three
companies had two separate Year 2000 compliance statements in existence,
which could create some ambiguity as to the meaning of results presented
to individuals or organizations.

Low Concern

At nine companies, there were specific concerns about the lack of
formality in the documentation of meetings and discussion on Year 2000
and the absence of design control process flow documents, action plans,
and written results of Year 2000 testing. However, these concerns were
determined to be relatively minor based on the overall awareness and
familiarity of the respective companies' management with Year 2000
compliance requirements and activities. In addition two companies did
not have a formal definition of Year 2000 compliance though technical
staff did have an adequate level of understanding of relevant issues.

### 4.1.3 Design Control and Configuration Management

**Strengths**

Overwhelmingly, the manufacturers' Year 2000 Project Methodologies were
well-defined and included project reporting, compliance definition,
outside-vendor contact, testing, remediation/validation,
de-commissioning, qualifications of required personnel, and adherence to
due diligence standards.

**Concerns**

High Concern

The one firm that received a High Concern rating did not have a
well-defined Year 2000 Project Methodology compliance definition.
Documentation regarding application software development did not have a
clearly stated purpose, had inadequate internal control, and provided
guidance that is open to interpretation by the user. The company also
did not have a repository for the documentation necessary to meet due
diligence requirements, did not have a formalized strategy or plan to
assess or define the scope of remediation for its device, and did not
have any provisions in place in the event of system failure.

Medium Concern

Only four medium level concerns were identified in the category of
Design Control and Configuration Management. One company did not have an
overall design control process flow document. However, that company did
have established standard operating procedures, pertaining to design
control and specifically mandated that Year 2000 compliance was required
for new products and product changes. Another did not address Year 2000
compliance as a special requirement. The third company did have a
process, but the application of the process to Year 2000 remediations
was not clearly defined and documented. The last had key configuration
status accounting information split between two sites, a situation that
would complicate compliance documentation.

Low Concern

For eight companies, the Examiners identified minor concerns, including
inventory records issues, compliance statements and software quality
manuals.

## 4.2 Risk Management

The objective of the risk management assessment area was to verify that
the manufacturers' risk analysis procedures were logical and adequate.
The Examiners used the following definitions when assessing the risk
management area.

-   Risk is the combination of the probability of occurrence of harm and
    the severity of that harm (ISO/DIS 14971).

-   Risk management is the systematic application of management
    policies, procedures, and practices to the tasks of analyzing,
    evaluating, and controlling risk (ISO/DIS 14971).

Looking at both risk analysis and risk control, the Examiners focused on
hazard analysis and risk estimation as well as the intent to eliminate
the hazard and to mitigate the estimated risk to an acceptable level.
Additionally, the assessment required viewing the firm's risk management
documentation, which included the description of the hazard, the
severity of the risk, the specific software cause, the risk control
method, the test or verification method, and the severity level after
mitigation.

In the risk analysis review, Examiners had to confirm that the
manufacturers assessed the probability of the occurrence of a Year 2000
event and the level of risk to a patient. Examiners identified devices
currently in service that were not Year 2000 compliant or potentially
not Year 2000 compliant. The Examiners also assessed the process and
testing methodologies used by the manufacturers to determine vendor
compliance for third party components.

Ratings

  --------------------------------------------- ------------- ------- --------- ---------
  **Assessment Area**                           **Concern**                     

                                                **HIGH**              **MED**   **LOW**

  **Risk Management:**                          **0**         **8**             **72**
  --------------------------------------------- ------------- ------- --------- ---------

**Strengths**

Examiners stated that 69 of 80 (86.3 percent) of the companies examined
either followed standard operating procedures suggested by the FDA/CDRH
or their own internal standard operating procedures in evaluating and
documenting risk to patients. Guidance from FDA may be found in the
document \"Information Paper on FDA Activities Related to the Year 2000
Date Problem and Medical Devices,\" dated December 18, 1997, page 3, in
the paragraph entitled \"Corrective Actions by Manufacturers.\"
Examiners specifically reported on, and found acceptable mappings
between, hazard severity and the probability of occurrence.

Risk management considerations included patient care, technical,
business risks, and costs. To eliminate the key risk for medical
devices, one manufacturer actually designed devices with Year 2000
compliance in mind. Most manufacturers conducted a risk analysis for all
their computer-controlled devices to identify any date-related
functionality and then documented the potential risk exposure to
operators and patients. Triage codes to determine the needs for fixes
from essential to lowest were instituted by four companies. Some
manufacturers whose medical devices have been assessed for Year 2000
compliance have the status and compliance level posted on the company
web site.

Many manufacturers initiated software revisions to remedy non-compliant
devices. Where software revisions were not available, workarounds were
identified and communicated for products requiring user intervention.

As in other assessment areas, the discovery of no date-related events
critical to the functioning of the devices or that no remediation or
modification was necessary for the devices made the concern for risk
management very low at 21 of the companies visited (26.3 percent). Some
companies conducted a very informal Year 2000 compliance process outside
of their rigorous risk analysis due to these very reasons as well as the
fact there was no identified risk to patients.

Examiners noted that device manufacturers made contacts with third party
vendors to obtain information on Year 2000 readiness of their components
with embedded clock chips.

**Concerns**

High Concern: None observed.

Medium Concern

A medium level of concern was noted for eight companies in the sample.
Four of these companies had not conducted a separate risk assessment of
Year 2000 failures. The others had done so, but had not separately
documented the results.

Low Concern

Specific issues were identified at 15 of the 72 firms that were rated as
Low Concern. Those issues were:

1.  Remediation planning and associated risk analyses were not conducted
    or were not documented by 13 companies for devices that tested Year
    2000 compliant.

2.  The Year 2000 was not actually listed as a "hazard."

3.  Risk analyses were not conducted on devices no longer being
    manufactured.

## 4.3 Corrective and Preventative Actions

This assessment area encompasses the practices used in the company to
ensure the manufacturer has comprehensive procedures for dealing with
customer interaction regarding changes in devices and/or warnings
regarding uses with the devices. Whether the company performed Year 2000
compliance checks, determined if devices were obsolete, and communicated
this information to customers is part of this assessment area as well.
This area includes any processes for logging, addressing and tracking
customer complaints and incident reports. The company may also use
customer communications to support corrective and preventative actions
and the Examiner was looking for evidence of such communications during
the site visit. Evidence of any remediation documentation or planning
for devices, rollback procedures, and/or contingency planning based upon
risk are part of this assessment area also.

Ratings

  --------------------------------------------- ------------- ------- --------- ---------
  **Assessment Area**                           **Concern**                     

                                                **HIGH**              **MED**   **LOW**

  **Corrective and Preventative Actions:**      **0**         **1**             **79**
  --------------------------------------------- ------------- ------- --------- ---------

This was the single strongest area found in this study. All but one
company received a Low Concern rating for this assessment area.

Strengths

Examiners specifically mentioned reviewing a standard operating
procedure for handling all customer requests for corrective and
preventative action in 63 companies (78.8 percent). Responsibility for
planning and implementation usually was at the senior manager level.
Most companies have extensive manuals outlining what actions and
responses are required regarding reported problems with devices, whether
discovered internally or through customers. Corrective Action Requests
and/or Engineering Change Proposals are standard practices in 17 of the
companies visited. Some Examiners cited the fact that companies follow
the FDA Guidelines to include the Quality System Regulation.

Many companies did use specific testing protocols exclusively for Year
2000 compliance on the devices. Devices with date-related functions at
19 companies had specific remediation plans created for tracking and
change verification/validation. Year 2000 project activities were well
documented, tracked, and monitored by specific managers at eight
companies. Twelve companies decided to use existing corrective and
preventative action processes that are well established rather than
creating specific Year 2000 procedures.

Remediation and validation plans were identified, designed, validated,
and documented for products that tested Year 2000 non-compliant in 35 of
the companies visited. Examiners identified 11 companies where specified
design controls and changes to the design are checked to determine if
they comply with design controls, including change verification and
validation. Four companies do not have remediation processes because
date-related functions are only convenience features and not integral to
the operation of their devices.

Communication on device events related to Year 2000 compliance are
commonly related through a website, a central phone number, technical
support services, and/or a specific e-mail address at 22 companies.

**Concerns**

High Concern -- None observed.

Medium Concern

Only one company, where remediation and validation plans have not been
identified, designed, validated or documented and no service levels
defined, received a Medium Concern in this assessment area.

Low Concern

Overall, the Examiners cited specific issues at only twelve of the 79
companies that were rated as a Low Concern. The issues raised were
considered minor and many companies acknowledged the weaknesses and
began working to correct the problems.

Although not all companies identify remediation planning and
documentation specific to Year 2000, the documentation available was
excellent for actions taken on their devices. Two companies had not
specifically identified that remediation actions taken were actually
Year 2000 compliance actions. One of those two companies also did not
provide any Year 2000 compliance information on its website. However,
both companies received Low Concern ratings because of the way they
handled compliance within their overall quality system.

Examiners noted that one company did not complete formal change
requests, but stated that the company was working on that issue. Two
companies did not have contingency plans based upon risk defined.

## 4.4 Test Planning and Procedures

This assessment area focused on the ability of the manufacturer to
ensure the adequacy of test plans and procedures.

Ratings

  --------------------------------------------- ------------- -------- --------- ---------
  **Assessment Area**                           **Concern**                      

                                                **HIGH**               **MED**   **LOW**

  **Test Planning and Procedures:**             **1**         **11**             **68**
  --------------------------------------------- ------------- -------- --------- ---------

### 4.4.1 Vendor Related

This sub-category of the assessment area focused on the manufacturers'
procedures for assessing the components purchased from their vendors. A
concern was the consideration of all of the components of the companies'
microprocessor-based devices. Examiners sought to determine if an
assessment of components had been carried out and documented. Part of
the review involved the examination of statements of compliance
solicited and received from suppliers. Also, the Examiners determined if
a single, consistent compliance definition was used on correspondence
with suppliers. Examiners did discover that not all companies assessed
use components from outside vendors.

**Strengths**

Examiners reviewed files of vendor related materials and sampled for
quality and completeness. The sizes of companies, the number of products
marketed, and the relationships with vendors ranged from small to very
large, few to many, and simple to complex. Sixty-eight companies (85
percent) receive supplies from outside vendors and had valid,
well-established processes in place. The processes included standard
contact methods, a means to qualify information and an on-going review
process to maintain the latest updated information. Generally,
statements from all vendors of critical components were on file.
Examiners found that after written notices were sent to suppliers, many
companies followed up with telephone calls to make sure the suppliers
were aware of compliance issues.

There were some companies that did not have typical vendor relationships
and had developed distinctive processes to deal with the vendor products
they are using. Seven companies had no date criticality in their vendor
products so supplier assessment issues focused mostly on software
compliance. Three companies, to ensure the software components were
compliant, took a unique approach by testing all the software components
internally and giving them compliance ratings. Most companies adapted
their purchasing processes to specify that all vendor supplied products
must be Year 2000 compliant. Statements of compliance for vendor
products were available at 55 of the firms assessed (68.8 percent).
These compliance statements were required to provide vendor contact
information, an information source, the date of the vendor statement,
model version or revision numbers, device associations, device failure
modes, suggested remediation for identified problems, and the status
(complete/incomplete) of the vendor review process for each item.

**Concerns**

High Concern: None observed.

Medium Concern

The Examiners found at three sites (all for the same firm) that the
process to upgrade Year 2000 non-compliant computers was unclear. The
firm considered such an upgrade to be the customer's responsibility when
the computer was not under a service contract. One company did not have
a well-developed vendor compliance process with standard contact
methods, qualification of information, and an on-going review process to
maintain the most up-to-date information. Five companies did not have a
consistent compliance definition to use in correspondence with vendors.
Two companies relied heavily on verbal reports of compliance from their
vendors. One company had a questionnaire for vendors that was too brief
and general to permit the identification of specific Year 2000
compliance or possible failure modes for specific devices. One company
did not solicit specific statements of compliance, but instead obtained
information from their suppliers' websites. That same company did not
inspect purchased items for Year 2000 compliance. One company had not
documented the decision model used to designate vendors as being
critical, and there is no certainty that all vendors have been rated
consistently. This is a fairly important part of the due diligence
documentation.

Low Concern

Of the 68 firms that received Low Concern ratings, fifteen firms did not
have structured vendor test plans, special procedures for assessing the
compliance of vendor-supplied components, or had not completed their
reviews of the products supplied by critical vendors. Four companies did
not define a consistent compliance definition to use in correspondence
with vendors. In one company a very minor concern existed due to the
lack of a structured vendor test plan, although much of the critical
software was developed in-house. In two companies some original test
results were missing. Subsequent to the assessment one of these two
companies initiated additional testing to correct the loss. Special
procedures for assessing Y2K compliance of components received from
vendors, on incoming inspection, were not in place in one company, but
the Examiner determined it was not significant due to strict supplier
approval procedures.

There were other, isolated concerns identified. One company was thought
to maintain vendor certifications for critical components at corporate
headquarters, but the site visited did not produce any documentation to
support the belief. In another company, although compliance definitions
from the vendor were relied upon for Year 2000 compliance, the
compliance statements were pulled from the vendors' websites rather than
from communications with the vendors. Finally, the vendor compliance
process at one company did not include an on-going review procedure to
maintain the latest updated information on the vendors.

### 4.4.2 Test Planning

In this sub-category assessment area, Examiners focused on the steps
involved in the review and approval to a test protocol, test results,
and the documentation required by the test plan. If any data and test
scripts were used to test devices for Y2K compliance, the Examiners
related those to risk management as well.

Part of the concern in this sub-category dealt with the dates examined
for Y2K vulnerability as well as the list of products that were to be
tested for Y2K. The focus was on the criteria, process, and approval for
eliminating devices from the list. Examiners also sought documentation
of the reasons required for removing specific devices from the list.

**Strengths**

In sixty-nine of the companies (86.3 percent) testing was indicated for
the medical devices and Examiners found test plans were well developed.
Testing documentation is based upon a standard template. The Examiners
reviewed all documents in the templates. Typically, these documents
contain test procedures addressing each of the major risks that were
identified during the hazard analysis activity. The person running the
test signed each step in the test procedures, ensuring full
accountability for the test results. Examiners found generally that all
high risk devices were Year 2000 tested with the exception of those with
no date/time function, those with no recording of date/time, or those
not being manufactured, but used only for concept exploration. Examiners
specifically cited 26 companies that used test planning from existing
software testing, while definitely making sure the Year 2000 critical
dates were addressed. Results for the devices tested were confirmed.

**Concerns**

High Concern

Only one company received a High Concern. The Examiner was concerned
with the types of testing and their definitions, the fact that some of
the tests failed, and that the errors noted on the tests were not
typical for acceptance testing. This was a major concern because this
level of testing is not acceptable for a mission critical application or
a PHRD. Further concerns were raised because unit testing, integration
testing, and system testing were not performed by a qualified person.
Also the planning of test cases did not include complete test scenarios
to encompass all possible conditions. Another item of concern was that
the Design Specifications used were outdated. Finally, the Examiner
found that a thorough testing methodology was not implemented.

Medium Concern

Although test procedures existed for all non-compliant devices, explicit
documentation of one or more \"standard\" Year 2000 test planning or
procedures documents or templates was not in evidence in four sites
visited. For another three companies the test results were not verified
and reviewed by an independent third party. Two companies had no formal
test planning, only a verbal request which did not encompass all the
Year 2000 possible failures. In one company an overall Year 2000 test
plan covering all devices was not identified; procedures for Year 2000
testing did not go through the normal Test Design Process and test
scripts were recorded and tracked elsewhere.

Low Concern

Of the 68 firms that received Low Concern ratings, Examiners noted
specific issues at only three companies. At two firms, there were no
plans for an independent verification/validation check on the testing
results. At the third firm, rollover testing had not been planned.

### 4.4.3 Execution of the Testing

For this sub-category assessment area, the focus was on the actual
conduct of the Y2K evaluation of devices through selection of a specific
PHRD device as an example. Examiners were looking at the detailed
testing in accordance with approved methodology, with or without user
groups or by proxy.

Documentation of test results, a list of Y2K problems that have been
identified in the devices, and a list available of the Y2K fixes that
have been implemented are sought in this area. Examiners also looked for
documentation of testing after fixes were complete, a regression test
analysis following the fix, and any independent verification and
validation.

**Strengths**

Testing performed by qualified personnel was completed on high-risk
devices and documented very well by the subject companies. Test results
were on file along with the remediation plans. For 71 of the companies
visited (88.8 percent), the testing for Year 2000 according to a testing
plan was successfully completed. Companies whose devices were Year 2000
compliant required no remediation or subsequent testing of upgrades.
Test results varied from 100 percent pass of devices showing no Y2K
problems to various problems identified for remediation. Examiners found
only one High and four Medium Concerns in this area.

**Concerns**

High Concern

The one company to receive a High Concern did not address critical
dates. Also, only tests that failed were retested after remediation
efforts. All test cases should have been re-tested to ensure that
changes made did not affect other areas or create new problems.

Medium Concern

At one company the Examiner was concerned that the testing was not
controlled enough to take into account all Year 2000 potential failures.
In another company, some test results were incomplete or were missing
signatures. No additional Year 2000 documentation above and beyond the
regular Engineering Change Order (ECO) process documentation had been
created to document the Year 2000 testing process in the third company.
At a fourth company a fix was still required for one PHRD.

Low Concern

Of the 68 firms rated as Low Concern, Examiners identified specific
issues for only one company. This company had no single document that
captured all the results of the Y2K tests.

### 4.4.4 Independent Verification and Validation

This sub-category of the assessment area is concerned with any work
groups outside the immediately responsible work group that reviewed the
Y2K assessment plan and participated in the Y2K testing. If there was an
external Independent Verification and Validation (IV&V) completed to
assist the company's internal audit process, the Examiner was interested
in confirming whether appropriate actions were taken and completed in
response to the findings and any recommendations from the internal or
external IV&V.

**Strengths**

Where no date-related functions were found and no testing actually
required, no IV&V was performed. Therefore the number of companies
assessed in this area depended on the types of device manufactured by
the company. For 61 companies (76 percent of the total) that identified
Year 2000 problems, internal audits were conducted. In addition, many
companies practice internal quality audits in accordance with ISO 9001
procedures. The Examiners reviewed the schedule of audits to be done,
saw documentary evidence that such audits had been done in the past,
that written reports had been produced and reviewed by management, and
that appropriate actions were taken. Twenty-four manufacturers (30
percent of the firms) have, in addition to internal corporate auditors
and audit procedures, obtained external audits by a third party company.

**Concerns**

High Concern: None observed.

Medium

Five of the sites did not identify specific requirements or explicit
procedures for IV&V, did not specifically audit the Year 2000 readiness
of products or devices, and did not participate in the Year 2000 test
planning or testing per se. Three of the companies (4 percent) had no
outside verification that the products were Year 2000 ready with one
company being dependent on another company's testing. Despite the
existence of a strong Quality Assurance department, no
[independent]{.underline} verification and validation (IV&V) was
performed at one company but with their strong test planning, the
assessment merited only a medium risk.

Low

Of those firms that were rated as Low Concern, fourteen companies did
not conduct an external Independent Verification and Validation (IV&V)
of their Year 2000 project. One company permitted part of their
development group to validate their process, so there was no truly
independent validation. One company received a Low Concern rating
because of the absence of IV&V in areas of code review and actual
testing.

## 4.5 Communications with Consignee

This assessment area examined the types of programs and services that
companies have provided to facilitate communication with consignees
regarding changes, notices, updates, and software upgrades for medical
devices. Examiners reviewed the methods used by the companies to
communicate the Year 2000 status of their devices, including those that
were obsolete, to the consignees of those devices. If an obsolete,
potentially high-risk device had a Year 2000 problem that posed a risk
to a patient, the Examiners determined if there were fixes and the
methods by which consignees could obtain the fixes. Included in the
review were the communications acknowledging receipt by consignees of
device status reports.

This assessment area also considered the companies' ability to support
requests for assistance in upgrading devices and in the verification
that fixes were applied correctly. Examiners checked for documentation
of efforts made to encourage the removal of obsolete PHRD devices from
the user community. Responses from PHRD users were reviewed to determine
the adequacy of the companies' efforts.

Ratings

  --------------------------------------------- ------------- ------- --------- ---------
  **Assessment Area**                           **Concern**                     

                                                **HIGH**              **MED**   **LOW**

  **Communications with Consignee:**            **0**         **4**             **76**
  --------------------------------------------- ------------- ------- --------- ---------

Seventy-six (or 95 percent) of the companies assessed received a Low
concern evaluation in this area. Most companies have strong programs in
place to permit effective communication with their consignees. Some
minor concerns were noted, but only four of the companies visited
received a Medium concern rating. Overall, this was the second strongest
assessment area for all companies. The medical device manufacturers
appear to have addressed the issue of communicating with their customers
very seriously.

**Strengths**

The Customer Contact Process is well established in all the companies
visited. One Examiner noted that the most efficient method of
communicating status to concerned parties appeared to be through the
Internet. Most companies have established websites that list Year 2000
information for the public to access. In addition, other, more
traditional means of communication are in existence and are working
well.

Most sites have a standard procedure and process in place to deal with
customer inquiries or concerns and an on-going process to distribute the
most current updates. Some of the following standard contact methods and
procedures were found at the companies visited:

-   A Year 2000 web site maintained with regular updates of the latest
    information, one even providing private access for clients.

-   Hotline and/or call center.

-   Mass mailing of an informational letter(s).

-   Questionnaire seeking information from consignees with respect to
    instruments in use and follow-up phone calls.

-   Technical support services with 1-800 numbers for assistance.

-   Informational newsletter.

-   Letter responses to inquiries.

-   Registered mass mailings with tracking for receipt and follow-up.

-   Shipped free upgrade software to customers as follow-up to
    non-responses.

-   Incentives for non-respondent customers.

-   Printed materials describing company's response to Year 2000 issues.

-   Letters of Year 2000 compliance for products, the company's internal
    business processes, and ability to deliver products to customers.

-   Distribution of an informational CD/ROM.

-   Postings on the FDA Clearinghouse.

-   Database of all consignees for all products worldwide.

-   Major advertisement campaign

-   Presentations to users' group.

    **Concerns**

High Concern -- None observed.

Medium Concern

The companies that were assigned a Medium Concern rating needed to
develop Customer Contact Processes and to implement them rapidly. Due to
the short amount of time remaining until the Year 2000, it was felt that
these companies would experience difficulty in contacting consignees and
completing the installation of upgrades. Nevertheless, the companies
involved had limited numbers of consignees and also were relatively
small, with agile organizations that could response as required.

Low Concern

The Examiners identified specific issues at only 13 of the 76 companies
that were rated as Low Concern. Most of these issues pertained to the
amount and accuracy of information available to consignees and the
methods through which it could be obtained. A very few companies did not
have a Customer Contact Process in place, one even relying on its
customers to make contact regarding upgrades. One company uses Field
Service Engineer visits to provide customers with Year 2000 compliance
information. Another company did not have a single source for
information for its customers, instead, maintaining the information in
separate departments.

Three companies visited do not have Year 2000 compliance information for
their products on their websites and one company's website was
undergoing modification, thus their Year 2000 compliance information
could not be accessed.

## 4.6 Implementation and Contingency Planning

This assessment area focused on the manufacturers' ability to ensure
that the necessary Year 2000 planning has been or will be successfully
executed. For intended Year 2000 upgrades, Examiners were looking to the
completed development of all upgrades according to an identified
schedule. If the company has not completed upgrades, the Examiners were
to ask for the plan for addressing the issue in sufficient time and for
communicating options to its customers. When a company has upgrades, the
Examiner needed to know if the company made the upgrades available to
its customers and how that company communicated the availability of
those upgrades.

Obsolete devices were also included in this assessment area. Of specific
concern were the plans that companies have to address the problem of
users who decide to continue to use an obsolete device. A more thorough
discussion of obsolete devices may be found in section 4.7 of this
document.

Finally, companies were expected to have plans in place for unexpected
Year 2000 contingencies and emergencies regarding their devices. The
Examiners sought evidence that those plans had been tested or rehearsed.
Specifically, in conjunction with the risk analysis, manufacturers
should have a plan for reacting quickly and efficiently to unexpected
problems reported by end users. Also, since many problems arising during
the century change timeframe potentially could be inaccurately
attributed to a perceived Year 2000 bug, the Examiners asked if the
companies would have additional trained staff available to respond to
reported problems. Any such staff would be expected to be able to
correctly identify bona fide issues and quickly respond to crises by
communications, directions, and actions.

Ratings

  --------------------------------------------- ------------- --------- ---------
  **Assessment Area**                           **Concern**             

                                                **HIGH**      **MED**   **LOW**

  **Implementation and Contingency Planning:**  **0**         **8**     **72**
  --------------------------------------------- ------------- --------- ---------

Strengths

The Examiners found no reason for concern in this assessment area at 50
of 80 companies. Twenty-three other companies received a low concern
rating, eight received a medium concern rating, and none received a high
concern rating. It was found that, in general, companies have completed
upgrades and tested them according to schedule. They have also provided
remediation solutions for their date/time sensitive PHRDs. All but four
of the companies have an inventory of devices that is kept current on an
ongoing basis as items are added, moved, modified, or taken out of
service (becoming obsolete). These databases are used as the repository
for electronic documentation of information related to the Year 2000
inventory items. The information from these databases is available on
some of the companies' web sites. In many cases, companies have
developed comprehensive steps on a corporate-wide scale to address
business continuity during the transition phase.

Concerns

High Concern: None observed.

Medium Concern

The primary concern for all eight companies that received a medium
rating is the absence of a formal plan to address unexpected Year 2000
emergencies. Special plans have not been put in place to train
personnel, assign additional staff to work or otherwise prepare for
emergencies during the transition period.

Low Concern

The Examiners expressed specific concerns about eight companies that had
no contingency plans and seven that were still planning contingency
work, because of the need to confirm that operational issues are
resolved before the end of the year. The Examiners had other concerns
about the absence of any planned post-Year 2000 support at four sites;
about the absence of plans at four sites to discourage the use of
obsolete devices by customers; and about plans at five sites to use
routine customer support and escalation procedures for unforeseen Year
2000 problems. One company had potential problems with support of
products released as late as September 1999 and another had not
addressed external dependency issues.

## 4.7 Obsolete PHRD Devices

Products become obsolete as a result of technology changes, customer
requirements for additional features, and the company's inability to
support it because of the lack of available parts. Product retirement is
driven by a business decision to move to a more advanced product with
improved functionality. Retired products are usually supported for a
lengthy period of time before they are officially obsolete as they
continue to consume disposable supplies. Some obsolete products are Year
2000 compliant. Companies perform a general risk assessment on affected
models and make a determination of the risk to patients. Consignees of
record are advised of the status of the devices. Compliance status and
remediation solutions, including upgrade replacement incentives are
provided to users.

Companies with no obsolete devices

Examiners found 22 companies that have no obsolete devices in service.

**Companies with no devices made obsolete**

In 38 companies no devices have been made obsolete because of Year 2000
non-compliance.

**Companies with obsolete devices**

Examiners found 20 companies with obsolete devices in service -- some
compliant, others non-compliant, and others not affected by the date
rollover. There were three companies with obsolete devices that are Year
2000 compliant found in this study. Companies with obsolete devices
demonstrated that they have communicated any Year 2000 issues associated
with non-compliant products to their customers. Examiner cited nine
companies that have posted all of their obsolete products on their Year
2000 web sites.

## 4.8 Federal Year 2000 Biomedical Equipment Clearinghouse

The examination confirmed the accuracy of the Federal Year 2000
Biomedical Equipment Clearinghouse for 54 of the 80 sites visited (67.5
percent). An additional 16 Clearinghouse entries referred to a page at
the corporate web site which contained accurate and timely information.
Clearinghouse entries for two companies needed review but contained no
errors of substance and two others had some data missing. Three
companies had inconsistencies between the data presented on their web
pages and the data in the Clearinghouse. The Clearinghouse entries for
two companies lack data on at least one major product, and one restricts
access to its Year 2000 data to clients and internal users.

## Conclusions

Based on the results of the assessments of the 80 medical device
manufacturers' Year 2000 programs, Battelle finds that the selected
manufacturers have quality systems that are adequate to deal with the
Year 2000 issue for their devices. Those firms have systematically
applied quality processes and procedures to determine the Year 2000
compliance of their products. Moreover, where Year 2000 compliance
problems existed, the companies have taken the necessary corrective
actions to remedy the problems. Since these results were obtained from a
representative, statistical sample of manufacturers, Battelle believes
that further assessments would produce similar results and the
additional results obtained would not offer any substantial benefit to
the FDA.

# Attachments

### Attachment 1

### Results of Telephone Calls

  ---------- ----------- ------------- --------- ---------- --------- ----------- --------- ------------ --------- --------- ------- -----------
  **Sample   **TOTAL**   **FALLOUT**   **% of    **VISITS   **% of    **NO\'S**   **% of    **No         **% of    **Dup             **Check**
  Number**                             Total**   SCHED**    Total**               Total**   Response**   Total**   Sites**           

  1          103         66            64.1%     28         27.2%     8           7.8%      1            1.0%      0         0.0%    103

  2          13          6             46.2%     5          38.5%     2           15.4%     0            0.0%      0         0.0%    13

  3          43          28            65.1%     13         30.2%     1           2.3%      1            2.3%      0         0.0%    43

  4          10          5             50.0%     4          40.0%     0           0.0%      0            0.0%      1         10.0%   10

  5          44          28            63.6%     9          20.5%     6           13.6%     0            0.0%      1         2.3%    44

  6          36          26            72.2%     7          19.4%     2           5.6%      0            0.0%      1         2.8%    36

  7          16          5             31.3%     3          18.8%     5           31.3%     0            0.0%      3         18.8%   16

  8          40          24            60.0%     6          15.0%     1           2.5%      3            7.5%      6         15.0%   40

  9          20          9             45.0%     5          25.0%     4           20.0%     0            0.0%      2         10.0%   20

  TOTAL      325         197           60.6%     80         24.6%     29          8.9%      5            1.5%      14        4.3%    325
  ---------- ----------- ------------- --------- ---------- --------- ----------- --------- ------------ --------- --------- ------- -----------

### Attachment 2

### Summary of Assessment Area Results

  ----------------------------------- ------------ ----------- -----------
  **Assessment Area**                 High         Medium      Low

  Executive Leadership and Control    1            8           71

  Risk Management                     0            8           72

  Corrective and Preventative Action  0            1           79

  Test Planning and Procedures        1            11          68

  Communication with Consignee        0            4           76

  Implementation and Contingency      0            8           72
  Planning                                                     

                                                               

  **Totals**                          2            40          438
  ----------------------------------- ------------ ----------- -----------