# An Innovative Approach to Functionality Testing of  Analysers in the Clinical Laboratory

Authors: Wolfgang Stockmann; Werner Engeldinger; Albert Kunst; Margaret McGovern
Journal: Journal of Automated Methods and Management in Chemistry
 (2008)

---

## Abstract

The established protocols for evaluating new analytical systems produce indispensable information with regard to quality characteristics, but in general they fail to ana¬lyse the system per¬formance under routine-like conditions. We describe a model which allows the testing of a new analytical system under conditions close to the routine in a controlled and systematic manner by using an appropriate software tool. Performing routine simulation experiments, either reflecting imprecision or method comparison characteristics, gives the user essential information on the overall system performance under real intended-use conditions.


---

## Body

## 1. Introduction

Conventionally, the evaluation of new
analytical systems is conducted on the basis of established protocols related
to analytical performance like those published by CLSI (NCCLS) [1], ECCLS [2],
or other national organizations. Sometimes additional exploratory testing is
performed in the hope of gaining some insight into the routine behaviour of the
system. While the standard protocols produce indispensable information with
regard to the quality characteristics, they fail to analyse the system performance
under routine conditions. Similarly, random testing only gives a chance
opportunity to detect system malfunctions.Obviously, there is no easy way to
experimentally test the course of events that lead to an erroneous assay result
and to verify its incorrectness under nonstandardized, that is, routine
conditions. This situation is caused by the increasingly complex interactions
of hardware, software, and chemistry which are found on modern analysis
systems. The manual generation of experiments that describe a sample sequence
with variable specimens and request patterns is feasible, but it is cumbersome
to produce and provides no information on the correctness of the measurements.
A better approach can be obtained by developing a software tool which generates
appropriate experimental request lists, allowing the testing of a new system
under conditions close to the routine in a controlled and systematic manner,
and which provides sufficient data reduction for the analysis of the results.


## 2. Methods

We have integrated this functionality in our evaluation
software tool Windows-based computer-aided evaluation (WinCAEv) [3, 4] in such a way, that the generation of
simulation experiments, the transfer of requests to the instrument, the on-line
data capture, and the result evaluation, can be easily achieved with the
available programme  functions [5]. The routine simulation (hereafter referred to as
RS) module allows for the definition and generation of typical test request patterns.A request list that reflects a routine
laboratory workload can be simulated by WinCAEv using appropriately defined
parameters. The required input data embraces typical test distributions, sample
materials, and sample request profiles. As an alternative to this programme
supported simulation, laboratory specific request lists captured electronically
from the laboratory information system or directly from the routine analysers are
automatically converted by WinCAEv to a corresponding worklist for the system
under evaluation.Three main types of RS experiments were
designed to allow systematic testing of an analytical system. In this way,
different types of routine situations can be modelled and the respective
performance situation evaluated.TheRS-Precision experiment type is used to test for systematic and/or random errors using the
imprecision characteristics of the system. The goal is to compare the analyte
recovery and precision generated during randomized processing with that produced
during batch analysis.  Pooled quality
control and pooled human materials are used as samples. A typical request list
is shown in Table 1.Table 1
Basic structure of a routine simulation precision experiment.
AssaysSample no.MaterialABCDEFGHIJBatch part1Pool Axxxxxxx2′′xxxxxxx⋯′′xxxxxxx⋯′′⋯⋯⋯⋯⋯⋯⋯⋯′′xxxxxxx11′′xxxxxxx12Pool Bxxxxx13′′xxxxx⋯′′xxxxx⋯′′⋯⋯⋯⋯⋯⋯′′xxxxx22′′xxxxx23Pool Cxxxx24′′xxxx⋯′′xxxx⋯′′⋯⋯⋯⋯⋯′′xxxx33′′xxxx⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯n⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯Random partn + 1Pool Bxxxn + 2Pool Axxxxxn + 3Pool Cxxxxn + 4Pool Bxxn + 5Pool Axxn + 6Pool Axxxn + 7Pool Bxxx⋯⋯xxxxx⋯⋯xxxxxx⋯⋯xx⋯⋯xxxxx⋯⋯xxx⋯⋯xxx⋯⋯x⋯⋯xx⋯⋯xxxx⋯⋯xx⋯⋯xxxx⋯⋯x⋯⋯xx⋯⋯xxx⋯⋯xxxxx⋯⋯xxxxn + x⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯⋯In repetitions of the same experiment, routine
provocations are introduced during the randomized processing to further
challenge the system's performance under various conditions. The type and
number of provocations depend on the system under evaluation, but generally
include items regularly encountered during operation in a routine laboratory,
like calibration and quality control measurements, reagent switchover or exchange, sample short,
STAT analysis, provocation of various data flags, sample reruns, and so forth.Errors related to instrument malfunctions or
chemistry problems can be deducted from the experimental data by comparing the
batch and random results. The mean, median, CV, relative 68%-median distance (md68%
describes a robust measure of variation) [6], and minimum and maximum of the random part are compared
with those from the batch part for every analyte measured. Random and/or systematic
errors will result in significant deviations like elevated CVs and differences
of the means. One can expect that the imprecision in a simulated routine run
will result in somewhat higher CVs due to more interactions of the analytical
system than during a standard batch run. Based on experience from various
system evaluations, we use the following expected CV in the random part: CVexp,rand = CVexp,ref + ΔCV, where we set ΔCV = 1/2(CVexp,ref); therefore CVexp,rand = 1.5 CVexp,ref;
CVexp,ref is the expected CV in
the reference part.Usually the routine simulation experiment is
performed with many different methods, and the high number of results produced has
to be assessed for relevant deviant results. This can easily be done by
comparing the CV and the relative 68%-median distance.The system handling of the routine provocations
is assessed for correctness, and the analytical results produced during and after
provocations are checked for marked deviations which may represent systematic
and/or random errors.Recently we extended this experiment in order
to run the routine simulation precision experiment via a host download
procedure (see below) so that the real routine request pattern is reflected and
a simulation by the WinCAEv software is not necessary.RS-Series 1/2 is used for the
comparison of randomized test processing in two runs. Fresh human specimens are
used as sample materials with request patterns reflecting the evaluation sites
typical routine workloads. The sequence of sample processing is identical in
both runs and the same samples are used, not placing fresh samples for the
second run.Random errors can be deducted from the
experimental data by comparing the deviation of the second from the first run
results. The results are grouped in 7 five percent categories between ±15%
deviation. Each sample pair is categorized; a summary shows the number of
samples per analyte in each category as well as a total statistic per category for
the complete experiment (see Table2 and Figure 1). Random errors will result in marked
deviations between both run results for one or more samples.Table 2
Routine simulation series 1/2, cobas 6000.
MethodNCategories– – –– ––++++++<−15%<−10%<−5%±5%>5%>10%>15%Na5252K6060Cl1111ALP_22626ALT364311AMYL_21010AST2525CK2121LDH_23434LIP871BIL-T365620311CREA_2351322GLUC_34444TP_26464UREA52502LACT_21010Ca1919Mg16151PHOS_21515CRP38371DIG913113Li1082AMYL_2 (urine)1010FT41010TSH862Pro-BNP981TNT918MYO_21019ßHCG1073TOTAL abs69770126521754TOTAL rel1%0%2%94%2%1%1%Figure 1
Routine simulation series 1/2, cobas 6000.RS-Method Comparison Download allows direct
comparison of the routine analyser methods (reference data) with those of the
instrument under evaluation processed in a randomized routine-like fashion. The
test results and sampling patterns from the routine laboratory analyser(s) are
electronically captured by WinCAEv via file import (host-download) or simply
with a batch upload. Using the host-download option, sample identification
numbers, requests, and results are exported from the laboratory host in a text
format file (e.g., comma separated values (CSV)) and then imported in WinCAEv.
No patient demographics are transmitted to WinCAEv. Method comparison
statistics and graphs are generated per analyte and comparison instrument.


## 3. Applications

Over the last decade, routine simulation
experiments have become an integral part of inhouse and multicentre system
evaluations at Roche Diagnostics. Here, we outline some typical areas of use
based on practical experience, as well as examples of errors difficult to
produce using conventional procedures yet observed using these experiments.RS-Precision is an extremely effective means of testing the interaction of software with all other system
components under stressed conditions. During a multicentre study of
Roche/Hitachi 917 in the early nineties for example, these experiments yielded
CVs of up to 4% for test applications using low-sample volume (2μL). An
example is shown in Figure 2 for cholesterol. Of the 48 runs performed during
this experiment, 83% (=40 series) were found with a CV higher than the
expected 2%; 24 series had a relative 68-median distance of more than 2%. The
difference between CV and md68% indicated that several series had clear deviant
results. Further inhouse investigations revealed that a software malfunction in
the sample pipetting process under certain conditions was the root cause for
these conspicuous results. After correction of the software and repetition of
this experiment, the CV of the cholesterol assay was in all cases below 2%.Routine simulation precision Roche/Hitachi 917, distribution of CV and md68% for
cholesterol (n = 48 series).
(a)(b)On Roche/Hitachi 912, we found that
introduction of STAT samples during operation led to intermittent incorrect data
flags on STAT sample results when a sample material other than serum was
selected. Investigation showed that if a STAT sample was requested by sample
disk position only on the analyzer, and the sample type downloaded from the laboratory
host is one other than serum or plasma, the sample was correctly measured for
the specified biological material but the data flagging was done as if the
sample was serum. In this case, an incorrect rerun of the sample was indicated
by the generated data flag.With the new generation of
Roche systems MODULARANALYTICS from
the late nineties, combining multiple
analyser modules, this experiment became indispensable and gained many new
areas of use. An Intelligent Process Manager distributes the sample carriers to
the various analyser modules in a way that ensures most efficient operation,
and background maintenance features on these systems allow the operator to
perform maintenance on one or more modules while continuing routine operation
on the other modules. The RS-Precision experiment allowed us to check these,
among other complex functions, in a systematic manner under numerous simulated
routine-like conditions. A typical provocation on such systems is the deactivation
and reactivation of a module during routine operation. The goal is to check
that the samples with requests for tests on the deactivated module(s) are
handled correctly, and that the reactivated module performs as expected after return
to operation.  During the MODULAR ANALYTICS SWA (serum work area) evaluation,
these provocations revealed sporadic errors after module reactivation like
wrong reagent inventory and bottle change-over to standby reagents although
current reagents were not empty.Also, RS-Precision is an effective tool to test
the interaction of reagents on a selective access analyser. During the recent
multicentre evaluation of cobas 6000, a new reagent carryover—not observed on other Roche analysers—caused by the reagent probe was found for the
creatinine enzymatic assay, when the creatine kinase reagent (CK) was pipetted
just before the creatinine assay. Creatine phosphate of the CK reagent (vial 2)
may partly be hydrolysed to creatine which can influence the creatine concentration
of the enzymatic creatinine assay, when contaminated by the reagent probe. As
shown in Table3, the CV changes from 0.9% in the batch part to 2.1% in the
random part. Looking to the single results, the creatinine concentration of
sample number 68 is increased and classified as an outlier (255 compared to the
median of 225 mmol/L, which is increased by more than the fivefold md68 [20 mmol/L]). By monitoring the “cuvette history” data base of the analyser, one
could confirm for that sample that the CK reagent was pipetted just before the
creatinine assay. Consequentially, an extra probe wash cycle was installed on
cobas 6000 for this reagent combination in order to avoid carryover. Table 3
shows also that the analyser works correctly after a provocation with an
expired reagent pack (example ß-HCG).Table 3
Routine simulation precision cobas 6000.
Test[Study unit]MaterialNMeanCV [%]CV diff. [%]RemarksNa[mmol/L]CSP144138.90.70.5K[mmol/L]CSP1445.00.80.5ALAT[U/L]CSP5388.61.00.3ASAT[U/L]CSP18105.80.90.1CK[U/L]CSP11308.41.10.3GGT_2[U/L]CSP12124.10.9−0.3CHOL_2[mmol/L]CSP703.60.90.3CREA_2 enz[μmol/L]CSP80225.02.11.2OutlierGLUC_3[mmol/L]CSP619.61.00.3TP_2[g/L]CSP5558.00.9−0.1UREA[mmol/L]CSP2315.10.5−0.3UA_2[μmol/L]CSP19449.31.00.2Ca[mmol/L]CSP392.91.10.6Fe_2[μmol/L]CSP7526.61.60.9CRP[mg/L]HSP7014.51.20.3IGG_2[g/L]CSP116.81.70.8IGA_2[g/L]CSP111.61.40.2IGM_2[g/L]CSP120.73.31.0RF_II[kU/L]HSP1019.22.41.0TRSF_2[μmol/L]CSP1224.41.30.2DIG[mmol/L]CSP104.62.21.0Li[mmol/L]CSP101.61.50.4CREA_2 enz[μmol/L]HUP197474.32.51.7TP_U-CSF_3[mg/L]HUP2099.42.71.4FT4[pmol/L]CSP2616.91.61.0TSH[mU/L]CSP311.21.60.6Pro-BNP[pmol/L]CSP10277.91.71.0TNT[μg/L]HSP100.32.00.9CKMBm[μg/L]CSP1029.21.60.8PSA[μg/L]CSP1020.91.30.9FPSA[μg/L]CSP105.11.30.4ßHCG[U/L]CSP0Flag“FL”**Flag “FL” for expired reagent provocation, flag excludes values from statisitical calculation.Conventionally used to test for reproducibility
of results in two runs with the goal to check for random errors, the RS-Series
1/2 experiment can also be easily adopted to address numerous system specific
functionalities. For modular systems,
this experiment is used to compare consistency between modules for example [7]. On the other hand, during the cobas Integra 800
evaluation [8], it was used to check the analyser's
clot and bubble detection function. Samples with clots and bubbles were
included in the first run and prior to the second run; bubbles were removed and
those with clots were centrifuged. The results were checked for correct
flagging of problematic samples in the first run and analyte recovery compared
with the second run.Having a practically automated procedure to immediately repeat the
workload from one or more routine instruments on the system under evaluation
and to directly compare the results, theRS-Method Comparison Download is an invaluable tool. It allows the investigator
to assess the new system from an analytical performance as well as from an overall
system performance perspective, under real life conditions.A total of 187 method comparisons were processed
for 50 analytes in nine laboratories under site-specific routine conditions
during the MODULARANALYTICS evaluation [9] for
example. Analysis of approximately 27 000 measurements for fresh human
specimens gave the final proof that the system was ready for market launch.


## 4. Conclusion

Conducting such routine simulation experiments
gives the manufacturer essential information on the overall system performance
under real intended-use conditions. This innovative, realistic, and thorough
approach to system testing has won the approval of principle investigators
during international multicentre evaluations over more than ten years.With the experience gained, we strongly recommend
to focus more on routine simulation type experiments during the performance
evaluation of a new analytical system, and above all to derive the various
quality characteristics like imprecision and method comparison from these
results.



---

*Source: 183747-2008-02-17.xml*