RIA
laboria

laboratoire de recherche
en informatique
et automatique

Computer component
and system testing
as a part of computer-aided
production and maintenance

M. Depeyrot.

Institut de Recherche
d'Informatique
et d'Automatique

Domaine de Voluceau
Rocquencourt
78150 - Le Chesnay
Tél. 954.90.20

Rapport de Recherche n° 83

Octobre 1974
COMPUTER COMPONENT AND SYSTEM TESTING AS A PART OF
COMPUTER-AIDED PRODUCTION AND MAINTENANCE

M. Depeyrot

Abstract :

Three major subsystems can be identified in the material aspect of a
production plant : Inventory Control, Process Control and Quality Control. Cer-
tainly what is supposed to maintain the smooth complementarity of these subsys-
tems is Product Scheduling together with some overall cost performance, so that
any separate evaluation of one of the subsystems should make explicit the inter-
face throughput and timing as well as the cost performance being optimized.
Moreover Cost Control should be viewed as an integral part of each of the separate
subsystems which should include "cost sensors" of its own. The same applies to
scheduling.

The aim of this survey is to sketch a definitional structure for the
Testing aspects- Testers and programs - of Quality Control so as to lead to a
clear understanding of their relationship with production. and maintenance optimi-
zation through computer aids.

KEY WORDS AND PHRASES : hardware test, tester, test process, computer-aided manufacturing, maintenance, progressive development, cost analysis, visibility, hierarchical system.

Sommaire :

Trois sous-systèmes essentiels peuvent être distingués sur le plan
matériel d'une usine de production : le contrôle d'inventaire, le contrôle de
processus industriels et le contrôle de qualité. Ce qui est censé maintenir une
complémentarité ordonnée entre ces sous-systèmes est une combinaison optimale
d'ordonnancement des produits et de gestion des coûts globaux, de sorte que toute
évaluation séparée d'un critère de performance pour l'un des trois sous-systèmes
devrait prendre en compte explicitement les débits et cadences sur ses interfaces
ainsi que la formule du coût optimisé. De plus la gestion des coûts devrait être
considérée comme une partie intégrante de chacun des trois sous-systèmes, qui
devraient inclure leurs propres "capteurs de coût". Il en est de même pour l'or-
donnancement.

Le but de ce survol est de poser une infrastructure de définitions et
principes de base sur les aspects-test (testeurs et programmes) du contrôle de
qualité, afin de conduire à une bonne compréhension de leurs interactions avec
l'optimisation de la production et de la maintenance assistées par ordinateur.
Table of Contents.

<table><thead><tr><th></th><th>page</th></tr></thead><tbody><tr><td>I. Prolegomena to a theory of test and diagnostic.</td><td>1</td></tr><tr><td>1. Types of defects aimed at.</td><td>2</td></tr><tr><td>2. Burn-in, wear-out and test.</td><td>3</td></tr><tr><td>3. Test programming languages.</td><td>4</td></tr><tr><td>4. The object of a test.</td><td>5</td></tr><tr><td>5. The operating modes.</td><td>6</td></tr><tr><td>6. Diagnostic kinds.</td><td>9</td></tr><tr><td>7. Frequency of use.</td><td>10</td></tr><tr><td>8. Test Program categories.</td><td>11</td></tr><tr><td>9. Tester System classes.</td><td>14</td></tr><tr><td>10. Translation and structural aspects.</td><td>16</td></tr><tr><td>II. Cost minimization and evolutivity of test systems.</td><td>18</td></tr><tr><td>1. Financial analysis of quality and test scheduling.</td><td>18</td></tr><tr><td>2. Reduction of test usage frequency through sampling.</td><td>23</td></tr><tr><td>3. Reduction of test pattern design cost.</td><td>25</td></tr><tr><td>4. Reduction of duration of undetected faults.</td><td>28</td></tr><tr><td>5. Reduction of redundant test investments.</td><td>30</td></tr><tr><td>III. A paradigm for financial optimization.</td><td>34</td></tr><tr><td>Bibliography on testing and related problems.</td><td></td></tr><tr><td>A. Test Systems.</td><td>38</td></tr><tr><td>B. Visibility.</td><td>40</td></tr><tr><td>C. Surveys.</td><td>41</td></tr><tr><td>D. Various Test Techniques.</td><td>43</td></tr><tr><td>E. Cost and Scheduling.</td><td>45</td></tr></tbody></table>
# I. PROLEGOMENA TO A THEORY OF TEST AND DIAGNOSTIC.

Test problems can be classified and presented according to a pleiad of points of view as the following instances :

- the defect type they aim at,

- the language used to specify them,

- the type of units they are applied to,

- their operating mode,

- the kind of diagnostic they provide,

- their frequency of use,

- their thoroughness, etc.

It appears that present technical and scientific literature does not permit an exhaustive coverage of Test and Diagnostic problems, both because of their magnitude and because of the lack of enthusiasm they have incited until now in most engineering circles. As a matter of fact Test theory is in a much infantile state as compared to Control theory and this might result from the added complexity of needing to first understand the structural nature of an object before starting to study the testing of this object. As an other example, it is striking to notice how little can be found about the interferences between Inventory Control or Product Scheduling on one hand and Quality Control on the other.

A more surprising fact is that no study whatsoever can be found that would interrelate the various circumstances of testing of a single object (be it an Integrated Circuit by itself, on a board or in a computer, or be it a peripheral device in manufacturing or in the field). This study of Test and Quality interrelations lays the foundations of the so-called VISIBILITY THEORY in emergence now.

This paper mainly considers the problems encountered with Data Processing Equipment but it does so in a way that should make it relevant for about any other technology of complex systems.
I.1. Types of defects aimed at.

The distinction between _deterministic_ and _random_ failures is most frequent although not most crucial. A deterministic failure for a logic gate can be expressed as "stuck-at"- zero or at-one or at power supply. A random failure is a failure with intermittent occurrence. In fact, automatic retry facilities as well as simultaneity or coincidence defects and sequential micro-electronic properties result in having deterministic failures at the finer structural level with a random appearance at the system level in operation. Conversely environment (temperature, vibrations, supply voltage or frequency) can render deterministic the occurrence of a random failure. [D.6].

Since most electronic equipment failures amount to a soldering or bonding defect, it is important to distinguish between _missing connections_ (stuck-at-ground,...) and _extra-connections_ due to smearing, drops or splashes.

A very common hypothesis is that of having to look for a _single_ fault, but such a single defect as having a connector linked to the wrong pin can result in _multiple_ faults.

The fundamental distinction is that between _error_ and _failure_, although it is less clear-cut in reality than it could appear. An error is a design fault, while a failure is a production defect or a component death in the field. An adjustment or adaption error however can at times induce failures of initially correct components that it is helpless to repair or replace.

As an illustration on the number v of potential failures of a given type (soldering here), a logic board with 16 and 24 pin components has $v \approx 2 \pi\gamma$ where $\pi$ is the average number of pins per component and $\gamma$ is the number of components. With 100 component locations, $\gamma = 80$ is common; hence $v = 3200$ is the likely number of potential soldering failures (assuming $\pi = 20$).

System _Reliability_ is concerned with failures only since it is the probability that a system satisfy a specific number of parameters defining the "model" under precise environment and operating conditions for a duration known as the _mission time_. [C.1]. But System _Efficiency_ takes errors into account as the probability that it will produce the results required by all user's programs within an allowable delay under normal conditions.
## I.2. Burn-in, wear-out and Test.

A **test** is an action aiming at efficiently revealing some discrepancy between expected behavior and actual behavior. A **diagnostic** can be deduced from a test and results in revealing that some discrepancy category has been encountered. Both tests and diagnostics are generally sequential activities amenable to computerization into **T and D programs**.

Although encountering a discrepancy also occurs during standard operation of any system, it cannot be considered as a test since it is not particularly efficient for discrepancy detection and it does not particularly ease diagnosis.

The set of failure detection aids (test programs and standard operation) helps exhibit the **survival rate** $\tau$ and the **mean-time-between-failures** MTBF or its inverse over thousand hours, the **failure rate** $\phi$. Figures 1-a and 1-b display typical shapes of these values as a function of time $t$ starting from production date. For non-repairable components, MTBF is to be replaced by **Mean-time To Failure (MTF)** upon which components must be discarded

<figure><img src="image_1.png" /><figcaption>Fig. 1.a.
The "survival plateau" $\tau(t)$.</figcaption></figure>

<figure><img src="image_2.png" /><figcaption>Fig. 1.b.
The "bath-tub curve" $\phi(t)$.</figcaption></figure>
Three phases can be distinguished on these figures : α, λ, ω.

The α → λ transition is called **burn-in** and the λ → ω transition is called **wear-out**; each phase can be grossly approximated as follows :

$$ \text{in } \alpha \text{ : } \tau(t) = \tau(t_{\alpha})e^{-t/t_{\alpha}}, $$

$$ \text{in } \lambda \text{ : } \tau(t) = \tau(t_{\lambda}) - k_{\lambda}(t-t_{\lambda}), $$

$$ \text{in } \omega \text{ : } \tau(t) = \tau(t_{\omega})e^{-k_{\omega}(t-t_{\omega})} \quad \text{and} \quad \varphi(t) = \varphi(t_{\omega})e^{+k_{\omega}^{-1}(t-t_{\omega})}. $$

Clearly a good test is supposed to be neither a burn-in, nor a wear-out action, but merely a detection tool. Testing of little durable equipment, like optical memories with a limited number of safe rewrites, can hence be problematic. On the other hand, repeated usage of a test program can turn it into an **endurance program** that can be used either for burn-in or to verify that the actual MTBF is larger than the minimal MTBF defined through the equipment quality level. In such a case thorough diagnostic message printing or storing is inhibited and replaced with failure histogram collection and averaging.

Burn-in or endurance programs are used to get rid of component infantile mortality (phase α) before system delivery to a customer, whenever failure occurrences are more costly there, but they are also used in the field for turning a random failure into a deterministic one.

## 1.3. Test programming Languages.

The most elementary test procedure is the sequence of phases : INITIALIZATION - ACTIVATION - COLLECTION corresponding to the selection of the initial state, of an input sequence and of an output subset in a sequential machine. For instance, initialization can be the positioning of a magnetic tape, activation can then be starting a block-read and write input-output program and collection can then be storing of input-output end-interrupt date of device and channel status bytes, together with essential transfer buffers.

This elementary test procedure must be followed by some form of a decision table branching to an analysis routine (COMPARISON phase) and eventually to the next test sequence. In that sense, the first elements of a Theory of
Testing are to be found in the Theory of Gedanken-Experiments originating from
the study of Physics. [D.1,13].

Test instructions proper to be found in the ACTIVATION phase can only
be specific since a test program is with respect to any system what a negative
is with respect to any film picture, or what a mould is with respect to a molded
object. More precisely, this remark is true for the whole (hardware + software)
test set whenever a special equipment is needed (logic board testing for instance).
Often a computer (or any complex equipment generally) contains powerful automatic
test instructions programmable but left unknown to the user.

Although test instructions must often remain specific, test program
languages have a characteristic "linguistic level". This level varies between
machine-languages and System-Writing Languages or Real-time languages. Translator-
Writing Systems and Decision Table Translators also are to be encountered, [D.12], as
well as "Fill-in-the-form programming" tools customary in Direct Digital Control.
It is customary to call "Test and Diagnostic Program Generator" a specialized
translator operating interpretively [TDPG].

### I.4. The object of a test.

The *complexity level* of the computer element under scrutiny varies
from that of the Reed switch level or the discrete triac up to that of the
input-output network, through those of the integrated circuit (of about one
square millimeter before its mounting on a plastic or ceramic package with an
average of twenty pins) and of the logic board (with about hundred integrated
circuits and more than two hundred connectors) and of the cabinets (with about
hundred logic boards) for control units and storage units or peripheral units.
Whenever one of these computer elements is being tested, the test program is
called *unit-test*. [D.2].

But the proper interconnection and cooperation between any two elements
at a given complexity level can also be the object of the test : hence the
*branch-impedance tests* for printed circuit boards or back panels and cables, the
_interface tests_ for input-output channels and storage links and the _simultaneity tests_ looking for diaphony, impedance mismatch, extrabonds, access rejection failures, buffer rythm errors, ...). [D.9].

At every stage of system assembly in production-line, it is thus necessary to associate a _hierarchy_ of test programs that make feasible the diagnosis of either a weakness of test programs at previous stages, or (infantile mortality) a failure of elements just assembled, or even an error in the assembly process control itself. From this complementarity point of view, an optimal organization of production test systems can be derived. But in maintenance on the field the object of testing is less precisely defined and more degrees of freedom are permitted in the test program organization. [A.13,B.1,2].

In all situations however it can be said that test problems must be divided according to the _analog_ or _digital_ nature of the object under scrutiny, as well as according to the electronical or electromechanical nature of its implementation. Hence central processors (electronical and digital) and peripheral subsystems are comprised within two very distinct broad categories, most failures being non-deterministic in the latter because of the continuous nature of encountered signals.

### I.5. The operating modes.

Much like the possibility of using a given program in different contexts leads to the introduction of the _process_ concept for each of its executions, it appears useful to distinguish between _test procedure_ (sequence of instructions ordered to the object under scrutiny) and _stimulus pattern_ (set of selected signals for one execution of a test program) so that a _test process_ is an execution of a test program with a given stimulus pattern resulting in a _response pattern_. In the example of INITIALIZATION, ACTIVATION and COLLECTION phases we have seen for a magnetic tape, the stimulus pattern comprises initialization parameters, like the input-output start date and the block address on tape, as well as the transfer data buffer. Analysis can then be performed by means of a comparison of the response pattern with a _reference pattern_ for the given test process, thus enabling the emission of a diagnostic. A _detection process_ is the resulting pair of a _test process_ and its response pattern in one execution instance. [C.14].
<figure><img src="image_3.png" /><figcaption>Fig. 2. Synopsis of the various time intervals: $\bar{d} \approx f/10$, $i \approx f$.</figcaption></figure>

Four characteristic durations must be taken into account for proper scheduling of check and repair operations :

- the **index time** i for initial positioning of the object under scrutiny, [E.11],

- the failure-free duration f of the test program,

- the average time to first detection d of any failure,

- the average time to diagnostic emission e for any detection.

Organizing elementary test procedures into a test and diagnostic program is complicated by the modes of stimulus-pattern inputting, response-pattern outputting, activation frequency, comparison rate and iteration predicates, as we are going to define them. Moreover, proper inference from a test process response requires that a given portion of hardware be operational, so that a test sequence is **truthful** only within a specific program context. This makes mandatory the elaboration of a **penetration graph** and the choice among [D.8] several test tactics like :
- "start-small" or **bottom-up** testing from a minimal guaranteed hardcore with halt-on-first-detection, [C.5],

- "start-big" or **top-down** testing from a high level with zooming effect for refinement-upon-failure-only,

- "multiple clue" or combinatorial analysis on failure vector gathered through thorough discrepancy-retrieval-until-last-process-response.

Because of the probability that any failure induces errors into the sequel of a test program, the frequency of discrepancy detection by a test program even in the single failure case will approximate that depicted on figure 3.
<figure><img src="image_4.png" /><figcaption>Fig. 3. Detection frequency along a "start small" test program.</figcaption></figure>

### I.5.1. Stimulus-pattern inputting.

Stimulus patterns can be presented to the object under test in two non-exclusive modes : P (pulse-input) and L (level-input), the latter being characteristic of asynchronous logic. In order to take into account the even-tuality of race conditions, two test modes are hence applicable : **natural**, whenever stimulus patterns are a subset of standard patterns, and **exclusively level-input**, when L stimulus patterns exclusively can be used (thus excepting clock signals from standard testing, ...). [C.15].

### I.5.2. Response pattern outputting.

With natural stimulus patterns, all response patterns from the object under test are worthwhile sampling, while with exclusively-level inputs, race conditions can occur and it is preferable to wait for response pattern sampling until output levels are stabilized. One generally admits this is the case for a logic board after ten thousand standard clock cycles.

### I.5.3. Activation frequency.

Testing is said to be performed **dynamically** whenever the object under scrutiny is exercised at its standard rate. It is said to be performed **statically** when the rate is slowed down so as to make feasible a stepwise comparison and analysis phase. It is **clockwise marginal** whenever the rate is slightly altered around its standard value.

### I.5.4. Comparison rate.

Analysis can be performed in real-time or after last process response. Similarly comparisons of response and reference patterns can be postponed or not
according to speed and memory constraints but it is also crucial to distinguish comparison with or without *backup*, i.e. the possibility for a stream of stimulus patterns to perform comparison merely at the end and eventually proceed through dichotomy (restarting the same stream and halting it in the middle for comparison, etc.) for zooming upon detection.

### I.5.5. Iteration predicates.

According to whether the result of a comparison phase can or not automatically guide the choice of the next process, the test program is *closed* or *open-loop*. Often branching is made feasible through the choice of parameters by the test-operator upon incidents with facilities like *halt-upon-error*, *iterate-on-failure*, *repeat in static mode* directives. But it is at times even more crucial to prevent any such options so that the test program will scan the penetration graph under the mere guidance of the system configuration chart.

## I.6. Diagnostic kinds.

The ability to have the test program automatically iterate the detecting process upon failure detection leads to the question of test program structuring: test programs must for that purpose be built as trees of successive and nested loops yielding a hierarchical organization, image of the architecture of the object under scrutiny. Innermost loops will generally-speaking correspond to most elementary elements of this object.

This structural decomposition can be of either of two types, *physical* (Cabinets, Racks, Boards, Packages) or *logical* (Processors, Instructions, Micro-instructions, order signals) and it makes it possible to reach various degrees of diagnostic through progressive refinements and correlations under the guidance of decision tables. [D.12].

Four kinds of diagnostic can be proposed:

a) **Functional** when the aim is to verify conformity of the object with equipment specifications at the standard operating interface (soft/hard).
b) **Parametric** when the aim is a set of performance measurements or transferometric evaluation.

c) **Referential** when the test and diagnostic program refers the operator to an intervention procedure guiding him through replacement or repair.

d) **Localizing** when the program specifies failure location up to the interchangeable element to be replaced or up to the adjustable element to be tuned. [C.3].

## I.7. Frequency of use.

Frequency of use of a test program (or subprogram) is a function of the check-out strategy in production and of the maintenance strategy in the field. For a computer system these strategies often make it possible to reuse the same program in various modes in diverse situations. [A.1].

### a. Preventive operation, before incident.

The test program is used periodically for detection purposes or upon reaching some quality threshold for diagnosis purpose. In the latter case, some on-line quality sampling is necessary to warn the test operator (frequency of automatic error correction on a tape drive for instance).

### b. Curative operation, after incident or modification.

The test program can be used as a repair or adjust tool while checking that none of these human interventions disturb equipment behavior around the concerned part.

### c. Acceptance operation, after production, installation or repair.

The test program is used for general validation of the object under scrutiny according to performance criteria and quality thresholds preestablished for both functional and technological purposes. This test operation subsumes various system processing power mixes characterizing the object speed [C.4]. At the level of integrated or discrete components, speed measurements or parametric tests (§.I.6) can lead to binning, i.e. cluster formation of more closely identical components, in order to selectively orient the use of components in accordance with their cluster average performance. For instance, faster than
normal integrated circuits can be reserved for central units. As an other instance, process control and testing can be tied together through binning in a production phase with *select-at-test* decision where a component with specific performance has to be selected for insertion into some unit according to parametric tests of this unit. [E.12.p. 108]. In all three of these operating situations, marginal testing can be efficient : non standard parameters (temperature, voltage, frequency) can induce a shift of correct functioning limits that reveal a weakening of quality of the object under test.

From the repetitive use of the same program in time and space, trend analysis and averaging can provide the producer with a reliable failure prediction and interpretation data base. [B.7].

## I.8. Test program categories.

Let us consider as an example the case of peripheral subsystem testing which encompasses all difficulties with analog, hybrid and digital as well as electromagnetic and electromechanic troubles. Peripheral subsystem test programs can be entered into diverse categories according to whether it is connected in standard operating conditions, as a whole or in parts.

If not, test is said to be *external*, but that may imply :

a) on a *tool-system* for the whole subsystem (controller, device and storage) as shown on figure 4.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Synoptic chart of present computer System linkage.</figcaption></figure>
b) on a **tool-controller** specific to production check-out for the electromechanical device and storage support (this controller might be already amortized and activated by a cheaper CPU).

c) on a **maintenance processor** isolated as a tester whenever the system in the field has a reconfiguration capability. [A.10,11,12].

d) on an **acceptance tester** for the storage support in the case of electromagnetic devices (discs and drums).

e) on a hardware-monitor for every electronic unit under performance measurement [C.6].

f) on a **subassembly exerciser** when racks or cabinets can be independently tackled for dynamic troubleshooting.

g) on a **board logic tester** for soldering misses or splashes and component deaths after mounting and possibly after burn-in through vibrations or in an oven.

h) on a **board branch-impedance tester** for connection errors and diode inversions after soldering of components. [C.9].

i) on a **back-panel bed-of-nails** for interconnection impedance checking of inner layers of printed circuits or externally wrapped wiring.

j) on a **power-supply tester** for equipment-specification conformity checking.

k) on a **cable-connector tester** for verifying proper isolation and linkage.

l) on a **position-tester** for automatic light-controlled comparison of component location and orientation on a board before soldering.

m) on a **bare-board bed-of-nails** for printed circuit single or multilayer interconnection and isolation checking.

n) on an **integrated circuit package tester** for equipment-specification conformity-checking on a sample basis or on an exhaustive basis. [C.11].

o) on a **silicon wafer tester** for localizing defectuous chips before cutting them from each other.

p) on a **discrete component tester** (relay, transistor, ...), etc.

Optimal use of so many testers raises a complex scheduling problem [E.8,9,13].

Otherwise, test is said to be **internal** whenever it is executed by a programmed processor residing for standard operation within the object under scrutiny :

1) An **autotest** is a test program for a central processing unit or for a micro-programmed controller checking themselves. [A.3].
2) A _microtest_ is a microprogram designed in some subunit for testing another one and emitting a microdiagnostic [A.4,5].

3) A _protest_ is a test implemented at the standard soft/hard code level.

Internal autotests tend to disappear and to be replaced by a combination of external unit microtests on a maintenance processor together with interface protests and simultaneity protests (§. I.4.). This eases resolution of the _hardcore dilemma_ described as follows : since any test can yield a meaningful diagnostic only if the hardware used by its processes is itself already checked, there is a minimal amount of pretested hardware required for starting, through autotesting, the penetration of any unit : it is called the unit hardcore, and the maintenance processor somehow transfers the hardcore out of the unit.

As a generalization it may be stated as the INVERSE POWER PRINCIPLE that the amount of control logic required in the tester or maintenance processor is inversely proportional to the logic capability or processing power of the object under check-out [B.2], i.e. to its gate/pin ratio (See figures 5 and 6).

It happens that the inverse power principle can be otherwise justified as follows : the amount of control logic required in the tester for a given technology is proportional to the transfer rate or cycle time of the object under scrutiny.

It is clearly uneasy to elaborate a Test Strategy with so many alternatives. The following figures are presented to visualize a way of clarifying the situation.

<figure><img src="image_6.png" /><figcaption>Fig. 5. Test and repair cost as a function of the system complexity level.</figcaption></figure>
Potential savings are derived from the DEPENDENCY PRINCIPLE according to which the set of attainable input (and output) patterns on a given element is reduced when this element is inserted into a broader unit : its gate/pin ratio is increased since pin signals become redundant because of their dependency from surrounding elements. Hence attainable stimulus patterns in the inserted case are a subset of those in the free-standing case.

<figure><img src="image_7.png" /><figcaption>Fig. 6. An approach to a rational coverage of failures at various production and maintenance levels : A + B + D + F.</figcaption></figure>

The paradigmatic test strategy displayed above in figure 6 results from a compromise between the need to test faults as early as possible in a product life as explained in figure 5 and the cost advantage in implementing tests in a reusable language.

## I.9. Tester System Classes.

One of the major bottlenecks in modern computer unit-testing is the storage space per time integral requirement. In order to efficiently save on both storage space and time waste, standard operating system and data base techniques are being more and more applied at various multiplexing levels, hence yielding hierarchical system structures [A.13;B.2,6,7,9,10]. Also intensive research has been done at the level of test process and test pattern generation algorithms so as to parallelize failure detection on independent logic paths,
whereby non interfering patterns are merged synchronously into a single test process, or asynchronous test processes on independent units are multiprogrammed.

Finally test systems can be ordered according to their degree of interference with standard operation of the object system user. Hence, a test operator for maintenance in the field essentially has at his disposal for a given peripheral subsystem one among the six following test classes :

a) STAND-ALONE for autonomous protests in privileged mode dedicated to one kind of objects to test like a decimal arithmetic Unit or a tape drive, and incorporating its own operating mode routines [D.2].

b) OFF-LINE for protests dedicated to specific objects to test but sharing common operating mode routines (like interrupt processing, file management, operator dialogue, iteration-on-error,...) in an _independent test monitor_ [A.1].

c) THROUGH-LINE for restricted protests with no other privileged capability than that granted by the standard Operating System to real-time users, hence granting under a _test processor_ system maintainability with no halt to Computer System operation in the field. [A.14].

<figure><img src="image_8.png" /><figcaption>Fig. 7. Synoptic view of the historic evolution of test operating classes.</figcaption></figure>

d) OUT-LINE for external microtest by maintenance processor, able or not to dialogue with the operating system or the service console, granted special hardware paths like the Diagnostic Shift Register [D.11] to force initialization and stimulus patterns and to collect response patterns.
e) ON-LINE for multiprogrammed execution of protests and user programs under operating system control with the former being granted secure use of privileged instructions by a *test monitor* supervising them as a real-time processor under the operating system. [A.2,7,9].

f) IN-LINE for multiplexed microtests capable of simultaneous execution with standard microprograms of the system instruction code, hence letting test and user operations be paralleled under operator control without operating system control. [A.6,8].

Here too a compromise has to be worked-out so as to trade-off technical efficiency and detection need coverage. Through-line and on-line protests are obviously cheaper to develop using multiprogrammed systems. Meanwhile system simultaneity protests at the level of a system integrated configuration are manually STAND-ALONE. Finally the THROUGH-LINE restrictions can be largely compensated by IN-LINE extensions and they together make lighter demand for facilities from the test processor.

## I.10. Translation and Structural aspects.

In all test classes overseen, test programs can be predesigned and compiled, or replaced by a conversational interpreter of test directives with appropriate parameter passing and subroutine linking facilities.

Advantage of the *compiled* form is its execution speed optimality after an initial translation investment, but any late evolution of the test, for extension purpose or for adaption to hardware change orders, is rendered difficult and costly by the need either to recompile or to modify the object-code through patching. We will reconsider the FOLLOW-UP problem of engineering change orders in the second half of this paper.

Advantage of the *interpreted* form with a file of prepared directives thus is the ease for modifying and extending by a maintenance specialist in the field in a high level language. The interpretive form can also be an optional complement to the compilative mode in manual intervention phases or special test cases.
Protest naming and file structure themselves can be organized through
a mapping from the hierarchical structure of the objects under test. For instance
peripheral subsystem tests can be decomposed into four test families of processes
independently of the type of peripheral.

a. Conformity checking of elementary functions of the standard input-output interface with functional diagnostic.

b. Testing of simultaneities internal to the subsystem with localizing diagnostic.

c. Compatibility-verifying of similar subsystems with performance evaluation (parametric diagnosis).

d. Interactive trouble-shooting with manual interventions of peripheral device adjustment and mounting facilities with referential diagnostic.

Interactions are possible with the compilative mode through control
parameter exchange with the operator via the service console, but availability
of the interpretive mode amounts to offering the operator an extra test family
with maximal interaction degree and minimal diagnostic aids.

Finally *Telemaintenance* is an option that makes possible the distinction
between rare and distant test specialists and local non-specialized system ope-
rators. Upon request from the operator a specialist can control through a main-
tenance data-line execution of a test program, can store in a centralized file
specific system log-outs and reliability histograms and refine the diagnostic
while the local operating system is still operational or not according to the
severity of the system failure under investigation.

It is the ultimate improvement in system serviceability i.e. the
amount of aids the system provides for recovery and availability improvement
whenever operator intervention is required for proper service to the user.
[B.3.4; C.13].
Having reached such a level of complexity in test strategy design, it is but natural to feel the system is quite intricate, or even heteroclitic and hence costly. This very human tendency amounts to confusing intrinsic complexity of a problem and extrinsic complexity of its presentation and to prefer the "ostrich policy". Lack of tests however, or worse, simplistic test strategies may amount to being merely more costly. The only rational approach thus consists in basing test strategy options on an explicit `RETURN ON INVESTMENT` evaluation. [E.2].

### II.1. Financial analysis of quality and test scheduling.

Instead of reasoning through uncertainty with probabilistic theories, our approach is to develop *reasoning through imprecision*, deriving from decomposing the problem into as many elementary parameters as possible— as rigorous as possible a conclusion. It is our experience that the couple of parameters required to specify Poisson processes, Gamma distributions, learning curves or linear growth rates, are generally at hand and make feasible the deduction of non intuitive conclusions.

Let us first consider the fact that product line quality ratios (MTBF for instance) vary along the lifetime of a product as a function of the number of copies being used by customers in the field: Conversely system *modifiability* — loosely defined as the inverse of the cost of application of an engineering change order — decreases while the number of product copies in the field increases, because of change-implementation cost, but also because of the change-design cost due to the aging of technological components and finally because of the progressive disappearance of the initial design team. This last fact is easily understood when one considers the total duration between initial objective specification and first customer shipment for a new range large computer is of *seven years*. System modifiability figures are nowadays available for some large software systems only [E.3,4 and 5]; they seem to fit closely patterns of more general system maintenance and growth problems.

For hardware systems, here, the following curves can be assumed for both errors and failures.
<figure><img src="image_9.png" /><figcaption>Fig. 8. Evolution of a product line quality ratio.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 9. Product range aging curve.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 10. Product range sale impact.</figcaption></figure>

Discrete time approximations of these families of curves can be obtained with functions $f_n = \alpha + \beta \gamma^{-n}$ and $g_n = \frac{\alpha}{\beta + n^2}$.
<figure><img src="image_12.png" /><figcaption>Fig. 11. Engineering change order rate.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 12. Cost growth rate and loss dynamics for change design and implementation</figcaption></figure>

The point of all this is that hardware modifications tend to render obsolete existing test programs so that along the life of a product the test system must be rewritten for a large part.

Let us now consider marginal profits that a system manufacturer may expect; they can be displayed in a simplified form as on figure 13. A more detailed study should distinguish between products being delivered on the basis of selling, renting or leasing.
<figure><img src="image_14.png" /><figcaption>Fig. 13. Marginal results expected from a product line.</figcaption></figure>

In order for the test system to be operational as soon as production is initiated, initial income is initially reduced by the amount of investment required for a new line start-up as shown on figure 14.a.

<figure><img src="image_15.png" /><figcaption>Fig. 14.a. Investment and Income catchup. Fig. 14.b. Profit and loss dynamics.</figcaption></figure>

Figure 14 is made more realistic in b through taking into account the evolution of n as a function of t (displayed on figure 10), the progressive creation of the initial test investment FT, [E.14], and the maintenance cost due to quality decay as presented in figure 9.

For rigidly defined products ($v_t = 0$), figure 14 b displays the likely profit and loss dynamics such that any company ideally tries to stop marketing a product line beyond point F where product aging renders the MTBF too costly to support in the field.
On the other hand, if system modifications through engineering change orders can occur during the product life as shown in figure 11, then a decline occurs in the profit increase slope as shown on figure 15.

<figure><img src="image_16.png" /><figcaption>Fig. 15. Profit and loss dynamics with modification cost decrement.</figcaption></figure>

We are thus reaching the conclusion that in order to save on the cost of system progressive redesign, it may be more profitable to delay initial fixed cost expenses of test design FT. This has to be based also on the fact that the small number $n_t$ of produced machines for $t = K$ makes tolerable the corresponding increase of check-out and maintenance proportional cost in the absence of a test system. Figure 16 presents the resulting gain. [E.6,7].

<figure><img src="image_17.png" /><figcaption>Fig. 16. Profit and loss dynamics with test development delay.</figcaption></figure>
This progressive development methodology can be expressed as a trade-off of variable cost for production check-out and maintenance against initial investment amortization. It helps fight product aging and postpone the renewal point from M to M* and thus it helps improve the final integrated profit to $I_M^*$.

<figure><img src="image_18.png" /><figcaption>Fig. 17. Comparative display of profit expectation.</figcaption></figure>

It is noticeable that test strategies are influential in a particularly sensitive phase of the profit dynamics evolution. Moreover the progressive development technology offers the advantage to help making the test system itself evolve during its implementation so as to optimize it thanks to the experience acquired with the first partial results. [D.3;E.10].

Before presenting four profitable PRINCIPLES OF TEST COST REDUCTION, let us emphasize that our estimation technique based on simple curves with one or two parameters to fit - as a replacement of decision in uncertainty by decision in imprecision - provides us however with a rigorous way to appreciate how crucial some of these elementary parameters could be in affecting any deduction. For doing so it suffices to construct an explicit mathematical expression of the predicted dynamics curves as displayed from figure 8 to figure 17 and to perform a SENSITIVITY STUDY through partial differential calculus along every one of the parameters.

## II.2. Reduction of test usage frequency through sampling.

**Inspection by attributes** is the procedure of testing for functional and parametric diagnostic through comparison of the unit under scrutiny with product specifications, whereby either the unit of product is classified simply
as defective or not (GO - NOGO), or else the number of defects in the unit of product is counted. This opens the way to applying in hardware test and diagnosis the statistical theory of tests; the extent of nonconformity i.e. diagnosis then is expressed either as

**Percent defectives** $\chi_u$ $\Delta$ 100 $\phi$/N $\%$ where $\phi$ is the number of defective units in the lot of N units inspected, or as

**Percent defect rate** $\chi_d$ $\Delta$ 100 $\phi$/N $\%$ where $\phi$ is the number of defects encountered in the lot (possibly several per unit).

The average number of defects per defective unit or **defect degree** is

$\Delta \Delta \approx \chi_d / \chi_u = \phi / \psi$.

Decision of acceptance or rejection based on inspection by attributes is based on a **sampling plan** trading-off exhaustive testing cost and inspection rejection failures according to a chosen **acceptance quality level** (AQL) defined as the maximal percent defective $\chi_u^M$ or as the maximal percent defect rate $\chi_d^M$ that can be optimal on an average basis. [E.1].

Inspection by attributes can be **open-loop** or **closed-loop** according to whether the sampling plan is not or is a function of first sample diagnostics with a feedback rule as indicated on figure 18. Open-loop inspection is said to correspond to **single sampling plans** (and closed-loop to multiple).

<figure><img src="image_19.png" /><figcaption>Fig. 18. The multiple sampling feedback rule.</figcaption></figure>

**Operating Characteristic** curves (OCC) are available in tabulated form to indicate the percentage $P_a$ of lots which may be expected to be accepted under various sampling plans for a given product quality $\chi$. They are based on binomial distributions but are approximated by Poisson distributions whenever AQL $\le 100$
or sample size is $N \ge 80$. For instance, the most commonly adopted AQL for integrated circuit lots is $0.65 = \chi_u^M$ and yields the OOC displayed on figure 19.

<figure><img src="image_20.png" /><figcaption>Fig. 19. Operating Characteristic Curve for single sampling plan.</figcaption></figure>

### II.3. Reduction of test pattern design cost.

Once a single test procedure has been programmed as described in paragraphs I.3. and 5., iterating on the choice of stimulus patterns can yield a quasi-infinity of test processes. It can hence be profitable to design a _pseudo-random_ generator for creating stimulus patterns and selecting efficient ones. In the case of sequential logic testing, circuit initialization often is the most costly [C.7,8,10], since it amounts to determining a _homing-sequence_ taking the circuit from any initial state into a specified state for starting tests. Such a sequence may even not exist in the absence of additional hardware granting _cut-points_ as exemplified by the Diagnostic-Shift Register [D.11]. It is usually feasible for logic board testing to actually use the same sequence generated by the pseudo-random generator, first for homing, then for testing. Such homing is accomplished through ignoring response patterns and with some exceptions (to take care of oscillators or periodic autonomous sequential subcircuits).

Comparing to a reference pattern the pseudo-randomly generated response patterns causes no problem in the example of magnetic tape input-output (§. I.3.5) because bytes of the transferred blocks are supposed to be the same for stimulus and response, but difficulties appear for condition codes, the interpretation of which requires analysis. More generally, whenever a logical operation is per-
formed upon stimulus patterns by the object under test to output response patterns, comparing to a reference pattern is more troublesome.

Response retrieval and failure detection are called *differed* or *immediate* according to whether several test processes can be executed in sequence before response pattern collection and analysis. Pattern storage constraints with random generation and differed analysis can be solved by the replacement of response pattern level collection with response signal *transition* count.

In the case of random failure search (§. I.1.) comparison to a reference is made easy by immediate collection of response patterns and comparison to previous responses taken as reference.

### II.3.1. Simulation.

For a well-defined small logic system, intellectual generation is well-adapted to the design of reference response patterns when stimulus patterns are selected in a more-or-less minimized number. But for a system complex and/or susceptible to engineering changes, it is preferable to develop beforehand a *predictive* simulation model that can be used for reference pattern file construction. This is particularly important for prototype design aid (See figure 20).

Several algorithms are known for automatic generation of test stimulus patterns at about the logic board level [Boolean differences, path sensitizing, ...]; they provide an *optimal cover* of the *fault set* defined by some failure hypothesis (§. I.1.).

<figure><img src="image_21.png" /><figcaption>**Fig. 20.** Synoptic view of the use of simulation for testing.</figcaption></figure>
Predictive simulation [D.5,10] can be economically replaced by
**real-time simulation** for production or maintenance purpose. It can be made
easy by system reconfiguration facilities like double-access on peripherals
or symmetry of right and left halves of an adder, so that hardware is consi-
dered all right whenever the same test process yields the same response pattern
through either side. Such a facility can be called **symmetrical redundancy** ; **asymmetrical redundancy** is also encountered as for floating-point operators that can
be simulated with binary operators (assuming they have been previously tested).

Predictive and real-time simulation can be optimally combined ; for
instance (See figure 2) test references can be precomputed only for the fault-
free situation (GO case), while on-the-spot simulation for fault diagnostic would
be performed upon fault detection exclusively (NOGO case).

II.3.2. Self-learning.

As soon as production is started, the company task can be considered
as a replication process of some reference hardware ; test programs then can be
used in **self-learning** whereby test processes no longer induce detection proces-
ses (with response pattern retrieval and comparison to a reference) but induce
reference **retrieval processes** which merely fill or modify files of test reference
patterns.

It is significant here to distinguish (§.I.4.) between the more deter-
ministic case of most digital units for which the stimulus-response relationship
is normally deterministic, and [C.12] the more random case of analog and electro-
mechanical units where adjustment shift margins render mandatory comparisons of
response patterns to reference patterns of several units. In the case of removable
magnetic discs for instance, and _itinerant stack_ is carried from system to system
in the field for peripheral subsystem calibration. It stores reference patterns,
and every time it is used and finds a subsystem acceptable, it augments the re-
ference file with the newly retrieved response patterns for later calibration
purposes. This itinerant stack calibration accuracy hence is a function of its
history, i.e. of the set of previously tested subsystems (to the point where a
subsystem can be accepted at one time, used for further reference, and rejected
later).
## II.4. Reduction of duration of undetected faults.

The cost of one system break-down in the field is a function of the **Mean-Time To Repair** (MTTR), and especially a function of the amount of jobs aborted and files destroyed. The cost of a destroyed file is the cost of reconstructing that file which at times may be an impossible task.

<figure><img src="image_22.png" /><figcaption>Fig. 21. Mean-time to Repair improving with the Test System Growth.</figcaption></figure>

System **Integrity** is the conditional probability that, in case of a failure among its elements, it will not lose information which has been entrusted to it. It requires mechanisms for failure **impact resiliency**. Considering that file **back-up** copies and instruction retry or error-correcting facilities help minimize the amount of destruction upon failure, nothing can replace as early as possible incident detection. This leads to replacing **periodic testing** by **continuous watching**. In that sense error-detecting codes are **dynamic test devices** as they check partial conformity of data pattern to a reference pattern associated with this data in the pattern itself and redundant with it (e.g. parity bits). This technique amounts to duplicating some fraction of the information contents, so that choice of a detecting code has to be based on the appropriate distribution of this fraction to optimally protect the information contents in some failure hypothesis [C.2]. Hence a cyclic code is preferred on a magnetic tape at block-end for instance. Notice however that dynamic test devices raise the recursive problem of test-tool testing.
If the code contains enough redundancy so as to make feasible some error correction, or else if the incident occurs at a time when it is still practicable to execute again the same operation that it had disturbed, without requiring operator or operating system intervention for user program restart, then some hardware process can automatically perform the operation *retry*. It is crucial however that a *log-out* be stored automatically so that trace can be kept of the system *degradation* (often called *graceful* when it remains transparent to the user), and that an *error journal* be displayed for the maintenance operator [D.7]. Retry statistics can then be compared periodically by the operating system with alarm thresholds, upon which the test system must be activated. This provides the system on one hand with maintenance intervention before customer operation be disturbed, on the other hand with *prognostic* data from stored log-outs indicating which test programs should be most relevant to activate [B.4.].

The counterpart of the concept of error journal and log-out statistics in maintenance is the concept of *visibility* in production, i.e. the set of facilities providing real-time audit, trend and correlation analysis about product line quality so as to trace defects of manufacturing and test processes as far back as possible in the assembly-line. [B. 5,8].

As we have seen in § I.4. and I.8., machine assembly is a hierarchical process, so that in order to avoid having to destroy previously invested workmanship through desassembling, it is a must to detect component and/or manufacturing deficiencies at their own assembly level. Since the number of possible failures includes unit deterministic failures plus simultaneity and random failures (diaphony,...), it grows more than linearly as a function of the number of interconnected components, so that failure localization is generally simpler when the assembly process is less advanced (this taking into account both the inverse power principle and the dependency principle seen in §.1.8.) [D.4.].

This is an incitement to trying to avoid the use on complex systems of long exhaustive test programs, as soon as the quality ratio revealed by analysis of assembly history of various interchangeable elements leads to expecting sufficiently low a percent defect rate $\chi_d$ (§. II.2.). Full application at the complex system level of the thorough test and diagnosis program would
not be economically justifiable. All this reasoning meanwhile subsumes a
real-time measurement control system about quality of all components and of
all manufacturing processes having taken part in system assembly [B.1,5]. This
requires an elaborate automated plant network of computers for Information
Control Trend Analysis and Decision making. But it is the rational approach to
ship computer extensions into the field with the same quality level as when
a whole computer configuration is first assembled and exercised in-plant, before
unmounting and shipment. What is more, this kind of in-plant merging can be
eliminated whenever quality criteria are satisfied so as to perform drop-shipment
of configuration parts (cabinets) followed by field-merge of whole computers.

## II.5. Reduction of redundant test investments.

A major consequence of production visibility and of test system evo-
lutivity with delayed investment (§. II.1.) is to help concentrate corrective
actions upon, on one hand these production line stations that are revealed
in real-time as week points of the manufacturing process through correlation
between defect rates among the set of assembled systems, on the other hand these
system units where the testing effort is most efficient in terms of the cost
of missing to detect a failure in time i.e.

$$C = \chi_d^p \$^p + \chi_d^f \$^f, \text{ where } \chi_d^p \text{ is the percentage of failure detection in-plant at a given assembly level and } \$^p \text{ the cost of detection and repair at this level, and } \chi_d^f \text{ is the percentage of failure detection at later levels (possibly in the field) and } \$^f \text{ the cost of detection and repair at these levels. This is why microdiagnostics to be developed firstly are those exercising these logic boards in a microprogrammable processor which are hardly testable on a board tester in static level-input mode.}$$

Side-effects of such a policy are multiple since it affects in-house
inventory control and component supplier attitude. For instance since part of
components (like relays) bought from outside suppliers and stored in-house tend
to fail at the beginning of their use (figure 1.b), it is better to burn them in
(§. I.2.) in early age on a specialized tester (vibrator) so as to get rid of
infantile mortality, rather than having them fail after having been mounted on a
line printer and having to localize them with an input-output program taking
processor time. If this burn-in is performed on a sample basis for inspecting
incoming lots, it helps refuse acceptance of a likely bad lot that would
burden the minimal in-plant stock of a rejection percentage margin and
minimize the RETURN-ON-ASSETS ratio. Such refusals of incoming lots is a
strong enticement for the component supplier to improve the quality of his
delivery to a specific customer. This policy also applies to larger system
units like peripheral devices which go through some acceptance exercising and
test called *dock-merging* before in-plant storage : insufficient test efficiency
then implies some increase of replacement unit stocks.

Finally, test program design investment must be rationally limited
in the three following ways : reusability as explained at the end of paragraph
I.8., limited exhaustivity and limited sensitivity as follows.

### II.5.1. Limited Exhaustivity.

Numerous authors have advocated the use of *fault simulation* techniques
[D.5,10] in order to estimate test system efficiency or even to facilitate dia-
gnostics through the association of a failure reference pattern. While this
exclusively applies to technological deterministic failures and not to random
failures or design errors, even if soldering or bonding defects only are taken
into account (excluding solder splashes, etc.) the number of failure types v
indicated in paragraph I.1., to be augmented with back-panel wiring and input-
output connectors, shows that even for a minicomputer the number of simulation
runs would have to be larger than ten thousand .

It is therefore crucial to distinguish between the number of possible
failures and the number of possible *symptoms*, i.e. of equivalence classes of
failures affecting identically a response connector of the object under test
for the set of all possible test processes. Symptoms can equivalently be viewed
as an equivalence class of hardware components which are indistinguishable
through any detection process when they are set into a faulty state indepen-
dently. It is also crucial to distinguish between the number of possibly emit-
ted diagnostics and the number of possible test process sequences, whereby dia-
gnostics are equivalent classes of symptom detections leading to the same main-
tenance intervention. This leads to considering globally diagnostic as equivalence
classes of test sequences leading to the same intervention and it can be factо-
rized as an abstract sequential machine as shown on figure 22.

<figure><img src="image_23.png" /></figure>

S* is the set of sequences of elements from set S.

Fig. 22. Symbolic representation of Test and Diagnostic algebra,

In the course of testing, a "clue vector" is elaborated by detection
processes and its analysis (§.I.5.) helps reduce it to a symptom. This exempli-
fies how fallacious it is to evaluate test system exhaustivity as a percentage
of actually implemented procedures over ultimately expected procedures at test
program completion, since some symptoms may never be detected (abstract machine
<u>unreachability</u>), while some procedures may be redundant (abstract machine
<u>non-minimality</u>). It is advisable to state that a test system is <u>exhaustive</u> if for
all possible symptoms (fault states) all diagnostics can be emitted (abstract
machine <u>observability</u>). Concretely this leads to elaborating a test system
backward from the abstract machine outputs toward its inputs, i.e. from the
intervention dictionary for some incident hypothesis, to the stimulus process set.

The *exhaustivity ratio* can now be measured (statistically) through
forcing failures (shorts, ...) and executing tests. This ratio is less than
optimal in the presence of unobservability and unreachability, i.e. if several
diagnostics are indistinguishable because they correspond to a single symptom
while distinct failures are involved (abstract machine *non-diagnosability*).

II.5.2. Limited sensitivity.

The previous point of view, external, can be completed by an internal
point of view based on the obvious fact that an elementary function is not tested
unless it is exercised by the test. For connections between integrated circuits
or between logic gates, a path is said to have been _sensitized_ when it has been submitted to level transitions from 0 to 1 and from 1 to 0.

The _sensitizing ratio_ can be measured statistically with a hardware-monitor (§. I.8.e) during the test process. It is supposed to be 100 % for generation algorithms like those mentioned in paragraph II.3.1. Such a ratio actually exists for any user program, but any program even though able to reveal the presence of some failure is not necessarily a test program. In fact the _detection ratio_, percentage of detected symptoms over the set of all possible symptoms, for any program ("exercising") not only must provide large enough an exhaustivity ratio, but above all must do so after short enough a detection duration.

Exhaustivity and sensitivity now appear as the basis for defining an AQL of test systems.
### III. A PARADIGM FOR FINANCIAL OPTIMIZATION.

We have seen in paragraphs I.8. and II.5. that any choice to invest some test effort at a given assembly level for a computer system depends on several opposite trends. It depends on the existence of a test technique able to reach diagnostic at this assembly level for a certain category of failures. In the search for a minimal cost with respect to testing at other assembly levels this usually leads to apply testing as much *upstream* as possible along the production line. Conversely this tendency has to be mitigated by the fact that the more a test implementation is *downstream* (applied on larger units) the more chances it has to be reusable at other levels, including in the field, inasmuch as failure hypothesis do not vary too drastically along the production line. Practically, operating modes could be different but it might suffice to have several test monitors or a more elaborate test monitor to adapt the same test programs to all operating situations.

Let us consider the problem of peripheral subsystem unit testing with program families as described in paragraph I.10. Figure 23 displays the historic trend in test monitor and program cost optimization and figures 24 and 25 present the basis for demonstrating choice optimality.

<figure><img src="image_24.png" /><figcaption>Fig. 23. Historic trend in test monitors and test programs.</figcaption></figure>
In the case of peripheral subsystems the need has been understood
to dispose of a set of test programs separating a common subset of ON-LINE or
THROUGH-LINE multiprogrammable protests, a subset of IN-LINE multiplexable
microtests and even a subset of OUT-LINE microtests applicable on test-allocated
configuration resources (like a reconfigurable input-output link, a main memory
block, etc.)

This strategy offers the extra advantage that product line visibility
is improved thanks to the use of a common test system through production and
maintenance.

<figure><img src="image_25.png" /><figcaption>Fig. 24. Comparative Cost display for peripheral subsystem test in test development

{ Man. hour cost/machine hour cost ≈ 20.
On-line machine hour cost/off-line
m.h. cost ≈ 25.</figcaption></figure>
A thorough cost analysis requires here the evaluation of reliability ratios derived from the usage ratio $u = MTBF/MTBF + MTTR$ expressing the proportion of system life-time during which it is operational. Its complement $r = 1 - u = MTTR/MTBF + MTTR$ is the repair ratio. On-line, through-line and in-line modes tend to make it transparent to the user in a way that down-time ($\Delta_i$) applies only to the subsystem ($S_i$) under scrutiny. The cost of down-time then becomes a function $\Sigma_i \Delta_i \delta_i$ where $\delta_i$ is the cost attached to the availability of subsystem $S_i$ in the total configuration. System maintenance ratio is $m = MTTR/MTBF$ such that $u = 1/m$ and $r = m$.

Now let $\mu_i$ be the MTBF for subsystem $S_i$ in the system configuration. Proportional costs in maintenance then can be measured and displayed as configuration averages like $\overline{\text{MTTR}} \propto \Sigma_i \Delta_i / \Sigma_i m$ and the time scale or number of test interventions can be measured as $\overline{\text{MTBF}} = \Sigma_i \mu_i / \Sigma_i m$.

Fixed costs resulting from all effects (psychological among others) of the need for preventive maintenance interventions taking the whole configuration from the system user, can be measured with the following coefficients for instance : when only OFF-LINE protests are available : 10 hours per week, when also ON/IN-LINE protests are available : 2 hours per week, when OFF and THROUGH-LINE protests are available : 5 hours per week.

<figure><img src="image_26.png" /><figcaption>Fig. 25. Comparative cost display for peripheral subsystem test in maintenance usage.</figcaption></figure>
As a matter of fact, notice one of the major areas of application of such a test system should be the in-plant hierarchical network of computers for integrated automation of inventory control, process control and quality control itself.
A. TEST SYSTEMS

[1] R. SUDA. "An application-oriented multiprocessing system. PART V. The Diagnostic Monitor". IBM Systems Journal. Vol. 6. N° 2.pp. 116-123. (1967).

[2] G.W. NELSON. "OPTS-600 — on-line Peripheral Test System". AFIPS. FJCC.33. pp. 45-50. (1968).

[3] M.S. HOROVITZ. "Automatic Check-out of small computers". AFIPS. SJCC. 34. pp. 359-365. (1969).

[4] N. BARTOW, R. MCGUIRE. "System/360 model 85 microdiagnostics". AFIPS.SJCC.36. pp. 191-197.a. (1970).

[5] A.M. JOHNSON. "The microdiagnostics for the IBM System/360 model 30". IEEE-C-20. N° 7. pp. 798-803. (July 1971).

[6] J.W. IRWIN, J.V. CASSIE, H.C. OPPEBOEN. "The IBM 3803/3420 Magnetic Tape subsystem". IBM J. of R. and D. Vol. 15. pp. 391-400. (September 1971).

[7] P.J. SCOLA. "On-line computer-Testing system". Second workshop on "Fault-detection and diagnosis in Digital Systems". LEHIGH University. pp. 103-114. (December 1971).

[8] G.R. AHEARN, Y. DISHON, R.N. SNIVELY. "Design innovations of the IBM 3830 and 2835 Storage Control Units". IBM. J. of R. and D. pp. 11-18. (January 1972)

[9] IBM Systems. "OS/VS OLTEP : On-line Test Executive Program". GC 28-0636-1. (1972).

[10] M.A. CALHOUN. "SYMBOL hardware debugging facilities". AFIPS.SJCC. 40. pp. 359-368. (1972).
[11] G. EDGE. "System 250-Diagnostics". pp. 201-212. in Conference on Computers : Systems and Technology. LONDON. IERE Conference Proceedings, n° 25. Middlesex Hospital Medical School. (October 1972).

[12] H.Y. CHANG, R.C. DORR, D.J. SENESE, T.L. SMITH. "Maintenance Techniques of a microprogrammed self-checking Control Complex for an electronic Switching System". IEEE-C-22. N° 5. pp. 501-512. (May 1973).

[13] D.R. FREESE. "Streamlining insertion of components in printed-circuit boards". ELECTRONICS. Vol. 47. N° 18. pp. 117-122. (September 5, 1974).

[14] P. DECITRE, M. DEPEYROT, A. JORRY, M. MOALLA. "The COSU through-line test-process interpreter". Rapport LABORIA. To be published. (September 1974).
B. VISIBILITY.

[1] D.M. BRADLEY. "The Systems Approach to Test Evaluation". IEEE-SSC-5, N° 3. pp. 223-230. (July 1969).

[2] J.E. STUEHLER. "An integrated Manufacturing Process Control System : Implementation in IBM Manufacturing". IBM J. of R. and D. Vol. 14. N° 4. pp. 605-613. (November 1970).

[3] D.L. DROULETTE. "Recovery through programming System/360-System/370". AFIPS. SJCC. 38. pp. 467-476. (1971).

[4] A.N. HIGGINS. "Error recovery through programming". AFIPS. FJCC. 33. pp. 39-43. (1968).

[5] A.S. KAMP, J.J. LYDON. "System for Utilization of Remote Equipment for Failure Investigation and Reliability Evaluation (SUREFIRE)". Annals of Reliability and Maintainability. pp. 279-289. (1970).

[6] F. BANCILHON et al. "Principes de Conception Automatique dans le Système SPECTRE". RAIRO. 7ème Année. B-3. pp. 31-62. (Octobre 1973).

[7] D.A. ENTREKIN. "Factory Automation and Data Collection". AFIPS. NCC. 42. pp. M.83-M.87. (1973).

[8] J.S. LAMB. "Computer-Aided Manufacturing : Today and Tomorrow". AFIPS. NCC. 42. pp. M.88-M.91. (1973).

[9] L.R. KNEPPELT. "A simple distributed systems approach to Manufacturing information systems". AFIPS. NCC. 43. pp. 485-490. (1974).

[10] S.J. OH, J.L. HAMMOND. "Computer Control in manufacturing — An annotated survey of the state of the art—". Joint Automatic Computer Conference. pp. 44-46. (1972).
C. SURVEYS.

[1] R.E. BARLOW, F. PROSCHAN, L.C. HUNTER. "Mathematical Theory of Reliability".
The SIAM Series in Applied Mathematics. WILEY. (1965).

[2] W.H. PIERCE. "Failure-Tolerant Computer Design". ACADEMIC PRESS. (1965).

[3] C.V. RAMAMOORTHY. "A structural Theory of machine diagnosis". AFIPS.SJCC. 31. pp. 743-756. (1967).

[4] J.M. SMITH. "A review and comparison of certain methods of computer perfor-
mance evaluation". The Computer Bulletin. pp. 13-18. (May 1968).

[5] J.J. DENT. "Diagnostic Engineering Requirements". AFIPS. SJCC. 32. pp. 503-507. (1968).

[6] M.E. DRUMMOND. "A perspective on system performance evaluation". IBM System Journal. Vol. 8. N° 4. pp. 252-263. (1969).

[7] H.Y. CHANG, E.G. MANNING, G. METZE. "Fault Diagnosis of Digital Systems".
N.Y. WILEY.' (1970).

[8] A.D. FRIEDMAN, P.R. MENON. "Fault-Detection in Digital Circuits". PRENTICE
HALL / Electrical Engineering Series. (1971).

[9] H.T. Mc Aleer. "A look at automatic testing". IEEE. SPECTRUM. pp. 63-78.
(May 1971).

[10] R.G. BENNETTS, D.W. LEWIN. "Fault Diagnosis of Digital systems - a review".
The British Computer Journal. Vol. 14. N° 2. pp. 199-206. (May 1971).

[11] F. VAN VEEN. "An introduction to IC Testing". IEEE.SPECTRUM. pp. 28-37. (December 1971).
[12] R.J. PREISS. "Fault Test Generation". Chap. 7 in "Design Automation of digital systems". Vol. I. Edited by M.A. BREUER. PRENTICE HALL. (1972).

[13] W.C. CARTER, W.G. BOURICIUS. "A survey of fault-tolerant Computer Architecture and its evaluation". IEEE.COMPUTER. pp. 9-16 (January-February 1972).

[14] J.J. HORNING, B. RANDELL. "Process Structuring". ACM Computing Surveys. Vol. 5. N° 1. pp. 5-30. (March 1973).

[15] M. DEPEYROT. "Sur la synchronisation et la fiabilité des automates". RAIRO. 5e Année, B-2. pp. 87-110. (Décembre 1971).
[1] E.F. MOORE. "Gedanken-Experiments on Sequential Machines". pp. 129-156. in "Automata Studies". Ed. by C.E. SHANNON, J. Mc CARTHY. Annals of Mathematics Studies. N° 34. PRINCETON UNIVERSITY PRESS. (1956).

[2] C.V. RAVI. "Fault-location in memory systems by program". AFIPS. SJCC. 34. pp. 393-401. (1969).

[3] M. COHN, G. OTT. "Design of Adaptive Procedures for Fault-Detection and Isolation". IEEE-R-20. N° 1. pp. 7-10 (February 1971).

[4] B.E. COWART, R. RICE, S.F. LUNDSTROM. "The physical attributes and Testing Aspects of the SYMBOL system". AFIPS. SJCC. 38. pp. 589-600. (1971).

[5] C.W. HEMMING, S.A. SZYGENDA. "Modular requirements for digital logic Simulation at a predefined functional level". Proceedings of the National ACM Conference. pp. 380-389. (August 1972).

[6] M.A. BREUER. "Testing for intermittent faults in Digital Circuits". IEEE-C-22. N° 3. pp. 241-246. (March 1973).

[7] S.A. SZYGENDA, M.J. FLYNN. "Self-diagnosis and self-repair in memory : an Integrated System approach". IEEE-R-22. N° 1. pp. 2-12. (April 1973).

[8] J.D. RUSSELL, C.R. KIME. "On the diagnosability of digital Systems". in Proceedings of the "Fault-Tolerant Computing Conference". pp. 139-144. (June 1973).

[9] M. DEPEYROT. "Design of interfaces as stochastic channels with memory". GRENOBLE International Workshop on Computer Architecture. (June 1973).

[10] C. KUBIAK, J. LE SAINT-MILON, M. PIGNAL. "Méthode de simulation de transition dans les réseaux logiques". AUTOMATISME. Tome XVIII. N° 11. pp. 399-409. (Novembre 1973).

[11] A. TOTH, C. HOLT. "Automated Data-Base driven Digital Testing". IEEE. COMPUTER. pp. 13-19. (January 1974).
[12] U.W. POOCH. "Translation of Decision Tables". ACM Computing Surveys. Vol. 6. N° 2. pp. 125-151. (June 1974).

[13] F. BANCILHON, M. DEPEYROT. "Identification des automates non-fiables". RAIRO.8e Année, B-2. pp. 127-153 (Juin 1974).
[1] MIL-STD-105D. "Military Standard : Sampling procedures and tables for inspection by attributes". Department of Defense - USA. (July 29, 1963).

[2] W.B. JOYCE. "Organizations of Unsuccessful R. and D. Projects IEEE-EM-18. N° 2. pp. 57-65. (May 1971).

[3] L.A. BELADY, M.M. LEHMAN. "An introduction to Growth Dynamics". Brown University Conference on Statistical Methods for the evaluation of Computer System Performance. (1971).

[4] L.A. BELADY, M.M. LEHMAN. "Programming System Dynamics or the metadynamics of Systems in Maintenance and Growth". IBM. Research Report. RC.3546. (September 17, 1971).

[5] R.E. MERWIN. "Estimating Software Development Schedules and Costs". IXth DESIGN AUTOMATION WORKSHOP. IEEE pp. 1-4. (July 1972).

[6] B.P. DZIELINSKI. "A guide to financial planning tools and Techniques. IBM Systems Journal. Vol. 12, N° 2. pp. 126-144. (1973).

[7] M. AISO. "Forecasting Techniques". IBM Systems Journal. Vol. 12. N° 2. pp. 187-209. (1973).

[8] A. JORRY. "Introduction to ORDON". Proceedings of the 4th International Workshop on Real-time Programming. IFAC-IFIP. BUDAPEST. (May 1974).

[9] A.R. PIERCE. "Bibliography on algorithms for shortest path, shortest spanning tree and related circuit routing problems, (1957-1973)". ACM-SIGDA-Newsletter. Vol. 4. N° 2. pp. 29-37. (June 1974).

[10] Significant Developments in Technology and Business : "Test and Measurement service concept developing". ELECTRONICS. Vol. 47. N° 14. pp. 3E-4E. (July 11, 1974).

[11] B. CHEW. "Scheduling an IC tester for maximum throughput". ELECTRONICS. Vol. 47. N° 17. pp. 96-99. (August 22, 1974).