December 1

Testing of Interaction-D

Manufacturing System

KC Morris

David Flater

Don Libes

Al Jones

Manufacturing Systems Integration Division

National Institute of Standards and Technology

Department of Commerce

NISTIR 6260

Testing Interaction-Driven Manufacturing Software 

Testing of Interaction-Driven Manufactu

Systems

Table of Contents

Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Abstract. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

The TIMS Project . . . . . . . . . . . . . . . . . . . . . . . . . . 

Manufacturing Interface Specifications . . . . . . . . . . 

Scope of Testing for TIMS . . . . . . . . . . . . . . . . . . . 

Manufacturing Software Environment . . . . . . . . . . . . . . . . . 

Interaction-driven Systems . . . . . . . . . . . . . . . . . . . 

Modeling System Behavior . . . . . . . . . . . . . . . . . . . 

Representation of Manufacturing Systems . 

Models for Evaluating Manufacturing System

Manufacturing Software Testing . . . . . . . . . 

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Testing of Interaction-driven Manufacturing Software. . . . . 

Component Testing . . . . . . . . . . . . . . . . . . . . . . . . . 

Manufacturing Interfaces . . . . . . . . . . . . . . . 

Existing Testing Efforts . . . . . . . . . . . . . . . . 

Test Techniques . . . . . . . . . . . . . . . . . . . . . . 

Future Directions . . . . . . . . . . . . . . . . . . . . . 

Integration and System Testing . . . . . . . . . . . . . . . . 

System-Level Manufacturing Specifications 

Existing Testing Efforts . . . . . . . . . . . . . . . . 

Integration / System Test Techniques . . . . . 
ii Testing Interaction-Driven Manufacturing Software 

Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Interoperability Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

The Reasons for Interoperability Testing . . . . . . . . . 

Public Relations and Other Political Implications . . 

Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Special Noteworthy Problems . . . . . . . . . . . . . . . . . 

Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Conclusions and Future Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Testing Interaction-Driven Manufacturing Software 

Abstract

As automation of manufacturing processes becomes more wides

turing systems are becoming more and more software dependen

time, the software used is becoming more and more modularized 

creation of customized systems consisting in large part of pre-ex

nents. This combination of factors leads to considerable flexibili

ing systems, but not without a cost. The reliability of those syst

due to the lack of experience in how to test such systems. To add

National Institute of Standards and Technology’s Manufacturing 

tion Division has undertaken a study of how to test “interaction-d

ing systems.” This paper contains an analysis of both the proble

solution space for testing these systems and is the first outcome 

Introduction

Manufacturing systems are inherently distributed and heterogene

designed and manufactured by a range of people with different s

ety of systems specialized for different functions. Not long ago, 

opment process (spanning the product life-cycle from design to 

characterized by islands of automation as automation was applie

computation-intensive tasks in a product’s development. More re

effort has gone into bridging the islands of automation and strea

opment process. In many industries this streamlining has becom

as to minimize the actual time from raw material to point-of-sale

are still working to reach this minimum.

In such an environment, system reliability is of utmost importan

can interrupt the entire chain of events and delay the delivery of 

ufacturers have traditionally planned down time for preventive m

machines and had back up machines for those in need of repair, 

environment, the physical machines are a lesser risk than the sof

the processes. Techniques for maintaining and repairing softwar

not as reliable as for their hardware counterparts. Furthermore, s
2 Testing Interaction-Driven Manufacturing Software 

interchangeable as hardware in that it may maintain knowledge about th

system which is in constant change, as well as historical knowledge abo

tem.

Maintenance and repair are one aspect of a bigger problem which involv

usability and reliability of an individual software component in a larger 

How does one diagnose failure of a software component? How is a com

impacted by the system of which it is a part and how does it impact the 

How much tolerance is there for component performance? What is the i

system failure? Can another component be swapped in when one fails? 

interaction between the components result in an unreliable system state

questions are all essential to implementing a reliable product developme

To answer these questions, one needs methods and a framework in whic

software components and the systems in which they operate.

The TIMS Project

To this end, NIST has initiated an effort to develop competence in the te

software systems. The Manufacturing Systems Integration Division (MS

focus more specifically on the testing of interaction-driven manufacturin

(TIMS). Interaction-driven manufacturing systems are those composed o

software components in which the interactions between those compone

automated. Automation is typically achieved through the definition of p

interfaces, both standard and proprietary, which allow the components t

interact without the need for human intervention at every step along the 

Today’s manufacturing software, and software systems in general, are b

ently than they were a decade ago. While NIST has established compete

traditional sequence of unit, integration, and system testing,2 what we o

“systems” are now single components in larger, integrated, distributed s

1. Such interfaces are often referred to as API’s, or Application Program Interfaces, where the 

is anything that happens to make use of the software component. API’s include the specificati

calls or commands as well as context-specific rules for the use of those commands. Together

the specifications governing the interactions of the system.

2. NBS Special Publication 500-98, Planning for Software Validation, Verification, and Testing

1982.
Testing Interaction-Driven Manufacturing Software 

The best of practice techniques3,4 have become dated, and the b

business are scrambling to come to grips with the reality of syste

ditional terminology, would have to be called “large and comple

tems of systems.”5

 We, too, must make this step and revise our t

Unlike manufacturing systems of the past, which were only desi

throughput, today’s systems are being designed for flexibility wh

mising throughput. Instead of building systems from scratch, ma

seeking to be able to integrate off-the-shelf software component

system with minimum expense on custom programming. To do 

be able to upgrade or replace individual components without bre

In reaction to this need, manufacturers are now paying more atte

tems and standards.6,7

Open systems are based on the premise that if interfaces are clea

defined (i.e., standard) then systems can be modified incrementa

newer, better functionality as it emerges in the form of improved 

nents. A host of software standards, both formal and de facto, ha

port these types of architectures.

Manufacturing Interface Specifications8

At the initiation of the TIMS project, two specifications had been 

able case studies for understanding the needs of testing interacti

facturing systems:

3. Boris Beizer, Software Testing Techniques (second edition). Van Nostrand Reinhol

4. Boris Beizer, Software System Testing and Quality Assurance. Van Nostrand Reinh

5. Genevieve Houston-Ludlam, in “Call for Papers, Testing Computer Software Con

<URL:news:comp.software.testing>, September 1, 1998.

6. Richard Kuhn, William Majurski, Wayne McCoy, Fritz Schulz, “Open Systems So

Concurrent Engineering,” Advances in Control and Dynamic Systems, vol. 62, Ac

7. Jeanine Katzel, “Moving Down the Path to Open Systems,” Plant Engineering On

<URL:http://www.manufacturing.net/magazine/planteng/archive/1997/ple0901.97

8. Names of companies and products are used in order to adequately specify the info

herein. In no case does such identification imply recommendation or endorsement 

of Standards and Technology.
4 Testing Interaction-Driven Manufacturing Software 

• STEP’s9 Data Access Interface (SDAI)10 and

• the emerging Process Specification Language (PSL).11

Subsequently a number of other manufacturing specifications have been 

as potential candidates for study:

• Manufacturing Message Specification (MMS),12,13

• the CIM (Computer-Integrated Manufacturing) Framework,14

• the Advanced Process Control (APC) Framework,15

• Object Management Group’s16 Product Data Management (OMG’s P

Enablers,17

• STEP’s Application Protocols (AP) using SDAI,

• CAM-I’s Application Interface Specification (AIS),18 and

• OLE (Object Linking and Embedding) for Design and Modeling.19

9. The Standard for The Exchange of Product Model Data (STEP) is a project of the Internatio

tion for Standardization (ISO) Technical Committee on Industrial Automation Systems and 

(TC184) Subcommittee on Industrial Data (SC4).

10.ISO 10303 Industrial automation systems and integration—Product data representation and 

Part 22: Implementation methods: Standard data access interface, International Organizatio

ization, Draft International Standard, 1998.

11.Process Specification Language home page. <URL:http://www.mel.nist.gov/psl/>.

12.MMS Information Server. <URL:http://litwww.epfl.ch/MMS/mms_main.htm>.

13.ISO/IEC 9506-1 Industrial automation systems - Manufacturing Message Specification, Par

definition and ISO/IEC 9506-2 Industrial automation systems - Manufacturing Message Spec

2: Protocol specification, International Standard, International Organization for Standardiza

14.SEMATECH’s CIM Framework Home Page, <URL:http://www.sematech.org/public/divisio

home.htm>.

15.Advanced Process Control (APC) Framework Initiative 1.0 Specifications, 1997. <URL:http

ech.org/public/docubase/abstract/3300aeng.htm>.

16.Object Management Group Home Page. <URL:http://www.omg.org/>.

17.Revised Submission (including errata changes) — PDM Enablers — Joint Proposal to the O

Response to OMG Manufacturing Domain Task Force RFP 1. <URL:http://www.omg.org/ar

02-02.pdf>, 1998.
Testing Interaction-Driven Manufacturing Software 

These specifications are intended to serve as standard definitions 

software supporting a manufacturing process. The specifications 

egories:

• infrastructures used to connect the system,

• applications used in the system, and

• the operation of the system itself.

Manufacturing systems rely on software specifications of many 

these specifications are generic in that they support many types o

not restricted to manufacturing. A 1995 report from MSID surve

vant to manufacturing systems integration.20 The report identifie

classes of standards: computing technology, industrial practices, 

ing equipment. The category of industrial practices contains the 

which are unique to interaction-driven manufacturing systems an

the focus for the TIMS project. While there are many infrastruct

dards (the computing technology category) relevant to manufact

facturing specific specifications in this category are MMS, poten

SDAI.

A specialized subsystem, such as a database, geometry engine, o

provides a particular function needed by one or more application

The subsystems typically provide an application program with in

their integration into the larger system. While many, if not most, 

systems provide application program interfaces, only a few of th

recognized as standards through an open consensus-based proce

interfaces include CAM-I’s AIS, OMG’s PDM Enablers, and OL

Modeling. Other public specifications in this category are expec

the near future. OMG is working on interfaces in the areas of M

18.Application Interface Specification (AIS), Version 2.1: Volume I, ‘Functional Spec

‘C Language Binding’, Report R-94-PM-01, Consortium for Advanced Manufactu

(CAM-I), Inc., Bedford, TX, USA (April 1994).

19.“What is OLE for Design and Modeling?” Design and Modeling Applications Cou

<URL:http://www.dmac.org/whatis/whatis.htm>. 1998.

20.Ed Barkmeyer, et al., SIMA Background Study, NISTIR 5662, September 1995.
6 Testing Interaction-Driven Manufacturing Software 

cution Systems, Enterprise Resource Planning, and Machine Control. IS

recently approved SDAI as a Draft International Standard, which is exp

result in implementations of Application Protocols for STEP using an in

driven interface.

Finally, specifications of the systems themselves guide how component

they be infrastructural or specialized, are supposed to interact to suppor

system. System specifications provide a framework for the appropriate s

of events in the system. Specifications of manufacturing systems includ

ECH’s CIM Framework and the APC Framework.

Scope of Testing for TIMS

Software testing has different meanings depending on its purpose. Its br

pose is to find problems in order to improve the robustness and reliabilit

ware product. It can be used for the sole purpose of improving products o

used as a means of evaluating products by distinguishing their differenc

distinguish three purposes of testing and the roles of the participants:

• Testing for product release: this type of testing is focused on finding 

for the purpose of debugging a product which is to be distributed to o

users. Both white box and black box testing can and should be used f

pose.

• Testing for product evaluation: this type of testing is done by an orga

the process of designing and implementing a system with specific fun

and performance requirements.22 In this case the testing of the softw

formed under the auspice of the organization that will be using the so

is not responsible for the development of the software.

21.Rick Hower published a rather exhaustive listing of the various types of testing at <URL:htt

www.charm.net/~dmg/qatest/qatfaq1.html#FAQ1_10>.

22.Kurt C. Wallnau, David Carney, Bill Pollak, “How COTS Software Affects the Design of CO

Systems,” SEInteractive, June 1998. <URL:http://interactive.sei.cmu.edu/Features/1998/Jun

COTS_Software/Cots_Software.htm>
Testing Interaction-Driven Manufacturing Software 

• Testing for product certification: this type of testing supports 

products. It is based on the premise that standards can be use

ware such that multiple vendors can provide software that can 

changeably by end-user systems. Testing against such standar

interface specifications, performance parameters, security pro

provides a measure of the software’s quality against an indepe

The verification that the software does support the standards 

confidence in the software.

As an independent organization, NIST is in a unique position to 

and tools which support all categories of testing and to address t

of testing specifically. The TIMS project will investigate what ty

and tools will be needed to support testing of interaction-driven 

systems and whether there is a role for NIST in certifying these 

The anticipated results of the project are methods and tools whic

those involved with software to support interaction-driven manu

The TIMS project examines testing from the perspective of syste

this context the goal of testing is focused on

• whether components reliably support the interface specificati

• whether the specifications are complete and consistent, and

• whether the system can be constructed using multiple instanti

ifications — in other words, using different vendors’ product

Much of the testing process can by executed by simulation of th

of the system in concert with actual components.

The remainder of this paper discusses some of the issues surroun

raised. It provides background material for the initiation of the p

with a description of characteristics of software supporting manu

followed by an analysis of how to test interaction driven manufa

We discuss three categories of testing — component, system, an

ity—that are understood within the new context of “large and co

systems of systems.” Each of these categories is described with r

ufacturing standards that fall within the category, the state of the 

these types of interfaces (often by analogy to similar interfaces i
8 Testing Interaction-Driven Manufacturing Software 

ware world), and finally a summary of the state of the art in testing techn

applies to the category, to the extent that we know it.

Manufacturing Software Environment

“The first line of defense against these bugs is the design. The first basti

defense is that there be a design for the overall software architecture. Fa

create an explicit software architecture is an unfortunate but common d

—Boris Beizer

The first step in testing an interaction-driven manufacturing system is to 

the components of the system. These should be clear from the system ar

The components in a typical manufacturing system will include several 

cial off-the-shelf (COTS) products which may support open or proprieta

faces. Components may be both infrastructural or specialized and range 

machine control devices through scheduling, inventory, and planning sy

Other components in the system will be custom built applications that in

with and bridge the gaps between these vendor-supplied products. Final

human or end-user can be considered a component in the system as wel

much of the system control is under user guidance. Simple component d

are often used to illustrate the system architecture. More formal Archite

Description Languages (ADL)23,24,25 are emerging. More detailed and f

system models can also be constructed using various aspects of the Unifi

ing Language (UML).26

The components in an architecture are connected by interfaces of variou

The TIMS project is concerned with the interfaces that can be automate

23.The RapideTM Language, Stanford University, <URL:http://pavg.stanford.edu/rapide/>.

24.The Acme Architecture Description Language, Carnegie Mellon University, <URL:http://ww

~acme/>.

25.SADL: A Structural Architecture Description Language, SRI Computer Science Laboratory

www.csl.sri.com/dsa/sadl-main.html>.

26.Unified Modeling Language, version 1.1. <URL:http://www.rational.com/uml/documentatio

tember 1, 1997.
Testing Interaction-Driven Manufacturing Software 

way or another. However, it is worth noting that a system archite

other more static interfaces such as the exchange of data based o

This is a very common type of interface in today’s manufacturin

system architecture can be divided into subsystems where these 

serve as borders. Components whose interfaces are all implemen

data exchange can be tested as stand-alone systems.27

Interaction-driven Systems

To facilitate the discussion of appropriate test methods for manu

we present an analysis of the style of interactions seen in such sy

lowed by some background on the modeling of system behavior

Just as data flow diagrams can be used to identify data sources a

kind of diagram can be used to identify sources and sinks for int

tivity is the activity of interacting, or the business of interaction. 

more than just reaction; hence, interactivity is more than just the 

sages. It represents a possible source of non-determinism in the 

critical factor in the system’s testability.

Flows of interactivity are used to show the sources of non-determ

ways that this non-determinism ripples through the system. By d

representing interactivity flow in one of the following diagrams, 

the component at the start of the arrow is perturbing the compon

the arrow in a partially or completely non-deterministic way. Th

could be a data flow, a control flow, a message, an event, or som

does not matter. These interactivity flow diagrams therefore diff

27.ISO 10303 Industrial automation systems and integration—Product data represen

Part 31:Conformance testing methodology and framework: General concepts, Inte

International Organization for Standardization, 1994.
10 Testing Interaction-Driven Manufacturing Software 

diagrams and similar diagrams because they contain information about t

of non-determinism and the parts of the system that are affected by it.

FIGURE 1. Simple interactivity diagram

In a classic client-server architecture such as the World Wide Web (mod

above), all components are either sources or sinks of interactivity. Becau

not have complex dependencies on other components, sources and sinks 

tested as stand-alone systems by replacing the components on the other 

interactions with a simple test harness.

The non-determinism in this system is that the arrival times of requests 

browsers and the contents of those requests are random as far as the web 

concerned. Because web servers are designed to be stateless, this would 

an insignificant observation. However, if we instead have a manufacturi

where orders for products and machine control commands are being ent

through web interfaces, the interactivity coming from the web browsers 

be sufficient to trigger timing-related failures in the system.

Browser

(source)

Browser

(source)

Browser

(source)

Web server

(sink)
Testing Interaction-Driven Manufacturing Software 

FIGURE 2. Hierarchical interactivity diagram

Components that act as both client and server create more difficu

system as a whole. However, some multi-tier systems are designe

be decomposed and tested as independent subsystems. The open 

archy shown above is an example of such a system. The interact

flowing downwards from the Guardian (a user interface compon

browser), so the behavior of each subsystem is determined by the 

test methods can therefore remain typical of those used by the so

for large software projects.

FIGURE 3. Cyclic interactivity diagram

Guardian

Controller Controller Con

Machine Machine Ma

Guardian

(source)

Databa

(sink

Shop

control

Workcell

control
12 Testing Interaction-Driven Manufacturing Software 

In this example, the shop and workcell controllers each have their own t

control and may initiate interactions at any time. When interactivity bec

cyclic as it has here, a new category of problems such as deadlocks, rac

tions, and inconsistent world views is introduced. These problems have 

explored extensively in network testing; special considerations for the c

interaction-driven systems will be described later in this document.

Systems having cyclic interactivity are inherently more difficult to test t

with acyclic interactivity because merely controlling the top-level sourc

activity is no longer sufficient to remove non-determinism from the syst

there may still be uncontrolled interactions at lower levels.

FIGURE 4. Chaotic interactivity diagram

Finally, there exist non-traditional architectures that use large numbers o

ing agents interacting chaotically to produce an emergent behavior. The

difficult to test because they are highly non-deterministic and the permis

spaces of the systems are not well defined.

Agent

Agent

Agent Agent

Agent

Agent
Testing Interaction-Driven Manufacturing Software 

Modeling System Behavior

Formal techniques for modeling the behavior of manufacturing s

in use for many years. These techniques are equally applicable f

software systems when the behavior of the software is fairly sim

software components are integrated into ever larger, interaction-

modeling becomes increasingly problematic. This section summ

approach for formal modeling of behaviors and common approa

such systems and explains the roadblocks to using it on complex 

systems.

Representation of Manufacturing Systems

• State Space Representation for Continuous Time Systems

Typically, we describe continuous time systems at any time t wit

x(t). x can be a single variable or a vector of variables. To mode

lution, we define a state transition function, f (•). This function u

rent state at time t, x(t), and the current control law, u(t), to predi

t+δ. That is

x(t+δ) = f (x(t), u(t)) t ∈[t 0, T]. 

The simplest and most widely used form for this function is line

x(t+δ) = Ax(t) + Bu(t)) t ∈[t 0, T].

The actual output from the system is called y(t), which is generat

function, g(t). This output function uses both x(t) and u(t),

y(t) = g (x(t), u(t)) t ∈[t 0, T]. 

Optimal control theory addresses the formulation of the “optima

for given problem and performance criteria. For example, a robo

located at a given position and orientation at time t0. The proble
14 Testing Interaction-Driven Manufacturing Software 

end effector at another location and orientation by time T. The performan

may be to minimize total energy consumption, minimize total time, etc. 

mal control policy would define the trajectory that is to be followed by t

effector. Subsequently, specifications are determined for managing the m

the various linkages in order to realize the desired trajectory.

As previously described two types of control policies exist: open loop a

loop. If the policy is open loop, the trajectory is initiated at time t0, term

time T, and does not change in between:

u(t+δ) = u(t) t ∈[t 0, T]. (3)

If the policy is closed loop, then feedback information, as denoted by y(

vided at a series of points, t=ti

, i=1,...k, between t0 and T. This feedback 

used to evaluate, and possibly change, the trajectory at any of these time

u(t+δ) = u (y(t), u(t)) t ∈[t 0, T]. (4)

At this point, we have said nothing about the parameters of the functions 

above. They may or may not have the time variable “t” as a parameter. I

not, the system is said to be time invariant. Most real-world systems, ho

time variant. If a system is time variant, both the state transition and out

tions will be dependent on time:28

x(t+δ) = f (x(t), u(t), t) and y(t) = g (x(t), u(t), t). t ∈[t 0, T].(5)

• State Space Representation for Discrete Time Systems

Many manufacturing systems have state variables that are not continuou

the state variables change only at discrete points in time. Two important 

are queuing systems and inventory systems. The state variables for thes

28.J. Reid, Linear System Fundamentals: Continuous and Discrete, Classic and Modern, McGr

Company, New York, New York, 1983.
Testing Interaction-Driven Manufacturing Software 

change only when there is a new arrival or a new departure. The d

equations (1) - (4) are given below.

x(tk) = f (x(tk-1), u(tk-1)) k=0,1,...,n 

y(tk) = g (x(tk-1), u(tk-1)) k=0,1,...,n 

u(tk) = u (y(tk-1), u(tk-1)) k=0,1,...,n 

Equation (8) represents the control policy, which can be either op

Open loop is a special case, since y(tk-1)=0, and u(tk)= u(tk-1) fo

Models for Evaluating Manufacturing Systems

• Simulation Models for Manufacturing Systems

Simulation software has been used to model different processes 

within the manufacturing enterprise for many years. The most im

in manufacturing simulation is state variable. State variables de

pening in the process or system at any point in time. Continuous 

are used for state variables that change continuously over time. T

mainly mathematical, differential, or difference equations that re

tion of some physical phenomena, which changes continuously 

are used primarily during product design and process selection. 

fluid and structural dynamics, stress analysis, heat transfer, and m

gram verification.

Event and process modeling methodologies can be used only for 

that change at discrete points in time. Examples of such variable

ber of jobs waiting in the queue in front of a machine, the status 

on the shop floor, and the location of each job in the factory. The 

els that are built using these methodologies are mainly flow mode

flow of entities through the factory. In the case of discrete-event 

tracking is done using times at which the various events occur. I
16 Testing Interaction-Driven Manufacturing Software 

tions, the tracking is done based on the resources utilized at the various s

process.29

• Finite State Machine Models for Manufacturing Systems

Another common way of modeling discrete time systems, and discrete a

tions to continuous time systems, is with a state machine. A state machi

defined using a 6-tuple,30

{I, O, S, Y, F, s0}

where I is the set of Input conditions, O is the set of Output events, S is 

all possible States, Y is a mapping from S × I → O, F is a mapping from 

and s0, s0 ∈ S, is the initial state. The set of possible states, S, can be fin

countably infinite. If the system is under a closed loop policy, then I typ

tains feedback information, F contains rules for the determining the nex

represents the new control policy, and O the actions required to implem

policy.

State machines are often visualized as a state table or state transition gra

example is provided in Figures 5 and 6. Given the current state, and the 

inputs, the next state and the output events can be determined completel

table or graph. Figure 7 shows how simply this can be implemented.31

Despite their simplistic nature, state machines can become unmanageab

applied to real world systems. The cardinality of S × I can grow very qu

the mappings Y and F can be quite complex. Frequently, the principles 

chy and abstraction are used to overcome these problems. We can aggre

29.Ibid.

30.B. Selic, G. Gullekson, and P. Ward, Real-time Object Oriented Modeling, John Wiley & So

York, New York, 1994.

31.The figures are reproduced from J. Albus, C. McLean, A. Barbera, and M. Fitzgerald, “An A

Real-time, Sensory-interactive Control of Robots in a Manufacturing Facility,” in Proceedin

ference on Information Control Problems in Manufacturing Technology, National Bureau of 

Gaithersburg, MD, pp. 81-89, 1982.
Testing Interaction-Driven Manufacturing Software 

into a single composite state using an abstraction process. This p

details of the original states and makes it possible to decompose 

machine into a hierarchy of simpler machines. This means that a 

description of a real-world system can be developed in stages. T

description can be initiated as a single, abstract machine; decom

archy of abstract machines; then gradually refined into the requi

level.

FIGURE 5. Example state transition graph
18 Testing Interaction-Driven Manufacturing Software 

FIGURE 6. Corresponding state-transition table
Testing Interaction-Driven Manufacturing Software 

FIGURE 7. Computing structure to execute state-transition tables

Manufacturing Software Testing

• Software Programs as State Machines

State machines can be used to model the behavior of software pr

ber of ways. In its simplest form, a state represents the valid valu

program variable. While this may be a valid, formal description 

is of little practical use. This is true because even for elementary 

few variables, there can be an exponential explosion in the size o

Therefore, it is necessary to have another, more abstract, definiti

which a change in the value of any variable does not automatical
20 Testing Interaction-Driven Manufacturing Software 

in state. Such a definition must capture the qualitative behavior of the pr

without direct reference to the quantitative values of specific variables.3

Hierarchical state machines can also be used to model certain aspects of 

programs. The resulting nesting of lower-level state machines within mo

states is similar to the nesting in block-structured procedural programm

guages. Moreover, the scoping rules are the same:

1. each nesting has a distinct lexical scope with its own variabl

declarations,

2. higher levels cannot access lower level scopes, and

3. lower levels can access all of their containing scopes.

There is one important difference between nested states and nested bloc

states are static, but nested blocks are dynamically created and destroye

program executes.

This notion of a software program as a state machine has been used, wit

success, as the basis for unit testing.33 A fixed set of inputs is generated

manually or automatically, for which the desired state transitions and ou

are known. Determining an appropriate set of inputs, based on some spe

of program behavior, is itself a complex problem. Nevertheless, given su

program errors are said to occur whenever the actual state transitions or 

ues differ from the desired ones. Whenever errors occur, the source is fo

corrected (another difficult problem), and further testing is done. This sc

continues until no further errors are detected, or until some predetermin

ping condition is met.

32.Selic, 1994.

33.David Banks, William Dashiell, Leonard Gallagher, Charles Hagwood, Raghu Kacker, Lyn

Software Testing by Statistical Methods - Preliminary Success Estimates for Approaches B

mial Models, Coverage Designs, Mutation Testing and Usage Models. NISTIR 6129, <U

sunsrv1.ncsl.nist.gov/~ftp/stsm/mar98ir.pdf>, March 12, 1998.
Testing Interaction-Driven Manufacturing Software 

• Systems of Systems

Problems arise when one attempts to use this approach to test ce

facturing software applications — namely, those that rely directl

“source” applications for their inputs and those with inputs that a

that happen in the real world. These are interaction-driven applic

quently, either we need the source application/real-world system 

testing, or we need substitutes that behave exactly like them. In 

substitutes themselves must be tested to ensure correct behavior

case, two scenarios arise.

If a source application provides the inputs, that application can b

finite state machine with its own set of transition points. If the in

real-world system, that system can be continuous time or discret

tem is continuous, its evolution is governed by equations (1-4); i

then equations (6-8) apply. This causes a problem known as raci

system being tested must race to complete its execution before th

impact of racing on testing is not known.

In many cases, outputs from the test system can be inputs to the 

or cause additional changes in the real world. This frequently oc

system and its source have a hierarchical relationship. When thi

put from the supervisor becomes input to the subordinate and fe

subordinate becomes input to the supervisor. In manufacturing, 

tion can have many subordinates and one or more supervisors. T

added complication to testing.

In addition to these interaction-driven systems, there is an increa

the use of software agents in manufacturing. These agents are ne

cedural programs nor typical object-oriented programs. They are 

emergent behavior, which means that their response to various in

as the agent becomes more “intelligent.” From a state-machine m

tive, this means that the state space changes over time, the state t

changes over time, and the outputs change over time. Furthermo

ships among these agents can change over time. This, again, has 

ing.34
22 Testing Interaction-Driven Manufacturing Software 

Summary

If composable/component-based software delivers on its promises, it ou

more amenable to modeling with hierarchical state machines than the av

interaction-driven software system. We should investigate whether we c

way alter our software architectures to make them tractable with the exi

eling techniques, and hence more testable. However, at this point in tim

able software is yet to be demonstrated on a large scale system, and even 

comes to fruition, the modeling of racing and nondeterminism remains 

Testing of Interaction-driven Manufacturing

Software

As described in the introduction, three categories for the testing of inter

driven manufacturing systems have been identified:

• component

• system

• interoperability

Based on a system architecture separable components can be identified 

nent testing, which addresses the question of whether a single compone

larger, distributed manufacturing system supports the interface and beha

described in its specification. When a component is based on a proprieta

face, testing will focus on performance and stress testing of the compon

determine if it meets the needs of the system in those respects. Function

while important for system integration, is typically left to the componen

the case of proprietary specifications; with public specifications a confo

testing program may be established to provide functional testing service

fication that a vendor’s products comply with the open specifications. A

tively, or in conjunction with conformance testing, interoperability testi

34.Selic, 1994.
Testing Interaction-Driven Manufacturing Software 

also be used to certify that a vendor’s product conforms well eno

compatible with other products.

A component based on an open specification is much more desir

manufacturing systems for several reasons:

• The same performance and stress tests can be performed agai

dor products.

• Third party organizations can be used to perform functional t

viding an objective measure of a systems conformance.

• The system architecture can be designed without predetermin

vendor’s solution.35

System testing concentrates on the interactions between the com

system. System testing answers the question: does the integrated 

designed? System testing may include a combination of simulat

system design is analyzed, integration testing which is used to br

line, and system monitoring which detects errors in the behavior 

Finally, to support open systems interoperability testing is perfo

nents supporting an open interface. Interoperability testing answ

could a component be replaced by another component without c

system? Where component testing is a systematic approach to fu

component interface, interoperability testing is testing by trial. T

exhaustive functional test of a component would answer the sam

interoperability testing. In practice, however, exhaustive testing 

cult, expensive, and frequently intractable given the wide range 

tems will allow. Interoperability testing is most often performed 

market place as they try multiple vendors products in working sy

be performed in a more controlled setting, as it often is through v

35.Wallnau et al., June 1998.

36.A case in point is programming language compilers: when software is ported betwe

in the program source code and the compiler may be found. Compiler bugs are rep

thus interoperability testing is taking place.

37.STEPnet, <URL:http://www.stepnet.org/>.
24 Testing Interaction-Driven Manufacturing Software 

Interoperability testing uses natural selection to limit the number of test

that are relevant to the products that need to be interchanged. Often a co

of functional component tests and interoperability testing yields the mo

effective results.38

The remainder of this section discusses each of these categories of testin

detail in terms of the open manufacturing interfaces that fall into the cate

existing test efforts that are underway, test techniques suitable for addre

category, and directions for future work.

Component Testing

As was discussed earlier, a system architecture is described in terms of t

ized subsystems it contains and the connections between those subsyste

closer examination one finds a number of different methods for designin

implementing the connections.39 For the purpose of this discussion, the 

tions supporting the connections in a system are referred to as infrastruc

ponents; the specialized subsystems are referred to as specialized compo

connections can be described both in terms of communication channels 

action flow. The channels are based on specifications, whether standard 

etary, formal or informal. The transaction flow is the series of events tha

during the operation of the system. While it may be repeated, the flow is 

the operation of a given system at a given time.

A specialized component supports a specific functional requirement of t

system. In a manufacturing system, specialized components will includ

such as databases, product data managers, schedulers, machine controlle

tion systems, and inventory systems to name a few. Testing of static inte

such as file exchanges, can be considered a sub-category of specialized c

testing. The interfaces to specialized components are usually defined eit

the context of a particular infrastructure or as a combination of an abstr

38.James Kindrick, John Sauter, Robert Matthews, “Improving Conformance and Interoperabil

StandardView, May 1996.

39. David Garlan and Mary Shaw, An Introduction to Software Architecture, CMU Software En

Institute Technical Report CMU/SEI-94-TR-21, ESC-TR-94-21, January 1994.
Testing Interaction-Driven Manufacturing Software 

face and a binding of that interface to a particular infrastructural 

implementation mechanism.

Infrastructural specifications provide a framework in which speci

interfaces can be defined and operate. Infrastructural component

ity with respect to integrating new specialized components into a 

Traditional software architectures use a tight integration of subs

shared libraries which are linked into run-time execution module

tems the common infrastructure is the programming language. D

(linking at run-time) is one step towards creating more flexibility 

based systems. Even so, connections between components tradit

reduced to file exchanges rather than direct interactions. Advanc

neering and software standards to support distributed processing 

more flexible integration of systems which are capable of direct 

ware standards which support distribution (including NFS/RPC,

IDL,43 and Java’s Remote Method Invocation and JavaBeans44) p

flexibility.

Infrastructural specifications address the coupling between spec

nents and provide a protocol (in the general sense) for the conne

the connections can be tested for consistency and completeness 

way that a compiler tests a program for consistent use of a progra

Infrastructural standards include programming languages, script

tribution protocols such as NFS/RPC and CORBA/IDL, messagi

as MMS, UNIX-style pipes, and sockets.

40.Network File System/Remote Procedure Call.

41.B. Callaghan, B. Pawlowski, P. Staubach, NFS Version 3 Protocol Specification, R

1813, Sun Microsystems, Inc., June 1995, <URL:http://www.cis.ohio-state.edu/ht

42.The NFS Distributed File Service, NFS White Paper - March 1995, <URL:http://w

white-papers/wp-nfs/>.

43.OMG, The Common Object Request Broker: Architecture and Specification. (COR

tion for the Interface Description Language (IDL). <URL:http://www.omg.org/cor

44.Java Technology Home Page, <URL:http://java.sun.com/>.
26 Testing Interaction-Driven Manufacturing Software 

Manufacturing Interfaces

This section describes manufacturing specific specifications for both the 

structural and specialized components of a system and the implications 

these have with respect to testing of interaction-driven manufacturing sy

• STEP

ISO 10303,45 a.k.a. STEP, is a family of specifications for product data 

covering the complete product life-cycle. STEP defines models of infor

be shared between systems and implementation methods for sharing the 

information models (STEP’s Integrated Resources and Application Prot

describe the content of the data; the implementation methods describe th

ing mechanisms to be used in sharing data. Together they define a varie

faces for components in a software system.

From a testing perspective, STEP is interesting because of the wealth of 

nent interfaces defined by combining the standard’s specifications and b

the rigor with which these standards are specified. Testing of the implem

methods can be defined either independent from or in conjunction with t

specifications. Two types of implementation methods are defined or em

within the STEP: an exchange file format46 (Part 21) and an application 

interface (SDAI). Each is discussed independently.

• STEP/Part 21

An interface based on a STEP AP and the STEP exchange file format is 

ple of a specialized component interface. In this case the interactivity qu

the components involved in the data exchange is very low and methods 

such interfaces are well established.47 This style of interface is mention

one end of the spectrum of interactions a component may have with a sy

45.Industrial automation systems and integration—Product data representation and exchange—

view and fundamental principles, International Standard, ISO, 1994.

46.Industrial automation systems and integration—Product data representation and exchange—

Implementation methods: Clear text encoding of the exchange structure, International Standa

47.The STEP Conformance Testing Project Home Page. <URL:http://www.erim.org/cec/stepte
Testing Interaction-Driven Manufacturing Software 

• STEP/SDAI

An interface based on SDAI supports a greater amount of intera

SDAI, the specialized components can be probed for data values 

data values can be altered in the underlying database system. SD

functional specification and a set of language bindings which inc

ming languages C and C++ and the interface definition language 

specification defines six implementation classes based on five ch

lowest implementation class provides minimal support for data e

essentially an API to an exchange file. The higher classes provid

more sophisticated features culminating in rich support for expr

which enables complete constraint checking and calculation of d

from the information models. The definition of implementation 

tems supporting the standard to evolve better and better support.

While supporting system evolution, SDAI’s implementation clas

guidance for software componentization. As applications adapt t

implementations, more and more of the functionality that they m

resident in the SDAI implementation. With this in mind, applica

SDAI should be designed such that they will not be adversely im

lution.

Test methods for SDAI implementations will need to address the 

mentation classes of SDAI’s functional specification as well as m

tation classes defined for each of the language bindings. Test me

using SDAI will need to further address the functional requireme

AP as well as the implementation classes defined that AP. Test m

implementations will need a strong framework in which to accom

variability imposed by the different specifications and the imple

involved.

• OMG/PDM Enablers

The OMG’s PDM Enablers specification48 defines twelve IDL m

eight conceptually separate PDM functions, or “enablers” of pro

ment. A conforming implementation need not support all of the 

the enablers; dependencies between the modules are spelled out 
28 Testing Interaction-Driven Manufacturing Software 

graph,49 and the modules required by each enabler are spelled out in an

table.50 There is also a mapping from eleven sub-processes of the produ

ment process to the enablers that they, in turn, require.51

The interfaces defined in the PDM Enablers specification provide a comb

data model and data access mechanisms. However, the behaviors exhibi

constraints enforced by conforming PDM implementations are not form

defined. Although there are UML diagrams and much explanatory text, 

IDL interfaces are considered normative.52 Since the PDM Enablers spe

is heavy on data modeling and light on interactivity, conflicting interpre

may not be as big a problem as they could be.

• CAM-I AIS

The CAM-I AIS interface specifies an API to a solid modeling system. T

can then be used as an engine to other software that needs solid modelin

ties. The interface is specified as both a file format and as a package of 

routines. While no testing activities specific to this interface have been i

methods for testing both file formats and software libraries are establish

• OLE for Design and Modeling

Object Linking and Embedding (OLE), a proprietary Microsoft infrastru

been used to define an interface for Design and Modeling.53 OLE for D

Modeling is an industry-led effort and it is an open interface. The interf

3-D graphical objects from one application to be included in (or embedd

another application. If the original application is changed, the referencin

48.Revised Submission (including errata changes) — PDM Enablers — Joint Proposal to the O

Response to OMG Manufacturing Domain Task Force RFP 1. <URL:http://www.omg.org/ar

02-02.pdf>, 1998.

49.Ibid., section 1.16.2.5 (“Module Interdepencies”).

50.Ibid., section 1.9 (“PDM Enablers”).

51.Ibid., section 1.12 (“Mapping the Product Development Process to the PDM Enablers”).

52.Ibid., section 1.16.2.1 (“IDL Specifications”).

53.OLE for Design and Modeling. <URL:http://www.intergraph.com/iss/technologies/jupiter/o
Testing Interaction-Driven Manufacturing Software 

tion will reflect those changes. Using this interface, interactivity 

cess. The client is able to read information from and create refere

and even override some of the server’s data on the client side, bu

able to affect any changes in the server.

• PSL

Not yet a mature standard, the Process Specification Language (

mechanism for applications of process information to exchange t

includes an ontology to support the exchange. It is envisioned th

tion will evolve to support interactivity between systems represe

ling manufacturing processes. A goal for the TIMS project is to 

guidelines for testability which will be used in the development 

• Summary

Table 1 summarizes the component interfaces for manufacturing 

ciates them with the infrastructure on which they are based and 

form of software. Testing techniques have been developed or are 

ment in association with the various kinds of software infrastruc

these are discussed further below.

TABLE 1. Infrastructures supporting Manufacturing Interfaces

Component Interface Software Infrastructure(s) Type of Software

AP xxxa

 + STEP Part 21

a. The notation xxx is used to indicate any AP.

none static file checker

AP xxx + SDAI C++ binding SDAI and C++ class library/ datab

AP xxx + SDAI C binding SDAI and C software library/ d

AP xxx + SDAI IDL binding SDAI and CORBA distributed invocat

AP xxx + SDAI Java binding SDAI and Java web/ Java/ databas

PDM Enabler CORBA distributed invocat

CAM-I AIS C software library

OLE for Design Microsoft COM/OLE distributed invocat

PSL tbd tbd
30 Testing Interaction-Driven Manufacturing Software 

Testing based on SDAI uses a multi-tiered infrastructure: one tier being 

guage binding and the other tier being SDAI. In Table 1 the column Typ

ware refers to general categories of software for which testing activities 

exist. When testing of the identified component interface is pursued, a m

depth investigation of these types of testing and associated tools will be 

Existing Testing Efforts

The only identified effort that directly addresses testing of specialized c

interfaces for manufacturing is that for STEP’s AP 20354 exchange file. 

activities are relevant in their analogy to manufacturing interfaces. Effor

ing generic components based on the infrastructural components used in 

turing include the 100% Pure Java testing program55 and NIST’s VRML 

Reality Modeling Language) testing program. NIST’s SQL (Structured 

Language) testing program addressed some of the challenges faced in te

ufacturing specifications.

• STEP/Part 21/AP 203

This testing effort generates test cases from the specification for AP 203

methodology used is rigorously defined within the context of STEP and 

used for other APs within STEP. One unique characteristic of the STEP 

ogy is that the standard includes the specification of abstract test suites 

AP. The standard abstract test suite for AP 203 formed the conceptual ba

executable AP 203 tests. While this method is extensively developed for 

functional requirements of an application, there has not been work in th

testing an interactive interface such as would be available with SDAI. P

methodology may be adaptable to testing implementations of STEP AP 

SDAI, as well as other APs with an SDAI interface.

54.ISO 10303 Industrial Automation Systems and Integration — Product Data Representation a

— Part 203: Configuration Controlled Design, International Standard, ISO TC184/SC4, 199

55.100% Pure Java Program Home Page, <URL:http://www.javasoft.com/100percent/cert.html

56.ISO 10303 Industrial automation systems and integration—Product data representation and 

Part 303/TR: Abstract test suite: Configuration controlled 3D Designs of Mechanical Parts a

blies, working draft, 1998.
Testing Interaction-Driven Manufacturing Software 

Additionally, while providing a different level of abstraction, wh

lap, the OMG PDM Enabler specification is consistent with AP 

for configuration control and thus provides an interactive interfa

which is not based on SDAI. The AP 203 test cases for file exch

applicable to testing the PDM Enabler interface as well.

• 100% Pure Java

The 100% Pure Java testing program certifies that Java classes su

interfaces.57 In this program the interfaces are not necessarily op

developer submits the classes to be tested along with the test sui

expected results of the tests. The testing program then verifies th

runs with the Java Virtual Machines from multiple vendors and t

expected results in each case. The testing program provides an i

ation of a program’s portability and an evaluation of run-time co

defined by the implementor, for Java applications.

Some aspects of this approach are of interest in the discussion o

facturing interfaces. The approach is generic to Java Class Libra

should be applicable to any manufacturing interface defined in J

ment of test suites by the vendors may be a practical approach to 

ment providing that someone is able to verify the reliability of th

NIST competence project in the Information Technology Labora

Testing by Statistical Methods, is studying methods for assignin

tainty to test suites.58 If successful, these methods could be appl

vendor developed test suites.

57.In this context public interface is the technical term defined for the Java programm

the availability of the interface for use by someone other than the class developer. 

ent than the less technical use of the term to mean that the interfaces that are publis

use by anyone.

58.Banks et al., NISTIR 6129, 1998.
32 Testing Interaction-Driven Manufacturing Software 

• VRML59

In this testing activity, NIST devised a way to evaluate the state of a VR

browser at run-time using features of the VRML language. Those states 

compared to the correct state as defined by a reference implementation 

mine the system’s correctness. The approach is to probe and monitor a s

ative to a reference implementation.

• SQL

The NIST SQL Test Suite60 addressed several of the challenges faced fo

manufacturing component interfaces, including multiple implementatio

in a specification, bindings to multiple programming languages, and acc

more than one process concurrently.

Both the SDAI and PDM Enablers specify interfaces that may be config

more than one way. In the SQL Test Suite, conformance tests were cata

according to the “profiles” that they tested. Some tests were designed to 

files singly while others tested them in useful combinations. A script to 

correct sequence of tests was then generated by a program according to 

files the vendor claimed to support. This same technique also helped to a

complexity of bindings to multiple programming languages.

SDAI and the PDM Enabler are both interfaces to systems which will b

more than one process. The SQL experience with issues of concurrent u

applicable here.

Test Techniques

Component testing is the most studied of the types of testing identified. 

there is some argument over whether interoperability or conformance te

better way to approach component testing, there is no doubt that the com

a complex distributed system of systems need to be tested in order to ha

59.NIST VRML Project Home Page <URL:http://www.itl.nist.gov/div897/ctg/vrml/vrml.htm>

60.NIST SQL Test Suite. <URL:http://www.itl.nist.gov/div897/ctg/sql_form.htm>, 1995.
Testing Interaction-Driven Manufacturing Software 

chance at success in complete system testing. Here we present a

some of the most pertinent aspects of component testing.

• Domain testing

A premise for theories of component testing is that exhaustive te

ble in a very limited number of cases. In most cases, testing is ap

strategy designed to produce optimal results; in other words, to fi

and the most serious bugs with the least number of tests. Much o

ware testing revolves around what is the best strategy. Theories a

box (e.g. debugging) and black box testing. The scope of the TIM

ited to black box testing.

Test coverage and software usage patterns are critical factors in 

Test coverage is an indication of how much of the software was 

Test coverage can be measured in two ways — in terms of how 

cation was exercised as a percentage of the complete specificatio

how completely the tests covered the domain of input values defi

fication. Tests which cover the complete range of input values ca

difficult to define and execute. For example, consider a program 

continuous function.

Since exhaustive testing is typically not practical, two non-exclu

are often used to limit the scope of the tests. One strategy is to a

terns of the component and use that analysis to prioritize the test

very similar to interoperability testing which achieves essentially 

a less formal way. Another strategy referred to as domain testing 

of input values and test the component with those values rather th

complete range of values. Beizer61 provides an in depth discussi

ing which describes systematic approaches to identifying an opt

values.

61.Boris Beizer, Software Testing Techniques, Chapter 6,Van Nostrand Reinhold, NY
34 Testing Interaction-Driven Manufacturing Software 

• The role of infrastructures

Testing of infrastructural components differs from testing of specialized 

nents in that, for the most part, infrastructural components are not manu

specific and can be exercised by generic types of test suites. The one ex

identified as a manufacturing specific infrastructure is MMS.

Generally speaking, infrastructures are most vigorously, thoroughly, and 

tested for conformance to a specification through use rather than by any 

testing activity. When there is a formal testing activity it generally come

fact and is designed to differentiate multiple vendor products supporting 

specification and highlight those areas of inconsistent support so that th

addressed.

On the other hand, infrastructural components are integral to the testing 

ized component interfaces since the specialized components are based o

structures. Many testing tools, and hence techniques, revolve around the 

particular infrastructure. Additionally, as is illustrated in Table 1, layere

structures are emerging (e.g., SDAI/a programming language, IDL/a pro

language, SDAI/IDL). Test methods based on infrastructures are essenti

aging the layers.

• Formal methods

The application of mathematical methods to software development has 

a discipline known as formal methods. The original emphasis for forma

was on software design and development. Formal methods are used to r

describe software specifications before any code is implemented. Some 

has been shown in applying formal methods in this manner. More recen

tion has been given to applying formal methods to the specification and 

standards62,63,64 and several standards for formal description techniques 

defined within ISO.65,66,67,68 The application of formal methods to stan

62.Richard Botting, Anthony Godwin, “Analysis of the STEP Standard Data Access Interface u

Methods,” Computer Standards and Interfaces, vol. 17, 1995.

63.Richard Sinnott, Kenneth Turner, “Applying Formal Methods to Standard Development: The 

uted Processing Experience,” Computer Standards and Interfaces, vol. 17, 1995.
Testing Interaction-Driven Manufacturing Software 

fers from the traditional use in software development since the in

ware implementations are not defined. The result of applying for

software standards is improvement in the standard itself, since it 

defined, and improvement in the ability to generate test software 

cation.

Future Directions

SDAI and PDM Enablers are both good examples of specificatio

driven manufacturing components which are intended to operate 

environment.

The SDAI specification provides the infrastructure for a series o

interaction-driven manufacturing systems. The SDAI approach i

ous methodology which has not yet been fully exercised. To date

for these types of interfaces have only been applied on individua

we have found no activities to systematically reuse the methods 

interfaces as would be possible by combining SDAI with STEP 

of different application areas.

The PDM Enablers use an infrastructure, namely CORBA, for w

ities to date have been ad hoc. Currently, several efforts are unde

testing methods using this infrastructure.

64.Kathy Liburdy, Martha M. Gray, and Lynne S. Rosenthal, “Formal Specification L

ance Testing.” Presented at the Eleventh International Software Quality Week 199

65.ISO/IEC 8807, Information Processing Systems — Open Systems Interconnection 

Description Technique based on the Temporal Ordering of Observational Behavio

zation for Standardization, Geneva, 1989.

66.ISO/IEC 8807, Information Processing Systems — Open Systems Interconnection — 

Description Technique based on an Extended State Transition Model, Internationa

dardization, Geneva, 1989.

67.IUT-T, Specification and Description Language, CCITT Z.100, International Cons

Telegraphy and Telephony, Geneva, 1992.

68.C. Ruggles (Ed.), Formal Methods in Standards: A Report from the BCS Working 

British Computer Society, 1990.
36 Testing Interaction-Driven Manufacturing Software 

Integration and System Testing

While many standards concentrate on individual software interfaces wit

ufacturing system, the concern of end users of manufacturing software i

the final integrated system performs as required. From both practical an

perspectives, verifying correct behavior of the entire system is quite diff

verifying the behavior of individual components. As Brooks observed, t

test is “unexpectedly hard,” and is often made harder by a lack of concep

rity between independently specified and/or developed components.69 T

cost of system level problems only becomes greater as industry moves a

monolithic, proprietary solutions and instead assembles systems from o

dard software components bought from completely different sources. Th

ual “components” that we are integrating today would have been consid

“systems” at one time, so “integration testing” and “system testing” nee

understood in this new, larger context.

Interaction-driven systems pose a special challenge because dynamic in

between disparate components is much more sensitive to errors than is s

exchange. If two components disagree on a syntax issue for file exchang

might be lucky enough to lose only a small part of the input; but if they 

about an interactive interface, it is almost certain to stop the show. Integ

system testing are therefore crucial to the reliability of interaction-drive

A publication from Rational Software Corporation defines integration a

testing as follows:

Testing a specific feature together with other newly developed fe

tures is known as integration testing. Testing the interface of two

components explores how components interact with each other.

Integration testing inspects the variables passed not only betwee

two components, but also the global variables. This test phase

assumes the components and the objects they manipulate have a

passed their local unit tests.

69.Frederick P. Brooks, Jr., The Mythical Man-Month, 1995 edition. Addison-Wesley.
Testing Interaction-Driven Manufacturing Software 

[...]

System testing is designed to reveal bugs that cannot be 

to individual components, or to the interaction among co

and other objects. System tests studies all the implement

aspects of the design similar to the customer’s environme

testing concerns issues and behaviors that can only be ex

testing the entire integrated system or a major part of it. 

testing includes testing for performance, stress, security, 

ability, configuration sensitivity, usability, data integrity, 

and recovery.70

By specifically excluding interaction-related faults, this definitio

ing suggests that serious problems such as deadlocks and race co

expected to have been found during integration testing, much as 

are expected to have been found during unit testing. This is cons

deal of our own precedent which steadfastly separates integratio

tem testing.71 However, for our larger distributed systems, the on

between these tasks are the invasiveness of the testing and the ty

we hope to find in each stage. For the purposes of this discussion

struggle to keep them separated, but assume that Brooks’ advice 

ponent at a time” will nonetheless be followed in the software d

cess.

System-Level Manufacturing Specifications

• SEMATECH CIM Framework

The SEMATECH Computer Integrated Manufacturing (CIM) Fr

cation72 is a specification for the software infrastructure within s

70.Laura Lee Rose, Getting the Most Out of an Automated Test Tool. Rational Softwa

<URL:http://www.rational.com/support/techpapers/pacorpwp/>, 1998.

71.NBS Special Publication 500-98, Planning for Software Validation, Verification, a

1982.
38 Testing Interaction-Driven Manufacturing Software 

tories. It provides interface definitions in OMG’s IDL, Harel Statecharts 

component states,73 Sequence Diagrams to show how components are m

interact,74 and prose explanations to explain the definitions and intent. S

ECH is no longer developing the CIM Framework per se, but the specifi

being used as an input to the Manufacturing Execution Systems / Machi

(MES/MC) Work Group of the OMG’s Manufacturing Domain Task Fo

where a widely accepted standard is expected to emerge. Concurrently, 

work is being pursued in the Semiconductor Equipment and Materials I

tional (SEMI)76 CIM Framework Task Force. SEMI is a global trade or

that publishes standards made by and for the semiconductor industry.

In their submission to the MES/MC Work Group,77 SEMATECH comm

the formal representations that they had used in the CIM Framework we

ble of capturing the level of semantics needed to enable higher levels of 

ability” and “weak in their representation of the behavioral semantics.” 

believe that they may have found a better way:

These problems led SEMATECH to contribute to the developme

of the BOCA [Business Object Component Architecture78] spec

cation in response to the OMG’s Business Object Facility RFP

[Request for Proposals79]. The BOCA meta-model and the rigoro

72.CIM Framework Home Page, <URL:http://www.sematech.org/public/division/fi/cim/cimhom

73.David Harel, “Statecharts: a visual formalism for complex systems.” Science of Computer P

pp. 231-274, July 1987.

74.UML Notation Guide, Version 1.1, “Section 7: Sequence Diagrams.” Rational Software Corp

<URL:http://www.rational.com/uml/html/notation/notation7.html>, September 1997.

75.Manufacturing Execution Systems / Machine Control Work Group Home Page, <URL:http:

www.omg.org/mfg/mfgmesmc.htm>.

76.SEMI OnLine, <URL:http://www.semi.org/>.

77.Response to the Manufacturing Domain Task Force RFI-3: Manufacturing Execution System

<URL:ftp://ftp.omg.org/pub/docs/mfg/98-05-03.pdf>, International SEMATECH and Fraun

May 1998.

78.Data Access Technologies, Inc., et al., Business Object Component Architecture (BOCA) Re

posal: revision 1.2, July 1998. <URL:ftp://ftp.omg.org/pub/docs/bom/98-07-01.pdf>.
Testing Interaction-Driven Manufacturing Software 

use of a Component Definition Language (CDL)80 offer 

capture a richer, more complete and more consistent spe

of the CIM Framework semantics.

• APC Framework

The Advanced Process Control (APC) Framework81 was once a

tem-level specification; it is now considered to be a subsystem o

work, specifically the Advanced Process Control Group of comp

implementation of this subsystem is now being commercialized

The CIM Framework itself may be viewed as the manufacturing 

system of the manufacturing enterprise, which would include ot

such as enterprise resource planning. This is noteworthy because 

issues arise at every level, and rigorous testing becomes less and 

the scope of the system is enlarged.

• MSI Control Entity Interface Specification

The Manufacturing Systems Integration (MSI) project at NIST8

ification for control entities in a distributed, hierarchical manufa

Despite its “interface” designation, the specification in fact prov

nitions, message definitions, state diagrams, and prose descriptio

behavior of components implementing planning and job control 

compose hierarchically to form an interaction-driven system.

79.Common Facilities RFP-4: Common Business Objects and Business Object Facili

www.omg.org/arch2/cf/96-01-04.pdf>, 1996.

80.Data Access et al., section 2.3: Component Definition Language Specification.

81.Advanced Process Control Framework Initiative (APCFI) 1.0 Specifications, 1997

www.sematech.org/public/docubase/abstract/3300aeng.htm>.

82.M. K. Senehi, S. Wallace, and M. E. Luce, “An Architecture for Manufacturing Sy

ceedings of the ASME Manufacturing International Conference, Dallas, TX, 1992

83.Sarah Wallace, M. K. Senehi, Ed Barkmeyer, Steven Ray, and Evan K. Wallace, M

Integration Control Entity Interface Specification. NISTIR 5272, September 1993
40 Testing Interaction-Driven Manufacturing Software 

A primary objective of the architecture was to incorporate provisions fo

recovery from anomalous situations at every level, so cyclic interactivity 

the system. The protocols and interfaces for distributed planning and co

would provide fertile ground for system level testing. While this specifi

se is clearly not the focus of industrial attention, it serves as an example 

defined hierarchical architecture for manufacturing, and its concepts and 

learned have contributed to several OMG specifications.

Existing Testing Efforts

Industry-driven efforts towards developing an open methodology for int

driven distributed system testing are in their infancy. OMG’s Test Speci

Group84 has as part of its mission to “offer guidelines to applications de

on how to perform appropriate testing of object-oriented and distributed 

tions” and to “encourage test tool vendors to create products that suppor

mate the testing process.” At this time the group has not finished produc

initial white paper. Similarly, the Information Technology Laboratory at 

identified “testing methods for object-oriented software, software compo

component interactions in an integrated software system” as one focus o

ware Testing by Statistical Methods project,85 but has not yet produced 

paper on the subject.

Although testing of these systems is done every day by companies using 

sense approaches (see “General technique,” below), these efforts are sel

cized. Conversely, a great deal of published work provides methods for 

of various distributed phenomena at a high level of abstraction, but it doe

the need for complete system testing.

Consider, for example, the common problem of different components giv

different interpretations to data that they exchange. Let us assume, as is u

case, that the components were separately specified and do not formally 

common ontology. Then this fault will probably not be detected in comp

84.Test Special Interest Group Home Page, <URL:http://www.omg.org/testsig/>.

85.Software Testing by Statistical Methods home page, <URL:http://www.itl.nist.gov/div897/c

stsm.htm>. April 1, 1998.
Testing Interaction-Driven Manufacturing Software 

ing because each component appears to satisfy its own specifica

not be detected by simulation and analysis of the component int

the correct protocol is being followed. In order to detect the fault

verify the contents of messages exchanged between components

message exchange itself or the components’ behaviors by thems

ily done using ad hoc techniques, but not so easily using most fo

which either abstract away the message contents in order to mak

protocol tractable, or abstract away the interactions in order to m

specification tractable. This disconnect carries through into avai

toolsets (e.g., ObjectGeode86 and the ObjectTime toolset87), wh

that use different languages (e.g., finite state machines versus cla

perform these interrelated tasks. Rapide88 tried to bridge this ga

putationally complete functions to an Architecture Description L

should investigate to see to what extent they were successful, an

resulting language is usable and understandable. (Of course, this 

simulation, not for testing of implementations.)

Integration / System Test Techniques

• General technique for locating faults and testing conforma

In system testing, a piece of bad data may propagate through sev

before a problem ever appears. Locating the fault in such cases c

good strategy to narrow the possibilities is to replace one or mor

nents with dummy components89 to see whether the problem goe

component anywhere on the path of the bad data will break the c

correct operation, so the fault is eventually located by walking b

source.

86.ObjectGEODE. <URL:http://www.verilogusa.com/solution/geode.htm>. 1998.

87.ObjectTime Developer toolset. <URL:http://www.objectime.com/>. 1998.

88.The Stanford RapideTM Project. <URL:http://pavg.stanford.edu/rapide/rapide.html

89.Frederick P. Brooks, Jr., The Mythical Man-Month, 1995 edition, p. 148. Addison-
42 Testing Interaction-Driven Manufacturing Software 

Along the way, swapping dummy components for real components may 

den deviations from the specification, where a system is working only b

off the components using a given interface use the same incorrect interp

and are hence “bug-compatible” by accident. Incorrect but functional us

interface can propagate through a software project like a virus because d

will copy or re-use working code. Once this problem is detected, the co

changed to match the documentation — or vice-versa, as is more often 

practice if there is no established standard. Clearly this could be a powe

nique for conformance testing as well.

• Network monitoring and capture/replay techniques

If some semblance of all of the necessary components for a networked d

system already exists, a capture/replay tool can be used to examine syst

ior and to emulate components. These commercially available tools beg

recording all network traffic during an actual run of the system. This rec

examined manually to insure that the messages exchanged between com

are what was specified. The replay tool can then replay segments of the 

traffic to emulate a component. With the use of “parameterization,” the r

can change key fields of generated messages in order to act out various t

narios.

Capture/replay tools are popular due to their simplicity, flexibility, and r

Because their interaction with the system is on an entirely syntactic leve

work to some degree with any networked system, and there is little or n

tion-specific scaffolding to build. Even opaque, COTS components with 

licly available specifications can be emulated without trouble. However, 

syntactic treatment of system interactions is also their greatest disadvan

tester is obliged to operate at the level of raw data and machine code to 

meaningful tests using snippets of captured traffic, which must be rever

neered to map the raw data fields to their counterparts in high-level lang

There is no way to “get inside” of the emulated component to add test s

and assertions. There is also no easy way to emulate components for wh

sonable facsimile does not already exist, and integration with semantics

mal methods is unlikely to happen in the near future.
Testing Interaction-Driven Manufacturing Software 

• Scenario based testing

While rigorously specifying the behavior of a distributed system 

difficult, specifying this behavior for a specific scenario is more 

demonstrated by the Component Interaction Specification (CIS) 

supported by the Manufacturer’s CORBA Interface Testing Tool

UML Sequence Diagrams,91 and Message Sequence Charts.92 C

tage of being directly translatable into test scaffolding for CORB

has disadvantages that will be discussed below.

CIS is a derivative of the integration testing method that was bei

Advanced Micro Devices to test components of the APC Framew

in turn, made use of ideas that are also used in UML Collaborati

A CIS interaction scenario consists of a tree of requests having s

outputs, and/or return values. The tree is rooted at a test client th

entire chain of events. In order to capture the tree structure of th

flat ASCII script, an outline numbering convention similar to tha

oration Diagrams is used:

 1 ... first request by testing client on server A ...

 2 ... second request by testing client on server A ...

 2.1 ... request by server A on server B ...

 2.2 ... request by server A on server C ...

 3 ... third request by testing client ...

90.MCITT home page. <URL:http://www.mel.nist.gov/msidstaff/flater/mcitt/>. 1998

91.UML Notation Guide, Version 1.1, “Section 7: Sequence Diagrams.” Rational Soft

<URL:http://www.rational.com/uml/html/notation/notation7.html>, September 19

92.ITU-TS Recommendation Z.120, “Message Sequence Charts (MSC).” ITU-TS, G

93.UML Notation Guide, Version 1.1, “Section 8: Collaboration Diagrams.” Rational 

<URL:http://www.rational.com/uml/html/notation/notation8a.html>, September 1
44 Testing Interaction-Driven Manufacturing Software 

In an actual CIS, the text comments shown above are replaced by mach

able syntax specifying the remote operations that are invoked and the in

puts, and/or return values that are expected.

Although the full extent of possible functionality is not supported by M

this time, this approach enables code generation for dummy component

matic generation of run-time assertions to verify that the inputs and retu

each interaction are as specified. The dummy components are useful in s

integration testing when some components are not available or not trust

conformance testing to provide a more controlled testing environment f

systems. One may also use dummy components to stress test a system; 

ple, if a shop controller is theoretically able to control up to N workcell

test the system with that many emulations.

Unfortunately, although the CIS syntax is expressive enough to describe 

tree of interactions through a distributed system, it assumes a single sou

interactivity. All interactivity is assumed to originate with the testing cli

is thus the question of how this approach can be extended to handle mor

interactivity. While it is trivial to extend the CIS syntax to specify cyclic 

interactivity, it is much more difficult to emulate or verify that behavior b

now require the capability to monitor and control the sequencing of eve

system level. For the currently supported hierarchical interactivity, the o

events is inherently deterministic, and it suffices to embed monitoring a

into the components of the system. But with cyclic or chaotic interactivi

lated component must somehow manage to generate requests in the ord

specified without the benefit of inherent determinism, and a total orderin

tem events cannot be derived without the aid of accurate synchronized c

separate network level monitoring tool.

In any case, the total ordering of events imposed by the CIS approach is 

what we want in systems having cyclic or (especially) chaotic interactiv

formal analysis techniques proposed by academia for use on distributed 

explicitly address the issues of concurrency and non-determinism and a

valid and invalid sequences of events to be identified; but there remains 

nect between those formal analysis techniques and the available technol

testing actual systems. Similarly, UML Sequence Diagrams and Messag

Sequence Charts are more powerful in being able to model cyclic intera
Testing Interaction-Driven Manufacturing Software 

are extensively used for simulation, but they have not (to our kn

adapted for concrete system implementation testing and run-tim

message contents as CIS has been. This may be due to the probl

and controlling global state, of specifying message contents with 

be able to interoperate with actual live components, or both.

Future Directions

A possible direction for future work would be to investigate how 

between abstract analysis and practical testing in the case of non

interaction-driven manufacturing systems. Historically, industry 

manufacturing systems to be as deterministic as possible in defe

principle (“Keep it simple, stupid”). If non-deterministic system

superior return on investment, then there is no motivation to pur

must also consider that the lack of good testing techniques is one 

that we might reduce if non-deterministic systems can provide s

mance, such as has been observed for some agent-based approac

deployments.

On a more concrete level, we should acquire hands-on experienc

that SEMATECH believes will solve their problems. At present, 

experience, it is not obvious to us how it would help with the coh

tem-level specification. We must gain competence with this eme

learn its relevance to the problems of specifying and testing inte

tems, and experiment with automated CDL tools as they become 

ADL tools, such as Rapide, might also prove helpful. Most ADL 

demic, showing wide variation even in the definition of what an A

is intended to accomplish. Nevertheless, we should investigate th

permit, particularly as a possible solution to the protocol versus 

omy that was discussed earlier.

There is, as of yet, no formal method to determine the best comb

ponents to test or to emulate in order to achieve a given conform

fault detection goal at the system level. By this we mean a forma

ducting the tests and a formal protocol for when to stop testing, 

discussed in reports from Information Technology Laboratory.94
46 Testing Interaction-Driven Manufacturing Software 

analogous method that would serve our purposes would be extremely co

because it would need to combine aspects of existing protocol specifica

test methods with aspects of existing interface specification and test me

These detailed formalisms would be needed to establish a rigorous conn

between specified requirements for the system and the testing that is per

Unfortunately, due to their complexity, combinatorics, and lack of cost 

tools, these formalisms have not exactly “taken industry by storm.”96 A 

fication is even less likely to be well received because the need to integr

ate methods is likely to require more simplifying assumptions that will 

damage the applicability of the resulting unified method. Nevertheless, 

remains a possible topic for future work.

Interoperability Testing

One flight test is worth a thousand simulations. —Hen

Overview

Interoperability is a term that generally refers to the ability of existing p

work together in practice as opposed to theory. The focus of interoperab

recognition that formal testing and analysis does not guarantee that con

products will actually work together. Indeed, practical operability some

requires explicitly ignoring specifications when they would otherwise p

interoperation, for example, with other non-conforming components.97 

ties make interoperability a very difficult field.

Testing of interoperability involves bringing together existing componen

exercising their interoperation. Ideally, as many functionally similar com

94.James Yen, David Banks, P. Black, L. J. Gallagher, C. R. Hagwood, R. N. Kacker, and L. S. 

Software Testing: Protocol Comparison. <URL:http://sdct-sunsrv1.ncsl.nist.gov/~ftp/stsm/

simulationmar98.pdf>, March 28, 1998.

95.Banks et al., NISTIR 6129, 1998.

96.Dan Craigen, Susan Gerhart, and Ted Ralston, “Formal Methods Reality Check: Industrial U

Transactions on Software Engineering, v. 21, n. 2, February 1995.

97.Networking Standards: A Guide to OSI, ISDN, LAN, and MAN Standards, William Stallings
Testing Interaction-Driven Manufacturing Software 

as possible are tested near simultaneously. Since the focus is on 

ity, the use of test scaffolding is discouraged. All of the compon

someone’s actual product. The greater the number of componen

expectation of the interoperability test being meaningful. On the 

operation between a large number of components guarantees no

component interoperation. It does, however, raise reasonable bel

tion.

Interoperability testing is very significant for interaction-driven 

the science of specification-based interaction-driven testing is so 

testing of such systems has relied heavily upon interoperability t

more formal testing such as component interface testing.

The Reasons for Interoperability Testing

Interoperability testing is not required in all situations, but any s

plex standard or interface brings with it the opportunity for error

incompleteness in the specification itself. Interoperability testing 

kinds of problems. Even with a formally verified specification, t

code may introduce problems such that the implementation is no 

(This can be avoided by reference implementations, but these inv

problems.)

Another need for operability testing is that conformance testing 

tive due to complexity and/or cost constraints. All but the simple

take an intractable amount of resources to be tested with all poss

of real-world data.

Interoperability testing also provides the opportunity for additio

performance and reliability without the expense and overhead of 

ing.
48 Testing Interaction-Driven Manufacturing Software 

Public Relations and Other Political Implications

Interoperability testing presents interesting semi- or non-technical oppo

• Implementations designed for interoperability testing tend to be muc

robust with the expectation of having to work with other implementa

have entirely different assumptions or interpretations of a specificatio

• Interoperability testing gives participants a good feel for the true state 

at a single instant in time.

• Nondisclosure agreements are easier to swallow “all around” in a neu

as compared to having to sign such an agreement when purchasing o

taking possession of a competitor's component in one's own workpla

Examples

Computer networking has used interoperability testing very successfully 

years. Indeed, many of the interoperability testing practices stem from e

with network interoperability testing. In the early days of the Arpanet (l

become the Internet), components were built by different contractors. T

mentors gathered together on a regular basis for interoperability testing

pants quickly realized the value of interoperability as a more important 

adherence to the specifications — to the point that for many years after, 

net was driven by implementations that resulted in specifications rather 

other way around. The interoperability tests became ritualized in a regu

“Interop” conference, now officially known as Networld+Interop.

There are now interoperability forums (i.e., vendor consortia that do int

ity testing) in many areas. These forums provide vendors the opportunit

and test their components for interoperation without having to purchase 

install) other vendors' products. For instance, forums exist for interoper

ing of video, modems, encryption algorithms, security systems, file serve

as a wide variety of networking and communications such as internet co

protocols and network management protocols. Of special note to the ma

ing domain is STEPnet,98 where CAD package vendors exchange files i

Part 21 format to test the STEP-enabled interoperability between their p
Testing Interaction-Driven Manufacturing Software 

Many organizations have taken advantage of Internet connectivi

interoperability on a more ad hoc basis. As an example, Bell Lab

HTTP (Hypertext Transfer Protocol)99 interoperability testing at 

tal.research.bell-labs.com:8000>. This URL provides access to a

server that provides extensive logging to assist testing. This serv

by anyone at any time and without any advance request.

Frameworks

Despite the use of interoperability testing as a complement to tra

component interface testing, interoperability testing can range fr

formal.

At its most informal, interoperability testing is little more than s

functional tests between interacting components. However, high

interoperability testing is possible as well. For instance, the “Int

Abstract Test Suite for PNNI (Private Network-Network Interfac

is a rigorous specification that describes the requirements, objec

and test set-up for carrying out such interoperability testing.100 T

suite is a good example of a dynamic interaction-driven specific

There is no de facto standard framework for interoperability test

each area specifies its own. For instance, in the field closely rela

ATM Forum has issued a series of its own specifications for inte

ing.101,102

However, there are many references that provide general conform

background. In particular, ISO/IEC 9646: The OSI Conformanc

98.STEPnet home page. <URL:http://www.stepnet.org/>. 1998.

99.Hypertext Transfer Protocol. <URL:http://www.w3.org/Protocols/>. 1998.

100.Interoperability Test for PNNI v1.0. <URL:ftp://isdn.ncsl.nist.gov/pubs/ATM_Fo

PNNI/atm97-0089.ps.Z>, 97-0089, February 1997.

101.ATM Forum Technical Committee, Interoperability Test Suite for the ATM Layer 

ftp.atmforum.com/pub/approved-specs/af-test-0035.000.pdf>, April 1995.

102.ATM Forum Technical Committee, Interoperability Test Suites for Physical Laye

Mbps MMF (TAXI), <URL:ftp://ftp.atmforum.com/pub/approved-specs/af-test-00
50 Testing Interaction-Driven Manufacturing Software 

ology and Framework describes terminology and specifications for gene

formance assessment.103 This certainly represents a an appropriate start

for interoperability testing as well.

Special Noteworthy Problems

Timing-related problems are among the most common errors exposed b

erability testing and deserve specific mention. Such problems include d

and race conditions. These are generally understood only by computer s

trained in operating system principles. Alas, communications protocols 

faces are often designed by personnel who are experienced primarily in 

cation field and to a much lesser degree in operating system principles i

Specific tools for modeling and simulating timing related issues (e.g., P

are available but little used. In reality, many of these errors are detected 

ing interoperability testing.

Future Directions

By its nature, interoperability testing is not something to which NIST, w

product to sell, stands to contribute, except possibly as an impartial thir

conduct the testing. However, we have identified the fact that implemen

designed for interoperability testing tend to be much more robust with th

tion of having to work with other implementations that have entirely dif

assumptions or interpretations of a specification. Interoperability testing 

therefore be used as a means for testing the robustness of manufacturing 

whether or not interoperability as such was a design goal. In systems no

to be interoperable, failures are inevitable, but it is nonetheless desirable 

failures to be detected and resolved in a safe fashion by the system.

103.ISO/IEC 9646: The OSI Conformance Testing Methodology and Framework, International 

for Standardization, 1992-95.
Testing Interaction-Driven Manufacturing Software 

Conclusions and Future Work

In our background study, we found that the art of component test

lished than the art of system testing, and that interoperability test

and well-established discipline. We will therefore take different 

different classes of problems in component and system testing, a

interoperability testing until such work appears to be necessary.

For manufacturing components, adapting and applying existing 

structure testing techniques in the new context of the interaction

should be a workable solution. We expect to show this using the 

Enablers specifications. A challenge with these specifications is 

which can be re-used as new specifications using the same infra

along. The specification of standards to support manufacturing i

ing at an astronomical rate. Standards are being produced at high 

tion and are intended to be used in combination with other speci

resulting in the problem of a combinatorial explosion of interface

methods developed to systematically handle testing of such inter

The growth of modern manufacturing systems into what are effe

of systems” has out-paced the available methods for specifying 

While established rigorous techniques can be used in the contex

ent design and development effort (i.e. a single system), the testi

are constructed by “gluing together” generic COTS software, sp

control software, and legacy systems is still an evolving art. For 

ing systems, we do not anticipate finding a complete solution ve

is promising new work to investigate:

• Composable / component-based approaches may simplify th

manufacturing systems to be more intuitive to specify and mo

existing specification and test methods. We must see what po
52 Testing Interaction-Driven Manufacturing Software 

ponent-based software work will extend to distributed, interaction-dr

tems.

• New specification languages like CDL and various ADLs may help 

certain system-level problems and assist with simulation and testing 

tems.

• Formal methods for testing of components and protocols suggest the 

of a system-level method to decide what combinations of components 

tested together, how they should be exercised, and when to stop testin

ever, first we would need to have sufficiently formal system-level spe

to enable such an analysis. With luck, these may become available th

above mentioned work with new specification languages.

In addition, we make the following general observations:

• Testability of many systems is hindered by the low quality or lack of 

tions for them. This problem has existed for many years, and we alre

that preaching at industry and/or standards bodies to use formal meth

not help. There are some techniques now that are attracting voluntary 

from industry, possibly because they have found an acceptable comp

between rigor and usability. We should concentrate our efforts on the

also gaining proficiency with general-purpose testing tools which can 

the absence of a formal specification.

• Just as Design-For-Manufacture considerations affect product design

For-Test considerations should influence system design. If a formal m

based approach is to be employed for testing, then its usage should s

initial requirements analysis. Furthermore, since utilities which prov

matic generation of tests and/or test scaffolding are often technology 

the availability of suitable utilities should be considered before syste

mentation language and environment are chosen. In addition, the test

the system may be enhanced through the choice of infrastructure tech

such as the Java language or CORBA, that natively provide introspec
Testing Interaction-Driven Manufacturing Software 

bilities. Early consideration of issues such as these is paramo

system testability.

• The problems caused by integrating components whose spec

share a common ontology are not easily revealed or diagnose

ods. Therefore, the testability of a system can be enhanced by 

mon ontology before the system is implemented. This can be 

adopting a set of definitions from a pre-existing standard.

• Existing tools for monitoring the communication between co

tributed system either operate at a very low syntactic level, wh

by inspection a thankless chore, or rely on application-specifi

We should look into the possibility of defining a standardized

inspection interface that components can support in order to p

to monitor their interactions at a higher semantic level, but sti

• The rigorous methods that we have for validating protocols a

when the state space is small. Therefore, increased testability 

among the many benefits of following the KISS principle in t

ufacturing systems.

Our continuing work in the area of testability will focus on findi

and techniques for constructing high-integrity specifications, defi

pose facilities to support the monitoring of interactions at a high

and gaining a deeper understanding of the design-for-testability 

manufacturing specifications. Complementing the testability wor

ing effort to develop competence in the testing itself through pro

ing of dynamic, interaction-driven components and systems. We 

tools and techniques where necessary to take advantage of the te

ments that we make, or where we may have a solution to an ope

wise, we will gain competence in using the existing tools and te

previously untested systems.

By attacking the problem at both ends, building up our testing c

simultaneously working to make systems more testable, we hop

compromise for improving the reliability of systems. Neither the 

development of systems should need to go to extremes if comple

ments are made to each. This more moderate approach may then 
54 Testing Interaction-Driven Manufacturing Software 

higher level of acceptance and adoption than extremely invasive testing 

extremely formal development processes have achieved