NBSIR 73-289

Characterization and Testing of

Interactive Graphics for

Computer-Aided Design and Engineering

Siegfried Treu

Computer Systems Section

Institute for Computer Sciences and Technology

National Bureau of Standards

Wastiington. D. C. 20234

30 June 1973

Interim Report for Period July 1972 to June 1973

Prepared for Contract No. 73 90042

U.S. Army Electronics Command

Fort Monmouth. Mew Jersey 07703

NBSIR 73-289

CHARACTERIZATION AND TESTING OF

INTERACTIVE GRAPHICS FOR

COMPUTER-AIDED DESIGN AND ENGINEERING

Siegfried Treu

Computer Systems Section

Institute for Computer Sciences and Technology

National Bureau of Standards

Washington, D. C. 20234

30 June 1973

Interim Report for Period July 1972 to June 1973

Prepared for Contract No. 73-90042

U. S. Army Electronics Command

Fort Monmouth, New Jersey 07703

U. S. DEPARTMENT OF COMMERCE. Frederick B. Dent, Secretary

NATIONAL BUREAU OF STANDARDS, Richard W. Roberts. Director

CHARACTERIZATION AND TESTING OF

INTERACTIVE GRAPHICS FOR

COMPUTER-AIDED DESIGN AND ENGINEERING

CONTENTS

Page

1. INTRODUCTION 1

1 . 1 Background 1

1 .2 Project Scope 1 1.3 Related Activities ^

1 .4 Overview 3

2. CHARACTERISTICS OF THE INTERACTIVE DESIGNER 4

2.1 Motivation 4

2.2 Stages in Graphical Interaction 5 2.3 Questions of Evaluative Interest 9

2.4 Definition of Desirable Features 10

2.5 Emphasis on Symbiosis 16

2.6 Efficiency and Effectiveness 18

3. EXPERIMENTS FOR THE MEDEA SYSTEM 18

3.1 Data Collection Requirements 18

3.2 Levels of Experimental Complexity 21

3.3 Description of Suggested Experiments Z3

3.3.1 Simplicity Z3

3.3.2 Responsiveness 27

4. CONCLUSIONS AND PLANS 30

REFERENCES

1. INTRODUCTION

1.1 Background

The material presented in this report is based upon studies conducted

over the period of 1 July 1972 to 30 June 1973 as part of a long-term

"Interactive Computer-Aided Techniques Study" initiated in August 1970.

This long-term effort of research, development and consultation by the

National Bureau of Standards is in direct support of the AMC program for

the implementation and promotion of the techniques of computer-aided

design and engineering (CAD/E).

The intent of this cooperative program between NBS and the Graphical

Systems and Technology Branch of the Electronics Command (ECOM), under

AMC, is to provide both general and specific assistance to ECOM in the

initiation and use of interactive graphics as a new and important tool

in CAD/E.

1 .2 Project Scope

ECOM has undertaken the development of a prototype system for the

support of CAD/E. The concept of this^ design terminal has been termed

MEDEA--Mul tidiscipl ine Engineering Design, Evaluation and Analysis. The

basic goals of the MEDEA system have been described as follows^:

1. Put the man in the computer loop as a problem solver not as

computer specialist.

2. Provide an interactive rate compatible either with the scientist/

engineer/designer process function or with the user's problem

solving rate.

3. Provide simple and unified access and use of analysis, evaluation

and design procedures and data for many technical disciplines.

1
4. Provide interactive, conversational use with interactive

graphics as the principal input/output medium.

5. Provide for the accumulation of and cross-referenced access to

user designated abstracts of use of the computer-aided system.

The role of NBS has been to review the design considerations for

the MEDEA system, taking into account the design constraints, the capabilities of local versus remote processors for the support of CAD/E,

and the trade-offs between hardware and software implementation of

graphics functions. A specific task for NBS has been to make recommendations regarding the evaluation of graphics systems for CAD/E and to

formulate a program for measuring the performance of the MEDEA system

with respect to the stated design objectives. '

This report deals specifically with techniques for evaluating the

efficiency and effectiveness of graphics systems for CAD/E, and the

assessment of user satisfaction. A discussion of the economic aspects

is presented in a separate document.

1.3 Related Activities ' - \

1 A subordinate role for NBS has been to provide ECOM with the means

of obtaining access to the facilities of the ARPA (Advanced Research

Projects Agency) Network, The ARPA Network, in turn, provides access

to a wide variety of computer systems, many of which are capable of

supporting CAD/E, either in a batch mode or interactively. NBS has

been established as a node on the ARPA Network and has a TIP (Terminal

Interface Message Processor) with a PDP-11 host processor. A communications protocol for use by ECOM in accessing the ARPA Network was

2 developed earlier and has been described in a previous report .

2
1.4 Overview

This report outlines the stages of development in the utilization

of interactive graphics as a tool for CAD/E. It presents a series of

characteristics which are of significance to the designers and users

of such systems and poses a series of questions of evaluative interest,

which are intended to delineate the extent to which a system under

examination achieves its stated design objectives. These characteristics are grouped in accordance with the nature and complexity of the

experiments which would need to be conducted to establish values for

them. The report then proceeds to select specific characteristics of

particular interest and suggests the design of experiments for examining

these characteristics in detail. As the MEDEA system evolves to a fully

operational status, the techniques outlined in this report can be applied

to assess the degree of user satisfaction achieved and to suggest lines

along which further development may be pursued.

3
2. CHARACTERISTICS OF THE INTERACTIVE DESIGNER

2.1 Motivation

The performance of interactive graphics systems can be tested and

evaluated in a variety of ways, ranging from very selective and objective consideration of specific hardware/ software functionings to a very

comprehensive and (in part) subjective assessment of the man-computer

relationships involved. A methodology exemplary of the latter is the

3 previously reported Symbiosis-Efficiency-Effectiveness (SEE) approach .

To develop an appropriate methodology and meaningful measures, a

characterization of the interactive graphics system is essential. If

it is viewed to be only an embodiment of graphics hardware and software

for enabling interactive design, independent of the man-computer interface questions involved, then the technical specifications and performance of the system can be analyzed in a purely objective manner without

the considerable complications arising from human user experiences and

judgments. If, however, the designer is considered to be an important

and integral part of a man-computer team, the difficult concerns about

4

human user characteristics must be confronted .

5

Dunn has recently stated that

...useful measures for interactive graphics systems are

dependent upon subjective judgments [which] are made by

both the users of the graphics consoles and the operators

and owners of the supporting systems. Hard-to-measure,

non-analytic factors, such as, "effectiveness," "productivity", "usefulness", and "satisfaction" are subjects

4
of these judgments. After all is considered, the reason

for interactive graphics is the human-in-the-loop.

As confirmed by the contents of this report, the "human-in-the

loop" requirement has been the focus of a considerable segment of our

work during the past year. It was not carried out to the exclusion

of concern about the characteristics and capabilities of the hardware/

software facility (MEDEA) itself, but rather with the intention of

creating testable hypotheses on the performance of the designer-system

partnership.

We cannot meaningfully test a man-machine relationship unless we

have some specific criteria or factors which are deemed to have some

significant bearing on good/bad performance. For example, the determination of "responsiveness" of a system would be useless without

clear indications that response time happens to be an important

6 7 feature ' which human users observe and which appears to have various

g interesting effects . But that is not the only feature or characteristic which is of significance to humans. This section is devoted to

development and description of a more complete profile of characteristics which are (or can be) hypothesized to be important to the human

designer who is attempting to take advantage of an interactive graphics

system such as MEDEA.

2.2 Stages of Graphical Interaction

9

Katter and McCarn have published an interesting description of the

reactions of interactive users of on-line retrieval systems. They

characterized three roughly sequential phases in the development of

user-system interaction. These can be briefly summarized as follows:

5
Phase 1. Confidenc e: The user shifts from an uncertain and

hesitant approach to a more relaxed attitude of

? ease and optimism about his ability to cope with

'

' ' I- : the system.

Phase 2. Insight : The user shifts from the somewhat unquestioning rote memorization and practice (in Phase 1) to learn-

' ing about more general underlying principles and

potentials of the system.

Phase 3. Incorporation : The user shifts from regarding the system ' as novel and supplementary (to his own resources/methods)

to incorporating its capabilities as an essential element

in his pertinent information seeking/using pattern.

Such phases are not peculiar to users of on-line retrieval systems.

Similar phases or stages undoubtedly exist in the progression of steps

taken by a novice towards becoming an experienced designer utilizing

interactive graphics. A suggested sequence is portrayed in Figure 1.

First, the uninitiated designer must learn the fundamentals. At

this stage, he may feel overwhelmed by procedural details and by having

to remember simple but necessary instructions pertaining to such things

as cursor (re)location and light pen operation. The learner may require

considerable personal (or on-line) assistance, and, in order to develop

self-confidence, he needs the reassurance which comes from repeated

successful experiences and observations that personal errors do not

cause system disasters.

6
PRODUCING

quality results

within the

known system

constraints

PROBING

into the more

subtle or difficult

features

PROGRESSING

to more

independent

use of the

system

EVALUATING

the graphic

system's capabilities

and performance

throughout the

successive

stages

Figure 1. Interaction Stages
After overcoming at least most of the hurdles confronted during

the learning stage, the interactive designer should then find himself

progressing to more independent status. Experience now has demonstrated

to him that he can remember and carry out the simple procedures and

operations in comparatively routine fashion, and hence he can start to

pay more attention to*the interactive design objectives themselves.

In so doing, and as he finds himself attempting more complicated

and advanced designs, he quite naturally moves to the next stage,

namely the probing. At this point, curiosity and initiative urge

the designer towards determining perhaps hidden system capabilities

which were neither apparent nor required during the more superficial

early stages. Here he might ascertain, for example, that some helpful

perspective views are possible through certain versatile manipulations

of the graphic objects on the display screen. On the other hand, he

may find out that the system is strictly limited in some respects,

such as in the precision obtainable from calculations returned from

supporting analysis software.

Having gained a reasonable level of experience with the graphics

system and a realistic assessment of what it can or cannot do for him,

the user can now settle down to a kind of "steady-state" phase in which

he performs various design tasks in an operational mode. He may have

resigned himself to living within the existing system constraints and,

in that context, taking advantage of the available graphics facility

to its fullest. While he may still need to learn more about it and

to progress further as an interactive designer, and while he may at

8
times continue to probe into accomplishing the tasks more effectively

and efficiently, he can nevertheless be considered a relatively

experienced designer who is able to produce on a regular and reliable

basis

As is also indicated in Figure 1, the designer can naturally be

expected to observe and evaluate the graphics system performance

throughout the learning, progressing, probing and producing stages.

This happens to be quite important because we want the human-i n-theloop to be ready and able to make evaluative judgments (see Section

2.1). But, in order to gather such data in a consistent and potentially

useful manner, a common basis for the selection and grouping of designers

according to levels of experience is highly desirable. The main purpose

of this section, therefore, has been to develop a rationale for distinguishing the backgrounds of persons who may become available for experimental (operational) testing of the MEDEA graphics facility. Depending

on whether a designer is at the learning, progressing, probing, or

producing stage, his behavior, expectations, requirements and evaluative

opinions may be quite different and should be interpreted accordingly.

2.3 Questions of Evaluative Interest

A specific question about the performance of a system can be viewed

as a criterion for system evaluation. If, for example, an interested

party wants to have an answer to the following question:

Is the MEDEA response time good enough to satisfy designer expectations?

then, assuming we know (or find out) what the designer expectations are and

how many of the local designers are to be satisfied, a "yes" or "no"

9
(qualified by 7o) should at least be possible. This answer can then be

regarded as one among numerous other possible factors carrying evaluative

information about good/bad MEDEA performance.

But, instead of attempting to compile an exhaustive list of

evaluative questions that could be asked about the MEDEA facility, it

is more manageable to identify a set of pertinent and desirable characteristics which would be inferred collectively. The above-asked question

clearly implies that "responsiveness" is an important system characteristic. Other questions, appropriately grouped, would similarly suggest

other features or characteristics which, taken in their adjectival

form, are not only descriptive and useful but also subject to experimental validation. The next subsection provides their definitions.

2.4 Definitions of Desirable Features

In the published paper^^ entitled "A Conceptual Framework for the

Searcher-System Interface", a portrayal of the system characteristics

desirable from the human user's standpoint was made. It was asserted

that these must be taken into account in designing the man-computer

interface for purposes of on-line information retrieval.

A similar characterization, appropriately re-oriented to graphicsaided design, has been carried out in preparation for testing the MEDEA

system. It seems only reasonable that, to develop useful experiments

and related measurements for evaluating/demonstrating the performance

of that system, we should first have an understanding of what features

or characteristics appear (or are hypothesized) to be good/bad or

10
advantageous/disadvantageous for the human designer wanting to employ

a graphics system for computer-aided design.

Following is a suggested list of such characteristics. They are

identified in alphabetical order but their respective definitions are

not given in dictionary style. Instead, because the general meaning

of each characteristic is commonly known, one selected question of

evaluative interest (along with some brief commentary) is used to

indicate the specialized meaning of each term.

INDICATIVE QUESTION

CHARACTERISTIC

1. Accessibility

O F EVALUATIVE INTEREST

Can the designer gain access to (i.e.

use) the graphics system hardware/

software when necessary/desirable?

If you can't get at the system (on

some schedule), you obviously can't

get your design task accomplished.

2. Accuracy Are the computations as well as

graphic positionings accurate?

The results of graphics-aided design

become understandably suspect if

numeric/positional errors (beyond

some marain) are detected.

Association Can objects and/or design parameters

be associated both explicitly (by graphic

11
4. Compatibility

5. Completeness ,

6. Continuity

connection) and implicitly (by proximity)?

The human mind quite naturally establishes

such association links. Graphics should

facilitate them.

Are the graphic display features (e.g.

screen size, character size, color, etc.)

and performance (e.g. interaction rate,

flickering) compatible with human capability? For better man-machine performance, the graphics machine should be

tailored to be conducive to and capitalize

on man's capabilities.

Does the repertoire of available graphic

manipulations include the normally expected move, rotate, zoom, erase, etc.? If

the graphics facility is incomplete or

comparatively primitive in the operations

enabled, much of its attraction is lost.

In moving/manipulating the displayed

graphic objects, is the sequence of

system actions non-sporadic? Highly

irregular or unpredictably spaced

motions or actions on the screen

become disturbing.

12
7. Control Does the designer maintain control

over the design process (e.g. can

he interrupt it)? Having no recourse

but to helplessly wait for the system

to do its thing is both unproductive

and undesirable.

8. Flexibility

9. Mobility

Can the graphic objects (including

characters) be modified in size and

shape? Dimensional restrictions

preclude many of the more interesting

and useful effects which interactive

graphics should provide.

Does the designer have enough freedom

to move the graphics objects on the

screen? If the display area itself,

the available operations, or the

interactive tools (e.g. light pen) ^

make movement cumbersome or restrictive,

the graphic environment is comparatively

static.

10. Order Are the command menu, the graphic

symbols menu, etc., displayed/

displayable in meaningful arrangements?

The human mind tends to find things

13
more easily when knowingly and sensibly

ordered (e.g. alphabetically).

1 1 . Power

12. Reliability

13. Responsiveness

14. Simplicity

Does the graphics system have enough

capacity and capability to handle

complex designs? If it can only be

used for very simple designs, its

value to the sophisticated designer

will become marginal

Is the graphic system reliable in

operation? If not, the designer's

confidence and interest in the system

will diminish.

Is the response time good enough to

satisfy designer expectations? To

close the man-machine communication

gap, the machine must be rendered

(appropriately) responsive.

Are the interactive language and

procedures simple enough to enable

easy learning? If the steps towards

becoming reasonably proficient are too

unwieldy and require too much memorization, the average designer will not

be served well

14
15. Spatial Reference Does the designer's sense of spatial

reference get adequate reinforcement

on the graphics screen? That is,

afterall, one of the important

justifications for graphics.

16. Support Can the designer request and get

(on-line) help when he needs it?

When puzzled about what action to

take next, some effective/friendly

form of assistance goes a long way

towards making the user more at ease.

17. Versatility Is the graphics system versatile in

enabling different types of design,

different I/O devices, etc.? If you

prefer and are used to a light pen,

you should not be forced to use a joystick or some other means.

Depending on precise interpretation of the above-listed characteristics,

others could undoubtedly be added. Furthermore, interrelationships obviously

exist. For example, a system cannot be adequately responsive if it lacks

the required power. It is also clearly impossible to achieve the espoused

flexibility and versatility unless a minimally complete repertoire of

graphic manipulations is available. Nevertheless, each of the characteristics

is important enough to warrant separate recognition.

15
Since our objective is to prepare for the testing of the MEDEA system

with respect to (at least) some of these characteristics, we should attempt

to relate them to the interaction stages described in Subsection 2.2. Does

the beginning interactive designer (Stage 1) perhaps view one or more of

the suggested characteristics differently than when he reaches a later

stage (e.g. Stage 4)? Table 1 portrays possible ranges of major applicability.

As is true for the characteristics themselves, the indicated ranges await

empirical verification.

2.5 Emphasis on Symbiosis

As stated in Subsection 2.1 and as confirmed by the list of desirable

features identified in the previous subsection, the main thrust of this

report is on the symbiotic relationship which may or may not exist between

the designer and his graphics system. All of the characteristics, from

accessibility to versatility, have one thing in common: If representative

evaluative questions must be answered in the negative, then the human user

is somehow detrimentally affected. Either he is consequently less satisfied

and hence (perhaps) less inclined to work well, or he may be performing

below his capabilities unknowingly due to certain system incompatibilities

or failures which do but need not exist.

The importance of the symbiosis aspect should therefore be clear.

Even though it represents only one of the three major aspects of the SEE

3 approach and even though it has more often than not been ignored or

conveniently set aside as too difficult to handle, symbiosis should

actually be of priority concern. Not only does it have obvious bearing

on the human-in-the-loop, but it also exerts substantial influence on

system efficiency and effectiveness.

16
Graphical Interaction

Characteristic

Stage 1 Stage 2 Stage 3 Stage 4

Accessibility

Accuracy *• '

'

Association '

Compatibility * ~ ~ '

Completeness "

Continuity *~" '

Control ' —

Flexibility *

;

Mobility " '

Order " ~~~ ———————

Power '

Reliability '

Responsiveness *

Simplicity ' ~^

Spatial Reference " ~ " ' —

Support "

Versatility * ~ ~ —

Table 1. Ranges of Major Applicability

17
2.6 Efficiency and Effectiveness

Just because the graphics system exhibits a certain symbiotic

characteristic, does that necessarily make it any more efficient or

effective? In some cases, especially reliability, one would certainly

expect so. In others, such as mobility and versatility, it is not as

clear. Expenditures for too much power and too many other nice capabilities can conceivably become luxurious in nature, particularly if

the users and their normal work assignments do not require such features.

Consequently, cost effectiveness may not be demonstrable for certain

profiles of graphic system characteristics. But the complex interrelationships existing between user satisfaction and motivation and

productivity, etc., must first be more thoroughly studied, tested, and

analyzed before reaching any general conclusions.

3. EXPERIMENTS FOR THE MEDEA SYSTEM

3.1 Data Collection Requirements

A list of characteristics asserted to be of significance to the

designer-system interface has been presented in Section 2. After

studying each of these very closely and after pinpointing the related

questions of evaluative interest, we can more easily determine what data

(and measures which meaningfully transform those data) are required to

arrive at evaluative answers.

Identification of the required data, in turn, enables a prescription

of what software and hardware mechanisms should be implemented to collect/

measure those data. After such implementation and data collection, the

data analysis and methodology for reaching evaluative conclusions, about

18
MEDEA will depend on whether the decision-oriented or threshold-oriented

3 approach is taken .

The above-indicated procedure for deciding on required data and data

collection techniques should be carried out with respect to each individualcharacteristic under consideration. For example, if reliability is to be

determined, we must ask what is necessary to find out whether MEDEA is

reliable. A similar question must be asked for any other characteristic

to be specifically studied. To preclude the possibility of duplicated

or redundant implementation/instrumentation efforts, the data collection

requirements can then be pooled for all the characteristics to be tested.

Simplicity and responsiveness are being recommended as appropriate

characteristics to be sought in the MEDEA system. Their selection will

be justified in the next subsection and respective experimentation is

described in Subsection 3.3 A system is implied to be "simple" if its

users are observed to require only short time periods for system-related

thinking, to issue infrequent requests for help, and to have learned

relatively little specialized technical information. A "responsive"

system, on the other hand, reacts to a user command after a time interval

which is compatible with the user-required delays for thinking and

psychological adjustment.

1. To ascertain whether MEDEA exhibits simpl ici ty , required data

include: 1

a. A record of the actual interaction sequence or dialogue

involved. \

b. A record of the "think times" as well as total times expended

in various phases of the design process.

19
iL' * , :ir c. A record of requests for assistance made by the designer.

d. A record of design opinions expressed during or after a /'r design session.

e. Any data to substantiate how much the designer had to learn

"

' ' ' - < in order to use MEDEA effectively.

To collect these data, the following tools/techniques are called for:

• a. An on-line graphics dialogue monitor and recorder.

b. Time tagging of all messages (or graphic display manipulations)

; c. An on-line opinion gathering mechanism.

d. A post-session questionnaire.

e. A special -purpose designer learning experiment (described in

Subsection 3.3)

.

2. To find out whether MEDEA exhibits responsiveness , required data include

a. A record of all response times.

; b. A record of the interaction sequence to determine individual

e design subtask completion.

c. A record of expressed designer opinions on response times.

; d. Any data to indicate just how often the designer waits for a late

system response or is unprepared to react to an early system

response. . .

To collect these data, essentially, the same types of data collection

tools/techniques are needed as for studying simplicity (see above).

However, a different special -purpose experiment is suitable (described

in Subsection 3.3).

20
3.2 Levels of Experimental Complexity

Although there may be various similarities among data collection

tools and techniques employable in testing for the previously described

system characteristics, that is not to say that the experimental complexity is about the same. In fact, depending on the type of characteristic and the nature of the data collection sources available and

applicable to it, the level of complexity in experimentation may vary

considerably.

A characterization of these differences is shown in Table 2. It

is intended to emphasize the existing spectrum of data collection sources,

ranging from highly objective and easily obtainable system records and logs

to the very complex psychological/physiological explanations of man's

interaction with a machine. The latter are obviously not as easily

dealt with and even understood. The categorization (in Table 2) of the

characteristics according to complexity levels is not asserted to be

definite or final. It merely represents current intuition-based interpretation of the meaning of each term and consequent expectation of the

difficulty in experimenting with/for it.

As previously stated,"responsiveness" and "simplicity" have been

selected for initial attempts at pertinent experimentation. According

to Table 2, these characteristics are considered to be at levels 2 and

3 respectively. Hence they should be representative of the more interesting and challenging characteristics to test for. Furthermore, because

both of them will require the availability of suitable experimental

subjects (i.e. human designers), the latter should be at appropriate

stages of graphical interaction (see Table 1).

21
' NATURE OF DATA

LEVEL COLLECTION SOURCES

Available records, logs,

^1 technical descriptions,

i

: direct observations

In addition to the above:

Subjective designer judg

ments and opinions

In addition to the above:

More complex psychological

and physiological

consi derations

APPLICABLE

CHARACTERISTICS

access i bi 1 i ty

accuracy

compl eteness

f 1 exi bi 1 i ty

rel i abi 1 i ty

versa ti 1 i ty

cont i nu i ty

control

mobi 1 i ty

power

responsiveness

support

association

compa ti bi 1 i ty

| order |

s i mpl i c i ty I

spatial references!

Table 2. Levels of Experimental Complexity

22
The simplicity (or learnabil ity) characteristic seems to be

especially important during early stages. Hence related experimentation

will be possible with the group of designers who will first be getting

access to and learning about the MEDEA facility as it becomes operationally available. Responsiveness, on the other hand, appears to be a

popular and logical follow-on in that it becomes more prominent beyond

the learning stage. Thus, perhaps the same group of designers could be

available for and willing to participate in both experiments.

Another important justification for the selection of simplicity and

responsiveness experiments is indicated in the next subsection.

3.3 Description of Suggested Experiments

In previously provided documentation about ECOM's Design Terminal

Concept and as recently restated by Gray^\ one of the design specifications is that

...the man that is put in the loop

be a problem solver, not a computer

specialist.

Among the performance objectives for the graphics terminal. Gray not

only included ease of use but also an interaction rate commensurate

with the user's activities.

These guidelines clearly lend themselves to corrobation of the

simplicity and responsiveness experiments described below.

3.3.1 Simplicity

To test and evaluate MEDEA with respect to this criterion, we should

design an experiment which may ascertain how easily the usage of MEDEA

can be learned. Does the human user have to become something of a computer

23
specialist before he can really understand and operate the MEDEA system

to advantage?

Hence, with regard to "simplicity," the following experiment is

outlined:

1. OBJECTIVE: To determine how frequently and under what circumstances

, the designer requests/requires specific assistance and/or

makes identifiable mistakes, and to observe the distributions of those numbers across n design sessions by each

user. Secondly, to study how many "computer specialties"

are learned by the uninitiated user between the start of

session 1 and the finish of session n.

2. PREREQUISITES: a. Availability and selection of a suitable group

. .;r v . of experimental subjects who are not experienced

with MEDEA (or other interactive graphics systems),

b. Identification of a design assignment, of representative complexity and reasonable duration,

which each of the subjects could learn to carry

; , out.

, -; , c. Possibly the same kind of special keying mechanism

required in conjunction with the responsiveness

experiment (Subsection 3.3.2), to enable the

user to indicate when assistance is required or

when a subtask is difficult to understand or

,

> , accomplish.

24
d. Clear identification of a manageable set of

" computer specialties ." or concepts/topics

which a non-computer specialist is not expected

to know about while a person with adequate

experience in the area should,

e. Carefully prepared pre- and post-experimental

questionnai res to probe into and distinguish

how much the user knows about the selected

"computer specialties" before and after the

sequence of n interactive graphics sessions.

f. Scheduling for and planning the control over

the experimental sessions,

INDICATOR DEVICE: User-indicated trouble spots requiring assistance

can of course be verbally communicated to an

available staff member. However, it would be

preferable to supplement the verbal mode by a

simple automatic logging procedure (which perhaps

could also activate a buzzer/light to call for

staff help). The keying device described in

Subsection 3.3.2 could serve this purpose.

GENERAL PROCEDURE: Prior to a new user's first design session,

he/she is properly briefed (including a

minimal demonstration) and then asked to

complete the pre-experimental questionnaire.

25
Then the user can proceed with the scheduled

number of design sessions for purposes of

accomplishing the assigned design task(s).

Throughout these sessions, requests for

assistance must be communicated (and keyed in)

.' ;. - 5y ij^Q Qf course anticipated and re-

' corded by available staff.

After the sequence of design sessions is finished

the user is asked to complete the post-experimental

- questionnaire and also debriefed to gain any

opinions about learning requirements or diffi-

... cul ties.

5. EXPECTED RESULTS: The data collected as described above and also by

. means of other tools (listed in Subsection 3.1),

should enable analysis for determining at least

the following:

a. (Individual and collective) distributions of

the number of requests for assistance and the

number of committed mistakes, from the first

to the nth sessions.

b. Average % of learning (or some other measure)

of the set of "computer specialties" over

the period of design sessions.

c. Dependencies or relationships (if any) of the

data considered in a. and b. with respect to

other recorded user characteristics or experience.

26
d. Comparison of the real-time user learning

performance with later user opinions

(gathered by questionnaire).

e. Feasibility of this technique.

3.3.2 Responsiveness

The testing and evaluation of the ECOM graphics facility to determine,

among other characteristics, its level of responsiveness, requires that

we employ several different data collection tools. These were outlined

in Subsection 3.1. To supplement the data on actual response times

experienced etc., an appropriate, specially designed experiment is

possible. It is outlined as follows:

1. OBJECTIVE: To determine how frequently and under what circumstances

the designer is mentally prepared to tackle the next

design step but is unable to do so because the graphics

system (ECOM's MEDEA) has not yet responded. Secondarily,

to consider the opposite problem of the response being at

times too fast for the designer's rate of assimilation or

preparation

2. PREREQUISITES: a. Availability and selection of a suitable group

of experimental subjects who have advanced

beyond the beginner's stage and can give serious

thought and attention to the overall interaction

flow.

b. Identification of a design assignment, of representative complexity and reasonable duration,

which each of the subjects could carry out.

27
c. Design and implementation of the facility for

enabling the designer to key in the desired

response time indications and for having these

properly time-tagged (along with the normal

^ interaction sequence).

.A d. Scheduling for and planning the control over

the experimental sessions.

3. VARIATION: It may be desirable not only to enable the designer to

- : indicate "too slow" and "too fast" response time, but

• 1'. also (especially for the former case) to let him qualify

the indication by distinguishing warranted from unwarranted

delay. If the designer is knowledgeable enough to recognize

v:. ' that the subtask being executed warrants a longer response

f'c/r-r - time, he should in effect be able to say: "I'm ready,

^ but I realize that the system is working on a complex

• subtask.

4. INDICATOR DEVICE: If the objective as stated and as qualified above

is to be accomplished, the keying in of userindicated time spots must be enabled. Possible

enabling devices include:

a. An autonomous little keyboard (such as the

Keyset used by Doug Englebart's group at

b. A grouping of (at least three) functions or

/ other special buttons in the existing keyboard.

28
c. A grouping of (at least three) selectable

dots displayed in some corner of the graphics

screen

.

In any case, the choice should make it as simple,

non-confusing, and non-disrupting to the designer

as is possible.

GENERAL PROCEDURE: Whenever the designer has completed his side of

a transaction and is consciously sitting there

and realizing that he is ready to undertake the

next step, he should be able to simply push/

select a special button to so indicate. An

associated button could be available in case he

recognizes that the delay is warranted.

Furthermore, if the response is perhaps faster

than the user can handle it (e.g. possibly

because of a need to reference some hard-copy

tables at that point), a third (labelled) button

could be pushed as soon as he is ready to assimilate the response.

EXPECTED RESULTS: If this experiment is properly carried out, we

should be able to analyze the data and ascertain

at least the following:

a. Average frequencies of excessive, appropriate,

and unnecessarily fast response times.

29
b. Correlation (if any) of those frequencies

- with different types of design subtasks

(and varying levels of complexity).

" c. Comparison of the real-time user indications

-i--- '' with later user opinions (gathered by questionnaire).

" d. Feasibility of this technique.

4. CONCLUSIONS AND PLANS

'

A detailed characterization of the stages of graphical interaction

and of the graphical terminal features which the human designer deems

to be important has produced the groundwork, specifically oriented to

the MEDEA system, on which experimental testing can now be based. The

two suggested experiments concerned with simplicity and responsiveness

reflect the progress made by ECOM towards bringing a symbiotici»lly designed

graphics facility to fruition.

We have reached a point at which we can focus on certain features

and plan for testing MEDEA with respect to their presence/absence or

level of performance. This means that the development of the design

terminal concept has gone far enough to enable our specification or

suggestion of those hardware/software devices or techniques which must

be incorporated in order to support the experimentation desired.

As the multi-user operating system software for MEDEA is completed

in the not too distant future, it will therefore become possible to

realize the data collection tools in operation. Our plan is then to

^ 30 '
continue the interaction between the staff of ECOM and NBS, not only in

terms of providing further guidelines for implementation/instrumentation

but also by means of assistance in the various preparatory, control, data

collection and analysis tasks involved in experimentation.

31
REFERENCES

1. Dunn, R.M., and Gray, B.H., An R&D Program for Interactive Graphics

Computer-Aided Design-Engineering Technology and Systems , Internal

Report No. 1-71, R&D Technical Support Activity, Mathematics/Computer

Aided Design Division, ECOM, February 1971.

2. Abrams, M.D., Hudson, J. A., Meissner, P., Pyke, T.N., Jr., Rosenthal,

R.M., and Ulmer, F.H., Use of Computer Networks in Support of Interactive Graphics for Computer-Aided Design and Engineering , NBSIR

73-217, Sponsored by U.S. Army ECOM, Contract No. 72-90013, June 30,

1972.

3. Pyke, T.N., Treu, S., and Hudson, J. A., Performance Measurement of

Interactive Graphics Systems in Support of Computer Aided Design

and Engineering , NBS Report 10 451, Sponsored by U.S. Army ECOM,

Contract No. 71-96088, June 1971.

4. Treu, S. , "Techniques and Tools for Improving the Interactive System

Interface," Interactive Bibliographic Systems , Ed. by M.B. Henderson,

AEC Symposium Series 28, April 1973, pp. 32-38.

5. Dunn, R.M., "Computer Graphics: Capabilities, Costs, and Usefulness,

Proc. of SHARE XL, Denver, Colorado, March 7, 1973.

6. Miller, R.B., "Response Time in Man-Computer Conversational Transactions," Proc. of FJCC, 1968, pp. 267-277.

32
7. Carbonell, J.R., Elkind, J. I., and Nickerson, R.S., "On the

Psychological Importance of Time in a Time Sharing System," Human

Factors , 10(2), April 1968, pp. 135-142.

8. Boehm, B.W., Seven, M.J., and Watson, R.A., "Interactive ProblemSolving - An Experimental Study of 'Lockout' Effects," Proc. of

SJCC, 1971, pp. 205-210.

9. Katter, R.V., and McCarn, D.B., "AIM-TWX - An Experimental On-Line

Bibliographic Retrieval System," Interactive Bibliographic Search:

The User/ Computer Interface , Ed. by D. Walker, AFIPS Press, 1971,

pp. 121-141.

10. Treu, S., "A Conceptual Framework for the Searcher-System Interface,"

Interactive Bibliographic Search: The User/ Computer Interface ,

Ed. by D. Walker, AFIPS Press, 1971.

11. Gray, B.H., "An Interactive Graphics Design Terminal Concept, Implementation, and Application to Structural Analysis," Proc. of Fourth

Annual Pittsburgh Modeling and Simulation Conference, April 1973.

12. English, W.K., Englebart, D.C., and Berman, M.L., "Display-Selection

Techniques for Text Manipulation," IEEE Transactions on Human Factors,

HFE-8, 1(5), 1967.

33

FOF<M NtlS-n4A n-71)

U.S. DEPT. OF- COMM. 1. I'PHI.IC 1 ION OR Rl.l'OK l' NO. 2. Cov't A< c <-..•; ion BIBLIOGRAPHIC DATA NI'.SiR 7< 7K') SHEET / - 7

3. Recipii mi's Aci is:, ion .No.

4. n i Ll-; AND SUH H I LI Characterization and Testing of Interactive GraphicG

for Computer- Aided Design and Engineering

5. Puhlu atioM D.ite 30 June 197 3

6, HeiforTTiiiiK Ori^anization C(.de

7, AUTTIOR(S)

Siegfried Treu

8. Fcrfornung ( tr^an i ^..i tion

9. PLRFOR.MING ORt, AN i /..M'lON NAM1-: AND ADDRICSS

NATIONAL HURF-AU Oi- STANDARDS

DEPARTMENT OF COMMl.RCE

WASHINGT ON, D.C. 20234

10. Proj'-ct/T ask/'i^ ork Unit 6505373

11. Contract/Grant No. 72-90013

12. Sponsoring Organization Name and Address U.S. .Army Electronics Command

AMSEL-GG-OD

Ft. Monmouth, New Jersey 07703

IJ. 1 ype ot Kepurt Si reriod Covered

Interim 7/72-6/73

14. Sponsoring Agency Code

15. SUPPLEMENTARY NO l ES

16. ABSTR.ACT (A 200-word or less factual summary of most significant information. If document includes a significant

bibliography or literature survey, mention it here.) This report presents material developed as part of a long-term

"Interactive Computer-Aided Techniques Study". The report outlines

the stages of development in the utilization of interactive graphics

as a tool for Com.put er-Aided Design and Engineering (CAD/E). A

series of characteristics are presented v;hich are of significance to

the designers and users of such systems and a series of questions of

evaluative interest posed. These questions are intended to delineate

the extent to which a system under exam.ination achieves its stated

design objectives. The characteristics are grouped in accordance with

the nature and complexity of the experiments v/hich would need to be

conducted to establish values for them. The report suggests selected

characteristics of particular interest and suggests the design of

experiments for examining them in detail. The report makes specific

reference to the MEDEA design terminal concept under development within

the Graphical Systems and Technology Branch of ECOM.

17. KEY uoRDs (Alphabetical order, separated by semicolons) Characteristics; c omput er- aided

design; interactive graphics; man-machine interaction; performance

measurement

18. AVAILABILITY ST ATEMENT 19. Si;CURlTY CLASS

(T HIS REPORT)

21. NO. OF PAGES

XX! UNLIMITED.

UNCLASSIFIED 33

1 ! EOR Ol-FICIAL DISTRIHUTTON. DO NOT RELI-ASE

TO NTIS.

20. SFXURITY CLASS

(IT US PAGE)

UNCI. ASSIITI-D

22. Price

USCOMM-DC 6et44-P7l

r