# FEASIBILITY ASSESSMENT OF AN

# INTERACTIVE RULES OF THE ROAD TESTER

Marvin C. McCallum, Battelle Seattle Research Center,

Myriam W. Smith, U.S. Coast Guard Research and Development Center, and

George Sandberg, Douglas Hard, and Richard Stewart, U.S. Merchant Marine
Academy

## ABSTRACT

This paper describes a project conducted to examine the issues
associated with the development of an Interactive Rules of the Road
Tester (IRORT). Such a test may potentially replace or augment current
United States Coast Guard (USCG) multiple-choice, paper and pencil
licensing exams. An experimental IRORT was successfully developed, using
commercially available desk-top simulator software. The test included
both multiple-choice and bridge operation measures. A systematic
methodology was established and applied in developing a series of
objective performance criteria for the set of bridge operation measures
and the test was administered to 100 merchant marine academy students.
Students' IRORT performance was compared to both their USCG test
performance and to a number of biographical measures obtained to assess
each student's knowledge and ability to apply the Rules of the Road
(ROR). Results of the study indicate that development of an IRORT is
technically feasible using commercially available hardware and software.
Analyses indicated that the interactive test measures real-time decision
making in a dynamic environment in addition to the factual knowledge
measured by multiple-choice test items. Development and implementation
of an IRORT should be considered for testing the proficiency and
competency of mariners for licensing.

## INTRODUCTION

The United States Coast Guard (USCG) examines and licenses merchant
mariners to establish standards of mariner proficiency and competency.
At the present time, most USCG examinations employ a paper and pencil,
multiple-choice format that is limited to the assessment of the
examinees\' factual knowledge. These paper and pencil tests do not
appear to measure the examinees' ability to make real-time decisions in
a dynamic operational environment. Recent improvements in the
accessibility and functionality of desk-top simulators provides an
opportunity to assess the feasibility of employing these simulators to
expand mariner proficiency and competency testing. The USCG is
sponsoring a preliminary study of an Interactive Rules of the Road
Tester (IRORT). The expectation is that an IRORT can allow the
measurement of an examinee's ability to apply Rules of the Road (ROR)
knowledge in a real-time operational context, which would enhance the
validity and diagnostic value of current USCG examinations. The purpose
of the study described here was to investigate the issues associated
with interactive testing and to determine whether the approach warranted
further, more complete consideration.

### Performance Measurement Issues

Simulators have been employed as a means of cost-effective training
since the 1940's. However, as simulators continue to evolve, fundamental
issues in determining how operator performance relates to real-world
performance have continued to be a barrier to their effective
implementation. In a general review of training simulators, Vruels and
Obermayer (1985) identified three fundamental issues in human-simulator
performance measurement: needs for assessing hidden knowledge and
embedded performance, establishing operational performance criteria, and
demonstrating the validity of the performance measures employed.

In discussing his experiences with measuring the effectiveness of
shiphandling simulator training, Gynther (1985) stressed the importance
of developing secondary performance measures, in addition to collision
avoidance measures. Specific measures recommended by Gynther included:
Closest Point of Approach (CPA), Range of Maneuver, Direction of Course
Change (Left/Right), Magnitude of Course Change, Number of Course
Orders, Number of Visual Bearings, Number of Radar Requests, and
Percentage of Correct Whistle Use. Jinseng and Fengchen (1992) have
addressed the issue of operational criteria for CPA measures by
employing a database comprised of at-sea anticollision actions. This
database provides a formal basis for the validity of their collision
avoidance performance criteria. However, a comprehensive set of
performance criteria for secondary measures is still required.

### Previous Interactive Tester Development Efforts

Simulators have been employed for training and certification throughout
the maritime industry for some time. Jenkins and Theedom (1993) discuss
the use of large-scale simulators for watchkeeping examination in
Canada. However, such systems continue to be very costly to procure and
operate, and cannot be employed economically for the thousands of
licensing examinations administered annually by the USCG. What is
required is a desk-top simulator that uses a personal computer for its
hardware components.

A prototype Mariner Licensing Device was developed through modification
of an existing desk-top simulator during an earlier USCG-funded effort
(Engineering Computer Optecnomics, Inc., 1987; Gardenier, Flyntz,
Spears, Willis, & North, 1987). This device provided displays for
visual, radar, and instruments; and controls for rudder, engine, radar,
and whistle signal actions. Evaluation of the prototype device included
the administration of an interactive test to 26 mariners. Bridge
operations performance was scored manually during this evaluation and it
was noted by the researchers that further development of standardized
and automatic scoring procedures was required. This initial effort was
successful in many areas; however, the further development of the
Mariner Licensing Device did not prove cost-effective at that time.

A recent review of an interactive desk-top bridge simulator by Hughes
(1993) describes the Officer of the Watch product designed by PC
Maritime, Ltd. to train procedures and strategies for minimizing the
risk of collision at sea. The Officer of the Watch system provides three
basic screens: bridge view, radar view, and chart view, each with
ship-handling controls concurrently displayed in the bottom portion of
the screen. The student can select each of the basic displays; as well
as select the direction of view from the bridge, use of a visual bearing
repeater, use of binoculars, engine telegraph controls, a whistle, and
helm controls in either manual or autopilot mode. All student control
actions are automatically recorded in files. This latter capability is
critical to successful development of an IRORT, since it provides the
basis for secondary performance measurement and analysis.

### Study Objectives

The basic objective of this study was to apply existing desk-top
simulator technology to develop an experimental IRORT. This development
effort was to provide the basis for assessing the feasibility of
developing an interactive tester and determine if further consideration
of such a system is warranted. Specific issues to be addressed during
the course of the feasibility study included:

> • Availability of affordable desk-top simulators with adequate
> functional capabilities;
>
> • Requirements for development of realistic scenarios that depict
> typical collision avoidance situations and require examinee actions
> that are comparable to those aboard ship;
>
> • Requirements for the development of performance measures and scoring
> criteria that correspond to appropriate collision avoidance actions
> aboard ship; and
>
> • The capability of the experimental IRORT to provide a valid
> assessment of examinee proficiency and competence.

## IRORT TEST DEVELOPMENT

Using Officer of the Watch desk-top simulator software and
IBM-compatible computers, U.S. Merchant Marine Academy (USMMA) faculty
developed a series of four IRORT examination scenarios. Use of Officer
of the Watch software does not constitute an endorsement by the USCG.
Examination scenarios and test items were modeled after a single version
of the USCG Third Mate's ROR examination, in an effort to reduce test
development costs and maintain consistency with current USCG test
objectives and emphasis. Development of the IRORT examination scenarios
was completed without encountering major technical difficulties.

IRORT examination scenarios were designed to assess factual knowledge;
recognition of traffic conditions, lights, and signals; and operational
performance. Factual knowledge items were formatted similarly to paper
and pencil, multiple-choice items; but these items were displayed on a
computer screen and responded to via cursor operation. Recognition items
were multiple-choice questions embedded within a dynamic operational
display. Finally, three dynamic interactive scenarios were developed to
provide an opportunity to assess examinees' ability to apply the ROR
during bridge operations. Separate scenarios presented examinees with
dynamic, open-sea vessel traffic situations, with between five and seven
other vessels, in day, night, and fog conditions (Stewart, Sandberg,
Meurn, and Hard, 1994).

A systematic methodology was established and applied in the development
of a series of bridge operation performance measures and performance
scoring criteria. These performance measures were developed by first
identifying three basic requirements of navigation law during watch:

> • Maintain a good lookout and determine if risk of collision exists;
>
> • Take appropriate action or maneuver to avoid collision; and
>
> • Determine if own ship's action or maneuver was adequate to avoid
> collision, and ensure that the action or maneuver does not put own
> ship in a close quarters situation with other vessels.

These general requirements were extrapolated into a series of bridge
operation measures. Proposed bridge operation measures addressed visual
search, binocular viewing, visual bearing, radar viewing, use of sound
signals, vessel speed, maneuver direction, projected CPA at time of
maneuver, and actual CPA. One of two general scoring criteria were
applied to each bridge operation measure. In most cases, a three-point
Proficiency Level procedure was applied, scoring performance as Expert,
Qualified, or Unqualified. In selected cases, a two-point Competency
Level criterion was applied, scoring performance as either Competent or
Incompetent.

For each bridge operation measure, a modified Delphi technique (Meister,
1985) was employed to define performance criteria corresponding to
specific Proficiency or Competency Levels. During this process, USMMA
expert mariners reviewed the operational situation for a given test
scenario then individually recommended performance criteria. The set of
recommended performance criteria were then reviewed by the group of
expert mariners until a consensus criterion was obtained for the
specific measure and scenario. This process was repeated for each bridge
operation measure in each scenario.

The development of the final set of IRORT measures and scoring criteria
represented a substantial accomplishment for the present project, and
demonstrated the general feasibility of applying a systematic
methodology to the development of quantitative criteria for secondary
performance measures. Subsequent efforts involved the development of
data reduction computer programs that were applied to the data saved
after test administration. This effort was generally successful; however
two measures, direction of turn and projected CPA, proved too difficult
to implement within the scope of the present project. Table 1 provides a
summary of the final set of paper and pencil test and IRORT measures.

#### Table 1. Summary of Paper & Pencil and IRORT Performance Measures

  ------------------- ---------------------------------------------------
  **Performance       **Description**
  Measure**           

  USCG                Sum of all correct paper & pencil multiple-choice
  Multiple-choice     items

  IRORT               Sum of all correct computer-based multiple-choice
  Multiple-choice     items

  Visual Search       Score based on expert mariners' expected time spent
                      visually searching each of four quadrants of view
                      during the first six minutes of each test scenario

  Binocular Viewing   Score based on expert mariners' expected frequency
                      of viewing each vessel through the binoculars
                      during the first six minutes of each test scenario

  Visual Bearing      Score based on expert mariners' expected frequency
                      of visual bearings of each vessel to be taken
                      during the first six minutes of each test scenario

  Radar Viewing       Score based on expert mariners' expected frequency
                      and total duration of radar viewings during the
                      first six minutes of each test scenario

  Maneuver Signal     Proportion of times a correct signal was sounded
                      within 30 seconds of a maneuver

  CPA                 Score based on expert mariners' expected minimum
                      closest point of approach to other vessels
                      throughout each test scenario

  Radar Maneuver      Score based on expert mariners' expected frequency
                      and total duration of radar viewings during the
                      first six minutes after the first own ship maneuver

  Fog Signal          Score based on expert mariners' expected time of
                      first sounding of the restricted visibility sound
                      signal during the fog scenario

  Fog Speed           Score based on expert mariners' expected time and
                      level of action to reduce speed following radar
                      failure during the fog scenario
  ------------------- ---------------------------------------------------

## TEST ADMINISTRATION

In order to examine the potential benefit of an IRORT, the
simulator-based test and the USCG multiple-choice exam were administered
to 100 U.S. Merchant Marine Academy midshipmen. The USMMA student
participants included 50 seniors, 25 juniors, and 25 sophomores. This
distribution of students was employed in an effort to obtain a broad
range of test scores. All students were paid a small remuneration for
their participation. It is noteworthy that all students at USMMA own a
relatively recent model of an IBM-compatible notebook computer, which
they use regularly to complete course assignments. Thus, all examinees
were considered to have basic computer skills.

The process of student testing is depicted in the flowchart in Figure 1.
Following agreement to participate, students were assigned to a specific
examination schedule, with approximately 50% of the students being
administered the USCG multiple-choice test first and the remaining
students being administered the IRORT exam first. At the time of their
first examination session, students completed a brief biographical
survey. Then they were administered the appropriate test. Administration
of the USCG multiple-choice test followed prescribed examination
procedures. Students were allowed up to 2 hours to complete the USCG
multiple-choice test, although few students took longer than one hour to
complete the test. IRORT examination sessions were conducted in group
sessions of between 6 and 8 students at individual computer stations. At
the beginning of each session, the test proctor lectured all students on
the operation of the OOW software, which included a demonstration on an
overhead projector of all functions available to the students. Then,
students completed a hands-on practice session, followed by the four
IRORT test scenarios. Each scenario was approximately 20 minutes in
length. Finally, after completion of the IRORT session, students
completed brief feedback surveys. The entire IRORT testing session
required approximately two hours.

Following completion of testing, additional information that was judged
to be relevant to ROR knowledge and application was obtained for each
student. Table 2 provides a summary of the individual biographical
measures that were obtained.

#### Table 2. Summary of Student Biographical Measures

  ---------------------- ----------------------------------------------------
  **Biographical         **Description**
  Measure**              

  Faculty−Knowledge      Faculty rating of the student\'s knowledge of the
                         ROR

  Faculty−Application    Faculty rating of the student\'s ability to apply
                         the ROR

  Faculty−Lookout        Faculty rating of the student's ability to maintain
                         a proper lookout and to apply proper bridge
                         watchstanding procedures

  Faculty−Shiphandling   Faculty rating of the student\'s shiphandling skills

  Faculty−Sleep          Faculty rating of their ability to sleep on board a
                         vessel they mastered, knowing that the student was
                         the Officer of the Watch

  Student−Year           Student's academy class at the time of study
                         participation (senior, junior, sophomore)

  Student−Classroom      Self-rating of ROR knowledge obtained from classroom
                         training

  Student−Simulation     Self-rating of ROR knowledge obtained from
                         simulation training

  Student−Sea            Self-rating of ROR knowledge obtained from sea
                         projects

  Grades−Freshman        A summary score based on grades from two freshman
                         courses relevant to ROR

  Grades−Sophomore       A summary score based on grades from one sophomore
                         course relevant to ROR

  Grades−Senior          A summary score based on grades from one senior
                         course relevant to ROR

  Grades−Recent          A single summary score based on the standardized
                         course grades from the student's most recent
                         relevant ROR course

  Grades−Simulator       A single course grade resulting from a structured
                         evaluation of students planning, teamwork, and
                         bridge operations during large-scale simulation
                         training
  ---------------------- ----------------------------------------------------

## RESULTS

The basic analysis strategy was to calculate a series of correlations.
These correlations provided an index of how well a set of
**performance** scores could be predicted on the basis of a set of
**biographical** scores. Correlation values range between 0.0 and 1.0;
and can be either positive or negative in value. A correlation of 1.0 or
-1.0 would be obtained if a plot of performance and biographical scores
fell on a single line. A correlation of 0.0 would be obtained if a plot
of performance and biographical scores represented a sphere. Two basic
questions were addressed by these analyses:

> • Do the separate test formats of USCG multiple-choice, IRORT
> multiple-choice, and IRORT bridge operation measure different aspects
> of performance?
>
> • Would the dynamic simulation and interactive characteristics of the
> IRORT increase the capability of licensing tests to identify
> proficient and competent mariners?

### Do The Separate Test Formats Measure Different Aspects Of Performance?

A series of analyses were performed to compare the correlations obtained
between two subsets of biographical measures and student scores obtained
from the three test formats, USCG Multiple-choice, IRORT
Multiple-choice, and IRORT Bridge Operations. The two subsets of
biographical measures were designed to separately represent students\'
ROR Knowledge and their Application of ROR. ROR Knowledge was comprised
of the Faculty−Knowledge and Student−Classroom biographical measures.
ROR Application was comprised of the Faculty−Application,
Student−Simulation, and Student−Sea biographical measures. Multiple
regression analyses were performed with the USCG Multiple-choice and
IRORT Multiple-choice scores, and first-root canonical correlations were
performed with the set of Bridge Operation measures to obtain the
correlation values.

The results of these analyses, presented in Table 3, were inconclusive.
Moderate correlations were obtained in all cases, supporting the
validity of all test scores. All correlations were statistically
significant, with the exception of the .47 correlation between Bridge
Operations and Application measures, which was slightly above a
probability level of .05. Although most correlations were significant,
the expected outcome of higher correlations between (1) the
Multiple-choice test format scores and the Knowledge biographical
measures and (2) the Bridge Operation scores and the Application
biographical measures were not obtained. These inconclusive findings are
likely the combined result of overlapping biographical measures and
commonalities in the skills and knowledge measured by the three test
formats.

#### Table 3. Correlations of Knowledge and Application Measures with Performance Test Scores

  ------------------------------ -------------------- --------------------
                                 **Biographical       
                                 Measures**           

  **Performance Measure          **Knowledge Subset** **Application
  Subsets**                                           Subset**

  USCG Multiple-choice           .53                  .45

  IRORT Multiple-choice          .40                  .39

  Bridge Operation Measures      .50                  .47
  ------------------------------ -------------------- --------------------

A second series of analyses was conducted to determine if the three test
formats measured different aspects of ROR knowledge and application.
Following the completion of the present testing sessions, academy
seniors completed a USMMA course that included weekly operation on a
full-scale bridge simulator. At the end of this training, students'
performance was assessed using a 47-item assessment instrument that is
used to score team and individual performance in the general areas of
Execution, Monitoring, Appraisal and Planning, and Organization and
Teamwork. This simulation score provided the best available measure of
students' ability to apply their knowledge in bridge operations. So, it
would be expected that the IRORT Bridge Operation scores would have a
stronger positive correlation with the simulation score. Figure 2
graphically presents the correlation values for these three comparisons.
Both multiple-choice test scores had negative, statistically
non-significant correlations with the simulation grade, -.10 for the
USCG test and -.11 for the IRORT test. These negative correlations
indicate that good simulation performance corresponded to somewhat
poorer multiple-choice test scores. In contrast, a moderate and
statistically significant positive correlation of .29 was obtained
between the simulation scores and Bridge Operation scores. This pattern
of results indicates that the multiple-choice and Bridge Operation
scores are measuring some different aspects of ROR knowledge and
application.

### Would The Dynamic and Interactive IRORT Test Formats Improve Licensing Examinations?

Given that the IRORT Bridge Operation scores apparently measure
different aspects of ROR Knowledge and Application, a next question
concerns the added benefit of the unique aspects of IRORT testing, as
compared to paper and pencil testing. There are two aspects of
performance measurement that are different: (1) examinees observe
dynamic vessel traffic situations; and (2) examinees must perform bridge
operations. In addition, computer-based testing, such as IRORT, affords
the test administrator the capability of maintaining computer-based test
files, which provide the opportunity of taking advantage of statistical
routines to optimize test items weights in discriminating between
licensing candidates on the basis of their proficiency and competency
levels.

A final series of correlational analyses were performed to assess the
potential benefits of these characteristics of IRORT. In all of these
analyses, a set of IRORT scores were correlated with the full set of
biographical measures, providing an indication of the degree of
association between the set of proficiency and competency predictors and
IRORT performance. Figure 3 depicts the results of these analyses. In
all cases, the correlations presented here were statistically
significant. Starting with the left-hand bar in Figure 3, the subset of
IRORT multiple-choice items presented in a static display that required
recall of factual knowledge had a moderate correlation with the
biographical measures of .55. Next, when the IRORT multiple-choice items
presented in a dynamic display requiring recognition are added to the
analysis, the correlation increases slightly to .59. When all IRORT test
scores are included in the analysis, the correlation increases to .68.
Finally, when statistically derived weights for all test items are
incorporated into the analysis, as could be done readily with this
computer-based testing, a relatively strong correlation of .79 is
obtained.

## DISCUSSION

Although the pattern of results discussed in the preceding section are
not definitive, they suggest the general relationships among the
different test formats and mariners' ROR knowledge and skills that is
depicted in Figure 4. In this Venn diagram, the large, dark circle
labeled *ROR Operational Knowledge and Application Skills*, represents
the domain of interest in the present research. The top-left circle
labeled *Current USCG Multiple-choice Test Performance* indicates that
this test measures a subset of ROR Operational Knowledge and
Application, but also measures other aspects of performance; likely
general paper and pencil test taking skills. The bottom-left circle
labeled *IRORT Factual and Recognition Test Performance* indicates that
these test items measure a comparable, but somewhat different, subset of
Operational ROR Knowledge and Application, plus some other aspects of
performance, likely test-taking and computer operation skills. Finally,
the larger, right-hand circle *labeled IRORT Bridge Operation
Performance* indicates that this aspect of the test measures a
substantial portion of mariners' ROR Operational Knowledge and
Application that is not assessed by either of the other test formats, in
addition to other aspects of performance. These additional aspects of
performance are likely computer operation skills and the capability to
respond operationally in a simulated environment.

![](media/image1.emf){width="3.0694444444444446in"
height="2.486111111111111in"}

#### Figure 4. General Relationships among the Different Test Formats and Mariners' ROR Knowledge and Skills.

Based on the IRORT development experiences and the pattern of obtained
results, the following conclusions have been made:

-   Commercially available desk-top simulators can support the
    development of an IRORT.

-   Employing a structured, modified Delphi technique to the
    specification of performance criteria provides a manageable means of
    defining scoring procedures for simulation testing.

-   Although current simulator capabilities did not provide adequate
    automatic scoring functions, the present effort demonstrated the
    feasibility of developing and implementing automated scoring
    routines for a wide variety of secondary performance measures.

-   Use of computer-based testing allows for the application of complex
    scoring algorithms that can further increase the diagnosticity and
    validity of an IRORT.

-   Although the academic setting of the present study limited both the
    range of subjects and the generality of the obtained findings,
    results indicate that a desk-top IRORT measures additional aspects
    of students' ROR-relevant skills and knowledge that are not measured
    by a paper and pencil test.

-   Further development of interactive certification testers is
    indicated; as the present results indicate they have the potential
    to increase the capability of paper and pencil exams to identify
    proficient and competent mariners.

## ACKNOWLEDGMENTS

The authors received assistance and advice from many individuals during
the current effort. Alvah Bittner provided consultation on statistical
analyses. Ann Sanborn and Robert Meurn of the USMMA assisted in the
definition of test measures and the collection of student biographical
measures. Daniel Baker provided programming assistance for data
reduction. The staff of PC Maritime and Houston Marine were very helpful
in supporting their products.

## REFERENCES

Engineering Computer Optecnomics, Inc., (1987). *Mariner Licensing
Device Final Report.* (Prepared for United States Coast Guard,
Washington, DC 20593, 27 May 1987.)

Gardenier, J.S., Flyntz, F.J., Spears, R.S., Willis, R.M., & North, D.C.
(1987). A desktop simulator for licensing mariners*.* Proceedings of
*The Fourth International Conference on Marine Simulation (MARSIM-87).*
Trondheim, Norway, June 22-24, 1987.

Gynther, J.W. (1985). Measuring the effectiveness of shiphandling
simulator based training*.* In *Proceedings of the Society for Computer
Simulation,* (pp. 53-58).

Hughes, T. (1993). Officer of the Watch simulator. *Seaways*, June, 3-5.

Jenkins, M.E. and Theedom, M.M. (1993). Use of simulators as a training
and examination tool. In *International Conference on Marine Simulation
and Ship Manoeuvrability (MARSIM-93)*. St. John's, Newfoundland, Canada,
September 26 - October 2, 1993.

Jingsong, Z and Fenchen, W. (1992). *The Examination in Simulator
Training*. Paper presented at the Seventh International Radar Navigation
Simulator Lecturers' Conference. U.S. Merchant Marine Academy, Kings
Point, New York, U.S.A., June 29 - July 3, 1992.

Meister, D. (1985). *Behavioral Analysis and Measurement Methods.* New
York: John Wiley & Sons.

Stewart, R.D., Sandberg, G., Meurn, R. and Hard, D. (1994). *Interactive
Rules of the Road Tester: Phase 1*. Prepared by the U.S. Merchant Marine
Academy for the U.S. Coast Guard Research and Development Center.

Vruels, D. and Obermayer, R.W. (1985). Human-system performance
measurement in training simulators. *Human Factors*, 27(3), 241-250.

*The opinions expressed in this paper are those of the authors and do
not constitute the policy of the U.S. Coast Guard or U.S. Merchant
Marine Academy.*