From: sbate@atl.ge.com (Stephen Bate)
Newsgroups: alt.consciousness
Subject: Mind/Machine Coupling (was NEXT)
Date: 31 Dec 92 15:10:16
Organization: GE Advanced Technology Laboratories
Distribution: world
Reply-To: sbate@atl.ge.com
NNTP-Posting-Host: mary.atl.ge.com

Using computers to expand human capabilities to store, retrieve, and
process data is very interesting. However, various people in this
conference have already described some the difficulties in
implementing such a system: limited human input/output (sensory/motor)
and no near-term technology for interfacing directly to the brain's
neural pathways. However, I've worked on another type of mind/machine
coupling that may be of interest and is more realistic technologically
(although still very challenging).  Prototypes of this system have
already been built and demonstrated to the Defence Advanced Research
Projects Agengy.

The system was called the Pilot's Associate (PA). The purpose of PA
was to provide decision aiding and automation support to a pilot in a
single seat fighter aircraft. The problem was (and still is) that
fighter jets are the most complex single-operator machine in
existence. As technology (sensors, smart weapons) advanced, pilot's
were quickly becoming overloaded with too much information to process
and too many activities to manage.

Our design philosophy was focused on expanding capabilities of the
pilot/aircraft system. We used a relatively simple capability model
(applicable to both humans and machines) that categorized various
functions as sensory (input), motor (output), or cognitive
(processing).  Fighter jets are designed to greatly extend the sensory
and motor capabilities. Our job was to extend the cognitive
capabilities. The ultimate "vision" was to extend the pilot's
capabilities in a way that was transparent to his consciousness. In
other words, rather than have the pilot control the machine, the pilot
and the machine would become a completely integrated *system*. Of
course, we knew we couldn't actually implement such a system fully,
but it gave us a philosophical basis for design.

Figure 1 is a greatly simplified diagram showing PA, the pilot, and
the aircraft. The key to the mind/machine coupling was a dynamic pilot
model maintained by the Pilot's Associate. Most interactive machines
are based on a very limited and static user model. The limited user
model usually results in an interface that requires the user to
explicitly communicate with the machine (e.g. press a button, move a
mouse). In our system, not only did the pilot have a mental model of
what PA activity, but PA also had a dynamic "mental" model of the
pilot's activitives. This pilot model included a representation of
current pilot activities, inferred pilot intent and anticipated
activities, and current/projected pilot workload. PA's intent model
allowed it to provide extensive real-time intelligent (AI) support to 
the pilot with a minimum of interference. (Who needs interference when
you're dodging missiles at high speeds and low altitude?)

   PILOT'S ASSOCIATE                                 PILOT
    |                                                  | Eyes, Ears... (S)
    |     (PA/Pilot Functions)                         | Hands, Legs... (M)
    |   Situation Assessment (C) ......................| 
    |   Automated Planning (C).........................|
    |   Plan Execution (C/M)...........................|
    |   Plan Monitoring (C/S)..........................|                  
    |   System Status (C/S/M)..........................|
    |   Pilot Modeling (C/S)- - - - - - - - - - - - - -| PA modeling (C/S)
    |   Display Management (C/M).......................|
    |                                                  |
    |                                               Displays/Controls
    |                                                  |  for Aircraft
    ----------------------- AIRCRAFT -------------------    and PA
       Sensors (S) 
       Low-level sensor integration (C)
       Weapon Systems (M)
       Countermeasures (M)
       Flight Control (M)

  LEGEND: (C) = cognitive function
          (S) = sensory function
          (M) = motor function
          .......  Similar function performed by pilot
          - - - -  PA/Pilot model synchronization

		Figure 1. PA/Pilot/Aircraft Functions

Intent modeling represents *implicit* mind/machine communication.
Intent inferencing was based on a cognitive model of the pilot's
behavior in various contexts ("scripts" in AI terminology).  Of
course, these intent models were ambiguous. Usually there were several
hypothetical intent models being monitored at any given time.  (Think
of the problem of being in the back seat of a car and trying to figure
out what the driver is doing -- without talking to him).

The pilot workload monitoring function predicted the pilot's available
sensory processing (aural, visual), and motor capabilites. This
information was used to manage displays and dynamically automate
certain aircraft functions during high workload situations. For
example, during low altitude flight, the pilot's visual system is
fully utilized to process information about the external situation
(keep from crashing into the ground).  Therefore, if PA had a choice,
it would use audio cues for pilot communication rather than a visual
display.  We viewed the pilot's sensor channels as a set of
information transducers.  The purpose of PA's dynamic display
management was to "optimally" couple PA's transducers (tone generator,
GUI, ...) to the the pilot's transducers (ears, eyes, ...).

The prototype was implemented in Lisp and C on a network of Unix
workstations and special-purpose LISP processors. It was demonstrated
in a real-time high-fidelity flight simulation facility at McDonnell
Aircraft Company. As it turned out, the computational requirements were
too much for current generation fighter aircraft and the funding was
eventually cut after 5 years or so of development.

Further questions and comments are welcome.

--
Stephen Bate                |
GE Advanced Technology Labs | Internet: sbate@atl.ge.com
300 Route 38, Bldg. 145-2   | Voice: 609-866-6603
Moorestown, NJ 08057        | Fax:   609-866-6397