X-Envelope-to: CSG-L@vmd.cso.uiuc.edu
X-VMS-To: @CSG
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Newsgroups: bit.listserv.csg-l
Date:         Thu, 19 Nov 1992 08:45:10 -0700
From:         "William T. Powers" <POWERS_W%FLC@VAXF.COLORADO.EDU>
Subject:      visual-linguistic-motor model

[From Bill Powers (921119.0700)]

RE: verbal & nonverbal control systems

I awoke this morning with a partly-formed idea about how words and
nonverbal experiences get coordinated. The first thought was "Maybe
Avery is right." It's possible to look for "a tree at the top of the
hill next to the path" by continually looking at the scene and asking
"Could that be called 'a tree at the top ....?'"

But it would also work if you had previously transformed the words into
a reference perception in visual terms, and continually compared the
current visual scene with the reference scene, in silence. My next
thought was "Maybe some people do it one way and other people the other
way."

Then I thought "Maybe we all can do it either way." That led to this
diagram:

        |      ^               ^      |
       word*   |               |    scene*
        |    word            scene    |
   --- COMP<---                 ---> COMP ---
  |            |    assoc      |             |
 SAY          perc <------->  perc          ACT
  |            |               |             |
  O  O------O  O               O  O-------O  O
                               |             |
  O            O               O             O
  |            ^               ^             |
  V            |               |             V|
    E   N   V   I   R   O   N   M   E   N   T

Note that the systems are drawn as mirror images to put the perceptual
functions next to each other. The "O"s are switches: you fill in the
gaps with a line, like this:

  O--O                   O  O
               or        |
  O                      O

On the left:

If "SAY" is not connected to anything and
  "perc" is not connected to anything

and on the right

  "perc" is connected to the environment and
  "ACT" is connected to the environment, or
   both are connected in the imagination loop, then

acting to produce the reference scene in nonverbal perception on the
right will produce the words associated with that scene in perception on
the left. This is the connection shown above.
Clearly this would also work if we interchanged "word" and "scene". So
this is a system that can be switched to allow the control of a
nonverbal perception to evoke a corresponding verbal perception, or vice
versa.

OK, put that half-finished idea aside. It's suggestive, anyway.

Then Avery's question about how you find the refrigerator floated to the
surface. Right after it came this:

         |           |              |                |
MEMORY   X* <------> Y* <-------> refrig* <-----> "refrig"*
         |           |              |                |
        PCO         PCO            PCO              PCO

Note that I have used an acronym for a "perceptual control operator"
which also happens to represent the Perceptual function, Comparator, and
Output function in the correct relationship for our standard diagram. A
perceptual control operator can be made of a sub-hierarchy of ECSs --
elementary control systems.

All the variables are now starred because they are reference variables.
According to BPC, these are evoked from memory by address signals, shown
as signals descending from Above. This addressing method of choosing
reference signals allows a higher system operating in terms of one kind
of perception to select a different kind of perception as a reference
signal for a lower system, memory providing the translation from one
logical type to another. An addressing dimension such as time would be
common to all perceptual types. There could bo others.

The horizontal double-headed arrows are suppose to indicate that all
these stored perceptions are linked associatively. When any one of them
is selected by a higher system, the others in this associative set are
also selected. An associative set is formed by perceptions recorded with
a common address -- say, time.

Now, suppose that you have looked in a direction X,Y and have seen a
certain configuration centered in the visual field and have heard the
word "refrigerator" spoken, all at the same time. This associative set,
once recorded, now contains two directional perceptions, a visual
configuration perception, and a verbal configuration perception.

If a higher system selects the X and Y angular coordinates as reference
signals, it will also select the reference signal for a certain
configuration, and another reference signal for a word. You will
"expect" to see a particular familiar object and you will "expect" to
hear it called "refrigerator." You will change your direction of looking
until the sensed direction matches the selected reference direction X*
and Y*, and you will see a refrigerator configuration matching the
reference configuration, correcting that error. Not hearing the
"refrigerator" configuration, you will imagine the word as you
experienced it the first time, spoken; this will correct the error in
the word-control system.
If a higher system selects the nonverbal configuration as a reference
signal, it will also, by association, select an X and a Y coordinate,
and will select a reference word. As above, you will look in the
required direction, see the required configuration, and imagine the
required word.

If a higher system selects the word-configuration "refrigerator" as the
reference signal, you will perceive that word via imagination (if you
don't say it aloud). At the same time, the visual refrigerator reference
signal and a pair of direction reference signals X and Y will be
selected associatively, so you will experience looking in the direction
X,Y and experience the expected refrigerator configuration. The looking
will correct the visual configuration error, as before.

Note that the configuration-PCO does not actually require any output
function. Its error is corrected by the X and Y control systems which
foveate the image of the refrigerator, thus supplying the reqested
perception of the refrigerator configuration.

Once you're looking at the refrigerator, it's a simple matter to
increase its retinal size by facing in the direction of looking and
moving forward, control a perception of a hand relative to the perceived
handle as in the Little Man model, and use the arm control system to
make the door perception (called "door") change to the open
configuration (called "open"). Similar associative sets of reference
signals would, I think, work here, too.

This arrangement requires that X and Y be defined in objective space, if
you are to be able to locate the refrigerator from any position in the
room and any initial orientation of the eyes, head, and body. Thus the
PCO for direction control must contain a perceptual map of the room that
is rotated and shifted according to the kinesthetically and visually
sensed relative direction of looking. The relationship between the
relative direction signals (eyes-head-shoulders-hips-knees-ankles-feet)
and the room map must be updated often to keep the corrections in
register. Wayne Hershberger, you may recognize elements of the
oculomotor study here.

Likewise, the refrigerator perceptual function has to be able to provide
a refrigerator perceptual signal with the refrigerator in a range of
orientations relative to you. At the same time, the orientation
perception of the refrigerator must remain in constant relationship to
the orientation perception of the room, if the same scene is to be
reproduced. These are all problems of configuration perception, which
remain to be solved.

I think this conception could be turned into a model. It breaks down
into several finite problems, which can be solved up to a point and from
there on stipulated. We might, for example, decide that instead of
angular directional reference signals we should have X*, Y*, and Z*
Cartesian locations in the room, and leave the problem of converting
from the sensed eyes-and-body configuration to objective space for
later. We obviously don't need to figure out how words are recognized or
produced; we can just say that they are. The associative part looks easy
to implement in a limited situation.
I think we could make up a working model that could do the following:
given the name of an object, turn and look at it; given a picture of an
object, name it and turn to look at its prototype in the room; given a
direction in which or a location at which to look, know that the object
seen is a familiar object and name it. We would have to stipulate object
recognition. Perhaps we could work in kinesthetic body configuration
sensing, but we would have to stipulate, for the time being, conversion
of visual images to objective space. But even with these limitations, I
think this might be an interesting model. I invite anyone who is
interested to be the first one to get such a model working.
-----------------------------------------------------------------
Best to all,


Bill P.
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU