Epistemic autonomy through adaptive sensing

Peter Cariani

Eaton Peabody Laboratory, Massachusetts Eye and Ear Infirmary

243 Charles St., Boston, MA 02114 USA. peter@epl.meei.harvard.edu

For the special session on the Semiotics of Autonomous Information
Systems,

Conference on Intelligent Systems and Semiotics (ISAS), September, 1998

1\. Abstract.

Sensors and effectors determine how events in the world at large are
related to the internal informational states of organisms and robotic
devices. Sensors determine what kinds of distinctions (perceptual
categories, features, primitives) can be made on the environment. By
\"evolving the sensors\" perceptual repertoires can be adaptively
altered and/or enlarged. To the extent that devices can adaptively
choose their own feature primitives for themselves, they gain a greater
measure of \"epistemic autonomy\" vis-à-vis their designers. Such
devices are useful in ill-defined situations where the designer does not
know a priori what feature primitives are adequate or optimum for
solving a particular task.

Several general strategies for adaptively altering or augmenting sensor
function are proposed: 1) prosthesis: adaptive fabrication of new
front-ends for existing sensors (e.g. telescopes), 2) active sensing:
using motor-actions to alter what is sensed through interaction (poking,
pushing, bending), 3) sensory evolution: adaptive construction of
entirely new sensors (adaptive antibody construction, Gordon Pask\'s
electrochemical device) and 4) internalized sensing: \"bringing the
world into the device\" by creating internal, analog representations of
the world out of which internal sensors extract newly-relevant
properties (perceptual learning). Since many neural sensory
representations appear to be analog and iconic in nature, neural
assemblies can be adaptively formed to function as internal sensors that
can switch behavior according to new perceptual categories.

2\. Autonomy of structure and function

Since antiquity, the functional organization of humans and animals has
been broadly conceptualized in terms of sensing, thinking, and acting.
Sensing is a process of gaining information about the external world,
acting the process of influencing the course of events in that world,
and thinking the process of choosing which actions to take given the
sensed appearance of the world.

The semiotics of these functionalities is fairly straightforward (Fig.
1). Strong parallels exist between the functional organization of
organisms as informational systems, the structure of scientific models,
and the operational structure of robotic devices (Uexküll, 1926; Pattee,
1985; Rosen, 1985; Cariani, 1989, 1992, in press)§. Sensors and
effectors mediate the kinds relationships that can exist between
internal functional states and the world at large. The sensors determine
the perceptual categories that are available, whilst effectors determine
the kinds of primitive actions that can be realized. Sensors and
effectors thus determine the nature of the external semantics of the
internal, informational states of organisms and robotic devices. For
sensing, the causal flow is from environment to organism; whereas, for
action, causation flows from organism to environment.

Mediating between sensing and acting is a coordinative functionality
that Aristotle called "the common sense." This deliberative faculty
realizes the percept-action mappings that govern the system's behavior
under different perceived circumstances. The mappings can be simple
reflexes that dictate particular responses given particular sensory
inputs, or they can be highly elaborated internal models that utilize
past experiences as guides for action.

To the extent that a system operates via simple reflexive input-output
mappings, the system's behavior is a slave to its (immediate) external
inputs. To the extent that more elaborated history-dependent
(memory-dependent) mappings are present, the system's behavior becomes
less dependent on its immediate past inputs, and more dependent upon its
entire history. This history-dependence is a form of memory, and to the
extent that percept-action coordinations are mediated by memory, the
present behavior of the device depends upon past inputs that are
increasingly distant in time and space.

To an external observer, a complex, history-dependent coordinative
process makes the behavior of the organism or device more difficult to
predict; the system appears to act more autonomously relative to its
immediate inputs, to depend more directly on its own internal processes
than on its external inputs. Our intuitive sense of animate vs.
inanimate trades heavily on whether an object appears to act
autonomously, whether it moves on its own, whether it initiates movement
on its own, whether it defies our simple predictions.

How is such behavioral autonomy achieved? In a word, it results from
mechanisms that permit the internal operation of the organism or device
to be altered by experience. If the device is to be adaptive, to improve
its operation over time, this alteration must be based in some way on
performance, however measured. Autonomy of function requires feedback
loops that connect evaluative mechanisms with the
percept-coordination-action cycles so as to evolve more optimal
percept-action mappings (Fig. 2). Subserving the functionalities in the
percept-action loop are material structures (sensors, neural mechanisms,
computational substrates, effectors). In material terms, functional
autonomy requires a degree of structural autonomy, a "feedback to
structure" in which material substrates are physically altered.

2\. Adaptive sensing

Classically, theories of learning and approaches to the design of
adaptive devices have focused on feedback to coordinative parts. It
almost invariably assumed that repertoires of percept- and
action-primitives are fixed and that learning entails
experience-dependent alterations in the coordinative part. For trainable
machines, this means that the primitive features and actions that are
available are fixed. A trainable machine learns to improve the
particular mapping of percepts to actions within those sets of percepts
and actions that were given at the start. To the extent that the
percept-action mappings are elaborated and made more dependent on the
remote history of the device, then the device attains some degree of
behavioral autonomy relative to its immediate inputs. In such cases, an
external observer would need more detailed information concerning the
history and current state of the system in order to render the system
predictable. Thus, to the observer the system appears to be more
"complex", to behave more autonomously relative to its surrounds.

A trainable machine can be no better than its inputs and its outputs,
however. At best trainable classifier can only achieve those
classifications that its feature primitives permit. Whatever parts and
functionalities of a machine that are fixed from the outset, i.e. are
not subject to adaptive modification, must be foreseen and specified by
its designer. If the designer of a classifier system fails to include
particular kinds of features that are necessary for solving the
classification problem at hand, such that the requisite information is
simply not in the feature set that was provided, then no amount of
computation on the provided features can correct the problem. In such
situations, the machine's human designer must recognize that the
features at hand are not up to the classificatory task, and come up with
a new, generally larger feature set. In terms of hardware, this means
altering pre-existing sensors or adding new ones. When an extra
primitive feature is added to the classifier, the dimensionality of the
feature space increases by 1; when existing sensors are altered or tuned
in new ways, the dimensionality of the space remains constant, but the
external semantics of the respective features are changed. Adding a new
independent feature or changing the semantics of an existing one
(functionally this means one has a different feature than one did
before) can drastically simplify and render tractable a classification
problem that was formerly not soluble (Fig. 2). Thus, changing the
features that are available by altering pre-existing

sensors or evolving new ones permits improvements in performance that
might not be possible under suboptimal sets of fixed features.

Similar kinds of advances are seen in biological realms. Throughout
biological evolution, new sense organs have evolved, and existing ones
have been adaptively modified. Various kinds of sensory distinctions
have been added to existing sensory systems over time (e.g. color
vision, echolocation, electroception, magnetoception). Those lineages
that survive and persist are able to do so in part because their sensory
systems permit them to detect of aspects of their environments critical
for survival and reproduction (e.g. detecting, recognizing, and locating
food, mates, and predators). In biological organisms, the structures of
sensory organs are specified genetically such that they can be
constructed through developmental processes. Selective pressures produce
surviving populations that tend to bear sensory systems that are better
adapted for particular niches. There is thus a "feedback to structure"
loop that connects evaluative mechanisms (natural selection) with the
mechanisms for constructing sensors (gene expression, developmental
processes) that result in adaptive alterations of percept-action
mappings. The nature of these adaptive alterations is important. The
basic sensory distinctions that are available to an organism form are
perceptual categories that constitute its Umwelt (Uexküll, 1926)§.
Sensory evolution is therefore a perceptual learning process (Gibson,
1991)§ by which the basic categories of perception are chosen and
refined.

Artificial devices can be designed and constructed so that they, too,
can evolve the sensing operations that they need to detect relevant
features of their surrounds. A traditional adaptive classifier can
control which inputs it pays attention to, but it cannot change the
nature of its inputs, their external semantics. When a system has the
capacity to adaptively modify or augment its sensors, then it in effect
can choose its own perceptual categories. The more degrees of freedom
that are available to a device in determining the structure of its own
sensors, the more control that it has concerning the types of empirical
information it can access. When the device chooses its own inputs in
this way, it attains some degree of epistemic autonomy relative to its
designer. The device no longer must depend solely on its designer for
the kinds of information (features, observables) it needs to solve a
problem, and the designer is no longer burdened with the problem of
foreseeing what perceptual categories are adequate and/or optimal. A
degree of epistemic autonomy means that the system itself can define the
terms of a problem in ways that were not anticipated by the designer;
the system can come up with more creative solutions that are not simply
logical combinations of pre-existing features.

Several strategies are available for altering existing sensing functions
or adding new ones: external prosthesis, active measurement, new sensor
construction, and internal sensing. As means of expanding the repertoire
of available sensing operations open to an organism or device, each of
these strategies confers upon a system some degree of adaptive control
over the nature of its inputs.

3\. Adaptive prosthesis

Perhaps the simplest means of altering the semantics of sensing
operations is to interpose external objects (filters or active devices)
between existing sensors and the world that modify the relationship
between them. The construction of prosthetic devices is strategy by
which we create devices to aid and extend our own senses. A geiger
counter with an acoustic output can be used to detect radiation by
organisms such as ourselves that can sense sound but not radiation.
Internal linkages in the device connect mechanisms sensitive to
radiation (to which we are insensitive) to those that produce physical
disturbances to which we are sensitive (visible pointer readings or
audible sounds).

While this strategy leaves the structural boundaries of our bodies
intact, the prosthesis effectively alters the semantics of some of our
perceptual states, such that those states are now linked in a different
way to the world beyond the prosthetic sensor. When we are using a
geiger counter, the clicks are no longer simply environmental sounds \--
they are linked to the presence of radiation in the vicinity of the
counter. Existing degrees of sensory freedom (e.g. auditory attention,
detection of particular sounds) are exchanged for the new ones afforded
by the prosthesis (detection of radiation). The linkage of the outputs
of the prosthesis with our own sensors moves the functional boundary of
the sensing operation, its point of contingency, outwards, to a point
distal to the prosthesis. Similarly when we wield a rigid stick, the
functional and experiential boundaries of our bodies extend to the end
of the stick (Churcher, 1982)§.

In the end, all technology is prosthesis, the extension and augmentation
of existing biological and informational capabilities. We put eyeglasses
in front of our eyes to bring the world into focus, telescopes and
microscopes to see those realms which our unaided eyes cannot fathom. We
use remote sensors to explore the planets, and these are all extensions
of the senses we have. All of our systems of writing and of calculation
are similarly extensions of memory systems and logical operations that
we can perform in our minds. Similarly the technological implements that
carry us from one place to the next, or that grow our food for us, or
that keep us warm are but extensions of our own bodies\' effectors.

Robotic devices can similarly be designed along these lines to
adaptively construct prosthetic devices for themselves (Figure 3). It is
not difficult to envision robotic systems endowed with pattern grammars
for assembling various kinds of external prostheses adaptively steered
using genetic algorithms evaluated using performance criteria.

4\. Active measurement

Active measurement is the process of acting on the world and sensing the
manner in which it changes contingent upon one's actions. For example,
one can poke an object to see if it moves (a strategy beloved by small
children). The interactive measurement conveys different information
from passive observation. Each distinct motor sequence sets up a
different measurement, a different observable. In the physicist's
operational terms, each motor sequence "prepares the system" in a
different way. Most experimental science is an active measurement
process, as elaborate motor-rituals are played out to set up physical
systems in different ways in order to make particular measurements. By
changing the motor plan that sets up the measurement, one changes the
reference states of sensors, thereby altering what is measured. Active
measurement requires additional coordinative and motor resources. For
active measurements motor degrees-of-freedom are exchanged for new
sensory degrees of freedom, as motor-programs involved with the active
measurement pre-empt other uses for their elements. If there is the
capacity for adaptive motor control, then these movements can be made
contingent on the quality of the information they bring in (relative to
some task), and the motor system in effect becomes a part of the
perceptual system. Possibilities of active measurement are only limited
by the kinds of physical actions that the actor-observer can carry out,
with or without the use of external tools (i.e. all of the manipulations
available through technology)..

5\. Adaptive construction of new sensors

While the strategies of prosthesis construction and active measurement
leave the original device sensors unchanged, it is also possible for a
device to adaptively construct new sensors (Figure 4). This is in effect
what the immune system does when mounting an immune response:
populations of molecular sensors (antibodies) are produced, those that
are effective (have higher antigen binding affinities) are selectively
produced, whist the hypermutation process alters genetic plans to
produce still greater variety of molecular sensors for selection and
production.

Use of prostheses and active measurements redeploy available sensory and
motor degrees-of-freedom for new sensory degrees-of freedom.
Construction of new, independent sensors in effect creates new sensory
degrees of freedom (feature spaces increase in dimensionality, and
percept-action mappings must also increase in dimensionality to
accommodate the new kind of information). Thus creation of a new sensing
operation not only requires that new sensor hardware be created, but it
also makes demands on the coordinative facilities that are available. In
order to make full use of the additional sensory distinctions that are
afforded by a new sensor, new coordinative capacity must also be
created.

Rudimentary devices that adaptively construct their own sensors de novo
have been built. In the late 1950's British cyberneticist Gordon Pask
fabricated electrochemical assemblages that in effect grew their own
sensors (Pask, 1959, 1961; Cariani, 1993)§. Pask\'s device is the only
artefact constructed thus far which adaptively finds its own \"relevance
criteria,\" the observables which are relevant to accomplish a task
(i.e. albeit in a very rudimentary way, the device solved the \"frame
problem").

Such artefacts that adaptively evolve their own sensors might be useful
as front-ends for trainable classifiers such as neural networks. The
self-organizing assemblage would optimize feature primitives while the
classifier would optimize feature-classification mappings. Whenever
desired levels of performance could not be achieved within the current
set of feature primitives, then a Pask assemblage would search for more
appropriate features and the cycle would begin anew.

6\. Internal sensing: bringing the environment within

Prostheses add new sensors distal to an organism or device's structural
boundary, while de novo creation of new sensory organs adds them at the
structural boundary. While construction of prostheses exteriorizes the
problem of making new linkages with the environment, this problem can
also be interiorized by bringing the environment into the device itself.
In part, Pask\'s device achieves this internalization by immersion in an
aqueous medium that is in contact with the external world.

Devices could thus be built in which complex analog interactions within
the device itself are adaptively converted into the discrete categories
of the feature vector by internal sensors (Figure 5). This shifts the
boundary between analog and digital encodings proximally, so that the
internal milieu is effectively an iconic, analog transformation of the
environment that preserves some of the richness and subtlety of its
dynamics. In contact with this internal milieu are various kinds of
internal detectors that are sensitive to particular motions of the
medium (of thinks of the immune system, or the cochlea, or the
vestibular apparatus in these terms). Various detectors within the
sensor are adaptively constructed and wired together in different ways
so as to register those complex motions of the internal medium that are
relevant for a particular classification or task. This arrangement has
several advantages. The sensory system can potentially access a wide
range of environmental properties, both simple and complex, but it need
not explicitly detect and discretely encode those aspects until they
become relevant for performing some task. In doing so, the designer is
freed from anticipating exactly which aspects will be most relevant to
particular tasks.

Many neural sensory systems can be seen in these terms. Rather than
producing discrete alternative response-states, the vast majority of the
responses of sensory neurons are graded in character (Mountcastle, 1967;
Perkell and Bullock, 1968; Rieke et al., 1997)§. (Even neurons thought
to function as "feature-detectors" rarely exhibit discrete response
states.) This generalization holds whether stimulus coding is based on
discharge rates or spike timings. In many sensory systems, such as
audition, mechanoception, and vision, the stimulus impresses its own
time structure on the timings of neural discharges ("stimulus-locking"
or "phase-locking"). In these systems, the times between spikes,
interspike intervals, can convey perceptually-relevant information about
stimulus qualities (Cariani, 1995; Cariani and Delgutte, 1996; Cariani,
1997a,b)§. Interspike intervals themselves can take on continuous ranges
of time duration; to the extent they reflect the stimulus time
structure, they constitute iconic, analog representations of the
stimulus. At more central stages of sensory processing, assemblies of
neural elements sensitive to particular classes of spatio-temporal
response patterns can be adaptively constructed to detect particular
activity patterns in the neural medium and to switch motor activity in a
discrete manner. When a neural assembly is formed, e.g. by changing
effective interneural connectivities, that functions as such a switch,
then in effect a new sensing function has been created on the internal
milieu. New feature-primitives (Schyns, Goldstone and Thibaut, in
press)§ and signal-primitives (Cariani, in press)§ can arise this way.
By virtue of correlation of the internal milieu with the external
environment, a new sensing function is realized on the external world.
In Gibsonian terms, the correlational structure of the environment has
been brought into the nervous system such that invariant properties of
neural activity that correspond to affordances in the environment can be
extracted by emergent assemblies of neurons. Perceptual repertoires can
thus be expanded, limited only by the ability of internal milieus to
capture the richness of their effective stimuli and by their ability to
exploit that richness by forming the appropriate neural assemblies.
Universal correlation-based mechanisms for representing sensory forms
and analyzing them permit neural assemblies to be constructed to extract
(almost) arbitrary sets of pattern invariances (Uttal, 1975)§. To the
extent that internal analog sensory representations and the formation of
neural assemblies are facilitated by attention- and motivation-mediated
processes, the organism gains control over which internal sensors are
created and extended.

7\. Conclusions

Together with other operations that create more powerful and flexible
coordinative capabilities and more elaborated internal anticipatory
models, adaptive sensing operations permit systems to determine for
themselves how they represent the world around them. Simple reflexive
reactivity can thus be replaced by a measure of epistemic autonomy.

8\. References

Cariani, P. 1989. On the Design of Devices with Emergent Semantic
Functions. Ph.D., State University of New York at Binghamton.