Report

FOR

NEURAL

NETWORKS

FUNCTION

DETERMINES

FORM

Francesca

Albertini

Eduardo

D.

Sontag

Department

of

Mathematics

Rutgers

University

New

Brunswick

NJ

E-mail

albertin@hilbert.rutgers.edu,

sontag@hilbert.rutgers.edu

ABSTRACT

This

paper

shows

that

the

weights

of

continuous-time

feedback

neural

networks

are

uniquely

identifiable

from

input/output

measurements

Under

very

weak

genericity

assumptions

the

following

is

true

Assume

given

two

nets

whose

neurons

all

have

the

same

nonlinear

activation

function

oe

if

the

two

nets

have

equal

behaviors

as

black

boxes

then

necessarily

they

must

have

the

same

number

of

neurons

and

except

at

most

for

sign

reversals

at

each

node

the

same

weights

Key

words

Neural

networks

realization

from

input/output

data

control

systems

Research

supported

in

part

by

US

Air

Force

Grant

Rutgers

Center

for

Systems

and

Control

May

FOR

NEURAL

NETWORKS

FUNCTION

DETERMINES

Francesca

Albertini

Eduardo

D.

Sontag

SYCON

Rutgers

Center

for

Systems

and

Control

Department

of

Mathematics

Rutgers

University

New

Brunswick

NJ

-LRB-908-RRB- 932-3072

E-mail

albertin@hilbert.rutgers.edu,

sontag@hilbert.rutgers.edu

ABSTRACT

This

paper

shows

that

the

weights

of

continuous-time

feedback

neural

networks

are

uniquely

identifiable

from

input/output

measurements

Under

very

weak

genericity

assumptions

the

following

is

true

Assume

given

two

nets

whose

neurons

all

have

the

same

nonlinear

activation

function

oe

if

the

two

nets

have

equal

behaviors

as

black

boxes

then

necessarily

they

must

have

the

same

number

of

neurons

and

except

at

most

for

sign

reversals

at

each

node

the

same

weights

Introduction

Many

recent

papers

have

explored

the

computational

and

dynamical

properties

of

systems

of

interconnected

neurons

For

instance

Hopfield

-LRB-

-LSB-

-RSB-

-RRB-

Cowan

-LRB-

-LSB-

-RSB-

-RRB-

and

Grossberg

and

his

school

-LRB-

see

e.g.

-LSB-

-RSB-

-RRB-

have

all

studied

devices

that

can

be

modelled

by

sets

of

nonlinear

differential

equations

such

as

xi

-LRB-

t

-RRB-

xi

-LRB-

t

-RRB-

oe

n

X

j

aijxj

-LRB-

t

-RRB-

m

X

j

bijuj

-LRB-

t

-RRB-

A

i

n

-LRB-

-RRB-

or

xi

-LRB-

t

-RRB-

xi

-LRB-

t

-RRB-

oe

n

X

j

aij

xj

-LRB-

t

-RRB-

A

m

X

j

bijuj

-LRB-

t

-RRB-

i

n

-LRB-

-RRB-

Here

each

xi

i

n

is

a

real-valued

variable

which

represents

the

internal

state

of

the

ith

neuron

and

each

ui

i

m

is

an

external

input

signal

The

coefficients

aij

bij

denote

the

weights

intensities

or

synaptic

strengths

of

the

various

connections

The

function

oe

IR

IR

which

appears

in

all

the

equations

is

called

the

activation

function

and

is

often

taken

to

be

a

sigmoidal-type

map

It

characterizes

how

each

neuron

responds

to

its

aggregate

input

Some

authors

assume

that

a

different

external

signal

can

control

each

neuron

but

it

seems

reasonable

from

a

systems-theoretic

perspective

to

expect

that

the

number

of

such

inputs

is

far

smaller

than

the

number

of

state

variables

that

is

m

o

n.

Electrical

circuit

implementations

of

these

equations

employing

resistively

connected

networks

of

n

identical

nonlinear

amplifiers

and

adjusting

the

resistor

characteristics

to

obtain

the

desired

weights

have

been

proposed

as

models

of

analog

computers

in

particular

in

the

context

of

constraint

satisfaction

problems

and

in

content-addressable

memory

applications

-LRB-

see

e.g.

-LSB-

-RSB-

-RRB-

We

also

assume

given

a

certain

number

p

of

probes

or

measurement

devices

whose

outputs

signal

to

the

environment

the

collective

response

of

the

net

to

the

stimuli

presented

in

the

research

was

supported

in

part

by

US

Air

Force

Grant

AMS

-LRB-

MOS

-RRB-

subject

classification

Primary

Secondary

channels

ui

Each

such

device

averages

the

activation

values

of

many

neurons

Mathematically

this

is

modelled

by

adding

a

set

of

functions

yi

-LRB-

t

-RRB-

n

X

j

cijxj

-LRB-

t

-RRB-

i

p

-LRB-

-RRB-

The

coefficient

cij

represents

the

effect

of

the

jth

neuron

on

the

ith

measurement

In

vector

terms

and

omitting

from

now

on

the

t

arguments

we

can

write

the

above

equations

-LRB-

-RRB-

and

-LRB-

-RRB-

respectively

as

x

x

oe

-LRB-

Ax

Bu

-RRB-

-LRB-

-RRB-

and

x

x

oe

-LRB-

Ax

-RRB-

Bu

-LRB-

-RRB-

where

A

is

an

n

n

matrix

and

B

is

an

n

m

matrix

and

where

we

use

the

notation

oe

-LRB-

x

-RRB-

to

denote

the

application

of

the

nonlinear

function

oe

to

each

coordinate

of

the

vector

x

We

also

write

the

output

equation

-LRB-

-RRB-

as

y

Cx

-LRB-

-RRB-

where

C

is

a

p

n

matrix

Note

that

once

the

nonlinearity

oe

is

specified

and

the

choice

of

model

-LRB-

-RRB-

or

-LRB-

-RRB-

is

made

-LRB-

that

is

once

the

architecture

of

the

network

has

been

fixed

-RRB-

the

triple

of

matrices

-LRB-

A

B

C

-RRB-

uniquely

determines

the

network

dynamics

The

entries

of

these

matrices

are

usually

referred

to

collectively

as

the

weights

of

the

network

Closely

related

to

the

above

models

are

those

for

which

the

neurons

evolve

according

to

x

oe

-LRB-

Ax

Bu

-RRB-

-LRB-

-RRB-

Again

with

outputs

-LRB-

-RRB-

they

are

a

very

natural

generalization

of

the

linear

systems

that

appear

in

control

theory

-LRB-

namely

those

for

which

oe

is

the

identity

mapping

-RRB-

Such

generalized

linear

systems

are

of

interest

for

many

reasons

besides

neural

network

applications

They

provide

a

class

of

semilinear

systems

for

which

one

might

expect

a

theory

closer

to

that

of

linear

systems

than

is

the

case

for

general

nonlinear

smooth

systems

Moreover

for

suitably

sharp

nonlinearities

oe

they

are

approximate

models

of

discontinuous

equations

such

as

x

sign

-LRB-

Ax

Bu

-RRB-

-LRB-

See

-LSB-

-RSB-

for

related

work

on

systems

that

mix

linear

dynamics

and

sign

functions

-RRB-

In

discrete-time

systems

of

the

type

-LRB-

-RRB-

have

been

recently

shown

to

be

at

least

as

powerful

as

any

possible

digital

computational

device

-LRB-

see

-LSB-

-RSB-

-LSB-

-RSB-

-RRB-

when

all

weights

are

rational

numbers

and

a

general

model

of

analog

computers

when

the

weights

are

allowed

to

be

real

-LRB-

-LSB-

-RSB-

-RRB-

Uniqueness

of

Weights

Stability

properties

memory

capacity

and

other

characteristics

of

the

above

types

of

systems

have

been

thoroughly

investigated

by

many

authors

see

for

example

-LSB-

-RSB-

-LSB-

-RSB-

and

references

there

In

this

paper

we

are

interested

in

studying

a

somewhat

different

issue

namely

To

what

extent

does

the

function

of

the

net

that

is

to

say

the

black

box

behavior

mapping

external

inputs

to

output

signals

uniquely

determine

the

coefficients

aij

bij

cij

defining

the

network

A

precise

formulation

is

as

follows

Assume

that

the

network

is

started

at

the

relaxed

state

x

-LRB-

-RRB-

and

an

input

signal

u

-LRB-

-RRB-

is

applied

Under

appropriate

technical

assumptions

a

solution

x

-LRB-

t

-RRB-

exists

for

the

differential

equation

respectively

-LRB-

-RRB-

-LRB-

-RRB-

or

-LRB-

-RRB-

and

an

output

signal

y

-LRB-

t

-RRB-

Cx

-LRB-

t

-RRB-

is

thus

generated

In

this

manner

and

for

any

fixed

architecture

for

each

triple

-LRB-

A

B

C

-RRB-

there

is

an

input-output

mapping

-LRB-

A

B

C

-RRB-

u

-LRB-

-RRB-

y

-LRB-

-RRB-

-LRB-

-RRB-

Our

main

question

is

to

decide

to

what

extent

are

the

matrices

A

B

C

determined

by

the

i/o

mapping

-LRB-

A

B

C

-RRB-

In

the

very

special

case

when

oe

is

the

identity

classical

linear

realization

theory

see

for

instance

-LSB-

-RSB-

Chapter

implies

that

generically

the

triple

-LRB-

A

B

C

-RRB-

is

determined

only

up

to

an

invertible

change

of

variables

in

the

state

space

That

is

except

for

degenerate

situations

that

arise

due

to

parameter

dependencies

-LRB-

non-controllability

or

non-observability

-RRB-

if

two

triples

-LRB-

A

B

C

-RRB-

and

-LRB-

A

B

C

-RRB-

give

rise

to

the

same

i/o

behavior

then

there

is

an

invertible

matrix

T

such

that

T

A

T

B

CT

C

-LRB-

-RRB-

This

is

the

same

as

saying

that

the

two

systems

are

equivalent

under

a

linear

change

of

variables

x

-LRB-

t

-RRB-

Tx

-LRB-

t

-RRB-

Conversely

still

in

the

classical

case

oe

identity

any

such

T

gives

rise

to

another

system

with

the

same

i/o

behavior

when

starting

with

any

given

triple

-LRB-

A

B

C

-RRB-

These

classical

facts

essentially

apply

only

when

oe

is

linear

as

we

will

discuss

in

this

paper

We

will

show

that

for

nonlinear

activations

-LRB-

under

very

weak

assumptions

-RRB-

the

natural

group

of

symmetries

is

far

smaller

than

that

of

arbitrary

nonsingular

matrices

being

instead

just

a

finite

group

We

will

prove

that

if

two

nets

give

rise

to

the

same

i/o

behavior

then

a

matrix

T

will

exist

satisfying

-LRB-

-RRB-

but

having

the

special

form

of

a

permutation

matrix

composed

with

a

diagonal

matrix

performing

at

most

a

sign

reversal

at

each

neuron

In

concrete

terms

this

will

mean

that

the

input/output

behavior

uniquely

determines

all

the

weights

except

for

a

reordering

of

the

variables

and

for

odd

activation

functions

sign

reversals

of

all

incoming

and

outgoing

weights

at

some

units

After

some

thought

this

result

is

not

surprising

see

the

intuitive

discussion

given

below

for

the

single-neuron

case

but

technically

it

requires

a

certain

amount

of

effort

to

establish

Intuition

Behind

the

Results

It

is

useful

to

consider

our

results

in

trivial

situations

-LRB-

one

neuron

-RRB-

in

order

to

develop

an

intuition

as

to

their

nature

Consider

first

the

case

of

two

one-neuron

one-input

one-output

systems

-LRB-

that

is

n

p

m

-RRB-

with

zero

A

matrix

as

follows

x

oe

-LRB-

bu

-RRB-

y

cx

x

oe

-LRB-

bu

-RRB-

y

cx

-LRB-

models

of

type

-LRB-

-RRB-

-RRB-

For

the

first

equation

the

zero-initial-state

i/o

behavior

takes

the

form

u

-LRB-

-RRB-

y

-LRB-

-RRB-

where

y

-LRB-

t

-RRB-

Z

t

coe

-LRB-

bu

-LRB-

s

-RRB-

-RRB-

ds

and

similarly

for

the

second

Note

that

in

the

case

in

which

oe

is

an

odd

function

-LRB-

which

is

the

case

studied

most

often

in

applications

-RRB-

reversing

the

signs

of

weights

leaves

the

i/o

behavior

invariant

Z

t

-LRB-

c

-RRB-

oe

-LRB-

bu

-LRB-

s

-RRB-

-RRB-

ds

Z

t

coe

-LRB-

bu

-LRB-

s

-RRB-

-RRB-

ds

for

all

u

-LRB-

-RRB-

Thus

both

systems

have

the

same

i/o

behavior

if

it

holds

that

b

b

c

c

-LRB-

-RRB-

Assume

conversely

that

the

zero-state

i/o

behaviors

coincide

we

wish

to

prove

now

that

either

both

systems

are

the

same

or

-LRB-

-RRB-

holds

Of

course

the

desired

implication

will

not

be

true

unless

suitable

nondegeneracy

assumptions

are

made

for

instance

if

c

c

then

both

give

the

same

i/o

behavior

but

b

and

b

may

be

very

different

Moreover

the

case

in

which

oe

is

linear

must

also

be

ruled

out

since

otherwise

coe

-LRB-

b

-RRB-

cboe

-LRB-

-RRB-

and

the

only

constraint

for

equality

is

that

cb

cb

Note

that

the

assumption

of

identical

i/o

behavior

is

equivalent

to

coe

-LRB-

b

-RRB-

coe

-LRB-

b

-RRB-

for

all

IR

-LRB-

-RRB-

-LRB-

equality

as

functions

IR

IR

-RRB-

If

oe

has

three

derivatives

at

zero

and

if

we

assume

that

-LRB-

-RRB-

and

-LRB-

-RRB-

-LRB-

-RRB-

then

taking

first

and

third

order

derivatives

with

respect

to

in

-LRB-

-RRB-

and

evaluating

at

we

conclude

that

cb

cb

and

-LRB-

-RRB-

from

which

it

follows

if

cb

that

and

hence

that

jbj

jbj

Thus

either

-LRB-

-RRB-

holds

or

b

b

c

c

as

desired

Instead

of

-LRB-

-RRB-

we

could

have

assumed

merely

that

some

derivative

oe

-LRB-

q

-RRB-

-LRB-

-RRB-

for

some

q

since

this

would

give

by

taking

enough

derivatives

in

-LRB-

-RRB-

that

cbq

cbq

and

hence

that

b

-LRB-

q

-RRB-

b

-LRB-

q

-RRB-

and

again

jbj

jbj

-LRB-

For

analytic

functions

oe

-LRB-

q

-RRB-

-LRB-

-RRB-

for

some

q

is

equivalent

to

nonlinearity

of

oe

-RRB-

For

models

of

type

-LRB-

-RRB-

the

equations

in

this

special

case

of

A

and

p

m

n

become

x

x

oe

-LRB-

bu

-RRB-

y

cx

Here

the

output

for

control

u

-LRB-

-RRB-

is

y

-LRB-

t

-RRB-

Z

t

et?scoe

-LRB-

bu

-LRB-

s

-RRB-

-RRB-

ds

so

again

the

assumption

of

identical

i/o

behavior

is

equivalent

to

-LRB-

-RRB-

and

the

result

is

as

before

The

same

result

is

valid

for

models

of

type

-LRB-

-RRB-

as

well

except

that

now

the

nondegeneracy

assumptions

must

be

slightly

different

since

merely

requiring

cb

is

not

sufficient

to

force

uniqueness

up

to

sign

Indeed

for

A

the

system

-LRB-

-RRB-

is

linear

and

thus

the

only

invariant

for

i/o

behavior

is

the

matrix

product

CB

so

B

and

C

are

highly

nonunique

For

instance

b

c

gives

the

same

zero

initial

state

i/o

behavior

as

b

c

for

x

x

oe

-LRB-

ax

-RRB-

bu

y

cx

-LRB-

-RRB-

when

a

But

under

suitable

nondegeneracy

of

A

the

result

is

still

true

As

an

illustration

take

the

p

m

n

case

but

now

with

a

in

-LRB-

-RRB-

Looking

at

the

derivative

-LRB-

-RRB-

we

know

that

two

systems

of

this

form

with

same

i/o

behavior

must

have

cb

cb

Arguing

purely

formally

now

take

an

impulsive

control

of

the

type

u

ffi

starting

at

x

-LRB-

-RRB-

This

results

in

x

-LRB-

-RRB-

b

and

x

-LRB-

-RRB-

b

Applying

now

a

further

control

identically

equal

to

zero

and

taking

derivative

with

respect

to

time

we

obtain

the

value

cx

-LRB-

-RRB-

coe

-LRB-

x

-LRB-

-RRB-

-RRB-

cb

cb

coe

-LRB-

b

-RRB-

for

the

first

system

and

similarly

for

the

second

This

implies

that

-LRB-

-RRB-

again

holds

and

the

argument

is

completed

as

above

Without

using

delta

functions

we

can

argue

as

follows

consider

the

control

u

-LRB-

t

-RRB-

and

the

derivative

at

t

of

the

resulting

output

y

-LRB-

t

-RRB-

Under

the

assumptions

that

oe

has

three

derivatives

oe

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

one

obtains

that

y

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

and

similarly

for

the

second

system

from

which

it

again

follows

that

jbj

jbj

as

desired

Other

Motivations

for

This

Study

There

are

many

motivations

besides

mathematical

interest

for

posing

the

questions

that

we

study

in

this

work

In

many

applications

feedback

neural

nets

or

recurrent

nets

as

they

are

also

called

are

used

as

models

whose

parameters

are

fit

to

input/output

data

-LRB-

The

purpose

may

be

to

use

these

models

instead

of

a

real

plant

for

purposes

of

control

or

for

predictive

purposes

-RRB-

This

is

done

for

instance

in

certain

approaches

to

grammatical

inference

and

speech

processing

see

for

instance

-LSB-

-RSB-

-LSB-

-RSB-

Typically

gradient

descent

algorithms

are

used

in

order

to

fit

parameters

thorugh

the

minimization

of

an

error

functional

that

penalizes

mismatches

between

the

desired

outputs

and

those

that

a

candidate

net

produces

-LRB-

the

term

continuous

backpropagation

is

sometimes

used

for

the

gradient

descent

procedure

-RRB-

Our

results

imply

that

a

dimensionality

reduction

of

the

parameter

space

as

is

the

case

with

linear

systems

where

canonical

forms

are

ubiquitous

in

identification

methods

is

perhaps

surprisingly

not

possible

for

neural

nets

In

terms

of

the

error

function

being

minimized

the

set

of

minima

is

finite

For

precisely

the

above

reasons

but

restricted

to

the

particular

case

of

feedforward

-LRB-

that

is

nondynamic

-RRB-

nets

the

question

of

deciding

if

the

only

possible

symmetries

are

indeed

the

ones

that

we

find

was

asked

by

Hecht-Nielsen

in

-LSB-

-RSB-

The

question

was

partially

answered

-LRB-

for

so-called

single-hidden

layer

nets

and

using

a

particular

activation

function

-RRB-

by

Sussmann

in

-LSB-

-RSB-

who

established

a

uniqueness

result

which

in

our

setting

would

apply

to

systems

of

the

special

type

x

oe

-LRB-

Bu

-RRB-

y

Cx

with

oe

tanh

-LRB-

x

-RRB-

-LRB-

That

is

there

is

no

A

matrix

the

result

does

allow

for

a

constant

bias

vector

inside

the

sigmoid

however

-RRB-

A

different

though

related

motivation

for

studying

the

problem

that

we

consider

in

this

paper

originates

from

synthesis

considerations

rather

than

identification

Given

a

specified

i/o

behavior

it

is

of

interest

to

know

how

many

possible

different

networks

can

be

built

that

achieve

the

design

objective

Our

results

show

that

there

is

basically

only

one

way

to

do

so

as

long

as

this

objective

is

specified

in

terms

of

a

desired

input/output

map

In

this

sense

structure

-LRB-

weights

-RRB-

is

uniquely

determined

by

function

-LRB-

desired

i/o

behavior

-RRB-

Remarks

Note

that

nonlinear

realization

theory

as

described

for

instance

in

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

can

be

also

applied

to

the

problem

considered

here

This

theory

would

allow

us

to

conclude

that

under

suitable

assumptions

of

controllability

and

observability

there

is

some

abstract

diffeomorphism

which

relates

two

networks

having

the

same

i/o

behavior

It

is

not

clear

how

to

exploit

the

special

structure

of

neural

nets

to

obtain

using

these

tools

the

more

precise

result

that

we

give

on

the

other

hand

the

techniques

that

are

used

in

the

standard

geometric

theory

of

nonlinear

systems

such

as

the

systematic

use

of

Lie

derivatives

are

also

central

to

the

proof

of

our

result

Definitions

and

Statements

of

Results

In

general

we

define

a

system

to

be

a

continuous-time

time-invariant

system

x

f

-LRB-

x

u

-RRB-

-LRB-

-RRB-

y

h

-LRB-

x

-RRB-

where

x

-LRB-

t

-RRB-

IRn

u

-LRB-

t

-RRB-

IRm

and

y

-LRB-

t

-RRB-

IRp

for

all

t

We

assume

that

f

is

at

least

differentiable

on

-LRB-

x

u

-RRB-

For

any

measurable

essentially

bounded

control

u

-LRB-

-RRB-

-LSB-

T

-RSB-

IRm

we

denote

by

OE

-LRB-

t

u

-RRB-

the

solution

at

time

t

of

-LRB-

-RRB-

with

initial

state

x

-LRB-

-RRB-

this

is

defined

at

least

on

a

small

enough

interval

-LSB-

-RRB-

For

the

systems

of

interest

in

neural

network

theory

f

-LRB-

x

u

-RRB-

is

always

Lipschitz

with

respect

to

x

so

T

-LRB-

All

results

that

we

use

on

existence

of

solutions

and

continous

dependence

are

included

in

standard

texts

such

as

-LSB-

-RSB-

-RRB-

For

each

control

we

let

-LRB-

u

-RRB-

-LRB-

u

-RRB-

be

the

output

function

corresponding

to

the

initial

state

x

-LRB-

-RRB-

that

is

-LRB-

u

-RRB-

-LRB-

t

-RRB-

h

-LRB-

OE

-LRB-

t

u

-RRB-

-RRB-

defined

at

least

on

some

interval

-LSB-

-RRB-

Given

two

systems

and

with

the

same

numbers

of

input

and

output

channels

i.e.

with

p

p

and

m

m

we

say

that

and

are

i/o

equivalent

if

it

holds

that

To

be

more

precise

we

require

that

for

each

u

the

domains

of

definition

of

-LRB-

u

-RRB-

and

-LRB-

u

-RRB-

coincide

and

their

values

be

equal

for

all

t

in

the

common

domain

As

mentioned

above

in

the

network

application

these

functions

are

most

often

everywhere

defined

Fix

an

infinitely

differentiable

function

oe

IR

IR

which

satisfies

the

following

basic

assumptions

oe

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

As

before

we

let

oe

-LRB-

x

-RRB-

denote

the

application

of

oe

to

each

coordinate

of

the

vector

x

We

let

-LRB-

n

m

p

-RRB-

-LRB-

respectively

-LRB-

n

m

p

-RRB-

and

-LRB-

n

m

p

-RRB-

-RRB-

denote

the

class

of

all

systems

of

type

-LRB-

-RRB-

-LRB-

respectively

types

-LRB-

-RRB-

and

-LRB-

-RRB-

-RRB-

with

fixed

n

m

p.

Given

two

systems

and

in

-LRB-

n

m

p

-RRB-

and

-LRB-

n

m

p

-RRB-

respectively

note

that

we

assume

m

m

and

p

p

defined

by

the

triples

of

matrices

-LRB-

A

B

C

-RRB-

and

-LRB-

A

B

C

-RRB-

we

will

say

that

and

are

sign-permutation

equivalent

if

the

dimensions

of

the

state

spaces

are

the

same

that

is

n

n

and

there

exists

a

matrix

T

such

that

-LRB-

-RRB-

holds

and

T

has

the

following

special

form

T

PD

where

P

is

a

permutation

matrix

and

D

diag

-LRB-

n

-RRB-

where

each

i

The

systems

and

are

just

permutation

equivalent

if

the

above

holds

with

D

I

that

is

T

is

a

permutation

matrix

We

also

define

equivalence

in

an

analogous

manner

among

systems

of

type

-LRB-

-RRB-

or

systems

of

type

-LRB-

-RRB-

Later

in

the

process

of

establishing

the

main

results

we

introduce

a

more

general

class

of

systems

which

includes

systems

of

the

types

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

we

will

also

define

a

notion

of

equivalence

oe-equivalence

which

will

encompass

both

permutation

and

sign-permutation

equivalence

We

call

a

subset

C

of

-LRB-

n

m

p

-RRB-

generic

if

there

exists

a

nonempty

subset

G

nm

mp

whose

complement

is

the

set

of

zeroes

of

a

finite

number

of

polynomials

in

nm

mp

variables

so

that

is

in

C

if

and

only

if

its

defining

triple

-LRB-

A

B

C

-RRB-

is

in

G.

Observe

that

such

a

set

G

is

open

and

dense

and

its

complement

has

measure

zero

-LRB-

In

the

results

to

be

given

the

polynomials

defining

the

complement

of

G

are

given

explicitely

and

they

correspond

to

the

vanishing

of

appropriate

determinants

-RRB-

We

also

define

generic

subsets

of

-LRB-

n

m

p

-RRB-

and

-LRB-

n

m

p

-RRB-

in

the

same

manner

Now

we

are

ready

to

state

the

main

results

We

assume

that

oe

IR

IR

satisfies

the

above

assumptions

-LRB-

-RRB-

and

also

that

there

exists

some

integer

q

such

that

oe

-LRB-

q

-RRB-

-LRB-

-RRB-

The

assumptions

are

extremely

weak

and

include

most

interesting

nonlinearities

Observe

that

for

instance

if

oe

is

odd

-LRB-

oe

-LRB-

x

-RRB-

oe

-LRB-

x

-RRB-

-RRB-

analytic

nonlinear

and

it

satisfies

-LRB-

-RRB-

then

it

satisfies

all

hypotheses

-LRB-

so

the

standard

nonlinearity

tanh

-LRB-

x

-RRB-

is

included

-RRB-

Theorem

Assume

that

oe

is

odd

For

each

m

n

p

there

exist

generic

subsets

-LRB-

n

m

p

-RRB-

-LRB-

n

m

p

-RRB-

-LRB-

n

m

p

-RRB-

-LRB-

n

m

p

-RRB-

-LRB-

n

m

p

-RRB-

-LRB-

n

m

p

-RRB-

such

that

for

any

two

systems

-LRB-

n

m

p

-RRB-

and

-LRB-

n

m

p

-RRB-

-LRB-

respectively

-LRB-

n

m

p

-RRB-

-LRB-

n

m

p

-RRB-

or

-LRB-

n

m

p

-RRB-

-LRB-

n

m

p

-RRB-

-RRB-

it

holds

that

and

are

i/o

equivalent

if

and

only

if

and

are

sign-permutation

equivalent

Theorem

Assume

that

oe

is

not

odd

Then

there

are

generic

subsets

as

in

Theorem

so

that

and

are

i/o

equivalent

if

and

only

if

and

are

permutation

equivalent

The

rest

of

this

paper

presents

a

far

stronger

technical

result

and

then

shows

how

to

derive

these

Theorems

as

simple

consequences

The

more

general

result

will

deal

also

with

systems

of

the

type

x

Dx

oe

-LRB-

Ax

Bu

-RRB-

Gu

y

Cx

The

General

Setup

We

consider

an

infinitely

differentiable

function

oe

IR

IR

which

satisfies

the

basic

assumptions

-LRB-

-RRB-

namely

oe

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

We

will

see

later

-LRB-

c.f.

Remark

-RRB-

that

for

our

purposes

we

may

always

assume

-LRB-

-RRB-

in

-LRB-

-RRB-

At

various

parts

we

will

also

impose

one

or

both

of

the

following

conditions

on

the

function

oe

the

second

of

which

is

assumed

for

Theorems

and

-LRB-

x

-RRB-

as

jxj

-LRB-

-RRB-

q

such

that

oe

-LRB-

q

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Remark

As

mentioned

earlier

any

function

oe

which

is

nonlinear

odd

and

analytic

-LRB-

such

as

tanh

-LRB-

x

-RRB-

-RRB-

satisfies

our

requirements

-LRB-

-RRB-

and

also

assumption

-LRB-

-RRB-

Property

-LRB-

-RRB-

is

often

satisfied

in

examples

of

interest

as

well

Remark

We

will

remark

later

that

for

most

results

only

infinite

differentiability

of

oe

in

a

neighbourhood

of

zero

is

needed

as

opposed

to

globally

If

oe

satisfies

-LRB-

-RRB-

then

the

following

property

is

an

immediate

consequence

If

oe

-LRB-

ax

-RRB-

aoe

-LRB-

x

-RRB-

for

all

x

in

a

neighbourhood

of

then

a

f

g

-LRB-

P

-RRB-

We

will

denote

by

oe

the

following

set

oe

f

IR

j

and

oe

-LRB-

x

-RRB-

oe

-LRB-

x

-RRB-

for

all

x

IRg

-LRB-

-RRB-

Note

that

oe

is

a

subgroup

of

-LRB-

IR

n

-RRB-

The

fact

that

inverses

are

again

in

oe

follows

from

the

defining

equation

in

-LRB-

-RRB-

applied

to

x

instead

of

x.

Moreover

when

property

-LRB-

P

-RRB-

holds

we

have

that

oe

Given

any

function

oe

IR

IR

and

any

positive

integer

n

we

define

oe

as

follows

-LRB-

more

precisely

we

should

denote

this

as

oen

but

we

omit

the

n

for

simplicity

-RRB-

oe

IRn

IRn

oe

-LRB-

x

-RRB-

col

-LRB-

oe

-LRB-

-RRB-

oe

-LRB-

xn

-RRB-

-RRB-

-LRB-

-RRB-

That

is

oe

-LRB-

n

X

i

aiei

-RRB-

n

X

i

oe

-LRB-

ai

-RRB-

ei

for

all

an

IR

-LRB-

-RRB-

where

eng

is

the

canonical

basis

in

IRn

From

now

on

we

fix

one

such

oe

Systems

We

will

study

continuous-time

systems

whose

state

space

input-value

space

and

output-value

space

are

IRn

IRm

and

IRp

respectively

and

for

which

the

dynamics

are

given

by

equations

of

the

type

-LRB-

x

f

-LRB-

x

u

-RRB-

Dx

oe

-LRB-

Ax

Bu

-RRB-

Gu

y

Cx

-LRB-

-RRB-

for

some

matrices

A

Rn?n

D

Rn?n

B

Rn?m

G

Rn?m

and

C

Rp?n

These

are

continuous

time

systems

in

the

sense

of

-LSB-

-RSB-

We

will

call

such

a

system

a

oe-system

and

denote

it

by

-LRB-

D

A

B

G

C

-RRB-

oe

Observe

that

in

the

special

case

in

which

oe

is

the

identity

or

more

generally

is

linear

we

have

a

linear

system

in

the

usual

sense

The

same

holds

if

oe

is

arbitrary

but

A

B

This

is

useful

to

keep

in

mind

when

understanding

the

conditions

to

be

imposed

later

in

order

for

the

various

conclusions

to

hold

Remark

Different

matrices

-LRB-

D

A

B

G

-RRB-

may

give

rise

to

the

same

function

f

-LRB-

x

u

-RRB-

Dx

oe

-LRB-

Ax

Bu

-RRB-

Gu

and

thus

to

the

same

system

-LRB-

C

on

the

other

hand

is

uniquely

defined

-RRB-

For

instance

if

oe

is

the

identity

f

depends

only

on

the

sums

D+A

and

B

G

But

for

the

type

of

functions

oe

that

we

consider

this

ambiguity

will

in

general

not

happen

Indeed

assume

that

-LRB-

C

-RRB-

oe

-LRB-

C

-RRB-

oe

Then

-LRB-

a

-RRB-

if

and

then

and

-LRB-

b

-RRB-

if

oe

satisfies

-LRB-

-RRB-

then

and

-LRB-

and

hence

also

and

-RRB-

To

prove

these

facts

we

can

argue

as

follows

Taking

x

ej

and

u

we

have

ej

oe

-LRB-

ej

-RRB-

ej

oe

-LRB-

ej

-RRB-

for

all

IR

which

implies

denoting

-LRB-

Al

-RRB-

i

j

by

ali

j

and

-LRB-

Dl

-RRB-

i

j

by

dli

j

j

oe

-LRB-

j

-RRB-

j

oe

-LRB-

j

-RRB-

-LRB-

-RRB-

for

all

real

and

all

indices

i

j

After

taking

the

derivative

with

respect

to

in

the

previous

equation

we

have

j

j

-LRB-

j

-RRB-

j

j

-LRB-

j

-RRB-

-LRB-

-RRB-

If

then

evaluating

-LRB-

-RRB-

at

and

we

have

j

-LRB-

-RRB-

j

-LRB-

-RRB-

which

implies

Denoting

-LRB-

Bl

-RRB-

i

j

by

bli

j

and

-LRB-

Gl

-RRB-

i

j

by

gli

j

and

using

x

and

u

ej

we

can

conclude

by

the

same

arguments

that

-LRB-

j

-RRB-

j

j

-LRB-

j

-RRB-

j

-LRB-

-RRB-

Thus

arguing

as

before

if

we

have

Note

that

in

general

when

-LRB-

-RRB-

holds

lim

-LRB-

afi

-RRB-

for

every

a

IR

-LRB-

either

a

and

this

is

identically

zero

or

we

use

-LRB-

-RRB-

-RRB-

So

by

taking

the

limit

as

in

-LRB-

-RRB-

and

in

-LRB-

-RRB-

we

conclude

and

if

-LRB-

-RRB-

holds

We

will

focus

our

attention

on

the

following

question

when

are

two

given

oe-systems

i/o

equivalent

-LRB-

in

the

sense

defined

in

Section

-RRB-

Remark

We

now

explain

why

-LRB-

-RRB-

can

be

replaced

by

the

stronger

assumption

that

-LRB-

-RRB-

without

loss

of

generality

Let

oe

IR

IR

be

a

differentiable

function

which

satisfies

the

basic

assumptions

-LRB-

-RRB-

and

let

a

-LRB-

-RRB-

Consider

the

function

oe

IR

IR

defined

by

oe

-LRB-

x

-RRB-

oe

-LRB-

x

a

-RRB-

Then

also

oe

satisfies

the

basic

assumptions

-LRB-

-RRB-

and

moreover

-LRB-

-RRB-

Now

if

-LRB-

D

A

B

G

C

-RRB-

oe

is

a

oe-system

we

may

define

the

new

system

having

-LRB-

D

A

B

G

C

-RRB-

oe

to

be

the

oe-system

with

A

aA

and

B

aB

Notice

that

if

u

-LRB-

t

-RRB-

is

a

control

function

defined

on

the

interval

-LSB-

T

-RSB-

and

x

-LRB-

t

-RRB-

is

the

corresponding

trajectory

of

starting

at

x

-LRB-

-RRB-

then

the

trajectories

and

outputs

in

corresponding

to

the

any

given

control

will

be

the

same

If

and

are

to

oe-systems

and

we

construct

and

in

this

manner

we

have

that

and

are

i/o

equivalent

if

and

only

if

and

are

i/o

equivalent

Since

our

interest

is

in

establishing

the

existence

of

various

linear

equations

relating

and

and

and

so

forth

and

since

these

equations

are

not

changed

under

multiplication

by

a

scalar

it

is

clear

that

we

can

assume

without

loss

of

generality

that

-LRB-

-RRB-

So

from

now

on

when

we

consider

a

differentiable

function

oe

we

implicit

assume

that

-LRB-

-RRB-

Equivalence

In

this

section

we

give

a

straightforward

sufficient

condition

for

equivalence

which

will

also

turn

out

to

be

necessary

in

a

generic

sense

We

again

fix

a

function

oe

IR

IR

though

for

now

we

do

not

need

to

assume

that

assumptions

-LRB-

-RRB-

hold

Let

ss

be

any

permutation

of

ng

and

P

be

the

permutation

matrix

which

represents

ss

i.e.

Pi

j

ffii

ss

-LRB-

i

-RRB-

-LRB-

-RRB-

where

ffii

k

is

the

Kronecker

delta

In

other

words

Pej

ess

-LRB-

j

-RRB-

for

the

canonical

basis

eng

in

IRn

Note

that

oe

-LRB-

P

x

-RRB-

P

oe

-LRB-

x

-RRB-

for

all

x

IRn

-LRB-

-RRB-

because

oe

-LRB-

P

x

-RRB-

oe

-LRB-

n

X

i

aiess

-LRB-

j

-RRB-

-RRB-

n

X

i

oe

-LRB-

aj

-RRB-

ess

-LRB-

j

-RRB-

and

P

oe

-LRB-

x

-RRB-

P

-LRB-

n

X

i

oe

-LRB-

aj

-RRB-

ej

-RRB-

n

X

i

oe

-LRB-

aj

-RRB-

ess

-LRB-

j

-RRB-

Let

Q

be

any

diagonal

matrix

such

that

Qej

j

ej

where

j

oe

for

all

j

n

-LRB-

-RRB-

Then

we

also

have

that

oe

-LRB-

Qx

-RRB-

Q

oe

-LRB-

x

-RRB-

for

all

x

IRn

-LRB-

-RRB-

We

let

noe

fT

Gl

-LRB-

n

-RRB-

j

oe

-LRB-

T

x

-RRB-

T

oe

-LRB-

x

-RRB-

for

all

x

IRn

g

The

following

Lemma

follows

trivially

from

-LRB-

-RRB-

and

-LRB-

-RRB-

Lemma

Let

oe

P

Q

be

as

above

Then

both

T

PQ

and

T

QP

are

in

noe

Observe

that

noe

is

a

subgroup

of

Gl

-LRB-

n

-RRB-

When

oe

is

linear

obviously

noe

Gl

-LRB-

n

-RRB-

Otherwise

this

group

is

proper

and

if

-LRB-

-RRB-

holds

it

has

cardinality

at

most

as

follows

from

the

following

result

Lemma

If

oe

is

differentiable

but

it

is

not

a

linear

function

then

every

element

of

noe

is

of

the

form

PQ

with

P

a

permutation

matrix

-LRB-

as

in

-LRB-

-RRB-

-RRB-

and

Q

Diag

-LRB-

n

-RRB-

with

each

i

oe

-LRB-

as

in

-LRB-

-RRB-

-RRB-

Proof

Assume

that

oe

-LRB-

T

x

-RRB-

T

oe

-LRB-

x

-RRB-

for

all

x

IRn

Consider

the

ffth-row

of

T

and

denote

this

as

-LRB-

tn

-RRB-

Then

oe

-LRB-

n

X

i

tixi

-RRB-

n

X

i

tioe

-LRB-

xi

-RRB-

-LRB-

-RRB-

for

all

-LRB-

xn

-RRB-

Since

T

is

invertible

there

exists

an

index

l

such

that

tl

Taking

the

derivative

in

-LRB-

-RRB-

with

respect

to

xl

we

have

-LRB-

n

X

i

tixi

-RRB-

-LRB-

xl

-RRB-

-LRB-

-RRB-

Picking

any

j

l

and

taking

the

derivative

in

-LRB-

-RRB-

with

respect

to

xj

we

get

tj

-LRB-

n

X

i

tixi

-RRB-

tj

-LRB-

xj

-RRB-

-LRB-

-RRB-

Multiplying

both

sides

of

-LRB-

-RRB-

by

tj

tl

and

comparing

-LRB-

-RRB-

with

-LRB-

-RRB-

we

have

tj

-LRB-

xj

-RRB-

tj

-LRB-

xl

-RRB-

This

must

hold

for

all

choices

of

xj

and

xl

in

IR

Since

oe

is

not

a

linear

function

is

not

constant

so

we

must

have

tj

The

index

j

was

arbitrary

thus

we

have

that

in

the

ffth-row

of

T

there

exists

only

one

index

l

such

that

tl

Since

also

ff

was

arbitrary

the

matrix

T

has

at

most

one

nonzero

entry

in

each

row

and

by

-LRB-

-RRB-

this

entry

is

in

oe

thus

the

conclusion

follows

Definition

Let

-LRB-

-RRB-

oe

-LRB-

-RRB-

oe

be

two

oe-systems

and

be

the

dimensions

of

the

state

spaces

in

respectively

We

say

that

and

are

oe-equivalent

if

n

and

if

there

exists

an

invertible

matrix

T

noe

such

that

T

T

T

T

The

next

property

is

trivial

but

we

state

it

for

further

reference

Proposition

Let

-LRB-

-RRB-

oe

-LRB-

-RRB-

oe

be

two

oesystems

If

they

are

oe-equivalent

then

they

are

also

i/o-equivalent

Proof

Denote

by

x

the

state

variable

for

and

by

z

the

state

variable

for

We

have

for

any

measurable

essentially

bounded

control

u

x

oe

-LRB-

-RRB-

oe

-LRB-

T

-LRB-

-RRB-

-RRB-

Thus

letting

z

T

we

have

z

T

-LRB-

oe

-LRB-

T

-LRB-

-RRB-

-RRB-

-RRB-

T

-LRB-

oe

-LRB-

-RRB-

-RRB-

oe

-LRB-

-RRB-

where

in

the

previous

equations

we

have

used

the

fact

that

T

noe

So

as

z

-LRB-

-RRB-

T

-LRB-

-RRB-

uniqueness

of

solutions

implies

that

z

-LRB-

t

-RRB-

T

-LRB-

t

-RRB-

for

all

t

for

the

trajectory

of

the

second

system

corresponding

to

the

same

control

which

implies

-LRB-

t

-RRB-

-LRB-

t

-RRB-

-LRB-

t

-RRB-

for

all

t.

Technical

Results

Again

we

fix

a

oe

IR

IR

assumed

differentiable

and

so

that

-LRB-

-RRB-

holds

more

precisely

so

that

oe

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

In

this

section

we

assume

that

two

fixed

systems

-LRB-

-RRB-

oe

and

-LRB-

-RRB-

oe

with

m

inputs

and

p

outpus

are

given

and

we

would

like

to

find

necessary

conditions

for

these

two

systems

to

have

the

same

i/o

behavior

when

both

are

initialized

at

Some

Basic

Identities

Let

ukg

be

control

values

i.e.

ui

Rm

for

each

i

and

let

ti

i

k

be

positive

real

numbers

Consider

the

piecewise

constant

control

u

-LRB-

t

-RRB-

on

the

interval

-LSB-

T

-RRB-

with

T

k

X

i

ti

which

is

equal

to

on

-LSB-

-RRB-

on

-LSB-

-RRB-

and

so

on

Denote

by

yij

-LRB-

t

-RRB-

the

jth-component

of

the

output

of

the

system

i

i

corresponding

to

the

control

u

-LRB-

t

-RRB-

and

by

X

iul

l

k

the

following

vector

field

X

iul

-LRB-

x

-RRB-

Dix

oe

-LRB-

Aix

Biul

-RRB-

Giul

-LRB-

-RRB-

We

denote

hij

-LRB-

x

-RRB-

-LRB-

Cix

-RRB-

j

The

following

is

a

well-known

formula

-LRB-

see

e.g.

-LSB-

-RSB-

page

-RRB-

@k

@tk

tk

yij

-LRB-

t

-RRB-

LXiuk

hij

-LRB-

-RRB-

-LRB-

-RRB-

for

all

IRn

where

LXh

denotes

the

Lie-derivative

of

the

function

h

along

the

vector

field

X

Thus

if

and

have

the

same

i/o

behavior

from

-LRB-

-RRB-

we

have

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

for

all

j

p

k

and

vectors

uj

Rm

The

following

technical

Lemma

will

be

used

later

to

derive

some

necessary

conditions

for

the

two

systems

to

have

the

same

i/o

behavior

If

h

is

a

smooth

function

from

IRn

to

IR

and

Z

is

a

smooth

vector

field

on

IRn

expressed

in

the

natural

coordinates

we

will

use

the

following

notations

rh

-LRB-

@h

@h

@xn

-RRB-

the

usual

gradient

-LRB-

Z

-RRB-

the

usual

Jacobian

matrix

of

Z

i.e.

if

-LRB-

Z

-RRB-

k

zk

then

-LSB-

-LRB-

Z

-RRB-

-RSB-

lk

@zk

@xl

When

we

omit

arguments

we

understand

all

Jacobians

and

gradients

as

being

evaluated

at

the

same

point

x.

Lemma

Let

h

be

a

smooth

function

from

IRn

to

IR

and

Zi

i

be

smooth

vector

fields

on

IRn

Then

for

all

k

the

following

formula

holds

LZk

rh

-LRB-

-RRB-

-LRB-

Zk

-RRB-

Zk

Gk

-LRB-

h

Zk

-RRB-

-LRB-

-RRB-

where

the

expression

Gk

-LRB-

h

Zk

-RRB-

which

is

a

scalar

function

of

x

is

a

sum

of

terms

each

of

which

is

a

product

of

k

factors

and

each

of

these

factors

is

a

derivative

of

order

at

most

k

either

of

the

function

h

or

of

the

components

of

the

vector

fields

Zi

Moreover

in

each

term

there

are

at

least

three

factors

where

the

derivative

is

of

order

or

and

at

least

two

of

them

are

of

order

Proof

We

will

prove

the

statement

by

induction

on

k.

Let

k

then

we

have

-LRB-

-RRB-

by

definition

of

Lie

derivative

So

in

this

case

the

Lemma

holds

with

-LRB-

h

-RRB-

To

give

an

idea

of

what

the

expression

of

the

function

Gk

is

before

dealing

with

the

general

induction

step

we

also

study

explicitly

the

case

k

In

this

case

we

have

n

X

i

@xi

-LRB-

-RRB-

-LRB-

-RRB-

i

rh

-LRB-

-RRB-

n

X

i

j

@xi

@xj

-LRB-

-RRB-

i

-LRB-

-RRB-

j

-LRB-

-RRB-

thus

letting

-LRB-

h

-RRB-

Pni

j

@xi

@xj

-LRB-

-RRB-

i

-LRB-

-RRB-

j

satisfies

the

desired

properties

and

so

-LRB-

-RRB-

holds

for

k

Let

now

k

By

inductive

assumption

we

have

LZk

-LRB-

LZk

-RRB-

r

-LSB-

rh

-LRB-

-RRB-

-LRB-

Zk

-RRB-

Zk

-RSB-

Zk

r

-LSB-

Gk

-LRB-

h

Zk

-RRB-

-RSB-

Zk

First

we

discuss

the

first

term

of

this

sum

Let

L

rh

-LRB-

-RRB-

-LRB-

Zk

-RRB-

a

row

vector

So

the

first

term

is

r

-LSB-

LZk

-RSB-

Zk

L

-LRB-

Zk

-RRB-

Zk

ZTk

-LRB-

LT

-RRB-

Zk

where

T

indicates

transpose

Thus

we

have

LZk

rh

-LRB-

-RRB-

-LRB-

Zk

-RRB-

Zk

Gk

-LRB-

h

Zk

-RRB-

where

Gk

-LRB-

h

Zk

-RRB-

ZTk

-LRB-

LT

-RRB-

Zk

-LCB-

z

-RCB-

r

-LSB-

Gk

-LRB-

h

Zk

-RRB-

-RSB-

Zk

-LCB-

z

-RCB-

-LRB-

-RRB-

Now

we

need

to

show

that

Gk

has

the

right

properties

It

is

clear

from

the

expression

of

Gk

using

rules

for

differentiation

of

products

that

it

is

a

sum

of

terms

in

which

each

term

is

a

product

of

k

factors

Moreover

the

terms

arising

from

in

-LRB-

-RRB-

involve

only

derivatives

of

at

most

order

two

and

those

coming

from

involve

derivatives

of

order

at

most

k

-LRB-

since

by

induction

Gk

has

derivatives

of

order

at

most

k

-RRB-

So

we

only

need

to

show

that

in

each

term

there

are

at

least

three

derivatives

of

order

or

and

two

are

of

order

Each

term

of

starts

and

ends

in

factors

of

order

zero

-LRB-

the

ones

arising

from

Zk

and

Zk

-RRB-

and

exactly

one

of

the

factors

in-between

includes

a

derivative

of

order

since

in

L

there

are

only

first

order

derivatives

For

the

terms

coming

from

notice

that

in

each

term

the

last

factor

is

of

order

zero

and

using

the

inductive

assumption

we

can

conclude

that

there

are

two

other

factors

one

of

order

zero

and

one

of

order

or

Notice

that

if

X

iu

is

any

vector

field

of

the

type

defined

by

equation

-LRB-

-RRB-

then

-LRB-

X

iu

-LRB-

x

-RRB-

-RRB-

Di

oe

-LRB-

Aix

Biu

-RRB-

Ai

-LRB-

-RRB-

where

for

any

vector

v

-LRB-

vn

-RRB-

oe

-LRB-

v

-RRB-

denotes

the

n

n

diagonal

matrix

oe

-LRB-

v

-RRB-

Diag

-LRB-

-LRB-

-RRB-

-LRB-

vn

-RRB-

-RRB-

Lemma

Let

-LRB-

D

A

B

G

C

-RRB-

oe

be

a

oe-system

and

pick

ui

IRm

i

k.

Denoting

by

Cj

the

jth-row

of

C

we

have

that

for

all

j

p

LXuk

Cj

-LRB-

x

-RRB-

Cj

-LRB-

D

oe

-LRB-

Ax

-RRB-

A

-RRB-

-LRB-

D

oe

-LRB-

Ax

Buk

-RRB-

A

-RRB-

-LRB-

Dx

oe

-LRB-

Ax

Buk

-RRB-

Guk

-RRB-

fkj

-LRB-

x

uk

-RRB-

-LRB-

-RRB-

where

fkj

is

a

function

which

is

identically

zero

if

k

or

and

for

k

satisfies

fkj

-LRB-

u

v

-RRB-

for

all

u

v

IRm

Proof

We

apply

the

result

in

Lemma

By

-LRB-

-RRB-

we

have

that

the

first

term

in

the

right

hand

side

of

-LRB-

-RRB-

is

exactly

the

first

term

in

-LRB-

-RRB-

If

k

or

then

we

get

the

desired

conclusions

from

equations

-LRB-

-RRB-

and

-LRB-

-RRB-

since

any

derivative

of

Cjx

of

order

greater

then

is

identically

zero

For

k

we

only

need

to

prove

that

Gk

-LRB-

Cjx

Xuk

-RRB-

-LRB-

-RRB-

at

x

when

uk

v

and

all

the

other

ui

's

but

one

equal

zero

Since

the

second

derivative

of

Cjx

is

zero

oe

-LRB-

Ax

Bu

-RRB-

jx

u

and

-LRB-

-RRB-

all

the

zero

and

second

order

derivatives

vanish

identically

for

entries

of

and

C.

Thus

-LRB-

-RRB-

holds

by

the

Lemma

Proposition

Let

and

be

the

two

oe-systems

If

they

are

i/o

equivalent

then

for

all

l

k

and

for

all

u

v

IRm

it

holds

that

-LRB-

-RRB-

l

-LRB-

oe

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

-LRB-

-RRB-

l

-LRB-

oe

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

-LRB-

-RRB-

Proof

The

statement

follows

from

Lemma

applied

to

the

sequence

of

control

values

-LRB-

-LCB-

z

-RCB-

l

v

-LCB-

z

-RCB-

k

u

-RRB-

since

oe

-LRB-

Ax

Bu

-RRB-

jx

u

I

and

fkj

-LRB-

v

u

-RRB-

Remarks

Let

and

be

as

before

Appling

-LRB-

-RRB-

to

the

sequence

of

control

values

-LRB-

-LCB-

z

-RCB-

k

u

-RRB-

we

have

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

-LRB-

-RRB-

for

all

u

IRm

Let

vl

be

the

lth-coordinate

of

v

IRm

Then

@vl

-LSB-

C

-LRB-

D

A

-RRB-

k

-LRB-

oe

-LRB-

Bv

-RRB-

Gv

-RRB-

-RSB-

C

-LRB-

D

A

-RRB-

k

-LSB-

oe

-LRB-

Bv

-RRB-

Bl

Gl

-RSB-

where

Bl

and

Gl

are

the

lth

columns

of

B

and

G

respectively

Thus

from

the

equality

-LRB-

-RRB-

and

evaluating

the

previous

derivative

at

v

we

have

that

-LRB-

-RRB-

k

-LRB-

-RRB-

-LRB-

-RRB-

k

-LRB-

-RRB-

-LRB-

-RRB-

for

all

integers

k

Finding

the

Equivalence

T

For

each

fixed

positive

integer

n

m

p

let

Sn

m

;p

f

-LRB-

A

B

C

-RRB-

j

A

Rn?n

B

Rn?m

C

Rp?ng

We

let

Scn

m

;p

be

the

subset

of

Sn

m

;p

consisting

of

those

triples

-LRB-

A

B

C

-RRB-

which

are

canonical

i.e.

observable

rank

-LSB-

CT

ATCT

-LRB-

AT

-RRB-

n

-RSB-

n

and

controllable

rank

-LSB-

B

AB

An

-RSB-

n

see

-LSB-

-RSB-

section

This

is

a

generic

in

the

sense

of

the

introduction

subset

of

Sn

m

;p

for

each

n

m

and

p.

Proposition

Assume

and

are

i/o

equivalent

Assume

also

that

-LRB-

Di

Ai

Bi

Gi

Ci

-RRB-

Scni

m

;p

for

i

Then

n

and

there

exists

a

unique

matrix

T

Gl

-LRB-

n

-RRB-

such

that

T

-LRB-

-RRB-

T

T

-LRB-

-RRB-

oe

-LRB-

-RRB-

T

-LRB-

oe

-LRB-

-RRB-

-RRB-

u

IRm

-LRB-

-RRB-

Proof

Note

that

equation

-LRB-

-RRB-

says

that

the

two

linear

systems

given

by

the

triples

-LRB-

Di

Ai

Bi

Gi

Ci

-RRB-

i

have

the

same

i/o

behavior

thus

since

they

are

both

canonical

we

know

by

the

linear

theory

that

and

there

exists

a

unique

invertible

matrix

T

that

satisfies

the

first

three

equations

of

-LRB-

-RRB-

Thus

in

particular

we

have

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

-LRB-

-RRB-

kT

-LSB-

oe

-LRB-

-RRB-

-RSB-

k

u

IRm

Combined

with

equation

-LRB-

-RRB-

this

gives

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

-LRB-

-RRB-

kT

-LSB-

oe

-LRB-

-RRB-

-RSB-

k

u

IRm

and

thus

by

the

observability

assumption

applied

to

the

pair

-LRB-

-LRB-

-RRB-

-RRB-

we

have

oe

-LRB-

-RRB-

T

-LRB-

oe

-LRB-

-RRB-

-RRB-

u

IRm

as

desired

Remark

Let

vl

be

the

lth-coordinate

of

v

IRm

Then

by

taking

the

derivative

with

respect

to

vl

in

the

last

equation

of

-LRB-

-RRB-

we

have

oe

-LRB-

-RRB-

-LRB-

-RRB-

l

-LRB-

-RRB-

l

T

-LRB-

oe

-LRB-

-RRB-

-LRB-

-RRB-

l

-LRB-

-RRB-

l

-RRB-

where

as

usual

-LRB-

Bi

-RRB-

l

and

-LRB-

Gi

-RRB-

l

i

are

the

lth

columns

of

B

and

G

respectively

This

implies

oe

-LRB-

-RRB-

T

-LSB-

oe

-LRB-

-RRB-

-RSB-

v

IRm

-LRB-

-RRB-

Proposition

Assume

that

satisfy

the

same

assumptions

as

in

Proposition

Then

the

invertible

matrix

T

satisfies

also

the

following

equation

-LRB-

oe

-LRB-

-RRB-

-RRB-

T

T

-LRB-

oe

-LRB-

-RRB-

-RRB-

u

IRm

-LRB-

-RRB-

Proof

Notice

first

that

by

-LRB-

-RRB-

we

have

-LRB-

-RRB-

l

-LRB-

-RRB-

lT

l

Now

we

apply

-LRB-

-RRB-

to

get

for

any

u

v

IRm

and

any

l

-LRB-

-RRB-

l

-LRB-

oe

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

-LRB-

-RRB-

lT

-LRB-

oe

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

Thus

by

the

observability

assumption

on

-LRB-

-RRB-

we

have

-LRB-

oe

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

T

-LRB-

oe

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

Applying

-LRB-

-RRB-

to

the

left-hand

side

of

this

we

have

-LRB-

oe

-LRB-

-RRB-

-RRB-

T

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

T

-LRB-

oe

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

Since

for

all

l

m

we

have

@vl

-LRB-

-RRB-

k

-LSB-

oe

-LRB-

-RRB-

-RSB-

v

-LRB-

-RRB-

k

-LSB-

-LRB-

-RRB-

l

-LRB-

-RRB-

l

-RSB-

where

-LRB-

-RRB-

l

and

-LRB-

-RRB-

l

are

the

lth

columns

of

and

respectively

we

can

conclude

that

for

all

k

-LRB-

oe

-LRB-

-RRB-

-RRB-

T

-LRB-

-RRB-

k

-LRB-

-RRB-

T

-LRB-

oe

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

k

-LRB-

-RRB-

This

equation

together

with

the

controllability

assumption

on

-LRB-

-RRB-

gives

-LRB-

-RRB-

Characterization

of

T

Now

we

want

to

prove

that

the

sufficient

condition

stated

in

Proposition

is

also

necessary

in

a

generic

sense

For

this

purpose

we

first

let

Boen

m

fB

IRn?m

j

bi

j

i

j

and

i

j

k

such

that

bi

k

bj

k

oeg

-LRB-

-RRB-

Notice

that

if

condition

-LRB-

P

-RRB-

holds

which

in

turn

would

be

the

case

if

-LRB-

-RRB-

holds

then

the

second

condition

in

the

definition

of

Boen

m

says

that

for

all

i

j

there

exists

k

such

that

jbi

kj

jbj

kj

As

an

illustration

if

-LRB-

P

-RRB-

holds

and

m

the

conditions

defining

Boen

say

that

all

the

components

of

the

vector

B

are

nonzero

and

they

have

different

absolute

values

We

let

Sn

m

;p

-LRB-

-LRB-

D

A

B

G

C

-RRB-

oe

B

Boen

m

rank

-LSB-

A

B

-RSB-

n

and

-LRB-

A

D

B

G

C

-RRB-

Scn

m

;p

-RRB-

-LRB-

-RRB-

Notice

that

Sn

m

;p

is

a

generic

subset

of

the

set

of

all

oe-systems

when

we

identify

the

latter

with

np

mn

c.f.

Remark

provided

that

oe

is

finite

Theorem

Let

i

-LRB-

Di

Ai

Bi

Gi

Ci

-RRB-

oe

Sni

m

;p

for

i

and

assume

that

oe

satisfies

-LRB-

-RRB-

Then

the

two

systems

are

i/o

equivalent

if

and

only

if

they

are

oe-equivalent

Proof

The

sufficiency

part

is

given

by

Proposition

thus

we

need

only

to

prove

the

necessary

part

From

Proposition

we

know

that

n

let

T

be

the

matrix

obtained

in

that

Proposition

By

equation

-LRB-

-RRB-

we

have

-LRB-

-RRB-

i

j

-LRB-

-RRB-

i

j

-LRB-

m

X

l

lul

-RRB-

-LRB-

-RRB-

i

j

n

X

k

ti

j

-LRB-

m

X

l

lul

-RRB-

For

a

control

value

of

the

special

form

u

-LRB-

v

-RRB-

with

v

IR

this

equation

becomes

-LRB-

-RRB-

i

j

-LRB-

-RRB-

i

j

-LRB-

-RRB-

-LRB-

-RRB-

i

j

n

X

k

ti

j

-LRB-

-RRB-

-LRB-

-RRB-

If

we

take

the

limit

as

v

then

since

all

the

entries

of

and

are

non

zero

and

-LRB-

-RRB-

holds

the

left-hand

side

of

equation

-LRB-

-RRB-

goes

to

zero

So

we

can

conclude

that

T

-LRB-

-RRB-

which

in

turn

implies

T

by

the

second

equation

in

-LRB-

-RRB-

Using

again

the

control

value

u

of

the

special

form

-LRB-

v

-RRB-

equation

-LRB-

-RRB-

becomes

-LRB-

-RRB-

i

j

-LRB-

-RRB-

i

j

-LRB-

-RRB-

-LRB-

-RRB-

i

j

n

X

k

ti

j

-LRB-

-RRB-

Taking

the

limit

as

v

we

have

as

before

that

the

left-hand

side

of

the

previous

equation

goes

to

zero

Thus

which

in

turn

implies

by

the

third

equation

in

-LRB-

-RRB-

Notice

for

further

reference

that

assumption

-LRB-

-RRB-

is

used

only

in

these

two

parts

of

the

proof

Thus

the

matrix

T

satisfies

all

the

desired

interlacing

equations

It

remains

to

show

that

T

noe

for

which

it

suffices

to

show

that

T

is

of

the

form

PQ

where

P

and

Q

are

defined

respectively

as

in

-LRB-

-RRB-

and

in

-LRB-

-RRB-

From

-LRB-

-RRB-

and

-LRB-

-RRB-

we

get

oe

-LRB-

-RRB-

T

T

oe

-LRB-

-RRB-

u

IRm

-LRB-

-RRB-

On

the

other

hand

since

we

have

proved

that

T

the

last

equation

in

-LRB-

-RRB-

becomes

oe

-LRB-

-RRB-

T

oe

-LRB-

-RRB-

u

IRm

-LRB-

-RRB-

Taking

the

partial

derivative

with

respect

to

ul

in

the

previuos

equation

we

conclude

for

all

u

IRm

oe

-LRB-

-RRB-

-LRB-

-RRB-

l

T

oe

-LRB-

-RRB-

-LRB-

-RRB-

l

where

-LRB-

Bi

-RRB-

l

indicates

the

lth

column

of

Bi

and

hence

also

oe

-LRB-

-RRB-

T

oe

-LRB-

-RRB-

Thus

using

the

other

interlacing

equations

we

conclude

oe

-LRB-

-RRB-

T

T

oe

-LRB-

-RRB-

-LRB-

-RRB-

for

all

u

IRm

From

-LRB-

-RRB-

and

-LRB-

-RRB-

we

have

oe

-LRB-

-RRB-

T

-LSB-

-RSB-

T

oe

-LRB-

-RRB-

-LSB-

-RSB-

Thus

by

the

full

rank

assumption

on

the

matrix

-LSB-

-RSB-

we

can

conclude

oe

-LRB-

-RRB-

T

T

oe

-LRB-

-RRB-

u

IRm

-LRB-

-RRB-

We

can

rephrase

-LRB-

-RRB-

as

follows

-LRB-

m

X

l

lul

-RRB-

ti

j

-LRB-

m

X

l

lul

-RRB-

ti

j

By

taking

u

of

the

special

form

wk

-LRB-

v

-RRB-

-LRB-

v

IR

in

the

kth-position

-RRB-

we

conclude

for

each

pair

i

j

ti

j

-RRB-

-LRB-

kv

-RRB-

-LRB-

kv

-RRB-

k

m

IR

-LRB-

-RRB-

Since

oe

-LRB-

-RRB-

we

may

integrate

and

conclude

that

koe

-LRB-

kv

-RRB-

koe

-LRB-

kv

-RRB-

for

any

such

i

j

and

for

all

v

or

equivalently

that

oe

-LRB-

sx

-RRB-

soe

-LRB-

x

-RRB-

for

all

x

where

s

k

k

that

is

ti

j

-RRB-

k

k

oe

k

m

-LRB-

-RRB-

For

i

n

let

Li

fj

j

ti

j

g

since

T

is

invertible

each

Li

Moreover

if

Li

Lj

for

some

pair

i

j

then

we

have

-LRB-

using

any

k

Li

Lj

-RRB-

l

l

oe

and

l

l

oe

l

m

Since

oe

is

a

subgroup

the

previous

equation

implies

l

l

l

l

l

l

oe

l

m

which

contradicts

the

assumption

that

Boen

m.

So

we

conclude

that

Li

consists

of

only

one

element

thus

for

each

i

n

there

exists

only

one

index

j

ss

-LRB-

i

-RRB-

such

that

ti

ss

-LRB-

i

-RRB-

Let

P

be

the

permutation

matrix

representing

ss

then

we

can

write

the

matrix

T

as

follows

T

PQ

where

Q

Diag

-LRB-

ss

-LRB-

-RRB-

tn

ss

-LRB-

n

-RRB-

-RRB-

To

complete

the

proof

we

need

to

see

only

that

ti

ss

-LRB-

i

-RRB-

oe

for

all

i

n

From

equation

-LRB-

-RRB-

and

since

we

have

that

for

each

u

-LRB-

um

-RRB-

IRm

oe

-LRB-

m

X

l

ti

ss

-LRB-

i

-RRB-

-LRB-

i

-RRB-

lul

-RRB-

ti

ss

-LRB-

i

-RRB-

oe

-LRB-

m

X

l

-LRB-

i

-RRB-

lul

-RRB-

for

each

i

n

Again

using

u

of

the

special

type

wk

-LRB-

v

-RRB-

-LRB-

v

IR

in

the

kth-position

-RRB-

we

get

oe

-LRB-

ti

ss

-LRB-

i

-RRB-

-LRB-

i

-RRB-

kv

-RRB-

ti

ss

-LRB-

i

-RRB-

oe

-LRB-

-LRB-

i

-RRB-

kv

-RRB-

v

IRm

As

-LRB-

i

-RRB-

k

this

means

ti

ss

-LRB-

i

-RRB-

oe

as

desired

Remark

We

can

conclude

the

same

results

if

only

require

that

the

function

oe

be

infinitely

differentiable

in

a

neighbourhood

of

zero

instead

of

on

all

of

IR

and

if

we

change

-LRB-

-RRB-

into

-LRB-

-RRB-

oe

-LRB-

IR

-RRB-

and

-LRB-

x

-RRB-

as

jxj

For

the

case

D

ffI

mentioned

in

Remarks

and

just

smoothness

about

x

is

sufficient

Remark

Assume

given

n

different

functions

oei

that

satisfy

our

assumptions

-LRB-

-RRB-

instead

of

a

fixed

oe

Define

oe

-LRB-

x

-RRB-

-LRB-

-LRB-

-RRB-

oen

-LRB-

xn

-RRB-

-RRB-

If

-LRB-

-RRB-

for

all

i

for

some

fixed

then

all

our

results

still

hold

for

two

generic

systems

of

the

same

dimension

Special

Cases

We

show

in

this

section

how

to

deal

with

a

few

special

cases

of

the

general

class

of

systems

considered

When

D

is

diagonal

and

G

Assume

now

that

we

fix

an

ff

IR

and

we

restrict

our

attention

to

the

subclass

of

systems

of

the

form

-LRB-

ffI

A

B

C

-RRB-

oe

where

ff

is

this

fixed

real

number

-LRB-

the

same

for

all

members

of

the

class

-RRB-

In

this

case

the

result

of

Theorem

applies

even

if

we

drop

the

assumption

-LRB-

-RRB-

In

fact

from

equation

-LRB-

-RRB-

since

ffI

and

we

can

conclude

without

using

assumption

-LRB-

-RRB-

that

T

and

oe

-LRB-

-RRB-

T

oe

-LRB-

-RRB-

u

IRm

As

assumption

-LRB-

-RRB-

was

used

only

in

this

part

of

the

proof

our

statement

follows

In

particular

this

applies

for

ff

which

gives

the

class

of

systems

in

equation

-LRB-

-RRB-

or

ff

which

gives

the

class

in

equation

-LRB-

-RRB-

Observe

that

in

this

case

-LRB-

D

diagonal

and

G

-RRB-

the

assumption

rank

-LSB-

A

B

-RSB-

n

is

redundant

as

it

follows

from

controllability

of

the

pair

-LRB-

A

ffI

B

-RRB-

or

equivalently

of

the

pair

-LRB-

A

B

-RRB-

this

is

just

the

case

ff

of

the

Hautus

condition

-LRB-

c.f.

-LSB-

-RSB-

Lemma

-RRB-

When

D

is

diagonal

and

B

Assume

that

we

fix

again

an

ff

IR

and

this

time

we

restrict

our

attention

to

the

subclass

of

systems

of

the

form

-LRB-

ffI

A

G

C

-RRB-

oe

where

ff

is

this

fixed

real

number

-LRB-

the

same

for

all

members

of

the

class

-RRB-

Notice

that

when

ff

this

class

of

systems

is

the

class

described

in

equation

-LRB-

-RRB-

A

typical

system

of

this

class

is

of

the

type

x

ffx

oe

-LRB-

Ax

-RRB-

Gu

y

Cx

-LRB-

-RRB-

In

this

case

we

can

not

appply

the

results

of

Theorem

directly

since

for

this

class

B

and

the

zero

matrix

does

not

belong

to

Boen

m

so

we

argue

as

follows

Given

as

before

we

consider

the

following

system

z

ffz

oe

-LRB-

Az

AGu

-RRB-

w

Cz

-LRB-

-RRB-

The

next

Lemma

is

easily

checked

Lemma

Let

u

-LRB-

t

-RRB-

be

an

input

function

defined

on

-LSB-

T

-RSB-

with

u

-LSB-

T

-RSB-

and

u

-LRB-

-RRB-

If

z

-LRB-

t

-RRB-

is

the

trajectory

of

corresponding

to

u

-LRB-

t

-RRB-

with

z

-LRB-

-RRB-

then

x

-LRB-

t

-RRB-

z

-LRB-

t

-RRB-

Gu

-LRB-

t

-RRB-

is

the

trajectory

of

corresponding

to

the

input

ffu

-LRB-

t

-RRB-

u

-LRB-

t

-RRB-

with

x

-LRB-

-RRB-

Let

A

be

the

following

class

of

control

functions

A

fv

j

v

-LSB-

T

-RSB-

for

some

T

v

-LRB-

-RRB-

g

Lemma

If

and

are

two

oe-systems

which

are

not

i/o

equivalent

then

there

exists

v

A

and

t

such

that

denoting

by

yi

i

the

respective

output

functions

corresponding

to

v

-LRB-

t

-RRB-

-LRB-

t

-RRB-

Proof

Given

an

admissible

control

u

-LRB-

-RRB-

we

can

find

a

sequence

un

-LRB-

-RRB-

A

such

that

the

controls

un

-LRB-

-RRB-

are

equibounded

and

converge

to

u

-LRB-

-RRB-

almost

everywhere

Now

we

need

only

apply

the

approximation

results

in

Theorem

of

-LSB-

-RSB-

to

conclude

the

desired

result

Let

and

be

two

systems

of

type

-LRB-

-RRB-

and

i

for

i

their

corresponding

systems

of

type

-LRB-

-RRB-

Proposition

If

and

are

i/o

equivalent

then

and

are

also

i/o

equivalent

Proof

We

first

notice

that

if

and

are

i/o

equivalent

then

for

any

value

v

IRm

we

have

-LRB-

-RRB-

Indeed

let

u

-LRB-

t

-RRB-

vt

be

a

linear-in-time

control

function

Then

for

t

small

enough

this

control

is

admissible

for

both

and

Moreover

denoting

by

yi

-LRB-

-RRB-

the

respective

output

functions

we

have

yi

-LRB-

-RRB-

CiGiv

which

given

the

i/o

equivalence

implies

-LRB-

-RRB-

We

now

prove

the

statement

by

way

of

contradiction

Assume

that

and

are

not

i/o

equivalent

Then

by

Lemma

there

exist

u

-LRB-

-RRB-

A

and

t

such

that

denoting

by

yi

-LRB-

-RRB-

the

respective

output

functions

corresponding

to

u

-LRB-

-RRB-

we

have

-LRB-

t

-RRB-

-LRB-

t

-RRB-

-LRB-

-RRB-

Since

u

-LRB-

-RRB-

A

by

Lemma

we

have

that

xi

-LRB-

t

-RRB-

zi

-LRB-

t

-RRB-

Giu

-LRB-

t

-RRB-

for

i

are

the

trajectories

in

i

corresponding

to

the

control

ffu

-LRB-

-RRB-

u

-LRB-

-RRB-

Moreover

their

corresponding

outputs

are

yi

-LRB-

t

-RRB-

yi

-LRB-

t

-RRB-

CiGiu

-LRB-

t

-RRB-

-LRB-

-RRB-

From

the

previous

equation

and

from

equations

-LRB-

-RRB-

and

-LRB-

-RRB-

we

have

-LRB-

t

-RRB-

-LRB-

t

-RRB-

which

contradicts

the

fact

that

the

systems

i

are

i/o

equivalent

The

proof

in

this

case

will

rely

upon

the

transformation

-LRB-

ffI

A

G

C

-RRB-

oe

-LRB-

ffI

A

AG

C

-RRB-

oe

This

will

require

reinterpreting

the

genericity

conditions

for

the

latter

system

in

terms

of

the

former

one

For

this

purpose

we

first

observe

that

for

any

two

matrices

A

IRn?n

and

G

IRn?m

it

holds

that

rank

-LSB-

A

AG

-RSB-

n

if

and

only

if

A

is

invertible

-LRB-

A

AG

-RRB-

is

a

controllable

pair

if

and

only

if

A

is

invertible

and

the

pair

-LRB-

A

G

-RRB-

is

controllable

The

first

property

follows

from

the

fact

that

-LSB-

A

AG

-RSB-

A

-LSB-

I

G

-RSB-

while

the

second

assertion

follows

from

-LSB-

AG

AnG

-RSB-

A

-LSB-

G

AG

An

-RSB-

Now

let

Sn

m

;p

-LRB-

-LRB-

ffI

A

G

C

-RRB-

oe

AG

Boen

m

A

invertible

and

-LRB-

A

G

C

-RRB-

Scn

m

;p

-RRB-

Theorem

Let

i

-LRB-

ffI

Ai

Gi

Ci

-RRB-

oe

Sni

m

;p

for

i

Then

the

two

systems

are

i/o

equivalent

if

and

only

if

they

are

oe-equivalent

Proof

We

need

only

to

prove

the

necessary

part

By

Proposition

if

and

are

i/o

equivalent

then

their

corresponding

systems

and

of

type

-LRB-

-RRB-

are

also

i/o

equivalent

It

is

easy

to

see

that

the

condition

i

Sni

m

;p

implies

i

Sni

m

;p

So

we

can

apply

Theorem

-LRB-

notice

that

this

Theorem

holds

without

assumption

-LRB-

-RRB-

by

the

remark

in

subsection

-RRB-

and

conclude

that

n

and

there

exists

T

noe

such

that

T

T

Since

the

matrices

Ai

are

invertible

the

last

two

equations

imply

T

as

desired

Yet

another

class

Under

a

change

of

coordinates

z

Ax

Equation

-LRB-

-RRB-

becomes

z

ffz

A

oe

-LRB-

z

-RRB-

y

-LRB-

-RRB-

where

AG

and

CA

-LRB-

asumming

invertibility

of

A

-RRB-

This

type

of

system

appears

often

in

the

literature

as

well

Again

the

uniqueness

result

holds

a

genericity

condition

can

be

derived

from

the

above

change

of

coordinates

Boen

m

A

invertible

and

-LRB-

A

-RRB-

Scn

m

;p

Several

other

classes

of

systems

can

also

be

treated

by

coordinate

transformations

on

the

classes

considered

here

Proof

of

Main

Results

Theorems

and

are

now

proved

as

follows

First

observe

that

since

property

-LRB-

P

-RRB-

holds

and

using

Lemma

every

element

of

noe

is

of

the

form

PQ

with

P

a

permutation

matrix

and

Q

a

diagonal

matrix

with

entries

in

f

and

ffl

When

oe

is

odd

noe

is

precisely

the

set

of

such

elements

ffl

When

oe

is

not

odd

oe

so

noe

is

exactly

the

set

of

permutation

matrices

The

results

now

follow

from

Theorem

via

the

remarks

in

subsections

and

Notice

that

for

Theorem

one

can

take

-LRB-

n

m

p

-RRB-

-LRB-

n

m

p

-RRB-

f

-LRB-

A

B

C

-RRB-

B

Boen

m

and

-LRB-

A

B

C

-RRB-

Scn

m

pg

and

-LRB-

n

m

p

-RRB-

f

-LRB-

A

B

C

-RRB-

AB

Boen

m

A

invertible

and

-LRB-

A

B

C

-RRB-

Scn

m

pg

Moreover

here

Boen

m

is

just

the

class

of

matrices

B

for

which

bi

j

for

all

i

j

for

each

i

j

there

exists

some

k

such

that

jbi

kj

jbj

kj

For

Theorem

the

sets

can

be

taken

in

the

same

way

except

that

now

the

set

Boen

m

consists

of

the

matrices

B

which

satisfy

that

all

entries

are

nonzero

and

this

property

holds

for

each

i

j

there

exists

some

k

such

that

bi

k

bj

k.

Universal

Inputs

for

Identification

The

results

in

this

paper

imply

that

i/o

experiments

completely

determine

the

internal

weights

of

networks

-LRB-

up

to

a

relabeling

of

units

and

sign

reversals

-RRB-

This

means

that

if

two

nets

are

not

equal

then

there

is

some

input

to

which

they

respond

differently

Given

any

pair

of

distinct

nets

and

assuming

the

genericity

conditions

apply

to

both

there

is

some

such

diagnostic

input

which

depends

on

the

pair

that

has

to

be

distinguished

In

principle

however

it

may

have

been

the

case

that

there

is

no

universal

input

which

given

any

pair

of

distinct

nets

serves

to

distinguish

this

particular

pair

It

turns

out

however

that

such

universal

testing

inputs

do

exist

and

in

fact

inputs

with

such

properties

are

generic

in

a

precise

mathematical

sense

This

is

summarized

in

the

following

statement

We

assume

that

oe

is

an

analytic

function

and

that

its

derivative

is

bounded

-LRB-

The

last

condition

is

imposed

simply

to

obtain

a

statement

not

involving

domains

of

definition

of

solutions

of

differential

equations

and

can

be

relaxed

the

analyticity

assumption

is

critical

on

the

other

hand

-RRB-

For

each

input

u

-LRB-

-RRB-

-LSB-

T

-RSB-

IRm

consider

the

mapping

u

y

-LRB-

-RRB-

where

y

-LRB-

-RRB-

-LSB-

T

-RSB-

IRp

is

the

-LRB-

zero-initial-state

-RRB-

output

of

when

the

input

u

-LRB-

-RRB-

is

applied

to

the

system

We

view

the

domain

of

u

as

the

union

of

the

sets

-LRB-

n

m

p

-RRB-

over

all

positive

integers

n

that

is

the

set

of

all

systems

of

the

first

class

and

with

m

input

channels

and

p

outputs

-LRB-

An

entirely

analogous

result

holds

for

all

the

other

classes

considered

in

this

paper

as

long

as

the

above

assumptions

on

oe

hold

-RRB-

Theorem

Let

T

be

arbitrary

Then

there

exists

an

input

u

-LSB-

T

-RSB-

IRm

such

that

u

is

one-to-one

This

result

can

be

interpreted

as

saying

that

it

is

theoretically

possible

just

from

the

information

contained

in

the

output

y

-LRB-

-RRB-

and

from

the

assumption

that

the

system

is

of

the

class

considered

to

uniquely

determine

all

the

weights

in

the

network

It

does

not

say

anything

about

actual

algorithms

for

determining

the

weights

from

this

output

data

of

course

this

latter

one

is

an

issue

for

much

further

research

Moreover

the

class

of

infinitely

differentiable

inputs

satisfying

the

conclusions

of

the

Theorem

can

be

shown

to

be

generic

in

the

sense

that

this

class

contains

a

countable

intersection

of

open

dense

subsets

in

the

standard

topology

of

the

set

of

all

smooth

controls

u

-LSB-

T

-RSB-

IRm

and

there

are

even

analytic

inputs

with

this

universality

property

-LRB-

Actually

the

class

of

good

inputs

is

extremely

large

being

defined

by

the

nonvanishing

of

certain

analytic

functions

so

it

is

fair

to

say

the

response

of

the

net

to

a

random

input

will

suffice

at

least

theoretically

for

determination

of

the

number

of

units

and

unique

identification

of

all

weights

-RRB-

The

proof

of

the

above

Theorem

is

immediate

from

the

general

results

for

control

systems

given

in

-LSB-

-RSB-

and

-LSB-

-RSB-

which

imply

that

identifiability

is

equivalent

to

single

experiment

identifiability

for

systems

defined

by

analytic

differential

equations

and

depending

analytically

on

parameters

-LRB-

here

the

weights

-RRB-

We

omit

details

of

the

application

as

this

would

involve

introducing

considerable

extra

terminology

but

the

proof

is

totally

routine

-LRB-

one

proves

the

result

for

each

fixed

n

and

then

takes

the

intersection

of

the

set

of

all

good

inputs

over

all

n

which

gives

still

a

generic

set

-RRB-

Less

Knowledge

of

Function

It

is

interesting

to

point

out

that

the

nontrivial

-LRB-

necessity

-RRB-

parts

of

the

results

presented

in

this

paper

often

do

not

really

require

precise

knowledge

of

the

nonlinearity

oe

but

merely

of

some

of

its

derivatives

or

as

discussed

later

slightly

weaker

results

hold

even

with

almost

no

information

at

all

In

this

section

we

sketch

these

generalizations

Assume

that

we

have

two

infinitely

differentiable

nonlinear

functions

and

such

that

oei

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

for

i

-LRB-

-RRB-

and

they

satisfy

assumption

with

the

same

q

i.e.

q

such

that

oe

-LRB-

q

-RRB-

i

-LRB-

-RRB-

ff

-LRB-

-RRB-

Notice

that

we

have

assumed

in

particular

that

oe

-LRB-

q

-RRB-

-LRB-

-RRB-

oe

-LRB-

q

-RRB-

-LRB-

-RRB-

Under

these

assumptions

it

is

easy

to

see

that

if

-LRB-

ax

-RRB-

-LRB-

x

-RRB-

for

all

x

in

a

neighbourhood

of

zero

then

a

f

g

We

let

n

-LRB-

T

Gl

-LRB-

n

-RRB-

T

PQ

where

P

is

a

permutation

matrix

and

Q

Diag

-LRB-

n

-RRB-

with

i

-RRB-

-LRB-

-RRB-

It

can

be

proved

that

arguing

as

in

Lemma

-LRB-

T

x

-RRB-

T

-LRB-

x

-RRB-

for

all

x

IRn

implies

T

n

We

denote

by

Fff

q

the

set

of

all

infinitely

differentiable

functions

that

satisfy

the

conditions

expressed

in

-LRB-

-RRB-

and

-LRB-

-RRB-

and

also

assumption

Now

we

give

a

definition

which

parallels

definition

Definition

Let

-LRB-

-RRB-

-LRB-

-RRB-

be

two

systems

and

be

the

dimensions

of

the

state

spaces

of

respectively

We

say

that

and

are

equivalent

if

n

and

if

there

exists

an

invertible

matrix

T

n

such

that

T

T

T

T

Given

the

previous

definition

we

would

like

to

conclude

that

-LRB-

again

generically

-RRB-

two

systems

that

are

i/o

equivalent

must

be

necessarily

equivalent

Notice

that

all

the

results

presented

in

Sections

and

are

based

on

the

properties

expressed

by

equation

-LRB-

-RRB-

for

the

functions

oei

Thus

it

is

easy

to

see

that

those

results

in

particular

the

equalities

expressed

by

equations

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

hold

if

we

substitute

oe

oe

and

oe

with

oei

oei

and

oei

in

a

consistent

way

Now

we

fix

two

values

for

the

parameters

ff

and

q

of

equation

-LRB-

-RRB-

and

we

define

two

sets

which

will

play

the

same

role

as

the

sets

Boen

m

and

Sn

m

;p

defined

in

equations

-LRB-

-RRB-

and

-LRB-

-RRB-

respectively

We

let

Bn

m

fB

IRn?m

j

bi

j

i

j

and

i

j

k

such

that

bi

k

bj

k

-LRB-

-RRB-

and

Sn

m

;p

-LRB-

D

A

B

G

C

-RRB-

oe

oe

Fff

q

B

Bn

m

rank

-LSB-

A

B

-RSB-

n

and

-LRB-

A

D

B

G

C

-RRB-

Scn

m

;p

-LRB-

-RRB-

Given

these

definitions

we

can

now

prove

the

following

result

which

corresponds

to

the

result

presented

in

Theorem

for

the

case

of

only

one

function

oe

-LRB-

Of

course

now

there

is

no

reason

for

equivalence

to

imply

i/o

equivalence

as

the

nonlinearities

may

be

very

different

-RRB-

Theorem

If

the

systems

i

-LRB-

Di

Ai

Bi

Ci

-RRB-

oei

Sni

m

;p

for

i

are

i/o

equivalent

then

they

are

equivalent

Proof

We

will

only

give

a

brief

sketch

of

the

proof

describing

only

the

critical

steps

following

the

proof

of

Theorem

Using

the

same

arguments

as

in

the

mentioned

proof

we

can

conclude

that

n

and

there

exists

an

invertible

matrix

T

which

satisfies

all

the

desired

interlacing

equations

Thus

it

remains

to

show

that

T

n.

Notice

that

the

following

equation

holds

-LRB-

-RRB-

T

T

-LRB-

-RRB-

u

IRm

So

we

find

ti

j

-RRB-

-LRB-

sv

-RRB-

-LRB-

v

-RRB-

where

s

k

k

and

the

previous

equation

holds

for

all

k

m

and

IR

Thus

we

have

ti

j

-RRB-

k

k

k

m

From

now

on

we

argue

as

in

the

last

part

of

the

proof

of

Theorem

For

i

n

let

Li

fj

j

ti

j

g

since

T

is

invertible

each

Li

Moreover

if

Li

Lj

for

some

pair

i

j

then

we

have

-LRB-

using

any

k

Li

Lj

-RRB-

l

l

and

l

l

l

m

Clearly

the

previous

equation

implies

l

l

l

m

which

contradicts

the

assumption

that

Bn

m.

So

we

conclude

that

Li

consists

of

only

one

element

thus

for

each

i

n

there

exists

only

one

index

j

ss

-LRB-

i

-RRB-

such

that

ti

ss

-LRB-

i

-RRB-

Let

P

be

the

permutation

matrix

representing

ss

then

we

can

write

the

matrix

T

as

follows

T

PQ

where

Q

Diag

-LRB-

ss

-LRB-

-RRB-

tn

ss

-LRB-

n

-RRB-

-RRB-

To

complete

the

proof

we

need

to

see

only

that

ti

ss

-LRB-

i

-RRB-

for

all

i

n

Notice

that

we

have

oe

-LRB-

ti

ss

-LRB-

i

-RRB-

-LRB-

i

-RRB-

kv

-RRB-

ti

ss

-LRB-

i

-RRB-

oe

-LRB-

-LRB-

i

-RRB-

kv

-RRB-

v

IRm

As

-LRB-

i

-RRB-

k

this

means

ti

ss

-LRB-

i

-RRB-

as

desired

No

Information

on

Derivatives

Finally

we

point

out

that

even

with

no

knowledge

of

derivatives

partial

results

can

be

obtained

Indeed

assume

we

are

given

a

function

oe

satisfying

the

following

properties

oe

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

a

oeq

-LRB-

-RRB-

b

-LRB-

-RRB-

together

with

assumption

Let

oe

-LRB-

t

-RRB-

roe

-LRB-

st

-RRB-

where

sq

a

b

and

r

as

Then

oe

q

If

-LRB-

D

A

B

G

C

-RRB-

oe

is

a

oe-system

then

the

oe-system

with

the

following

matrices

D

D

A

A

B

B

G

rG

C

C

has

the

same

i/o

behaviour

Assume

now

that

we

are

given

two

systems

i

with

i

and

with

functions

oei

satisfying

the

assumptions

of

equation

-LRB-

-RRB-

-LRB-

now

with

possibly

different

parameters

a

and

b

-RRB-

Then

the

following

Proposition

is

a

direct

consequence

of

Theorem

and

of

the

previous

observation

Proposition

Let

i

-LRB-

Di

Ai

Bi

Ci

-RRB-

oei

be

two

systems

as

before

Assume

for

i

that

Bi

Bni

m

rank

-LSB-

Ai

Bi

-RSB-

ni

-LRB-

siri

Ai

Di

Bi

riGi

Ci

-RRB-

Scni

m

;p

If

the

two

systems

are

i/o

equivalent

then

n

and

there

exists

a

matrix

T

n

such

that

T

T

T

T

Final

Comments

We

have

proved

that

generically

on

nets

the

i/o

behavior

uniquely

determines

the

internal

form

up

to

simple

symmetries

The

sets

where

this

conclusion

does

not

hold

are

thin

in

the

sense

that

they

are

included

in

sets

defined

by

algebraic

equalities

-LRB-

Note

that

the

particular

sets

S

-LRB-

n

m

p

-RRB-

that

we

construct

may

be

slightly

larger

than

strictly

needed

for

the

conclusions

to

hold

-RRB-

The

results

imply

unique

identifiability

of

parameters

under

all

possible

input/output

experiments

We

also

gave

a

result

showing

that

single

experiments

are

-LRB-

generically

-RRB-

sufficient

for

identification

in

the

analytic

case

Finally

the

last

section

showed

that

if

the

precise

nonlinearities

are

not

known

a

result

can

still

be

obtained

essentially

providing

uniqueness

up

to

four

parameters

References

-LSB-

-RSB-

Cleeremans

A.

D.

Servan-Schreiber

and

J.L.

McClelland

Finite

state

automata

and

simple

recurrent

networks

Neural

Computation

-LRB-

-RRB-

-LSB-

-RSB-

Cohen

M.A.

and

S.

Grossberg

Absolute

stability

of

global

pattern

formation

and

parallel

memory

storage

by

competitive

neural

networks

IEEE

Trans

Systems

Man

and

Cybernetics

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Cowan

J.D.

Statistical

mechanics

of

neural

nets

in

Neural

Networks

-LRB-

E.R.

Caianiello

ed

-RRB-

Springer

Berlin

pp.

-LSB-

-RSB-

Hecht-Nielsen

R.

Theory

of

the

backpropagation

neural

network

in

Proceedings

of

the

Int

Joint

Conf

on

Neural

Networks

Washington

IEEE

Publications

NY

-LCB-

-LSB-

-RSB-

Hirsch

M.W.

Convergent

activation

dynamics

in

continuous-time

networks

Neural

Networks

-LRB-

-RRB-

-LSB-

-RSB-

Hopfield

J.J.

Neurons

with

graded

responses

have

collective

computational

properties

like

those

of

two-state

neurons

Proc

of

the

Natl.

Acad

of

Sciences

USA

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Isidori

A.

Nonlinear

Control

Systems

An

Introduction

Springer-Verlag

Berlin

-LSB-

-RSB-

Michel

A.N.

J.A.

Farrell

and

W.

Porod

Qualitative

analysis

of

neural

networks

IEEE

Trans

Circuits

and

Sys.

-LRB-

-RRB-

-LSB-

-RSB-

Nijmeijer

H.

and

A.V.

Van

der

Schaft

Nonlinear

Dynamical

Control

Systems

SpringerVerlag

New

York

-LSB-

-RSB-

Robinson

A.J.

and

F.

Fallside

Static

and

dynamic

error

propagation

networks

with

application

to

speech

coding

in

Neural

Information

Processing

Systems

-LRB-

D.Z.

Anderson

ed

-RRB-

American

Institute

of

Physics

NY

pp.

-LSB-

-RSB-

Schwarzschild

R.

and

E.D.

Sontag

Algebraic

theory

of

sign-linear

systems

in

Proceedings

of

the

Automatic

Control

Conference

Boston

MA

June

-LSB-

-RSB-

Siegelmann

H.T.

and

E.D.

Sontag

Turing

computability

with

neural

nets

Appl

Math

Lett

-LRB-

-RRB-

-LRB-

-RRB-

-LSB-

-RSB-

Siegelmann

H.T.

and

E.D.

Sontag

On

the

computational

power

of

neural

nets

in

Proc

Fifth

ACM

Workshop

on

Computational

Learning

Theory

Pittsburgh

July

-LSB-

-RSB-

Siegelmann

H.T.

and

E.D.

Sontag

Analog

computation

neural

networks

and

circuits

submitted

-LSB-

-RSB-

Sontag

E.D.

On

the

observability

of

polynomial

systems

SIAM

J.Control

and

Opt.

-LRB-

-RRB-

-LSB-

-RSB-

Sontag

E.D.

Mathematical

Control

Theory

Deterministic

Finite

Dimensional

Systems

Springer

New

York

-LSB-

-RSB-

Sussmann

H.J.

Single-input

observability

of

continuous-time

systems

Math

Systems

Theory

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Sussmann

H.J.

Existence

and

uniqueness

of

minimal

realizations

of

nonlinear

systems

Math

Sys.

Theory

-LRB-

-RRB-

-LSB-

-RSB-

Sussmann

H.J.

Uniqueness

of

the

weights

for

minimal

feedforward

nets

with

a

given

input-output

map

Neural

Networks

to

appear