Problems of Performance Measurement in Locally-Organized Systems

Vadim L. Stefanuk

Institute for Information Transmission Problems

Russian Academy of Science

Bolshoy Karetny per. 19,

Moscow, 101447, Russia

 

This research was partially supported by grant from the program “Mathematical modeling, Intellectual computer systems and

non-linear mechanical systems control” of Russian Academy of Sciences, project # 2.24, and by grant from Russian Fund for

Basic Research, project #02-01-00955.

ABSTRACT

It is shown that many of the modern intelligent systems belong to a

wide class of distributed systems. The external behavior of such a

system is governed with criteria which induce only partial ordering

among various systems. This partial order does not allow building an

analytical metrics in the space of such systems, the fact making

systems largely indistinguishable. This explains the existence of

many versions of AI systems made for the same purpose, like Expert

Systems shells, which are frequently differentiated only through their

secondary properties. The situation may be compared with that of the

use of Pareto sets in theory of games where all different solutions

belonging to a Pareto set are considered to be intrinsically similar.

KEYWORDS: distributed systems, performance metrics

1. INTRODUCTION

Practical considerations do require introduction of some

metrics to measure system performance. If a scalar value is

used to estimate the performance it is quite natural to define a

metrics based on this scalar. In some cases there are some

serious grounds for this. The information throughput [1], an

average income obtained in the system for Stock Exchange

trend forecast may be considered as examples of the scalar

valued system.

However when the performance is measured by a vector

value it obviously brings some additional problems. In the

simplest case a weighted vector is used reducing the problem

to the scalar type formulation. However in many cases it is

impossible to find weights uniformly suitable for the whole

performance space like the distance measure in an Euclidean

space. It is for this reason some other “optimality”

considerations are used like Pareto sets in Game Theory [2]

and in many other applications (in particular, in Mathematical

Economics.)

2. DISTRIBUTED SYSTEMS

Another wide area where the search for a performance metrics

is doomed to failure, and which will be discussed in this

presentation, is the area of distributed systems, referred here to

as the locally-organized systems (LOS) [3, 4]. We use the

latter term in order to emphasize the fact that in such systems

a unique common measure defining the whole system

performance does not exist. Instead, one has a collection of

local criteria for the system components [5], which are used

concurrently and individually and hence can not be reduced to

a unique scalar or vector value.

The exact meaning of notions, used in our description of

locally organized systems depends on the subject domain.

However it is possible to demonstrate some classes of models

for which the situation is known in advance:

• The subsystems are deterministic or probabilistic finite

automata with the deterministic or probabilistic

interaction (collective behavior, collective behavior of

automata, automata games )

• Subsystems are automata with the continuous sets of

actions with the interaction of deterministic type (multiple

access communication systems, sociological and

economics models).

• Subsystems are finite automata with a fuzzy interaction

(Expert Systems).

• Subsystems are the enterprises or individuals who are

involved in a complex monetary and commodity

circulation.

• Inhomogeneous technical systems, where people are also

involved (man-machine systems).

• Subsystems are interacting programming modules

(interactive high level languages, Artificial Intelligence

systems).

Anyway, probably due to some practical considerations,

many distributed systems of this kind are treated as centralized

ones allowing some metrics for the whole system
performance. Thus, in the majority of multi-agent systems [5]

there exists a central point collecting all the information from

agents and assigning tasks to the individual agents. It is

important to note that such a centralization is out of question

for LOS.

The proposed presentation will be organized in the

following way. First, we will introduce a formal definition for

LOS. Then we will list the reasons why LOS becomes a

necessity in many applications, later describing the methods

typically used for analyses and study of LOS.

We still are able to speak about “goals” for LOS.

However the goal of the locally organized system (LOS) is the

provision of a normal functioning of its subsystems, instead of

reaching a certain system goal. That is why the criteria for the

choice of an adequate system version may include such

considerations as expediency, survivability, openness,

elasticity and etc., which have been studied in many

applications and in many subject domains [6] and which will

be briefly reviewed in our presentation. Any of the above

criteria may take only one of two values Yes/No or 1/0.

Obviously, the induced partial order gives only limited

possibility to compare systems.

The mentioned criteria are subject to changes from one

group of problems to another still having the property that

they do not allow to compare systems by introducing any

reasonable metrics to supplement the above step-wise partial

order. A number of examples will be listed - from collective

behavior to manipulator control and interacting programming

modules of an Artificial Intelligence system. This list and

other considerations will demonstrate that the fraction of

systems belonging to the class of LOS will probably only

grow with time.

Finally some mathematical models of an intelligent

warehouse [7] are used to illustrate the whole approach in a

step by step manner.

3. CONCLUSION

As many of the modern intelligent systems are organized in a

local way, being collectives of interacting components each

having its own goal and behavior, our results show that it is

not simple to find a suitable metrics for measuring system

performance to be used for making a comparison among

intelligent systems with respect to each other. Probably this

explains the existence of many versions of AI systems made

for the same purpose, like Expert Systems shells, which are

frequently differentiated only through their secondary

properties.

Our analyses is formal and strict. However it is

supplemented with a practical example of an intelligent

warehouse which admit different type of organization of

storage place.

4. REFERENCES

[1] Fano R.M., Transmission of information. A statistical

theory of communication., MIT Press, N.Y., 1961.

[2] Luce R.D. & Raiffa H. Games and Decisions. Introduction

and critical survey, Harvard University, N.Y., 1957.

[3] Stefanuk V.L. Collective Behavior of Automata and the

Problems of Stable Local Control of a Large Scale System,

Second International Joint Conference on Artificial

Intelligence (IJCAI), London, pp. 51-56, 1971.

[4] Stefanuk V.L. “General Approach to Locally-Organized

Systems”, Proceeding of Advanced Summer Institute 1999

(ASI’99) of the Network of Excellence in Intelligent Control

and Integrated Manufacturing Systems, Leuven, Belgium. pp.

39-45, 1999.

[5] Proceedings of IJCAI-01, “Multi-agent systems”, pp.

1017-1224.

[6] Stefanuk V.L., Stability of Local Control in a System with

Contralinear Interaction of Subsystems, Proceedings of

European Control Conference (ECC'93), vol. 1, Groningen,

pp. 117- 119, 1993

[7] Stefanuk V.L., Algorithms to Optimize Retrieval in a

Distributed Memory System, Proceedings of RAN: Theory

and Control Systems, #4, 1999. - pp. 93-99.