Measuring Performance and Intelligence of Intelligent Systems

Workshop Schedule

General Chair – Elena Messina

Program Chair – Alex Meystel

August 14 - 16

The Workshop opens in Lecture Room A, Bldg. 101

Afternoon Plenary Lecture will be conducted at Green Auditorium

N I S T

Gaithersburg, MD

2000
2

The Schedule of a Session

• Each Session is allotted 2 hours.

• It is expected that a speaker will use a slot of 25 minutes for his/her

presentation (20 minutes) and answering questions (5 minutes).

• The remainder of time (20 minutes) should be used for a general

discussion and combining the Final Recommendations of the Session.

• The Final Recommendations of all sessions will be integrated into

Final Recommendations of the Workshop.

• The results of Each Day are discussed at the evening Plenary Discussion
3

1

st Day, Monday, August the 14th

Lecture Room A

8.30 AM Introduction by Workshop Organizers : A. Meystel, J. Evans, E. Messina

8:45 AM J. Evans, E. Messina, The Present and The Future of Measuring Performance of

Intelligent Systems

PLENARY LECTURE – 9 AM – 10 AM - Lecture Room A

H. Szu, Machine IQ with Stable Cybernetic Learning With and Without a Teacher

Coffee Break: 10 AM-10.30 AM

Sessions: 10.30 AM – 12.30 PM

______________________________________________________________________________________

Lecture Room A

I Day, morning A: Features of Industrial Intelligent Systems,

Co-Chairs: M. Cotsaftis, W. H. VerDuin

• M. W. Bailey, W. H. VerDuin, FIPER: An Intelligent System for the Optimal Design of Highly

Engineered Products

• S. A. Wallace, J. E. Laird, K. J. Coulter, Examining the Resource Requirements of Artificial

Intelligence Architectures

• C. Peterson, A Metric for Monitoring and Retaining Flight Software performance

• M. Cotsaftis, On Definition of Task Oriented System Intelligence

Lecture Room B

I Day, Morning B: Features of Living Intelligent Systems

Co-Chairs: K. Bellman, C. Joslyn

• K. Bellman, Understanding and Its Behavioral Correlates”

• C. Joslyn, Toward Measures of Intelligence Based On Semiotic Control

• H. Sarjoughian, B. Zeigler, Model-based Design and Measurement of Intelligence

• T. Chmielewski, P. Kalata, Biometric Techniques: The Fundamentals of Evaluation

Lecture Room C

I Day, Morning C: Special Issues of Evaluating Intelligence

Co-Chairs: R. Sanz, A. Wild

• R. Sanz, I. Lopez, Minds, MIPs, and Structural Feedback

• A. Wild, Using the Metaphor of Intelligence

• R. Garner, R. N. Bishop, Applied Applications for Mimetic Synthesis: The AAMS Project Summary

• H. M. Hubey, General Scientific Premises of Measuring Complex Phenomena
4

Lunch 12.30 PM – 2 PM – Back of Cafeteria

PLENARY LECTURE – 2 PM-3 PM - Green Auditorium

G. Saridis, Definition and Measurement of Machine Intelligence

Coffee Break: 3 PM-3.15 PM

Sessions: 3.15 PM – 5.15 PM

Lecture Room A

I Day, Afternoon A: Metrics and Comparison of Alternatives: General Issues

Co-Chairs: L. Pouchard, W. C. Stirling

• L. Pouchard, Metrics for Intelligence: the Perspective from Software Agents

• J. Spall, et al, Towards an Objective Comparison of Stochastic Optimization Approaches

• W. C. Stirling, R. L. Frost, Intelligence with Attitude

• S. Lee, W.-C. Bang, and Z. Z. Bien, Measure of System Intelligence: An Engineering Perspective

____________________________________________________________________________________

Lecture Room B

I Day, Afternoon B: Metrics and Comparison of Alternatives: Case Studies

Co-Chairs: R. Finkelstein, E. Grant

• E. Grant, G. Lee, Properties of Learning Knowledge Based Controllers

• V. Grishin, A. Meystel, Using Visualisation for Measuring Intelligence of Constructed Systems

• R. Finkelstein, A Method for Evaluating the IQ of Intelligent Systems

• L. Polyakov, In Defense of the Additive Form for Evaluating Vectors

PLENARY DISCUSSION - 5.15 PM – 6.15 PM Green Auditorium

Panel: K. Bellman, M. Cotsaftis, R. Finkelstein, E. Grant, C. Joslyn, C. Peterson,

L. Pouchard, W. C. Stirling, A. Wild

8 PM – Meeting of the Advisory Board (at the Holiday Inn)
5

2

nd Day, Tuesday, August the 15th

PLENARY LECTURE – 9 AM-10 AM - Green Auditorium

J. Albus, Features of Intelligence Required in Unmanned Autonomous Vehicles

Coffee Break: 10 AM-10.30 AM

Sessions: 10.30 AM – 12.30 PM

______________________________________________________________________________________

Lecture Room A

II Day, Morning A: Measuring performance

Co-Chairs: A. Sanderson, T. Samad

• T. Samad, Technologies for Engineering Autonomy and Intelligence

• A. Sanderson, Minimal Representation Size Metrics for Intelligent Robotic Systems

• J. Zhang, A Formal Method to the Performance Metrics for Engineering Systems

• R. Yager, A Hierarchical Framework for Constructing Intelligent Systems Metrics

Lecture Room B

II Day, Morning B: Modeling and Measuring Machine Intelligence

Co-Chairs: P. Davis, T. Whalen

• P. Davis, Exploratory Analysis Enabled by Multiresolution, Multiperspective Modeling

• M. Jabri, Measuring intelligence: a neuromorphic perspective

• I. Nourbakhsh , Two measures for measuring the `intelligence' of human-interactive

robots in contests and in the real world: perceptiveness and expressiveness

• T. Whalen, What is the Value of Intelligence and How Can It Be Measured?

Lecture Room C

II Day, Morning C: Evaluating Factors of Intelligence in Systems

Co-Chairs: J. Hernandes-Orallo, C. Peterson

• J. Hernandes-Orallo, On the Computational Measurement of Intelligence Factors

• A. Wild, Heterogeneous Computing

• J. Bryson, et al, Hypothesis Testing for Complex Agents

• T. Balch, Hierarchic Social Entropy: An Information Theoretic Measure of Robot Group Diversity
6

Lunch 12.30 PM – 2 PM – Back of Cafeteria

PLENARY LECTURE – 2 PM – 3 PM - Green Auditorium

S. Grossberg, Some Constraints on Intelligent Systems:

 Autonomous Computation in a Changing World

Coffee Break – 3 PM – 3.15 PM

Sessions: 3.15 PM – 5.15 PM

___________________________________________________________________________________

Lecture Room A

II Day, Afternoon A: Measuring of Intelligence of Multiagent Networks

Chair and Organizer: S. Phoha

• R. R. Brooks, STIGMERGY: A measure of intelligence for emergent distributed behaviors

• S. Phoha, D. Friedlander, Goodness of Fit Measures for Intelligent Behaviors of Interacting Machines

• M. E. Cleary, M. Abramson, M. B. Adams, S. Kolitz. Metrics for Embedded Collaborative Intelligent

Systems

• D. Friedlander, S. Phoha, A. Ray, Domain Independent Measures of Intelligent Control

• S. Perraju Tolety, G. Uma, On Measuring Intelligence in Multi-Agent Systems

__________________________________________________________________________________

Lecture Room B

II Day, Afternoon B: Evaluating Intelligent Systems by Testing and Competition:

Benchmarks

Co-Chairs and Organizers: A. Schultz, R. Murphy

• A. Schultz, Evolution of Metrics for Mobile Robots

• A. Jacoff, E. Messina, J. Evans, A Standard Test Course for Urban Search and Rescue Robots

• R. Murphy, J. Casper, M. Micire, J. Hyams, "Assessment of the NIST Standard Test Bed for Urban

Search and Rescue Competitions"

• T. Balch, Performance/N is the Wrong Metric for Multirobot Teams

• S. K. Agrawal, A. M. Ferreira, S. Pledgie, Performance Evaluation of Robotic Systems: A Proposal for

a Benchmark problem

Lecture Room C

II Day, Afternoon C: Measuring Intelligence of Distributed Systems

Co-Chairs: R. Fakory, W. J. Davis

• W. J. Davis, Evaluating Performance of Distributed Intelligent Control System

• R. Fakory, M. Jahangiri, Real Time Distributed Expert System for Automated Monitoring of Key

Monitors in Hubble Space Telescope

• X. Qin, A. E. Aktan, Distributed Internet-Based Multi-Agent Intelligent Infrastructure System

• D. P. Gravel, W. S. Newman, Flexible Robotic Assembly
7

Plenary Discussion- 5.15 PM – 6.15 PM - Green Auditorium

Panel: T. Balch, P. Davis, W. J. Davis, R. Fakory, J. Hernandes-Orallo, R. Murphy, S. Phoha,

T. Samad, A. Sanderson, A. Schultz, T. Whalen

Evening: COCKTAILS AND B A N Q U E T

– 6.45 PM at the Holiday Inn

L. Zadeh,

Banquet speech "The Search for Metrics of Intelligence -- A Critical View."

3

rd Day, Wednesday, August the 16th

PLENARY LECTURE – 9 AM – 10 AM - Green Auditorium

W. Freeman, The neurodynamics of intentionality in animal brains provides a basis

 for constructing devices that are capable of intelligent behavior

Coffee Break: 10 AM-10.30 AM

Sessions: 10.30 AM – 12.30 PM

______________________________________________________________________________________

Lecture Room A

III Day, Morning A: Measuring Intelligence Taking in Account Linguistic, Psychological and

Biological Factors

Co-Chairs: L. Reeker, A. Meystel

• L. Reeker, Theoretical Constructs for Measurement Performance and Intelligence

• A. Meystel, Generalizing Natural Language Representations for Measuring the Intelligence of Systems

• P. Wang, Machine Intelligence Ranking

• A. Treister-Goren, J. Dunietz, The AI Language Development Metric

Lecture Room B

III Day, Morning B: Measuring Intelligence of Systems with Autonomy and Mobility

Co-Chairs: G. S. Sukhatme, J. Weng

• G. S. Sukhatme, Measuring Mobile Robots Performance: Approaches and Pitfalls

• L. E. Parker, Evaluating Success in Autonomous Multi-robot Teams: Experience of ALLIANCE

 Architectures Implementation
8

• A. Lacaze, S. Balakirsky, Search Graph Formation for Minimizing the Complexity of Planning

• J. Weng, Automatic Mental Development and Performance Metrics for Intelligent Systems

Lunch 12.30 PM – 2 PM - Back of Cafeteria

PLENARY LECTURE – 2 PM – 3 PM - Green Auditorium

A. Meystel , Evolution of Intelligent Systems Architectures:

 What Should Be Measured

_____________________________________________________________________________________

Coffee Break – 3 PM – 3.15 PM

Afternoon Session – 3.15 PM – 5.15 PM

PLENARY PANEL - 3:15 PM - 5:15 PM - Green Auditorium

Perspectives of Governmental Programs on Measuring Intelligence

Panel organizers – J. Albus, J. Blitch, J. Evans

• J. Albus, NIST

• J. Blitch, DARPA

• B. Bialczak, SESI, Fort Knox

• J. Evans, NIST

• S. Sastry, DARPA

• C. Shoemaker ARL,

• M. Swinson, DARPA

• C. Weisbin, NASA

PLENARY PANEL – 5: 15 PM – 6:15 PM - Green Auditorium

General Discussion of the Workshop Results

Panel: J. Albus, J. Evans, E. Messina, A. Meystel, L. Reeker, G. S. Sukhatme, J. Weng

The Meeting Closes: 6.15 PM