PERMIS-2007

PerMIS'07 will be the seventh in the series that started in 2000, targeted at defining measures and methodologies of evaluating performance of intelligent systems. The workshop has proved to be an excellent forum for discussions and partnerships, dissemination of ideas, and future collaborations in an informal setting. Attendees usually include researchers,

graduate students, practitioners from industry, academia, and government agencies.

PerMIS 2007 

PERFORMANCE METRICS

FOR 

INTELLIGENTSYSTEMS WORKSHOP

Courtyard Gaithersburg Washingtonian Center, Gaithersburg, Maryland USA

August 28 - 30, 2007
FOREWORD

Welcome to the 2007 Performance Metrics for Intelligent Systems (PerMIS) Workshop! 

PerMIS’07 is being held at the Courtyard Gaithersburg Washingtonian Center from August 28–30. This is the 

seventh installment of PerMIS that started in 2000 targeted at defining measures and methodologies of evaluating performance of intelligent systems, and it focuses on applications of performance measures to practical 

problems in commercial, industrial, homeland security, and military applications. The cardinal theme of this 

year’s workshop is the interplay between autonomy and intelligence, i.e. how does autonomy influence 

intelligence and vice versa. Topic areas include, but are not limited to:

Defining and measuring aspects of a system:

• The level of autonomy

• Human-robot interaction

• Collaboration

Evaluating components within intelligent system

• Sensing and perception

• Knowledge representation, world models, ontologies

• Planning and control

• Learning and adaption

• Reasoning

Infrastructural support for performance evaluation

• Testbeds and competitions for intercomparisons

• Instrumentation and other measurement tools

• Simulation and modeling support

Technology readiness measures for intelligent systems

Applied performance measures in various domains, e.g.,

• Intelligent transportation systems

• Emergency response robots (search and rescue, bomb disposal)

• Homeland security systems

• De-mining robots

• Defense robotics

• Hazardous environments (e.g., nuclear remediation)

• Industrial and manufacturing systems

• Space/Aerial robotics

• Medical Robotics & Assistive devices

We have assembled an exciting program for this year consisting of four plenary addresses, one featured 

presentation, four special sessions, and two panel discussions. In addition to these, there will be five general 

technical sessions. All of these presentations will address, in one way or another, performance metrics, 

PerMIS 2007
evaluation, and analysis of intelligent systems in diverse domains ranging from space robotics to manufacturing, from mobile robotic systems to human machine interaction, to name a few.

PerMIS’07 is sponsored by NIST with technical co-sponsorship of the IEEE Washington Section Robotics and

Automation Society Chapter and in-cooperation with the Association for Computing Machinery (ACM) Special

Interest Group on Artificial Intelligence (SIGART). We would also like to acknowledge the financial support of 

the IEEE Washington Section. 

There are several firsts to this year’s workshop. The proceedings of PerMIS’07 will be indexed by INSPEC, Ei 

Compendex, ACM’s Digital Library, and are released, as in previous years, as a NIST Special Publication. 

These indexing services will enable your work to reach a wider audience for increased references and citations. Springer Publishers will be hosting a booth on the last two days of the workshop during which time 

some of the displayed books will be raffled off. We thank Springer for their participation and hope that this is 

the beginning of many years of their support.

We would like to thank all members of the PerMIS′07 Program Committee, and the reviewers for contributing 

to the success of the workshop. Most importantly, we thank all authors for their valuable submissions and the 

attendees for their participation. We sincerely hope that you enjoy the presentations and ensuing discussions,

while forging new relationships and renewing old ones. It is our great pleasure to host all the attendees. Enjoy 

the workshop!

Raj Madhavan Elena Messina

Program Chair General Chair

SPONSORS

 

 

PerMIS 2007
PROGRAMCOMMITTEE

General Chair:

Elena Messina (Intelligent Systems Division, NIST, USA) 

Program Chair:

Raj Madhavan (Oak Ridge National Laboratory/NIST, USA)

R. Bonneau (AFRL USA)

S. Balakirsky (NIST USA)

G. Berg-Cross (EM & I USA) 

J. Bornstein (Army Res. Lab. USA)

S. Carpin (UC Merced USA)

J. Evans (USA)

D. Gage (XPM Tech. USA)

J. Gunderson (Gamma Two, Inc. USA)

L. Gunderson (Gamma Two, Inc. USA)

A. Jacoff (NIST USA)

S. Julier (Univ. College London UK)

T. Kalmar-Nagy (Texas A&M USA)

R. Lakaemper (Temple Univ. USA)

L. Latecki (Temple Univ. USA)

M. Lewis (Univ. of Pittsburgh USA)

A. del Pobil (Univ. Jaume-I, Spain)

L. Reeker (NIST USA)

C. Schlenoff (NIST USA)

A. Schultz (Navy Res. Lab. USA) 

M. Shneier (NIST USA)

R. Smith (OSD USA)

R. Tilove (General Motors USA)

E. Tunstel (Jet Propulsion Lab. USA)

PerMIS 2007
ABSTRACT

Fully repeatable and controllable 

experiments are essential to enable 

a precise comparison of multi-robot 

systems. Using different case 

studies, we describe a general 

methodology for conducting 

experimental activities for multirobot systems. This is a first step 

toward the goal of fostering the 

practice of replicating experiments 

in order to compare different 

methods and assess their strengths 

and weaknesses.

In the first case study, we examine 

the problem of building a 

geometrical map of an indoor 

environment using multiple robots. 

The map is built by integrating 

partial maps made of segments 

without using any odometry 

information. We show how to 

improve the repeatability and 

controllability of the experimental 

results and how to compare different mapping systems.

We then present a case study of auction-based methods for the 

allocation of tasks to a group of robots. The robots operate in a 

2D environment for which they each have a map. Tasks are 

locations in the map that must be visited by one robot. Robots 

bid to obtain tasks, but unexpected obstacles and other delays 

may prevent a robot from completing its allocated tasks. We 

show how to compare our experimental results with other 

published auction-based methods.

BIOGRAPHY

Maria Gini is a Professor at the Department of Computer 

Science and Engineering of the University of Minnesota. Before 

joining the University of Minnesota, she was a Research 

Associate at the Politecnico of Milan, Italy, and a Visiting 

Research Associate at Stanford University. Her work has 

included motion planning for robot arms, navigation of mobile 

robots around moving obstacles, unsupervised learning of 

complex behaviors, coordinated behaviors among multiple 

robots, and autonomous economic agents. She has coauthored 

over 200 technical papers. She is currently the chair of ACM 

Special Interest Group on Artificial Intelligence (SIGART), a 

member of the Association for the Advancement of Artificial 

Intelligence (AAAI) Executive Council and of the board of the 

International Foundation of Autonomous Agents and Multi-Agent 

Systems. She is on the editorial board of numerous journals, 

including Autonomous Robots, the Journal of Autonomous 

Agents & Multi-Agent Systems, Electronic Commerce Research 

and Applications, Integrated Computer-Aided Engineering, and 

Web Intelligence and Agent Systems.

ABSTRACT

This talk first describes several 

approaches to measure the 

performance of ground robots. It is 

easy enough to measure quantities 

such as speed and reliability. It is 

more challenging to define metrics 

for perception, planning, and 

autonomy. The talk then presents 

selected results of applying the 

approaches to systems developed 

by several Government programs.

BIOGRAPHY

Dr. Krotkov is the President of Griffin 

Technologies, a consulting and software firm specializing in robotics and 

machine perception. Before founding 

Griffin, he worked in industry as an 

executive in a medical imaging technology start-up, in government as a 

program manager at DARPA, and in 

academia as a faculty member of the 

Robotics Institute at Carnegie Mellon 

University. Dr. Krotkov earned his Ph.D. degree in Computer and 

Information Science in 1987 from the University of Pennsylvania, 

for pioneering work in active computer vision.

 ABSTRACT 

Designing human-robot collaboration 

systems is an inherently multidisciplinary endeavor aimed at providing 

humans with rich, effective and satisfying interactions. Over the past ten 

years, my laboratory has focused on 

educational collaboration, wherein 

the purpose of the interaction is to 

provide measurable learning for humans through exploration and discovery. We propose that the creation 

of a successful human-robot collaboration system requires innovation in 

several areas: robot morphology;

robot behavior; social perception;

interaction design; human cognitive 

models and evaluation of educational 

effectiveness. Our iterative process 

for collaboration design extends 

evaluation techniques from the informal learning field together with underlying technical advances in robotics. This talk describes our research methodology, technical contributions and experimental outcomes for three fielded robot systems that push on developing a generalizable, formal approach 

to educational human-robot collaboration. For the past several 

months, our group has been laying the groundwork for largescale dissemination of our technology and curricular instruments. 

PLENARY SPEAKER

Prof. Illah 

Nourbakhsh

Carnegie Mellon

University, USA

Formalizing 

Educational 

Human-Robot

Collaboration

Wed. 08:30

PerMIS 2007 

PLENARY SPEAKER

Prof. Maria Gini

University of 

Minnesota, USA

Methodology 

for 

Experimental 

Research in 

Multi-robot 

Systems with 

Case Studies

Tue. 08:30

PLENARY SPEAKER

Dr. Eric Krotkov 

Griffin 

Technologies, 

USA

Measuring 

Ground 

Robot 

Performance

Tue. 14:00
I will describe the robot “community” we wish to help spawn, 

and the ingredients that may help to catalyze a broad form of 

technologically empowered community, including the Telepresence Robot Kit and the Global Connection Project. 

BIOGRAPHY

Illah R. Nourbakhsh is an Associate Professor of Robotics and 

head of the Robotics Masters Program in The Robotics Institute 

at Carnegie Mellon University. He was on leave for the 2004 

calendar year and was at NASA/Ames Research Center serving 

as Robotics Group lead. He received his Ph.D. in computer 

science from Stanford University in 1996. He is co-founder of the 

Toy Robots Initiative at The Robotics Institute, director of the 

Center for Innovative Robotics and director of the Community 

Robotics, Education and Technology Empowerment (CREATE) 

lab. He is also co-PI of the Global Connection Project, home of 

the Gigapan project. He is also co-PI of the Robot 250 city-wide 

art+robotics fusion program in Pittsburgh. His current research 

projects include educational and social robotics and community 

robotics. His past research has included protein structure 

prediction under the GENOME project, software reuse, 

interleaving planning and execution and planning and scheduling 

algorithms, as well as mobile robot navigation. At the Jet 

Propulsion Laboratory he was a member of the New Millenium 

Rapid Prototyping Team for the design of autonomous 

spacecraft. He is a founder and chief scientist of Blue Pumpkin 

Software, Inc., which was acquired by Witness Systems, Inc. Illah 

recently co-authored the MIT Press textbook, Introduction to 

Autonomous Mobile Robots.

ABSTRACT

Commercial applications for the 

everyday deployment of autonomous 

systems based on robotic and 

intelligent systems technologies 

require the highest levels of 

performance, reliability and integrity. 

The general public expects intelligent 

machines to be fully operational 

100% of the time. People expect 

autonomous technologies to operate 

at higher levels of performance and 

safety than people themselves 

exhibit. For example smart car 

technologies are expected to cause 

ZERO accidents while human errors 

kill more 150,000 people on our roads 

every year! This talk will describe the 

design principles that have been 

developed over of the last 10 years 

through exhaustive trial and error 

testing to underpin autonomous 

systems that are suitable for realworld deployment. Currently, it is not 

yet possible to realise an autonomous 

system that doesn't fail periodically. Even if the mean rate 

between failures is days or weeks, a single failure could have 

catastrophic consequences. The approach we have adopted to 

address this situation has been to build-in monitoring systems 

that continually check all key system parameters and variables. If 

the monitored parameters move outside tightly defined bounds 

the system will safely shutdown, and alert the human supervisor. 

The failure conditions are logged and then further testing and 

debugging is performed. The value and appropriateness of our 

approach will be shown by a number of real-world studies. We 

will show that how it is possible to design computer vision 

systems for human-machine applications can operate with over 

99% reliability, in all lighting conditions, for all types of users 

irrespective of age, race or visual appearance. These systems 

have been used in automotive and sports applications. We have 

also show how this approach has been used to design field 

robotic systems that have deployed in automobile safety 

systems and 24/7 mining applications.

BIOGRAPHY

Dr. Alex Zelinsky is a well-known scientist, specialising in 

robotics and computer vision and is widely recognised as an 

innovator in human-machine interaction. Dr. Zelinsky is currently 

Group Executive, Information and Communication Sciences and 

Technology, and Director, CSIRO Information Communication 

Technology (ICT) Centre. Before joining CSIRO in July 2004, Dr. 

Zelinsky was CEO of Seeing Machines, a company dedicated to 

the commercialisation of computer vision systems. Dr. Zelinsky 

co-founded Seeing Machines in June 2000, the company is now 

publicly listed on the London Stock Exchange. The technology 

commercialised by Seeing Machines was developed at the 

Australian National University where Dr. Zelinsky was Professor 

and Head of the Department of Systems Engineering (1996

-2000). Prior to joining the Australian National University, Dr. 

Zelinsky worked as an academic at the University Wollongong 

(1984-1991) and as a research scientist in the Electrotechnical 

Laboratory, Japan (1992-1995). Dr. Zelinsky is an active member 

of the robotics community and has served on the editorial boards 

of the International Journal of Robotics Research and IEEE 

Robotics and Automation Magazine, he also founded the Field & 

Services Robotics conference series. Dr. Zelinsky's contributions 

have been recognised by awards in Australia and internationally. 

These include the Australian Engineering Excellence Awards, US 

R&D magazine Top 100 Award and Technology Pioneer at the 

World Economic Forum.

PerMIS 2007

PLENARY SPEAKER

Dr. Alex Zelinsky

CSIRO ICT

Centre, 

Australia

Building 

Autonomous 

Systems of 

High 

Performance,

Reliability 

and Integrity

Thu. 08:30 
 ABSTRACT

After spectacular successes, in 

1970s-1980s, in the use of robotics in

highly structured environments - e.g. 

automotive assembly, welding, and 

painting lines - the penetration of 

"serious" robots (those large and 

powerful enough to be harmful) into 

new applications has slowed down 

markedly. User manuals of most robot arm manipulators warn that under no circumstance can people enter the workspace of an operating 

robot. The reason is simple - due to 

intended use these robots are strong 

enough to endanger a human, yet 

their sensing and intelligence is "too 

dumb" to be trusted for human 

safety. In the roboticists' parlance, 

today's robots are not designed to 

operate in unstructured environments, that is settings not created 

specifically for the robot's operation. 

It is not the function the robot is built 

for that is the problem - it is the robot's interaction with its environment.

The problem is lesser with robot rovers 

but quite pronounced with arm manipulators.

The way to break this barrier is to design robots fully capable of 

operating in an unstructured environment, in places where things

are unpredictable and must be perceived and decided upon on 

the fly. This is a new terrain - the required hardware and intelligence are to be more complex and sophisticated than what we 

know today. In this talk we will review related technical and scientific issues.

BIOGRAPHY 

Dr. Vladimir Lumelsky is the head of the Laboratory of Robotics 

for Unstructured Environments at NASA-Goddard Space Center, 

and is Adjunct Professor of Computer Science at the University 

of Maryland-College Park. The long-term goal of the laboratory is

to develop robots capable of operating in the uncertain and 

changing settings likely to arise in future NASA missions. This 

work builds upon Dr. Lumelsky's work on large sensitive robot 

skin systems prior to joining NASA in 2004, as a professor at Yale

University and later at the University of Wisconsin-Madison 

(where he was The Consolidated Papers Professor of Engineering). Dr. Lumelsky is the author of three books and over 200 professional papers covering the areas of robotics, computational 

intelligence, human-machine interaction, human spatial reasoning, massive sensor arrays, bio-engineering, control theory, 

kinematics, pattern recognition, and industrial automation. He 

has held a variety of positions in both the public and private sectors: he was Program Director at the National Science Foundation, and has led large technical projects, including development 

of a universal industrial robot controller at General Electric (GE 

FEATURED 

PRESENTATION

Dr. Vladimir 

Lumelsky

NASA-Goddard 

Space Center, 

USA

Human-Robot

Interaction in 

Physical 

Proximity: 

Issues and 

Prospects

Wed. 14:00

Research Center), and a joint robot skin development effort with 

Hitachi Corporation. Dr. Lumelsky also has held temporary positions at the Science University of Tokyo (Japan), Weizmann Institute (Israel) and US South Pole Station, Antarctica. He is the 

founding Editor-in-Chief of the IEEE Sensors Journal, and has 

served on editorial boards of other professional journals. He has 

been guest editor of special issues at professional journals; 

served on the Administrative Committees of IEEE Robotics Society and Sensors Council; chaired technical committees and 

working groups; and chaired and co-chaired major international 

conferences, workshops and special sessions. Dr. Lumelsky has 

served as a technical expert in legal cases, including multinational litigation. He frequently gives talks at US and foreign 

universities, government groups, think tanks, and in industry. He 

is a member of several professional societies, and is a Fellow of 

IEEE. 

PerMIS 2007
PerMIS 2007 

T

U

E

S

D

A

Y

2

8

August

08:15 Welcome & Overview

08:30 Plenary Presentation:

Maria Gini 

Methodology for Experimental Research in Multi-robot 

Systems with Case Studies

09:30 Coffee Break

10:00 TUE-AM1 Mobile Robot Performance Evaluation I

Chairs: C. Schlenoff & M. Childers

• Evaluation of Navigation of an Autonomous Mobile Robot

[N. Muñoz, J. Valencia, N. Londoño]

• Assessing the Impact of Bi-directional Information Flow in UGV 

Operation: A Pilot Study

[M. Childers, B. Bodt, S. Hill, R. Dean, W. Dodson, L. Sutton]

• Standard Operator Control Unit Color Scheme for Mobile Robots

[M. Shneier, R. Bostelman, J. Albus, W. Shackleford, T. Chang, 

T. Hong]

• How DoD's TRA Process Could be Applied to Intelligent Systems 

Development [D. Sparrow, S. Cazares]

• A Brief History of PRIDE [C. Schlenoff, Z. Kootbally, R. Madhavan]

12:30 Lunch on your own

14:00 Plenary Presentation:

Eric Krotkov

Measuring Ground Robot Performance

15:00 Coffee Break

15:30 TUE-PM1 Mobile Robot Performance Evaluation II

Chairs: S. Balakirsky & C. Lundberg

• Assessment of Man-portable Robots for Law Enforcement 

Agencies [C. Lundberg, H. Christnesen]

• Performance Metrics and Evaluation of a Path Planner based on 

Genetic Algorithms [G. Giardini, T. Kalmar-Nagy]

• The Evolution of Performance Metrics in the RoboCup Rescue 

Virtual Robot Competition [S. Balakirsky, C. Scrapper, S. Carpin]

• Robot Simulation Physics Validation 

[C. Pepper, S. Balakirsky, C. Scrapper]

• Design and Validation of a Whegs Robot in USARSim

[B. Taylor, S. Balakirsky, E. Messina, R. Quinn]

18:30 Reception

PROGRAM PERMIS
PerMIS 2007 

August

08:15 Welcome & Overview

08:30 Plenary Presentation: 

Maria Gini 

Methodology for Experimental Research in Multi-robot 

Systems with Case Studies

09:30 Coffee Break

10:00 TUE-AM2 Special Session I: Autonomy Levels for Unmanned Systems

Organizer: Hui-Min Huang (NIST)

• Autonomy Levels for Unmanned Systems (ALFUS) Framework: 

Safety and Application Issues [H-M. Huang]

• Evaluation of Autonomy in Recent Ground Vehicles Using the 

Autonomy Levels for Unmanned Systems (ALFUS) Framework

[G. McWilliams, M. Brown, R. Lamm, C. Guerra, P. Avery,

K. Kozak, B. Surampudi]

• A Methodology for Testing Unmanned Vehicle Behavior and 

Autonomy 

[D. Gertman, C. McFarland, T. Klein, A. Gertman, D. Bruemmer]

• Standardizing Measurements of Autonomy in the Artificially 

Intelligent [A. Hudson, L. Reeker]

• Autonomous Level Density And Distribution [D. Brayman]

12:30 Lunch on your own

14:00 Plenary Presentation:

Eric Krotkov

Measuring Ground Robot Performance

15:00 Coffee Break

15:30 TUE-PM2 Special Session II: Human Robot Interface Issues

Organizers: Salvatore Schipani & Brian Antonishek (NIST)

• Maze Hypothesis Development in Assessing Robot Performance 

During Teleoperation [S. Schipani, E. Messina]

• Human System Performance Metrics for Evaluation of

Mixed-Initiative Heterogeneous Autonomous Systems

[L. Billman, M. Steinberg]

• Concepts of Operations for Robot-Assisted Emergency Response

and Implications for Human-Robot Interaction

[J. Scholtz, B. Antonishek, B. Stanton, C. Schlenoff]

• Multimodal Displays to Enhance Human Robot Interaction

On-the-Move* [E. Haas, C. Stachowiak] 

18:30 Reception

PROGRAM PERMIS

T

U

E

S

D

A

Y

2

8

*A multi-modal information system will be demonstrated.
PerMIS 2007 

W

E

D

N

E

S

D

A

Y

2

9

August

08:15 Overview

08:30 Plenary Presentation:

Illah Nourbakhsh

Formalizing Educational Human-Robot Collaboration

09:30 Coffee Break

10:00 WED-AM1 Autonomy Vs Intelligence

Chairs: J. Gunderson & J. Evans

• Autonomy (What's it Good for?) [J. Gunderson, L. Gunderson]

• Definitions and Measures of Intelligence in Deep Blue and the 

Army XUV [J. Evans]

• Automotive Turing Test [S. Kalik, D. Prokhorov]

• Autonomous Robots with Both Body and Behavior 

Self-Knowledge [B. Gordon]

• A Cognitive-based Agent Architecture for Autonomous Situation 

Analysis [G. Berg-Cross, W-T. Fu, A. Kwon]

12:30 Lunch on your own

14:00 Featured Presentation:

Vladimir Lumelsky

Human-Robot Interaction in Physical Proximity: 

Issues and Prospects

15:00 Coffee Break

15:30 WED-PM1 Human Machine Interaction

Chairs: N. Dagalakis & A. Steinfeld

• Evaluation of an Integrated Multi-Task Machine Learning System 

with Humans in the Loop 

[A. Steinfeld, S. Bennett, K. Cunningham, M. Lahut, 

P-A. Quinones, D. Wexler, D. Siewiorek, J. Hayes, P. Cohen, 

J. Fitzgerald, O. Hansson, M. Pool, M. Drummond]

• Survey Measures for Evaluation of Cognitive Assistants 

[A. Steinfeld, P-A. Quinones, J. Zimmerman, S. Bennett, 

D. Siewiorek]

• Development of Tools for Measuring the Performance of Computer

Assisted Orthopaedic Hip Surgery Systems

[N. Dagalakis, Y. Kim, D. Sawyer, C. Shakarji]

• Haptic Feedback System for Robot-Assisted Surgery 

[J. Desai, G. Tholey, C. Kennedy]

19:30 Banquet (Adam Jacoff, NIST)

PROGRAM PERMIS
PerMIS 2007 

August

08:15 Overview

08:30 Plenary Presentation:

Illah Nourbakhsh

Formalizing Educational Human-Robot Collaboration

09:30 Coffee Break

10:00 WED-AM2 Panel Discussion I: Can the Development of Intelligent 

Robots be Benchmarked? Concepts and Issues from Epigenetic 

Robotics (Moderator: Gary Berg-Cross, EM & I)

• Douglas Blank, Bryn Mawr College

• James Marshall, Sarah Lawrence College

• Lisa Meeden, Swarthmore College

• Charles Kemp, Georgia Tech.

• Chad Jenkins, Brown University

12:30 Lunch on your own

14:00 Featured Presentation:

Vladimir Lumelsky

Human-Robot Interaction in Physical Proximity: 

Issues and Prospects

15:00 Coffee Break

15:30 WED-PM2 Special Session III: Space/Aerial Robotics

Organizer: Edward Tunstel (JPL)

• Prototype Rover Field Testing and Planetary Surface Operations 

[E. Tunstel]

• Planning to Fail - Reliability as a Design Parameter for Planetary 

Rover Missions [S. Stancliff, J. Dolan, A. Trebi-Ollennu]

• A Decision Space Compression Approach for Model Based 

Parallel Computing Processes [R. Bonneau, G. Ramseyer]

• TBD [E. Atkins]

19:30 Banquet (Adam Jacoff, NIST)

PROGRAM PERMIS

W

E

D

N

E

S

D

A

Y

2

9
PerMIS 2007 

T

H

U

R

S

D

A

Y

3

0

August

08:15 Overview

08:30 Plenary Presentation:

Alex Zelinsky

Building Autonomous Systems of High Performance, 

Reliability and Integrity

09:30 Coffee Break

10:00 THU-AM1 Performance Assessment of Algorithms

Chairs: R. Lakaemper & S. Spetka

• Analyzing the Performance of Distributed Algorithms

[R. Lass, E. Sultanik, W. Regli]

• An Agent Structure for Evaluating MAS Performance 

[C. Dimou, A. Symeonidis, P. Mitkas]

• Information Management for High Performance Autonomous 

Intelligent Systems [S. Spetka, S. Tucker, G. Ramseyer, 

R. Linderman]

• Efficient Monte Carlo Computation of Fisher Information Matrix 

using Prior Information [S. Das, J. Spall, R. Ghanem] 

• Performance of 6D LuM and FFS SLAM -- An Example for 

Comparison using Grid and Pose Based Evaluation Methods 

[R. Lakaemper, A. Nuchter, N. Adluru, L. Latecki]

12:30 Lunch on your own

14:00 THU-PM Panel Discussion II: (Re-)Establishing or Increasing Collaborative Links Between Artificial Intelligence and Intelligent Systems

(Moderator: Brent Gordon, NASA-Goddard)

• James Albus, Senior Fellow, Intelligent Systems Division, NIST

• Ella Atkins, Associate Professor, University of Michigan

• Henrik Christensen, Director, Center for Robotics and Intelligent 

Machines, Georgia Tech.

• Larry Reeker, Computer Scientist, Information Technology 

Laboratory, NIST

15:30 Coffee Break

16:00 Adjourn

PROGRAM PERMIS
PerMIS 2007 

August

08:15 Overview

08:30 Plenary Presentation:

Alex Zelinsky 

Building Autonomous Systems of High Performance,

Reliability and Integrity

09:30 Coffee Break

10:00 THU-AM2 Special Session IV: Smart Assembly Systems

Organizers: Robert Tilove (GM) & John Slotwinski (NIST)

• Smart Assembly: Industry Needs and Challenges

[R. Tilove, J. Slotwinski]

• Hybrid Emulation for Manufacturing Automation Controls: Needs 

and Challenges [S. Biller, F. Gu, C. Yuan]

• Science based Information Metrology for Engineering Informatics 

[S. Rachuri]

• Evaluating Manufacturing Control Language Standards: An 

Implementer's View [T. Kramer]

• Interoperability Testing for Shop-Floor Inspection

[J. Falco, J. Horst, T. Kramer, J. Michaloski, F. Proctor, W. Rippey]

• A Virtual Environment-Based Training Systems for Mechanical 

Assembly Operations

[S. Gupta, D. Anand, J. Brough, R. Kavetsky, M. Schwartz]

12:30 Lunch on your own

14:00 THU-PM Panel Discussion II: (Re-)Establishing or Increasing Collaborative Links Between Artificial Intelligence and Intelligent Systems

(Moderator: Brent Gordon, NASA-Goddard)

• James Albus, Senior Fellow, Intelligent Systems Division, NIST

• Ella Atkins, Associate Professor, University of Michigan

• Henrik Christensen, Director, Center for Robotics and Intelligent 

Machines, Georgia Tech.

• Larry Reeker, Computer Scientist, Information Technology 

Laboratory, NIST

15:30 Coffee Break

16:00 Adjourn

PROGRAM PERMIS

T

H

U

R

S

D

A

Y

3

0
digests
TUE-AM1

Mobile Robot Performance Evaluation I

Chairs: C. Schlenoff & M. Childers
TUE-AM2

Special Session I: Autonomy Levels for Unmanned Systems

Organizer: H-M. Huang
TUE-PM1

Mobile Robot Performance Evaluation Il

Chairs: S. Balakirsky & C. Lundberg
TUE-PM2

Special Session Il: Human Robot Interface Issues

Organizers: S. Schipani & B. Antonishek
WED-AM1

Autonomy Vs Intelligence

Chairs: J. Gunderson & J. Evans
WED-AM2

Panel Discussion I: Can the Development of Intelligent Robots be Benchmarked?

Conepts and Issues from Epigenetic Robotics

Moderator: G. Berg-Cross, EM&I
WED-PM1

Human Machine Interaction

Chairs: N. Dagalakis & A. Steinfeld
WED-PM2

Special Session III: Space/Aerial Robotics

Organizer: E. Tunstel
THU-AM1

Performance Assessment of Algorithms

Chairs: R. Lakaemper & S. Spetka
THU-AM2

Special Session IV: Smart Assembly Systems

Organizers: R. Tilove & J. Slotwinski
THU-PM

Panel Discussion II: (Re-)Establishing or Increasing Collaborative Links Between

Artificial Intelligence and Intelligent Systems

Moderator: B. Gordon, NASA-Goddard
Restaurants

Red Rock Canyon Grill – American

100 Boardwalk Place

(240) 631-0003

California Pizza Kitchen – American

136 Boardwalk Place

(301) 330-0681

Hamburger Hamlet -- American

9811 Washington Blvd., Rio Center

(301) 417-0773

Joe's Crab Shack - American

221 Rio Blvd

(301) 947-4377

Potbelly’s – American

128 Boardwalk Place

(301) 963-4055

Corner Bakery – American

1 Grand Corner Ave

(301) 721-1634

Romano's Macaroni Grill - Italian

211 Rio Blvd

(301) 963-5003

Taco Bar - Mexican

10003 Fields Rd

(301) 987-0376

Guapo's Cantina-- Mexican

9811 Washington Blvd., Rio Center

(301) 977-5655

Rio Grande – Mexican

231 Rio Blvd

(240) 632-2150

Tara Thai – Asian

9811 Washingtonian Blvd, Rio Center

(301) 947-8330

Asian Bistro - Asian

(301) 670-2870

Café Spice - Indian

9811 Washingtonian Blvd, Rio Center

(301) 330-6611

Village Green Grill House of Kabob -

Mediterranean

9811 Washingtonian Blvd, Rio Center

(301) 519-4220

Maggie Moo’s Ice Cream – Specialty

215 Boardwalk Place

(301) 977-6667

Swensen’s Ice Cream – Specialty

Grand Corner Ave

(301) 417-6710

Starbucks Coffee – Specialty

213 Boardwalk Place

(301) 216-9635
ACKNOWLEDGMENTS

These people provided essential support to make this event happen. Their 

ideas and efforts are very much appreciated.

Website and Proceedings

Debbie Russell (Chair)

Local Arrangements

Jeanenne Salvermoser (Chair)

Jennifer Peyton

Conference and Registration

Kathy Kilmer (Chair)

Teresa Vicente

Mary Lou Norris

Angela Ellis

Finance

Betty Mandel (Chair)

PerMIS 2007 

Intelligent Systems Division 

Manufacturing Engineering Laboratory 

National Institute of Standards and Technology

100 Bureau Drive, MS-8230

Gaithersburg, MD 20899 

http://www.isd.mel.nist.gov/

Thank you 

PerMIS 

attendees!