PerMIS 2010 

PERFORMANCE METRICS

for

INTELLIGENT SYSTEMS (PerMIS) 

WORKSHOP

Hyatt Regency, Baltimore, Maryland USA

September 28 - 30, 2010
FOREWORD

“It is both in a spirit of scientific enquiry and for pragmatic motivations that we embark on the quest for metrics for intelligence of constructed systems.”

From the White Paper explaining the goals of the Workshop: “Measuring Performance and Intelligence 

of Systems with Autonomy: Metrics for Intelligence of Constructed Systems,” Messina, E. and 



Meystel, A., Editors, Measuring the Performance and Intelligence of Systems: Proceedings of the 2000 

PerMIS Workshop, Gaithersburg, MD, August 14-16, 2000, NIST Special Publication 970.

As the new millennium was upon us in 2000, a group of researchers gathered for the first time seeking to address several issues pertaining to intelligent systems:

• How can we measure the current state of the science and assess progress in the field?

• How can users select among different candidate systems and decide which system will be most 

 suited to their application?

• How can we break the cycle of re-invention and constant initiation of project with blank slates and 

 find ways to reuse existing components?

The first Performance Metrics for Intelligent Systems brought together researchers, developers, and users 

from disparate academic disciplines and domains of application to share ideas about how to tackle the multifaceted challenges of defining and measuring intelligence in artificial systems. The intelligent systems could 

take numerous forms: robots, factory or enterprise control systems, smart homes, decision support systems, 

etc. A community was formed, which evolved over the years. The workshop series carried on and became an 

annual event (with the exception of 2005).

Intelligent systems are becoming more of a reality with each passing year and the questions raised in the first 

workshop are still relevant. Additional questions have been raised, such as “how does one specify the requirements for the performance of an intelligent system?” and “how can concrete performance goals and 

good measures of performance help spur and focus innovation?” Over the years, the center of gravity of the 

program shifted more towards applied measures, rather than theoretical discussions about the general nature 

of intelligence. Many communities have availed themselves of the special sessions to focus on their particular 

interests and create mini-workshops. The concept of performance evaluation being an integral part of any research and acquisition program has become accepted. Many of the papers published in the PerMIS proceedings have been highly-referenced and provide the communities with good starting points for establishing 

measurements for new projects and programs. We are extremely grateful to the numerous colleagues who 

have supported PerMIS throughout the years. Without their dedication and hard work, this series would not 

have survived for a decade.

In this 10th workshop, we focus our attention to systems which are designed to work closely with humans. 

The theme of PerMIS’10 is key role of performance assessment in developing intelligent systems that 

can co-exist with humans towards improving the quality of our lives intertwined with automation. Adaptability to human-centered collaboration, the ability to cope with unstructured, dynamic environments, and keeping humans out of harm’s way have been widely accepted as critical prerequisites. Designing such flexible, 

PerMIS 2010
smart, and safe systems requires that their performance be quantifiable thereby facilitating emerging technologies and societal acceptance. 

PerMIS’10 is sponsored by NIST, DARPA and NSF, with technical co-sponsorship of the IEEE Washington 

Section Sensors Council Chapter, and in cooperation with the Association for Computing Machinery (ACM) 

Special Interest Group on Artificial Intelligence (SIGART). The Defense Advanced Research Projects Agency 

Information Processing Technology Office graciously provided funding to help support the workshop. Special 

thanks are due to the National Science Foundation for providing funding to allow undergraduate and graduate 

students to participate in a special poster session this year. We also thank Professor Holly Yanco of the University of Massachussetts – Lowell for organizing the student poster grants program. We gratefully acknowledge the support of our sponsors.

We thank the special session organizers for proposing interesting topics and assembling researchers related 

to their sessions. These focused sessions provide an opportunity to delve deeper into specialized topics and 

to hear from experts in the field. Our thanks are also due to the Program Committee members for publicizing 

the workshop and the reviewers for providing feedback to the authors, and for helping us to put together an 

exciting program. 

The proceedings of PerMIS will be indexed by INSPEC and Compendex and will be available through ACM’s 

Digital Library, as well as being released as a NIST Special Publication. Outstanding papers from this year’s 

proceedings will be considered for inclusion, in an expanded form, in a special issue of the International Journal of Intelligent Control and Systems.

It is our sincere hope that you will enjoy the presentations, the social programs, renew old relationships, and 

forge new ones at PerMIS’10!

Elena Messina 	 	 Raj Madhavan

General Chair 		 	 Program Chair

PerMIS 2010
SPONSORS

 

 

 

PerMIS 2010
PROGRAM COMMITTEE

General Chair:

Elena Messina (Intelligent Systems Division, NIST, USA) 

Program Chair:

Raj Madhavan (UMD-CP/ISD-NIST, USA)

Publications Chair:

B. Weiss (Intelligent Systems Division, NIST, USA)

Publicity Chair:

B. Grabowski (MITRE, USA)

Poster Session Chair:

H. Yanco (U. of Mass-Lowell, USA)

S. Balakirsky (NIST, USA)
 
 
 
 


G. Berg-Cross (EM & I, USA)

 
 
 


G. Blankenship (UMD-CP, USA)

F. Bonsignorio (UC3M, Spain)

R. Bostelman (NIST, USA)

M. Childers (ARL, USA)

M. Fields (ARL, USA)

A. Godil (NIST, USA)

J. Gunderson (GammaTwo, USA)

L. Gunderson (GammaTwo, USA)

S. K. Gupta (UMD-CP, USA)

A. Kleiner (UFreiburg, Germany)

R. Lakaemper (Temple, USA)

M. Lewis (UPitt, USA)

A. del Pobil (UJaume-I, Spain)

E. Prassler (UAppSci-BRS, Germany)

F. Proctor (NIST, USA)

D. Prokhorov (Toyota, USA)

C. Schlenoff (NIST, USA)

C. Scrapper (MITRE, USA)

J. Shi (GM, USA)

M. Shneier (NIST, USA)

E. Tunstel (JHU-APL, USA)

PerMIS 2010
PLENARY SPEAKERS

Prof. Gregory Dudek, McGill University, Canada

Building Interfaces for Robotic Data Collection and Human-Robot Collaboration Underwater 

and Outdoors 

Tue. 8:30 am 

ABSTRACT

In outdoor environments robots are ready to serve as tools for scientific data collection as well as assistants for human operators. 

Typically, however, a robotic system is subservient to a human operator and needs to respond to commands and constraints that may 

be issued in the field. On the other hand, keyboard entry and reprogramming is not appropriate user-interface mechanisms, even for 

technical users, when they are on a field expedition. Our lab has been developing human-robot interaction methods that allow a scuba 

diver to interact with a robotic assistant while underwater. This has entailed both the development of an amphibious robotic device 

with emphasis on gait selection, locomotion modes and software infrastructure, and on a communication language based on optical 

sensing of fiducial markers. A key part of this has been measuring the performance of the vehicle underwater and, more significantly, 

attempting to develop a communication paradigm for use underwater that is at once both expressive but also manageable. Due the 

substantial logistic overheads in doing work underwater, developing terrestrial surrogate tests to evaluate out work has also been an 

important requirement. Finally, we are working towards integrating the activities of our underwater vehicles with an amphibious mode 

of operation, a robotic boat, and a fixed-wing robotic aircraft. While we have made some progress towards our objectives, several 

challenges remain.

BIOGRAPHY 

Gregory Dudek is a Professor with the School of Computer Science and a member of the McGill Research Centre for Intelligent Machines (CIM) and an Associate member of the Dept. of Electrical Engineering at McGill University. In 9/2008 he became the Director of 

the McGill School of Computer Science. He is the former Director of McGill's Research Center for Intelligent Machines, a 25 year old 

inter-faculty research facility. In 2002 he was named a William Dawson Scholar. In 2008 he was made James McGill Chair. In 2010 he 

was awarded the Fessenden Professorship in Science Innovation. In 2010 he was also awarded the Canadian Image Processing and 

Pattern Recognition Award for Research Excellence and also for Service to the Research Community. He directs the McGill Mobile 

Robotics Laboratory. He has been on the organizing and/or program committees of Robotics: Systems and Science, the IEEE International Conference on Robotics and Automation (ICRA), the IEEE/RSJ International Conference on Intelligent Robotics and Systems 

(IROS), the International Joint Conference on Artificial Intelligence (IJCAI), Computer and Robot Vision, IEEE International Conference 

on Mechatronics and International Conference on Hands-on Intelligent Mechatronics and Automation among other bodies. He is 

president of CIPPRS, the Canadian Information Processing and Pattern Recognition Society, an ICPR national affiliate. He was on 

leave in 2000-2001 as Visiting Associate Professor at the Department of Computer Science at Stanford University and at Xerox Palo 

Alto Research Center (PARC). During his sabbatical in 2007-2008 he visited the Massachusetts Institute of technology and co-founded 

the company Independent Robotics Inc. He obtained his PhD in computer science (computational vision) from the University of Toronto, his MSc in computer science (systems) at the University of Toronto and his BSc in computer science and physics at Queen's 

University. He has published over 170 research papers on subjects including visual object description and recognition, robotic navigation and map construction, distributed system design and biological perception. This includes a book entitled "Computational Principles of Mobile Robotics" co-authored with Michael Jenkin and published by Cambridge University Press. He has chaired and been 

otherwise involved in numerous national and international conferences and professional activities concerned with Robotics, Machine 

Sensing and Computer Vision. He research interests include perception for mobile robotics, navigation and position estimation, environment and shape modelling, computational vision and collaborative filtering. He grew up in Montreal and favors light food. With his 

children he is re-discovering model rocketry, rollerblading, and has discovered he's not good at surfing but loves it.

PerMIS 2010 
Prof. Herman Bruyninckx, Katholieke Universiteit Leuven, Belgium

Benchmarking Reusability and Composability in Complex Software Systems−The Open 

Source Opportunity

Tue. 2:00 pm

ABSTRACT 

Designers of current and future robot systems are confronted with an increasing amount of complexity, not only with respect to the 

richness of the desired end-user functionalities in these systems, but also with respect to the hardware and software infrastructure 

required to realise these functionalities. The domain of robotics has passed the tipping point beyond which it is not possible anymore 

for one single organization or company to develop robot systems completely in house. 

Hence, system designers must find ways to integrate third-party components into their designs, reliably, predictably and effectively.

This talk defines the concepts of reusability and composability on this context of complex systems design in a multi-sourcing world, 

and discusses a dozen or so aspects that system designers can use to benchmark reusability and composability of components, their 

own as well as those from third-party providers. Throughout the presentation, the role and importance of free and open source software will be motivated, and illustrated via a number of (un)successful real-world examples.

BIOGRAPHY

Dr. Bruyninckx obtained the Masters degrees in Mathematics (Licentiate, 1984), Computer Science (Burgerlijk Ingenieur, 1987) and 

Mechatronics (1988), all from the Katholieke Universiteit Leuven, Belgium. In 1995 he obtained his Doctoral Degree in Engineering 

from the same university, where he is now professor with research interests in on-line Bayesian estimation of model uncertainties in 

sensor-based robot tasks, kinematics and dynamics of robots and humans, and the software engineering of large-scale robot control 

systems. In 2001, he started the Free Software ("open source") project Orocos, to support his research interests, and to facilitate their 

industrial exploitation.

Prof. Ken Goldberg, University of California, Berkeley, USA

Putting the Turing into Manufacturing: Recent Developments in Algorithmic Automation

Wed. 8:30 am 

ABSTRACT

Automation for manufacturing today is where computer technology was in the early 1960’s, a patchwork of ad-hoc solutions lacking a 

rigorous scientific methodology. CAD provides detailed models of part geometry. What's missing is formal models of part behavior, 

frameworks for the systematic design of automated systems that handle (e.g. assemble, inspect, sort, feed) parts, and tools for rigorous specification, analysis, and synthesis.

In 1937, Alan Turing introduced an elegant model of computing with precise vocabulary and operations that formalized concepts of 

equivalence, correctness, completeness, and complexity. Can we develop similar models for manufacturing?

“Algorithmic Automation” introduces abstractions that allow the functionality of automation to be designed independent of the underlying implementation and can provide the foundation for formal specification and analysis, algorithmic design, and consistency checking. Algorithmic Automation can facilitate integrity, reliability, interoperability, and maintainability and upgrading of automation. Researchers are developing a variety of algorithmic models. I'll present results from my lab and others on specific problems in part feeding and fixturing, including a framework for fixturing deformable parts and new geometric primitives for vibratory bowl feeders, and 

propose open problems for future research..

BIOGRAPHY

Ken Goldberg is Professor of IEOR, EECS, and the iSchool at UC Berkeley, and craiglist Distinguished Professor of New Media. He 

served two terms as Vice-President of Technical Activities for the IEEE Robotics and Automation Society. His research addresses 

robot manipulation, geometric algorithms for automation, and networked robots. More information on his work is available at: 

http:// goldberg.berkeley.edu/

PerMIS 2010
Dr. Jonathan A. Bornstein, Army Research Laboratory, USA

ARL Autonomous Systems Enterprise

Wed. 2:00 pm 

BIOGRAPHY

Dr. Bornstein has been intimately involved in robotics for over a decade. He has served as Chief of the Autonomous Systems Division, 

Vehicle Technology Directorate, Army Research Laboratory since January 2010. He has responsibility for a group of approximately 30 

Government and contractor personnel conducting research in perception and intelligence research for unmanned vehicle systems and 

a micromechanics group. He is the Collaborative Alliance Manager for the new Robotics CTA and has responsibility for coordination 

of autonomous systems research throughout ARL. He previously served as Chief of the Army Research Laboratory Robotics Program 

Office (RPO) and Collaborative Alliance Manager for the Robotics Collaborative Technology Alliance (CTA) (since April 2006) and as an 

engineer in the RPO since 1997. From 1995 through 1996 he served as a Program Manager at the Defense Advanced Research Projects Agency (DARPA) with responsibility for the Demo II Unmanned Ground Vehicle Program. 

Dr. Bornstein received his PhD. in Aeronautics & Astronautics from the Polytechnic Institute of New York in 1976. From 1975 through 

1985 he was a member of the Fluid Mechanics Group at the Corporate Research Labs of Brown, Boveri & Cie, AG in Baden, Switzerland. In 1985, he joined the Fluid Physics Branch of the U.S. Army Ballistics Research Laboratory conducting research in projectile 

launch dynamics. With the formation of the Army Research Laboratory in 1992, his technical focus shifted from dynamics to weapons 

systems and ultimately robotics technology. He received a U.S. Army Research & Development Achievement Award in 1989 and Army 

Superior Civilian Service Award in 1997. He is a registered Professional Engineer in Maryland. 

Ms. Helen Greiner, CEO of CyPhy Works and Founder of iRobot, USA

Thur. 8:30 am

BIOGRAPHY

Helen Greiner is CEO of CyPhy Works, Inc, a startup company whose mission is to be a “SkunkWorks” for robotics. She is a cofounder of iRobot, a ~$300Million business and the global leader of practical robots. Ms Greiner served as President of iRobot until 

2004, Chairman until October 2008, and currently serves on the iRobot Board. While at iRobot, she developed the strategy for and led 

iRobot's entry into the military market place. She served as the Principal Investigator on the DARPA program that created the original 

PackBot Tactical Mobile Robot, of which over 3,000 have now been deployed. At iRobot, she helped create a culture of practical innovation and performance that led to the creation of the iRobot Warrior, PackBot EOD, SUGV, and successful participation in many other 

DARPA, Army and Navy research programs. Ms. Greiner also ran iRobot's financing projects which included raising $35M venture 

capital and a $70M initial public offering. Before starting her new venture, she led iRobot's investment in a deployable Flash LADAR 

and acquisition of Nekton, an Unmanned Underwater Vehicle (UUV) company. Greiner holds a bachelor's degree in mechanical engineering and a master's degree in computer science, both from MIT. She was presented with an honorary PhD by WPI in 2009.

Ms. Greiner is highly decorated for her contributions in technology innovation and business leadership. She was named by the Kennedy 

School at Harvard in conjunction with the U.S. News and World Report as one of America's Best Leaders and was honored by the Association for Unmanned Vehicle Systems International (AUVSI) with the prestigious Pioneer Award. She has also been honored as a 

Technology Review Magazine "Innovator for the Next Century," invited to the World Economic Forum as a Global Leader of Tomorrow, 

and has been awarded the DEMO God Award at the DEMO Conference. In 2003, she was named one of the Ernst and Young New 

England Entrepreneurs of the Year and has been inducted in the Women in Technology International (WITI) Hall of Fame. Her 20+ 

years of experience in robotic technology includes work at NASA's Jet Propulsion Laboratory and MIT's Artificial Intelligence Laboratory. 

Ms. Greiner is a Trustee of the Boston Museum Science, Massachusetts Institute of Technology (MIT), National Defense Industrial Association (NDIA), Autonomous Unmanned Systems Vehicle International (AUVSI), the Massachusetts Technology Leadership Council 

(MTLC), and the US Army War College Board of Visitors. Ms. Greiner serves as the elected President and Board Member of the Robotics Technology Consortium (RTC) - a 180 member industrial/academic group. 

PerMIS 2010
Time Tuesday September 28 Wednesday September 29 Thursday September 30

8:00-8:30 Welcome/Opening Remarks Overview Overview 8:30-9:30 Plenary 1: Gregory Dudek Plenary 3: Ken Goldberg Plenary 5: Helen Greiner

9:30-10:00 Coffee Break Coffee Break Coffee Break 

10:00 -12:30 

TUE-AM1: 

Measures and 

Metrics

TUE-AM2: 

Performance 

Metrics for Mixed 

Palletizing 

Operations

WED-AM1: 

Testing and 

Evaluation of 

Intelligent Systems

WED-AM2: 

Evaluation of Human 

Detection and 

Tracking for Robot 

Safety, Collaboration 

and Interaction

THU-AM1: 

Human-Machine 

Interaction and 

Collaboration 

THU-AM2:

Evaluation of 

Sensors for Object Pose Estimation in 

Manufacturing 

Applications

12:30-14:00 Lunch Lunch Lunch 

14:00 - 15:00 Plenary 2: Herman Bruyninckx Plenary 4: Jon Bornstein (14:00-16:00) 

THU-PM1: 

Integrated 

Performance 

Assessment Through 

Experimentation

(14:00-16:00) 

THU-PM2: 

Performance 

Evaluation for 

Mapping & 

Navigation in 

Unstructured 

Environments

15:00-15:30 Coffee Break Coffee Break 

15:30 -17:30 

TUE-PM1: 

Interoperability and 

Sustainability

TUE-PM2: 

Unmanned and 

Autonomous 

System Test 

Technology

WED-PM: 

Co-X Panel Discussion Coffee Break 

16:00-16:30

Adjourn 

16:30 (18:30-20:30) 

Reception & Poster Session

 (18:30 – 22:00) 

Banquet (Dinner Cruise)
PerMIS 2010 TUESDAY 28

September

08:00 Welcome & Opening Remarks - Howard Harary, Acting Director, MEL, NIST

08:30 Plenary Presentation: 

Gregory Dudek

Building Interfaces for Robotic Data Collection and HumanRobot Collaboration Underwater and Outdoors

09:30 Coffee Break

10:00 TUE-AM1 Measures & Metrics

Chairs: Hui-Min Huang and Seungbin Moon

• Visual Metrics for the Evaluation of Sensor Data Quality in Outdoor Perception 

[Christopher Brunner, Thierry Peynot]!!!!!!!!!!!!!!!!

• Proposals for New UGV, UMV, UAV, and HRI Standards for Rescue Robots [Robin Murphy] 

• Metric Selection for Evaluating Human Supervisory Control of Unmanned Vehicles

[Birsen Donmez, M. L. Cummings]""""""""""""""""

• Performance Measures Framework for Unmanned Systems (PerMFUS): Models for 

Contextual Metrics [Hui-Min Huang, Elena Messina, Adam Jacoff, Robert Wade, 

Michael McNair]""""""""""""""

• Towards Standardization of Metrics for Evaluation of Artificial Visual Attention

[M. Zaheer Aziz, Bärbel Mertsching]

• Performance Evaluation Procedure for Vision Based Object Feature Extraction Algorithms

[Minku Kang, Wonkook Choo, Seungbin Moon]

• Benchmarks, Performance Evaluation and Contests for 3D Shape Retrieval 

[Afzal Godil, Zhouhui Lian, Helin Dutagaci, Rui Fang, Vanamali ThiruvadandamPorethi, 

Chun Pan Cheung]

12:30 Lunch

14:00 Plenary Presentation: 

Herman Bruyninckx

Benchmarking Reusability and Composability in Complex 

Software Systems!The Open Source Opportunity

15:00 Coffee Break

15:30 TUE-PM1 Interoperability & Sustainability

Chairs: John Horst and Ani Hsieh

• Modeling and Simulation Analysis Types for Sustainable Manufacturing 

[Deogratias Kibira, Guodong Shao, Tina Lee]"""""""""""""""""""""""""""""

• Metrics for the Cost of Proprietary Information Exchange Languages in Intelligent Systems 

[John Horst, Nathan Hartman, George Wong]

• Component Models in Robotics Software [Azamat Shakhimardanov, 

Nico Hochgeschwender, Gerhard Kraetzschmar]"""""""""""""""""

• Benchmarking Production System, Process Energy, and Facility Energy Performance 

Using a Systems Approach [Jorge Arinez, Stephan Biller, Kevin Lyons, Swee Leong, 

Goudong Shao, B.E. Lee, John Michaloski]!!!!!!!!!!!!!!!!!

• Complexity Measures for Distributed Assembly Tasks [Ani Hsieh, Joshua Rogoff]

• Advanced Sensing Towards Improved Forklift Safety [Roger Bostelman, Will Shackleford]

18:30 Reception & Student Poster Session

 PROGRAM PERMIS
PerMIS 2010 

September

08:00 Welcome & Opening Remarks - Howard Harary, Acting Director, MEL, NIST

08:30 Plenary Presentation: 

Gregory Dudek

Building Interfaces for Robotic Data Collection and HumanRobot Collaboration Underwater and Outdoors

09:30 Coffee Break

10:00 TUE-AM2 Special Session I: Performance Metrics for Mixed Palletizing 

Operations

Organizers: Stephen Balakirsky and Henrik Christensen

• Mixed Palletizing in Transformed Supply Chains: Operational 

Requirements and Technical Challenges

[Larry Sweet]

• Industrial Robots in Warehousing and Distribution Operations

[Don Faulkner and Sean Murphy]

• Planning in Logistics: A Survey

[Pushkar Kolhe, Henrik Christensen]

• Metrics for Mixed Pallet Stacking

[Stephen Balakirsky, Tom Kramer, Fred Proctor]

• Panel Discussion

12:30 Lunch

14:00 Plenary Presentation: 

Herman Bruyninckx

Benchmarking Reusability and Composability in Complex 

Software Systems!The Open Source Opportunity

15:00 Coffee Break

15:30 TUE-PM2 Special Session II: Unmanned and Autonomous System Test 

Technology

Organizer: Robert Heilman

• Unmanned and Autonomous Systems Test Technology [Rob Heilman]

• A Multi-Vehicle Testbed for Underwater Motion Coordination 

[Nitin Sydney, Seth Napora, Derek Paley]

• Measurement of Autonomous Operations [Bill Hamel]

• DCF® – A JAUS and TENA Compliant Agent-based Framework for

UAS Performance Evaluation [Nicholas Lenzi, Benjamin Bachrach,

Vikram Manikonda] 

• Testing and Evaluation Aspects of Integration of Unmanned Air Systems 

into the National Air Space [Mauricio Castillo-Effen, Nikita Visnevski]

18:30 Reception & Student Poster Session

PROGRAM PERMIS

TUESDAY 28
PerMIS 2010 WEDNESDAY 29

September

08:15 Overview

08:30 Plenary Presentation: 

Ken Goldberg

Putting the Turing into Manufacturing: Recent

Developments in Algorithmic Automation

09:30 Coffee Break

10:00 WED-AM1 Testing & Evaluation of Intelligent Systems

Chairs: Brian Weiss and James Gunderson

• Evaluating Intelligent Systems with Performance Uncertainty in Large 

Test Spaces [Miles Thompson]!!!!!!!!!!!!!!!!!

• The Multi-Relationship Evaluation Design Framework: Creating 

Evaluation Blueprints to Assess Advanced and Intelligent Technologies 

[Brian Weiss, Linda Schmidt]!!!!!!!!!!!!!!!

• Implementation and Application of Maximum Likelihood Reliability 

Estimation from Subsystem and Full System Tests

[Coire Maranzano, James Spall]""""""""""""""""

• “What do you do with a drunken robot?” In Situ Performance 

Measurements of Intelligent Mobile Robots

[James Gunderson, Louise Gunderson]""""""""""""""""

• Comprehensive Standard Test Suites for the Performance Evaluation of 

Mobile Robots [Adam Jacoff, Hui-Min Huang, Elena Messina, 

Ann Virts, Tony Downs]

• Towards a Standardized Test for Intelligent Wheelchairs [Joelle Pineau, 

Robert West, Amin Atrash, Julien Villemure, Francois Routhier]!!!!!!!!!!!!!!!!

• Evaluation of Ultra-Wideband Technology for Use in 3-D Locating 

Systems [Adam Kopp, Kamel Saidi, Hiam Khoury]

12:30 Lunch

14:00 Plenary Presentation:

Jon Bornstein

ARL Autonomous Systems Enterprise

15:00 Coffee Break

15:30 WED-PM Co-X Panel Discussion

18:30 Banquet

Dinner Cruise

 PROGRAM PERMIS
PerMIS 2010 

September

08:15 Overview

08:30 Plenary Presentation: 

Ken Goldberg

Putting the Turing into Manufacturing: Recent

Developments in Algorithmic Automation

09:30 Coffee Break

10:00 WED-AM2 Special Session III: Evaluation of Human Detection and 

Tracking for Robot Safety, Collaboration and Interaction

Organizers: Tsai Hong and Roger Eastman

• An Overview of Human and Machine Performance on Face Recognition 

[Jonathon Phillips]

• Detecting Humans under Partial Occlusion using Markov Logic 

Networks [Raghuraman Gopalan, William Schwartz]

• Transitional or Partnership Human and Robot Collaboration for 

Automotive Assembly [Jane Shi, Roland Menassa]

• Performance Assessment of Face Recognition Using Super-Resolution 

[Shuowen Hu, Robert Maschal, Susan Young, Tsai Hong, 

Jonathon Phillips]

• Inexpensive Ground Truth and Performance Evaluation for Human 

Tracking using Multiple Laser Measurement Sensors

[William Shackleford, Tsai Hong, Tommy Chang]

• Development of Performance Metrics and Test Methods for First 

Responder Location and Tracking Systems 

[Francine Amon, Camillo Gentile, Kate Remley]

• Automated Gross and Sub-pixel Registration Accuracy of Visible and 

Thermal Imagery 

[Stephen Won, S. Susan Young, Gunasekaran Seetharaman]

12:30 Lunch

14:00 Plenary Presentation:

Jon Bornstein

ARL Autonomous Systems Enterprise

15:00 Coffee Break

15:30 WED-PM Co-X Panel Discussion

18:30 Banquet

Dinner Cruise

PROGRAM PERMIS

WEDNESDAY 29
PerMIS 2010 THURSDAY 30

September

08:15 Overview

08:30 Plenary Presentation: 

Helen Greiner

09:30 Coffee Break

10:00 THU-AM1 Human-Machine Interaction and Collaboration

Chairs: Craig Schlenoff and Paul Oh

• An Indoor Study to Evaluate a Mixed-Reality Interface For Unmanned 

Aerial Vehicle Operations in Near Earth Environments 

[James Hing, Justin Menda, Kurtulus Izzetoglu, Paul Oh]"""""""""""""""""

• A Network-based Approach for Assessing Co-Operating Manned and 

Unmanned Systems (MUMS) [Lora Weiss]!!!!!!!

• Lessons Learned in Evaluating DARPA Advanced Military Technologies

[Craig Schlenoff, Brian Weiss, Michelle Steves]!!!!!!!!!!!!!!

• Modified Cooper Harper Scales for Assessing Unmanned Vehicle 

Displays [Birsen Donmez, M. L. Cummings, Amy Brzezinski, 

Hudson Graham]""""""""""""""""

• Using the “Negative Attitude Towards Robots Scale” with Telepresence 

Robots [Katherine Tsui, Munjal Desai, Holly Yanco, Henriette Cramer, 

Nicander Kemp]"""""""""""""""""

• Teams Organization and Performance Analysis in Autonomous HumanRobot Teams [Huadong Wang, Michael Lewis, Shih-Yi Chien]

• Intentions and Intention Recognition in Intelligent Agents

[Gary Berg-Cross, Christopher Crick]

12:30 Lunch

14:00 THU-PM1 Special Session V: Integrated Performance Assessment 

Through Experimentation

Organizer: Marshal Childers

• Evaluating Hierarchical Planner Performance through Field Experiments 

and Simulation [Juan Pablo Gonzalez, Marshal Childers, Barry Bodt]

• Evaluating the Performance of Unmanned Ground Vehicle Water 

Detection [Arturo Rankin, Tonislav Ivanov, Shane Brennan]

• Autonomous Mobility for Areas with Large Number of Pedestrians 

[Alberto Lacaze, Karl Murphy, Nenad Uzunovic, Joseph Putney]

• Observations on Single Operator Performance Controlling Two UGVs

in a Field Assessment [Susan Hill]

• Assessing Unmanned Ground Vehicle Tactical Behaviors Performance 

[Marshal Childers, Barry Bodt, Richard Camden]

16:00 Coffee Break

16:30 Adjourn

 PROGRAM PERMIS
PerMIS 2010 

September

08:15 Overview

08:30 Plenary Presentation: 

Helen Greiner

09:30 Coffee Break

10:00 THU-AM2 Special Session IV: Evaluation of Sensors for Object Pose 

Estimation in Manufacturing Applications

Organizers: Roger Eastman, Tsai Hong, and Hui-Min Huang

• 6-DOF Pose Estimation: The Need for Standardization in Industrial 

Applications 

[Shaun Edwards, William Flannigan, Paul Evans]

• Flexible Robotic Assembly in Dynamic Environments

[Jane Shi, Roland Menassa]

• Smart Sensing for Real-Time Pose Estimation, Assembly and Inspection 

using 3D Laser Scanning Systems

[Chad English]

• Dynamic Performance Evaluation of 6D Laser Tracker Sensor 

[Kam Lau, Yubing Yang, Yuangun Liu, Henry Song]

• Methodology for Evaluating Static Six-degree-of-freedom (6DoF) 

Perception Systems 

[Tommy Chang, Tsai Hong, Mike Shneier, Mili Shah, Roger Eastman]

• Panel Discussion

12:30 Lunch

14:00 THU-PM2 Special Session VI: Performance Evaluation for Mapping & 

Navigation in Unstructured Environments

Organizers: Rolf Lakaemper and Raj Madhavan 

• Evaluation Criteria for Appearance Based Maps 

[Gorkem Erinc, Stefano Carpin]

• Evaluation of Maps using Fixed Shapes: The Fiducial Map Metric

[Sören Schwertfeger, Adam Jacoff, Chris Scrapper, Johannes Pellenz, 

Alexander Kleiner]

• The Platform- and Hardware-in-the-loop Simulator for Multi-Robot 

Cooperation [Tomonari Furukawa, Lin Chi Mak, Kunjin Ryu,

Xianqiao Tong]

• World Modeling for Autonomous Navigation in Unstructured and 

Dynamic Environments [Rolf Lakaemper, Raj Madhavan]

• Discussion

16:00 Coffee Break

16:30 Adjourn

PROGRAM PERMIS

THURSDAY 30
AUTHOR INDEX

Amon, F. .................WED-AM2

Arinez, J. .................TUE-PM1

Atrash, A. ...............WED-AM1

Aziz, M.Z. ................TUE-AM1

Bachrach, B. ...........TUE-PM2

Balakirsky, S. ...........TUE-AM2

Berg-Cross, G. ........THU-AM1

Biller, S. ...................TUE-PM1

Bostelman, R. ..........TUE-PM1

Donmez, B. .............TUE-AM1

Bodt, B. ...................THU-PM1

Bodt, B. ...................THU-PM1

Brennan, S. .............THU-PM1

Brunner, C. ..............TUE-AM1

Brzezinski, A. ..........THU-AM1

Camden, R. .............THU-PM1

Carpin, S. ................THU-PM2

Castillo-Effen, M. .....TUE-PM2

Chang, T. ................WED-AM2

Chang, T. .................THU-AM2

Cheung, C.P. ...........TUE-AM1

Chien, S.Y. ..............THU-AM1

Childers, M. ............THU-PM1

Childers, M. ............THU-PM1

Choo, W. .................TUE-AM1

Christensen, H. .......TUE-AM2

Cramer, H. ...............THU-AM1

Crick, C. ..................THU-AM1

Cummings, M.L. ......TUE-AM1

Cummings, M.L. .....THU-AM1

Desai, M. .................THU-AM1

Donmez, B. .............THU-AM1

Downs, T. ...............WED-AM1

Dutagaci, H. ............TUE-AM1

Eastman, R. ............THU-AM2

Edwards, S. ............THU-AM2

English, C.................THU-AM2

Erinc, G. ..................THU-PM2

Evans, P. .................THU-AM2

Fang, R. ...................TUE-AM1

Flannigan, W. ..........THU-AM2

Furukawa, T. ...........THU-PM2

Gentile, C. ..............WED-AM2

Godil, A. ..................TUE-AM1

Gonzalez, J.P. .........THU-PM1

Gopalan, R. ............WED-AM2

Graham, H. .............THU-AM1

Gunderson, J. ........WED-AM1

Gunderson, L. ........WED-AM1

Hamel, B. ................TUE-PM2

Hartman, N. .............TUE-PM1

Heilman, R. ..............TUE-PM2

Hill, S. ......................THU-PM1

Hing, J. ....................THU-AM1

Hochgeschwender, 

N. .............................TUE-PM1

Hong, T. ..................WED-AM2

Hong, T. ..................WED-AM2

Hong, T. ..................THU-AM2

Horst, J. ...................TUE-PM1

Hsieh, A. ..................TUE-PM1

Hu, S. .....................WED-AM2

Huang, H.M. ............TUE-AM1

Huang, H.M. ...........WED-AM1

Ivanov, T. .................THU-PM1

Izzetoglu, K. ............THU-AM1

Jacoff, A. .................TUE-AM1

Jacoff, A. ................WED-AM1

Jacoff, A. .................THU-PM2

Kang, M. ..................TUE-AM1

Kemp, N. .................THU-AM1

Khoury, H. ..............WED-AM1

Kibira, D. .................TUE-PM1

Kleiner, A. ................THU-PM2

Kolhe, P. ..................TUE-AM2

Kopp, A. .................WED-AM1

Kraetzschmar, G. .....TUE-PM1

Kramer, T. ................TUE-AM2

Lacaze, A. ...............THU-PM1

Lakaemper, R. .........THU-PM2

Lau, K. .....................THU-AM2

Lee, B. E. .................TUE-PM1

Lee, T. ......................TUE-PM1

Leong, S. .................TUE-PM1

Lenzi, N. ..................TUE-PM2

Lewis, M. ................THU-AM1

Lian, Z. ....................TUE-AM1

Liu, Y. ......................THU-AM2

Lyons, K. .................TUE-PM1

Madhavan, R. ..........THU-PM2

Mak, L.C. ................THU-PM2

Manikonda, V. ..........TUE-PM2

Maranzano, C. ........WED-AM1

Maschal, R. ............WED-AM2

McNair, M. ...............TUE-AM1

Menassa, R. ...........WED-AM2

Menassa, R. ............THU-AM2

Menda, J. ................THU-AM1

Mertsching, B. .........TUE-AM1

Messina, E. ..............TUE-AM1

Messina, E. ............WED-AM1

Michaloski, J.		.........TUE-PM1

Moon, S. ..................TUE-AM1

Murphy, K. ..............THU-PM1

Murphy, R. ...............TUE-AM1

Murphy, S. ...............TUE-AM2

Napora, S. ...............TUE-PM2

Oh, P. ......................THU-AM1

Paley, D. ..................TUE-PM2

Pellenz, P. ................THU-PM2

Peynot, T. ................TUE-AM1

Phillips, J. ...............WED-AM2

Phillips, J. ...............WED-AM2

Pineau, J. ...............WED-AM1

Proctor, P. ................TUE-AM2

Putney, J. ................THU-PM1

Rankin, A. ...............THU-PM1

Remley, K. ..............WED-AM2

Rogoff, J. .................TUE-PM1

Routhier, F. .............WED-AM1

Ryu, K. ....................THU-PM2

Saidi, K. ..................WED-AM1

Schmidt, L. ............WED-AM1

Schwartz, W. ..........WED-AM2

Schwertfeger, S. .....THU-PM2

Schlenoff, C. ...........THU-AM1

Scrapper, C. ............THU-PM2

Seetharaman, G. ....WED-AM2

Shackleford, W.........TUE-PM1

Shackleford, W. ......WED-AM2

Shakhimardanov, A. TUE-PM1

Shao, G. ..................TUE-PM1

Shao, G. ..................TUE-PM1

Shi, J. .....................WED-AM2

Shi, J. ......................THU-AM2

Shah, M. .................THU-AM2

Shneier, M. ..............THU-AM2

Song, H. ..................THU-AM2

Spall, J. ..................WED-AM1

Steves, M. ...............THU-AM1

Sweet, L. .................TUE-AM2

PerMIS 2010 
Sydney, N. ...............TUE-PM2

ThiruvadandamPorethi, 

V. ..............................TUE-AM1

Thompson, M. ........WED-AM1

Tsui, K. ....................THU-AM1

Tong, X. ...................THU-PM2

Tzvetkova, G. ..........TUE-PM1

Uzunovic, N. ...........THU-PM1

Villemure, J. ............WED-AM1

Visnevski, N. ............TUE-PM2

Virts, A. ...................WED-AM1

Wade, R. ..................TUE-AM1

Wang, H. .................THU-AM1

Weiss, B. ................WED-AM1

Weiss, B. .................THU-AM1

Weiss, L. .................THU-AM1

West, R. ..................WED-AM1

Won, S. ...................WED-AM2

Wong, G. .................TUE-PM1

Yanco, H. ................THU-AM1

Yang, Y. ...................THU-AM2

Young, S. ................WED-AM2

Young, S. ................WED-AM2

PerMIS 2010
ACKNOWLEDGMENTS

These people provided essential support to make this event happen. Their 

ideas and efforts are very much appreciated.

Website and Proceedings

Debbie Russell

Local Arrangements

Jennifer Peyton

Conference and Registration

Mary Lou Norris

Angela Ellis

Kathy Kilmer

PerMIS 2010 

Intelligent Systems Division 

Manufacturing Engineering Laboratory 

National Institute of Standards and Technology

100 Bureau Drive, MS 8230

Gaithersburg, MD 20899-8230 

http://www.nist.gov/mel/isd/permis2010.cfm