DYNAMIC PERCEPTION 

WORKSHOP REPORT: 

Requirements and Standards for 

Advanced Manufacturing

Edited by: 

Roger Eastman 

Elena Messina 

Tsai Hong 

Hui-Min Huang 

Mike Shneier

U.S DEPARTMENT OF COMMERCE

National Institute of Standards and Technology 

Intelligent Systems Division 

Gaithersburg, MD 20899-8230 

and 

Jane Shi

James Wells 

GM Research and Development Center

Warren, MI 48090

NISTIR 7664
 

NISTIR 7664

DYNAMIC PERCEPTION 

WORKSHOP REPORT: 

Requirements and Standards for 

Advanced Manufacturing

Edited by: 

Roger Eastman 

Elena Messina 

Tsai Hong 

Hui-Min Huang 

Mike Shneier

U.S DEPARTMENT OF COMMERCE

National Institute of Standards and Technology 

Intelligent Systems Division 

Gaithersburg, MD 20899-8230 

and 

Jane Shi

James Wells 

GM Research and Development Center

Warren, MI 48090

January 2010

U.S. DEPARTMENT OF COMMERCE 

Gary Locke, Secretary 

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY 

Patrick Gallagher, Director
1

DYNAMIC PERCEPTION WORKSHOP REPORT: 

(1-14-2010) 

Requirements and Standards for Advanced Manufacturing 

Roger Eastman, Elena Messina, Tsai Hong, Hui-Min Huang 

and Mike Shneier 

National Institute of Standards and Technology (NIST) 

Jane Shi and James Wells 

Manufacturing Systems Research Lab 

GM Global Research & Development Center 

Held in conjunction with 

International Robots, Vision & Motion Control Show 

Rosemont, Illinois 

Tsai Hong (Chair) and Hui Huang (Coordinator) 

National Institute of Standards and Technology (NIST) 

Steering Committee (in alphabetical order)

Roger Eastman, Loyola 

Jeff Fryman, RIA 

Tsai Hong, NIST 

Hui Huang, NIST 

Frank Maslar, Ford 

Elena Messina, NIST 

Jane Shi, GM 

Michael Shneier, NIST 

Dana Whalls, AIA 

June 11, 2009 
2

Disclaimer........................................................................................................................... 3 

I. Introduction.................................................................................................................... 4 

II. Workshop Presentations............................................................................................... 5 

II.1 An Example Process Model ................................................................................. 5 

II.2 Background Presentation...................................................................................... 7 

II.3 Discussion Session Theme and Process............................................................... 8 

III. Initial definition of scenarios...................................................................................... 9 

IV. Refinement of material manipulation scenarios...................................................... 12 

V. Refinement of Automated Guided Vehicle (AGV) scenarios.................................... 14 

VI. Requirements for scenarios and sensor components............................................... 15 

VII. Summary................................................................................................................... 15 

Acknowledgments........................................................................................................... 18 

Appendix.......................................................................................................................... 19 

A. Call for Participation............................................................................................... 19 

B. Background Presentations........................................................................................ 19 

B.1 Measuring the Performance of Response Robots .............................................. 19 

B.2 Dynamic Perception Workshop ......................................................................... 26 
3

Disclaimer

Certain trade names and company products are mentioned in the text or identified in 

certain illustrations. In no case does such an identification imply recommendation or 

endorsement by the NIST, nor does it imply that the products are necessarily the best 

available for the purpose. 

The opinions expressed in this Workshop Report are those of the workshop participants 

and are not the official opinions of NIST. 
4

I. Introduction 

The Dynamic Perception Workshop was held on the morning of June 11th, 2009, in 

conjunction with the International Robots & Vision Motion Control Show in 

Rosemont, Illinois. The stated objective of the workshop was: 

To facilitate highly flexible automation in next generation manufacturing by: 

• Developing standard metrics and test methods for the performance evaluation of 

advanced sensor and perception systems. 

• Providing users, developers, integrators and researchers with quantitative measures of 

capabilities of perception systems. One example of a user-stated need is the 

following quote. 

“GM, as a sensor end user, will benefit from this standardized 

performance evaluation by being able to utilize sensors that have 

demonstrated their capabilities as defined by NIST standards first rather 

than going through an intensive trial-error learning process utilizing GM’s 

resources” 

 — Roland Menassa, General Motors, April 2008 

The workshop was organized to solicit community input on how to define and develop 

the test methods and metrics for perception systems. Attendees came from the automotive 

industry, the sensor and metrology equipment industries, the robotics industry, the US 

Post Office, and academic institutions. 

The workshop objectives grew out of community input gathered at the Smart Assembly 

Workshop and the Dynamic Measurement and Control for Autonomous Manufacturing 

Workshop. The Smart Assembly Workshop, held in Gaithersburg, Maryland in October 

2006, was organized to identify key characteristics of smart manufacturing assembly, 

scientific and infrastructure challenges to deploying smart assembly, and opportunities to 

accelerate the development of smart assembly. A full report on the smart assembly 

workshop is available at http://smartassembly.wikispaces.com/. Among other findings, 

intelligent, flexible automation was identified as key to smart assembly. 

An enabling technology for intelligent automation is dynamic perception, defined as the 

ability to sense the changing manufacturing environment for the intelligent control of 

automation equipment. To further refine the requirements of dynamic perception, the 

Dynamic Measurement and Control for Autonomous Manufacturing Workshop was held 

in Columbia, Maryland in October 2007. This workshop was organized to identify 

requirements for next generation robotics in dynamic environments, identify perception 

needs for visual servoing in autonomous assembly, and identify specific requirements for 
5

safe operation of robot arms and vehicles. The workshop report is available as a NIST 

internal report, NISTIR-75751. 

Attendees at the 2007 Dynamic Measurement and Control for Autonomous 

Manufacturing Workshop concluded that, to advance beyond the current generation of 

machine vision systems on the manufacturing floor, the perception systems must: 

• Be comprehensive, pervasive, and redundant. 

• Be advanced by well-defined terminology, requirements, and scenarios. 

• Be supported by reference standards for calibration and dynamic measurements. 

• Be integrated into a community of successful commercial interests. 

• Be validated for software and hardware safety and reliability by processes, test beds, 

and high fidelity simulations. 

The present workshop, Dynamic Perception workshop, was organized to further refine 

these conclusions by looking more closely at terminology, requirements and scenarios. 

II. Workshop Presentations 

The workshop began with a set of presentations to explain its goals, scope, and approach, 

as described in the following sections. The workshop kicked off with welcoming 

remarks by Jeff Burnstein, President of the Robotic Industries Association (RIA). This 

was followed by presentations meant to set the stage for the active discussion. 

II.1 An Example Process Model 

The first presentation was entitled, “Standard Performance Test Method Development 

Process for Response Robots: Foundations for Progress in Manufacturing.” This talk 

described a successful project in which NIST has coordinated the development of 

requirements, metrics, and test methods aimed at producing performance standards for 

robots applied to urban search and rescue (US&R)2

. The process model used in this 

work was proposed as a straw man to be adapted for use in developing requirements and 

standards for dynamic perception. 

The model is a user-driven process as depicted in Figure 1. For the emergency response 

robots case, the effort started by inviting end users, i.e., the Department of Homeland 

Security (DHS), Federal Emergency Management Agency (FEMA) and Urban Search 

and Rescue (US&R) Task Forces, to a series of requirement-generation workshops. The 

task force members described US&R operations and identified areas where they wanted 

the assistance of robots. Most responders were not at all familiar with robots and their 

 

1

 Hong, Tsai H., Eastman, Roger, Bostelman, Roger V., Huang, Hui-Min, McMorris, B., 

“Proceedings of the Dynamic Measurement and Control for Autonomous Manufacturing 

Workshop,” NISTIR 7575, May 2009. 

2

 Messina, E., “Performance Standards for Urban Search & Rescue Robots: Enabling 

Deployment of New Tools for Responders,” Defense Standardization Program Office 

Journal, July/December 2007, pp. 43-48. 
6

present or potential capabilities. This posed the additional challenge of trying to envision 

what roles the robots could realistically be expected to take on during US&R missions. 

Figure 1: The Process Model for Standardizing Emergency Response Robots Performance 

Evaluation 

The results of the workshops were sets of robotic performance requirements, expressed in 

operational terms. The following are the categories developed for US&R requirements: 

 Terminology 

 Human-Robot Interaction (HRI) 

 Logistics 

 Safety/Operating Environment 

 Robotic Subsystems 

- Mobility/Maneuvering 

- Energy and Power 

- Sensing 

- Communications 

- Manipulation and Other Payloads 

- Chassis 

Test methods are being developed to evaluate how well candidate robots meet the 

requirements. For example, the mobility test methods include how wide a gap a robot 

can cross, how steep staircases a robot can climb, and what types of terrain a robot can 

traverse, etc. 
7

The essential elements of a test method are purpose, apparatus (physical test method 

setups), performance metrics, and evaluation procedure. For this particular project, the 

test methods become standards through a consensus balloting process within ASTM’s 

E54.08 Homeland Security Applications, Operational Equipment Subcommittee. 

The designed test methods must be thoroughly evaluated and validated. Periodically, 

NIST organizes large-scale response robot exercises, which bring together responders 

(users) with robot developers (manufacturers and researchers) at FEMA training 

facilities. All robots run through all relevant draft test methods, which enables the users 

and developers to provide feedback on the relevance and fairness of the tests. Training 

scenarios available at the FEMA facilities are used for exploring how robots can be 

utilized within search missions. This educates both responders and developers and 

allows for a common understanding of the goals of incorporating robots into US&R 

missions. 

Competitions such as Robocup Rescue League3

 are also being utilized for “testing the 

tests.” At these events, robots have to search arenas (also known as test courses) within a 

fixed amount of time, locate victims, and map the area. The arenas are comprised of test 

method elements, such as mobility, sensor, and manipulation tests. Research-grade 

robots that come to the competitions are often developed specifically to be able to 

complete the test courses. This stimulates innovative technology. Having a high volume 

of traffic and robot diversity running through the test courses helps shake out problems 

with the test methods. Verification and validation of the test methods are also done 

within test methods, located at NIST and elsewhere, where commercial robots are openly 

invited to participate. By repeatedly running the robots through the test courses, each of 

the essential elements of the designed test methods is being evolved to maturation. 

The key concluding remarks of this presentation noted that it is important, for a new 

technology to become useful and usable, to: 

• Measure performance in reproducible, repeatable ways that can correlate 

to use in the field. 

• Develop concepts of operation and match the right characteristics to 

different deployment needs. 

• Achieve buy-in from all the stakeholders, so that they perceive the benefits 

of this effort and support its advancement. 

II.2 Background Presentation 

The second presentation covered previous work done in the “Metrology and Standards 

for Advanced Perception Systems Project” by NIST and collaborators. The presentation 

covered previous workshops along with experiments conducted to develop metrology 

 

3

 http://www.robocuprescue.org/ 
8

techniques to evaluate dynamic perception systems. The presentation material on 

previous workshops has been summarized in the introductory section and will not be 

repeated here. 

The Metrology and Standards for Advanced Perception Systems Project develops 

quantitative, reproducible test methods to evaluate the robustness, accuracy and 

performance characteristics of advanced perception systems for general assembly and 

other macro scale manufacturing operations. For advanced use on the manufacturing shop 

floor, perception systems need to sense three-dimensional structure in a dynamic, 

uncontrolled environment. In particular, they must be able to accurately determine the six 

degree of freedom (6DOF) position and orientation of parts, people, robots and other 

objects in motion. This will enable new applications of automation, including continuous 

assembly where a robot interacts with a moving assembly line. A primary objective of the 

project is to develop dynamic 6DOF metrology that can provide calibration and ground 

truth measurements for perception systems. 

The project encompasses a number of efforts, including 

providing dynamic metrology support for NIST work in 

AGV navigation evaluation and robotic arm calibration. 

This work is done as part of the NIST robot test bed, which 

includes a robot arm mounted on an overhead linear rail, 

AGVs, conveyer belts, sensors and a laser tracker for 

calibration and precise metrology. 

The project has conducted collaborative experiments with 

the Robot Vision Laboratory of Purdue University in which 

the NIST laser tracker was used to provide ground truth for 

evaluation of the Purdue “Line Tracking for Assembly-onthe-fly” system. This system uses video cameras to track a 

part and direct a robot arm to insert a peg in a hole while 

the part is moving. The NIST work was able to produce measurements of the position and 

orientation of the part to 50 micro meters at 30 frames per second. The project is also 

working with the perception company BrainTech, Inc. to develop protocols for evaluating 

camera calibration and object pose algorithms. 

II.3 Discussion Session Theme and Process 

The workshop discussion was organized into two sessions, held sequentially. Each 

session was structured with an initial period for participants to generate ideas on note 

cards, a period to collect, post and sort the initial ideas, and a discussion period to refine, 

enhance and prioritize the initial ideas. The sessions were: 

Session 1 - Manufacturing Perception Tasks and Scenarios 

Facilitators: Elena Messina and Hui Huang 

Objective: To develop a representative list of manufacturing scenarios 
9

Issues: 

Are there representative sets of manufacturing scenarios or tasks that can be identified as 

candidates for employing sensing to advance robotic capabilities? 

Which of these manufacturing tasks may be the “low hanging fruit” in terms of 

perception system applications, i.e., where current or near-term perception technology, 

metrics, and standards can readily make an impact? 

Concrete outcome: a list of near term scenarios where improved sensing would have an 

impact. 

Session 2 - Dynamic 3D Sensor Requirements 

Facilitators: Roger Eastman and Tsai Hong 

Objective: To develop requirements for sensor performance 

Issues: 

What expanded capabilities, enhanced performance, and new information are needed 

from sensor and perception systems to advance manufacturing robotics? 

What are the essential metrics for the sensory and perception systems (e.g., precision, 

response time, weights, working volumes, lighting requirements)? 

Concrete outcome: a categorized set of requirements for advanced dynamic sensors. 

III. Initial definition of scenarios 

The workshop discussions started with each participant individually listing on note cards 

what they considered important requirements and scenarios for the future application of 

advanced perception systems on the manufacturing floor. The topics concentrated on 

discrete material transport, handling, and processing. They were organized, in a group 

effort, into two general categories: 

• Scenarios on the manipulation of materials and parts for logistics and processing. These 

scenarios focused on how parts would be loaded onto and unloaded from pallets and bins 

and how the parts would be fed into and handled during assembly and processing steps. 

Particular examples were: 

i) Load a kit of parts through picking from small bins. (A kit is a technical 

term for a fixture that holds a number of parts needed for an assembly 

task. See Figure 2. ) 

ii) Pick from a bin on a cart containing loose parts in a consistent orientation. 

iii) Unload and separate parts from a mixed-content pallet. 

iv) Pick a part from a kit and present it to an operator or load it into a 

machine. 

v) Build a logistics pallet with mixed stock items. 

vi) Mate rigid or flexible parts onto an assembly. 

vii) Process parts without using fixtures (machining, cutting, welding). 
10

viii) Unload boxes as they arrive at a facility. 

ix) Adapt quickly to production changeovers in low volume systems. 

Figure 2. An example of filled kit with seven parts for a car door assembly task. 

• Scenarios on the transport of parts through a facility. These scenarios focused on the use 

of automated guided vehicles (AGVs) and automated guided carts (AGCs) to move 

materials inside a facility. Particular examples were: 

i) Locate an AGV on the factory floor. 

ii) Navigate an AGV through human traffic. 

iii) Respond to visual signals from a human operator. 

iv) View a signpost and deal with blocked or obscured signposts. 

v) Move heavy objects to designated areas without human interaction. 

During discussions on these initial scenarios, the groups observed a few patterns and 

implications. First, most of the scenarios dealt with material handling: getting parts and 

material to the right place and in the right orientation. Only one of the initially-offered 

scenarios dealt with processing such as milling, finishing, cutting, and welding. The 

general observation was that, operating in an unstructured environment, the key new 

difficulty is getting a part from a kit, bin, or cart into position for a process. The flexible 

automation system must handle the job of fixturing a part for a processing operation, and 

then the processing operation will have its own automation procedure. The sense from the 

group was also that, for near term advances in flexible automation, material handling is 

key and success would lead to the next step of automating flexible processing. 

Second, several of the part manipulation scenarios dealt with integrating robots into 

existing human-scale, manual processes. A kit is a bin designed to hold a set of parts for 

easy access as a person works on an assembly, and traditionally, filling and using a kit 

has been a manual process. The scenarios listed to build a kit and to pick from a kit are 

examples of human-robot collaboration where the robot assists by doing tasks it performs 

well. These are assumed to be simply handling objects, and then passing parts to a human 

operator so he or she can carry out joining tasks, which are assumed to be more 
11

challenging and best done by a person. Increased use of human-safe, flexible robotics 

would enable mixed use of humans and robots, and imply that the processes must 

generally be at a speed and scale appropriate for people. 

Participants noted advantages in automating kit-oriented tasks that can be integrated with 

manual processes. Currently, robots are fenced off due to safety concerns and human 

operators can only enter a robot’s work area by shutting down the work cell. This means 

that all steps in the work cell must be automated, since humans can’t be in the loop, and 

that the automation must be highly reliable and rarely require human intervention since 

that is very costly in time lost. Integrating human-safe robots into manual processes 

would allow people back into the work cell, enable them to intervene or take over when a 

robot malfunctions, and help adapt the technology to low or medium volume processes so 

small manufacturers could afford automation. Integration of robots into manual 

processes contrasts with re-engineering and advancing processes for fully automated 

work cells performing tasks not previously automated. 

• Issues for specialization and customization of scenarios. 

Ideally, manufacturing scenarios used to develop testing protocols and standards would 

be as general as possible and have wide application to manufacturing industries. 

However, participants noted that current uses of perception on the manufacturing floor 

are highly specialized and require considerable customization to adapt to a specific plant 

or production line. Over time, as machine and computer vision applications on the plant 

floor develop, they should become more robust and general. However, the technology 

remains much less adaptable than human vision, so in the short run many solutions will 

remain constrained to specific applications and particular requirements. To drive 

technologies into useful solutions, general scenarios may need to be specialized for 

different application areas. The implications include: 

 1) Being clear if a scenario and standard is aimed at evaluating current vision 

systems for today’s processes, or encouraging development of emerging sensors, 

techniques, and systems that enable new solutions for processes. The two types of 

scenarios, validating current technology or encouraging new approaches, may lead to 

different conversations between vendors and end-users. Those that encourage new 

technologies will have to be adaptable and frequently updated. 

 2) Developing metrics and evaluation procedures that as much as possible apply 

to both general scenarios and to specific applications. It would be useful for 

manufacturers to have metrics that apply to their specific needs, but are not so specialized 

that they do not apply to other applications. 

 3) Taking into account the ability of a vendor to optimize or customize their 

system to perform a standardized task. This could be part of a metric to evaluate a 

perception system. For current machine vision applications, the vendor often invests 

considerable time in customizing and optimizing their system for an end user task. For 

tests against a standardized scenario, the permitted efforts for customization should be 

made explicit (whether permitted or not, and how much effort is permitted.) If 

customization efforts are permitted, the effort involved should be tracked and logged so it 
12

can be used in evaluation. A system that can perform adequately out of the box may be 

considered more valuable than one that works, but requires considerable effort to finetune. 

 4) Insuring that calibration issues are included in metrics. Calibration can be 

critical for success. A system that works well when calibrated, but does not maintain 

calibration well, is less valuable than one that maintains calibration well, or that performs 

self-calibration while in operation. Metrics for evaluation may need to consider whether a 

perception system is robust against poor calibration, whether it gives feedback on the 

quality of current calibration, whether it has diagnostics for calibration issues, whether it 

self-calibrates under operation, and whether the degree of difficulty of calibration and 

operator training needed for calibration. 

IV. Refinement of material manipulation scenarios

Most of the workshop time was spent on refining the material manipulation scenarios, 

generally defined as pick and place operations. The basic task was defined as identifying, 

locating, gripping, and orienting an item for transfer from a container to a destination. 

This could be done for logistics, assembly, or processing. The group considered the 

characteristics of materials and parts, how parts are stored and transported, environmental 

conditions, and other aspects of the task. The general purpose of the discussion was to 

define the dimensions along which the basic pick and place task varied, so a hierarchy of 

challenging tasks could be defined. There was a working assumption that the materials or 

parts could be treated as discrete, solid units rather than continuous or fluid materials. 

The aspects of the problem considered were: 

• Item count. The number of objects to be selected from – whether parts are 

presented one at a time or in larger numbers. 

• Item geometry. The shape of the object, which determines the number of stable 

positions, the number and nature of gripping points, the symmetries, and the 

nature of visual features. 

• Item reflectance/emissivity and texture. The visual/radiometric surface of the 

object. 

• Item visual features. The number and nature of distinctive features that could be 

used by a vision system, which are a result of geometry and reflectance. 

• Item consistency. Whether the parts are all one kind or mixed and the allowable 

normal variability in a single part type (which may be higher if man-made.) 

• Item flexibility. Whether the part is rigid or flexible/compliant and whether it 

includes hanging subcomponents like a wiring harness or a bundle of hoses. 

• Item grip points. Where and how the part can be gripped, including the choice of 

end effector needed and how small and/or forgiving the grip points are. 
13

• Item size. Whether the part is very large and heavy, which introduces a number of 

constraints. 

• Container type. The nature of the container from which parts are picked. For 

example, a bin could be a cardboard shipping container, a pallet, a rigid plastic 

bin, or a segmented kit. 

• Container interference. If the container has a height or shape that occludes views 

of the parts and interferes with parts as they move out of the container. 

• Container arrangement. How the parts are arranged for the robot to pick up. The 

parts could be individually presented, either flat on an assembly line or in some 

more general configuration such as in a fixed position kit, where each item has a 

nesting spot, in a bin or cart in a loose but consistent orientation, or in a bin in a 

random pile. 

• Container motion and transport. How the container is presented – is it on a cart or 

other conveyor, is it stable, is it moving? 

• Container internal stability. Whether the parts are stable within the container and 

whether they are likely to move or settle as they are being picked. 

• Pick destination. Where the material or part is to be placed after the pick, whether 

in another bin, in a kit, in a chuck or fixture, or joined into an assembly. 

• Pick destination rigidity. How stable and rigid the pick destination is, which 

influences whether perception is needed to check the pick destination before 

moving a part to it. 

• Pick gripper stability/perception. If the part maintains its pose in the gripper, and 

if the sensor system can sense the gripped part. 

• Pick destination planning. Whether the placement of the part requires 

considerable path planning to carry out, such as stacking containers or inserting a 

part inside a varying assembly, and whether the required motions avoid collisions 

and singular configurations. 

• Environmental conditions. Lighting, noise, vibration, background motion, EM 

interference, humidity, temperature, building/sensor mount point stability, and all 

other factors external to the task itself. 

While the group did not consider at length domain-specific applications and 

scenarios, there were a few observations about common situations. First was the 

ubiquity of objects shrink-wrapped for transport, which presents particular issues in 

sensing the objects and in determining safe grip points. Second was the palletizing of 
14

large products or bulky items, where the stacking of multiple layers of parts means 

that layers can drift from rigid positions and require sensing to detect offsets. 

Once the group had listed a number of dimensions for the problem, it identified a 

number of requirements for the general task. 

• Correctness of part identification. Can the sensor correctly identify parts? 

• Accuracy of part location. Can a sensor system accurately locate an object? What 

is the quality of the sensed pose? 

• Correctness/accuracy of grip point identification. Are the grip points on the object 

properly and accuracy located? Does the object need to be re-gripped? 

• Accuracy of part placement. Can the robot/sensor system accurately place the 

object? 

• Calibration of robot/sensor system. Can the sensor be easily and quickly 

calibrated to the robot? 

• Anomaly handling. Can the sensor detect defective parts or anomalies in the 

container position? 

• Speed/frequency of sensing. How quickly does the sensor system operate? 

• Number/reality of candidates. Given a scene with multiple possible objects, how 

many candidates does the sensor system perceive, and how many of those 

candidates are false positives. How many objects does it miss? 

An additional sensor issue identified was the relative importance of contact and noncontact sensing – what is the importance of touch and force sensing, and compliant 

grippers, relative to vision sensors? 

V. Refinement of Automated Guided Vehicle (AGV) scenarios

In refining the AGV scenarios, the basic task was defined as the use of an autonomous 

vehicle to move parts or materials from place to place inside a facility. The discussion 

focused on the basic tasks of navigation, responding to guiding signals in the facility, 

avoiding obstacles and pedestrians, and the ability of the vehicles to carry different types 

of loads. The AGV discussion focused more on the capabilities of AGVs rather than on 

categorizing the aspects of the task. The capabilities considered were: 

• Localization. How to localize an AGV on the shop floor. 

• Navigation. Planning and executing a path through the facility. 
15

• Scene mapping. How to use 2D or 3D imaging to map a scene for navigation or 

localization. 

• Reading signposts. Ability to locate and interpret signposts, and to handle them 

when the signs are blocked or obscured. 

• Responding to visual signals. Ability to perceive and respond to signals from a 

human operator. 

• Detection and avoidance of obstacles. Ability to perceive and avoid obstacles, 

moving or static, including pedestrians and other AGVs. 

• Nature of AGV. The type of AGV –a forklift, an AGV, an AGC (autonomously 

guided cart), or some other form of transport. 

• Nature of cargo. What the AGV can transport and in what volume or weight. 

• Environmental conditions. Lighting, noise, vibration, background motion, 

electromagnetic interference, humidity, temperature, floor condition and all other 

factors external to the task itself. 

The requirements discussed were: 

• Safety. Can the AGV operate safety without damage to humans, self or other 

elements of plant? 

• Speed. How fast can the AGV operate and safely manage point-to-point delivery? 

• Gripping requirements: Levels of difficulty? Loading and unloading speed. How 

quickly can the AGV be loaded and unloaded? 

VI. Requirements for scenarios and sensor components

At the end of the workshop, a discussion was started, but not completed, on how to 

properly organize a test for evaluating data from range sensors. The discussion started 

with requirements for possible artifacts or targets, issues of resolution and accuracy, and 

the nature of an object’s visual characteristics such as reflectance, emissivity and selfocclusion. The latter was defined as the characteristic of an object to hinder types of 

sensors, such as triangulation sensors, that require simultaneous views from two 

directions. This discussion was short and few conclusions were reached during the 

workshop. A decision made to pursue the issue post-workshop in follow-up discussions. 

VII. Summary 

The Dynamic Perception workshop was intended to further the development of 

standardized, reproducible, and portable test methods that can advance the technology of 
16

sensors and perception systems for new, flexible robotic and automation applications in 

manufacturing. The purpose of the test methods is to provide demonstrations and metrics 

for sensor performance that manufacturing firms can use to evaluate sensor-based 

products for deployment on the factory floor. 

A central issue is finding universal, abstract test methods that are representative of 

factory floor conditions in smaller, summary formats, so that success in the test method 

predicts success in the factory. Manufacturing is a vast and complex domain, difficult to 

capture in limited scope test methods, and to advance robot and sensor technology it 

would be useful to identify and prioritize scenarios in manufacturing that can be 

translated into effective test methods. To this end, the workshop addressed four questions 

about advancing the use of perception systems in manufacturing robotics and automation: 

1) Are there sets of manufacturing scenarios or tasks that can be identified as candidates 

for employing sensing to advance robotic capabilities? 

2) Which of these scenarios or tasks are the low hanging fruit that can have a near-term 

impact? 

3) What expanded capabilities and performance are needed to advance perception 

systems in manufacturing robotics? 

4) What are essential metrics for evaluating perception systems in these scenarios? 

The workshop discussions carefully addressed questions 1, 2 and 4, with question 3 on 

expanded capabilities implicitly covered by requirements developed in answers to the 

other questions. 

A) The participants endorsed using a framework along the lines of the methodology 

used in the US&R standards project. In that project, test methods and standards 

evolve thorough strong communication and feedback among NIST and industry 

personnel. Rather than a fixed set of artifacts and test procedures, the test methods 

are based on flexible, adaptable test arenas and techniques that can be modified as 

industry participants advance the technology, and can be used to model a wide 

range of domain scenarios. This comes from the expectation that the technology is 

in a period of rapid development and broadening applications, and fixed test 

methods will eventually become obsolete. 

B) The participants generally identified material handling tasks as the best first 

priority for candidate scenarios. During the initial phase in which participants 

individually listed possible scenarios, only one participant mentioned part 

processing such as machining, welding, cutting and painting while one other 

mentioned part joining. Other participants listed variations on the perception of 

objects for a broad range of picking and placing operations, and the open 

discussion centered on defining and refining pick and place scenarios. We 

concluded from this that perception during transformative operations, like cutting, 

welding or milling, is a lower priority at this time, and that initial success at 

perceiving objects for grasping and placement would be a natural first step to 

loading and handling objects during processing. 
17

C) The participants spent the greatest time on characterizing pick and place 

operations that require perception systems to find, identify, locate precisely and 

assist in grasping parts and objects under very general conditions. The workshop 

produced 18 characteristics, as documented in section IV, to refine the definition 

of a pick and place operation. In short, the characteristics covered the nature of 

the item to be picked, the container from which the part is picked, and the 

destination. For example, the standard bin-picking operation would be defined as 

picking from a bin with many parts and placing the part in a known orientation at 

the destination. In the most general case, perception is required to locate and 

grasp the part, find obstacles as the part is moved to a new position, ensure the 

part is properly grasped, and analyze where the part is to be placed. 

D) The 18 characteristics defined for pick and place operations cover considerable 

variations and do not narrow the pick and place scenario to concrete cases. The 

question is how to develop a sequence of concrete scenarios that require an 

increasing complexity of sensor and perception performance, advancing the 

technology step by step. In discussions during and following the workshop, the 

following two scenarios gained some attention. 

a) Managing part shipments as they arrive (or depart) a factory, requiring 

perception of boxed and shrink-wrapped objects. 

b) Kit loading and unloading as defined in Section IV.

The first scenario, managing shipping packages, has the advantage of being a 

limited but complete domain. Packages are typically of rectangular shape and of 

fixed reflectance patterns (cardboard or plastic, with tape and labels), and they are 

present in fixed orientations. The scenario has room for evolution as the nature of 

the packages can become more complex over time, and the scenario can lead to 

opening packages to unload, or loading packages. 

The second, kit loading and unloading, has the advantage of offering welldefined, transportable test artifacts. A manufacturing assembly challenge might 

consist of a suitcase-style case that folds open with hinges that allow the two sides 

to be disconnected. In one half would be a set of parts of specified sizes and 

shapes in holders designed for each. The other half would be a bin. There could 

also be a mat with outlined spots for each part. 

The challenge kit would support a number of tasks. All the parts could be dumped 

in the bin side and then picked and placed onto the kit side in the designated 

locations. Alternatively, the kit side could be unloaded part by part into the bin. 

For evaluating robot precision in grasping, the kit side could be unloaded onto the 

mat to match the outlines (there are no supporting sides to guide a part.) The parts 

could be varied to provide both a recognition task and an orientation and gripping 

task, and the parts could vary in difficulty of recognition and gripping. For mobile 

manipulators, the bin and kit could be separated. The parts could be designed with 
18

different geometries, such as prismatic, cylindrical, or ovoid, so that some are 

easier for pose calculations and some are harder. For single-arm robot systems, 

the assembly should either have, or come with, a stable base to build on. 

Beyond simple parts handling, the kit could be assembled with or without human 

help. The parts could be designed for various assembly operations, such as peg-inhole with gravity to hold them in place, screw threads, etc., and with different 

levels of difficulty. Increased dimensional tolerance could intentionally be added 

to the base parts to make assembly easier, but also to prevent the robot using dead 

reckoning for assembly. Over time, the kits could be made more complex. 

E) With less time dedicated to perception for AGV tasks, the participants came up 

with a shorter list of characteristics and requirements than for manipulation, and 

the list should be reviewed for expansion. The list emphasized AGV navigation, 

cargo capabilities, performance in differing environments, and response to visual 

signals from signs and humans. The participants did not develop concrete 

scenarios. 

Acknowledgments 

We wish to thank Jeff Burnstein for the wonderful introductory words and overall 

support for the workshop, and Jeff Fryman for the workshop arrangements including the 

conference room. We thank William Townsend from Barrett Technology Inc., Otto 

Geiseman from Automated Precision Inc., Mike Boudreau from Technology Brewing 

Corporation, Joyce Guthrie from United States Postal Service, Jane Shi and James Wells 

from GM, Gino Mastrippolito from US Army, German Holguin from Purdue University, 

and Brian McMorris from SICK all for their excellent ideas and fruitful discussions that 

contributed to this report. 
19

Appendix 

A. Call for Participation 

Dynamic Perception Workshop 

Requirements and Standards for Advanced Manufacturing 

Users, developers, and integrators of advanced sensory and perception systems for 

manufacturing have the following common interests: Users need to have full 

confidence in the capabilities of the systems before investing in them. Developers 

need to understand the user requirements in order to develop saleable products. 

Integrators need a common framework to be able to select the best products for their 

application and know they will work well together. To meet these needs, standard 

metrics, evaluation processes, and calibration methods are vital. Existing robot standards 

cover topics including terminology, safety protocols, and performance evaluation for 

robot manipulators and vehicles, but there are no corresponding standards for sensing 

systems used in manufacturing. Integration of sensors into robotic systems will help 

manufacturers attain higher levels of adaptability and reliability in their operations. We 

thereby invite interested parties to participate in defining these standards. 

The issues for workshop discussion include: 

• Are there sets of manufacturing scenarios or tasks that can be identified as candidates 

for employing sensing to advance robotic capabilities? 

• Which of these manufacturing tasks may be the “low hanging fruit” in terms of 

perception system applications, i.e., where current or near-term perception technology, 

metrics, and standards can readily make an impact? 

• What expanded capabilities, enhanced performance, and new information are needed 

from sensor and perception systems to advance manufacturing robotics? 

• What are the essential metrics for the sensory and perception systems (e.g., precision, 

response time, weights, working volumes, lighting requirements)? 

B. Background Presentations 

B.1 Measuring the Performance of Response Robots 

Presentation on US&R program given by Elena Messina. 
20
21
22
23
24
25
26

B.2 Dynamic Perception Workshop 

Introductory presentations on workshop purpose and background. 
27
28
29
30
31
32