NISTIR 7876

Technology Readiness Levels for

Randomized Bin Picking

Performance Metrics for Intelligent Systems 

(PerMIS) 2012 Workshop Special Session

Workshop Summary and Proceedings 

Jeremy Marvel

Roger Eastman 

Geraldine Cheok

Kamel Saidi

Tsai Hong

Elena Messina 

http://dx.doi.org/10.6028/NIST.IR.7876
NISTIR 7876

Technology Readiness Levels for 

Randomized Bin Picking

Performance Metrics for Intelligent Systems 

(PerMIS) 2012 Workshop Special Session

Workshop Summary and Proceedings

Jeremy Marvel

Geraldine Cheok

Kamel Saidi

Tsai Hong

Elena Messina

Intelligent Systems Division

Engineering Laboratory

Roger Eastman

Loyola University Maryland

http://dx.doi.org/10.6028/NIST.IR.7876

August 2012

U.S. Department of Commerce 

Rebecca Blank, Acting Secretary

National Institute of Standards and Technology 

Patrick D. Gallagher, Under Secretary of Commerce for Standards and Technology and Director 
Technology Readiness Levels for Randomized Bin Picking

Performance Metrics for Intelligent Systems (PerMIS) 2012 Workshop 

Special Session

Workshop Summary and Proceedings

Session Organizers

Jeremy Marvel, National Institute of Standards and Technology (NIST), University of Maryland

Roger Eastman, Loyola University Maryland

Geraldine Cheok, NIST

Kamel Saidi, NIST

Tsai Hong, NIST

Elena Messina, NIST

Expert Panel (in alphabetical order)

Bob Bollinger, Procter & Gamble (P&G)

Paul Evans, Southwest Research Institute (SwRI)

Joyce Guthrie, United States Postal Service (USPS)

Eric Hershberger, Cognex

Carlos Martinez, ASEA Brown Boveri (ABB)

Karen McNamara, National Aeronautics and Space Administration (NASA)

James Wells, General Motors (GM)

Contributors (in alphabetical order)

Stephen Balakirsky, NIST

Joe Falco, NIST

Frank Maslar, Ford

Mark Rice, Maritime Applied Physics

Brian Weiss, NIST

Session Date: 21 March, 2012
Disclaimer

Certain trade names and company products are mentioned in the text or identified in certain illustrations. 

In no case does such an identification imply recommendation or endorsement by the National Institute of 

Standards and Technology (NIST), nor does it imply that the products are necessarily the best available for 

the purpose.

The opinions expressed in this Workshop Report are those of the workshop participants and are not the 

official opinions of NIST.
Background

Trends in manufacturing technologies and initiatives are placing increased importance on the integration of 

robot technologies in manufacturing facilities. Despite years of considerable progress in improved sensing 

capabilities, 3D pose estimation systems, and vision-guided robotics, a limiting factor in realizing full 

system automation is the challenge of acquiring components from a randomized bin of parts, i.e., bin 

picking. A large part of this challenge is identifying and assessing technologies that enable randomized bin 

picking application solutions. Various metrics can be defined and applied to the bin picking problem, but 

no single metric captures the complexity of a given application, nor are all metrics applicable to all bin 

picking problems. In order to address these issues, one must understand the challenges facing the 

development of generalized bin picking solutions and identify methods for communicating the maturity of 

technologies integral to such solutions.

One can divide the metrics of bin picking into two generalized categories: metrics that define bin picking 

composition, and metrics for bin picking performance. The metrics for bin picking set-up attempt to 

capture the complexity of a given application domain, and include measurable factors such as scene 

complexity, degrees of freedom for part poses, and part feature strength (i.e., how distinct and easily 

identifiable the image features on the parts being acquired are for the task of pose estimation). In contrast, 

performance metrics describe the capabilities of a specific instance of a robot configuration. Such metrics 

include picking speed, throughput, and pick success rate.

A significant gap exists between knowing how complex a given bin picking application is and knowing 

how well a given solution performs. Specifically, knowing what technologies to actually implement and 

the limitations of their capabilities is an issue not addressable by either set of metrics. Matching a user’s 

needs to vendor system capabilities is a challenge that involves identifying all possible subcomponents of 

potential systems (e.g., sensors, algorithms, manipulators, and grippers), assessing a given technology’s 

state of development reliability, and then deciding to move forward with technologies determined to be 

sufficient for the task. One possible structure for this decision-making process is the Technology 

Readiness Level (TRL) hierarchy. Although not designed with manufacturing applications in mind, the 

TRL structure provides a rigid (though somewhat subjective), tiered ranking of technology maturities prior 

to mission integration.

The question as to whether TRLs are appropriate for manufacturing technologies such as bin picking is still 

unanswered, but there is consensus that some form of structured hierarchy should be developed that will 

address the issues of communicating and encapsulating technology readiness. An important first step 

toward this goal is the identification and documentation of the technological and practical challenges 

associated with randomized bin picking. From that point, a full understanding of the TRL structure’s 

limitations, alternatives, and standardization efforts would enable the development of a technology maturity 

assessment process for industrial and manufacturing applications.

I. Introduction

The special session on Technology Readiness Levels (TRLs) for Randomized Bin Picking was held during 

the morning session of the 2012 Performance Metrics for Intelligent Systems (PerMIS) workshop, 21 

March, 2012. The stated objective of the special session was to discuss the state of the art and metrics of 

TRLs for bin picking solutions that are robust against random pose and part variations (see Appendix A.1

for the advertised session description). We sought to address maturity indicators for overcoming 

challenging factors in bin picking applications, including shape variations, pose and orientation uncertainty, 

weak or no distinguishing image features, and limited grasping options.

The special session was given a semi-structured open discussion format that featured a panel of experts 

from academia, industry, and government in order to solicit information from both the panel and the 

audience regarding the utility and assessment of bin picking technology maturity (see Appendix A.2 for the 
session agenda handed out at the workshop). Each panel member was identified as an expert in his or her 

respective field regarding the use or development of bin picking applications or TRLs.

Manufacturing technologies have witnessed a veritable boom in robot integration and improved sensing 

modalities for safety and task automation. Worldwide manufacturing initiatives stress the integration of 

robot technologies in modernized manufacturing facilities and push the boundaries of both productivity and 

innovation in an ever-increasingly competitive market. Despite years of considerable progress in 3D pose 

estimation systems and vision-guided robotics, one of the greatest challenges to manufacturing automation 

is the task of component acquisition from a randomized bin of parts.

The principal goal of the special session was to establish a common understanding of how to match the 

robotic bin picking perception requirements of manufacturers against the current capabilities of vendor 

systems. Further, we intended to determine the best mechanisms for advancing the capabilities and, 

therefore, the greater deployment of robotic bin picking. This could be through an advanced perception 

TRL framework or other common set of metrics and evaluation criteria that can be developed by the user, 

vendor, research, and government communities through a consensus standardization process.

We discussed the requirements and processes involved with the grading of different levels of bin picking 

difficulty, and the feasibility of establishing a set of standardized artifacts for bin picking solution 

validation. Additional topics of discussion included the challenges inhibiting solution integration and 

opportunities for advancement in next-generation manufacturing environments. 

II. Summary of the Workshop Presentations

II.1 Technology Readiness Levels

The first presentation (the slides for which are given in Appendix B.1) was entitled, “NASA Technology 

Readiness Levels: Relevance to Manufacturing,” and was presented by Karen McNamara from NASA 

Johnson Space Center. This talk described the structure and intent of TRLs at NASA, which is where the 

concept was originally developed, and how other U.S. and international agencies utilized similar 

technology maturity assessment and reporting schemas. 

The presentation began with a broad overview of TRLs, indicating that they are predominantly used by 

domestic and international agencies for evaluating the maturity of technologies for aerospace and 

aeronautic interests. Other agencies and users, however, have adapted the TRL language to better suit their 

own technologies, production patterns, or management structures. Across domains, the TRL structure 

provides risk assessment metrics for inserting new technologies into new missions or mission elements, and 

is seen as a critical communication tool for the agency when conversing with partners, suppliers, and 

customers. The TRL structure is most commonly a 9-stage hierarchy as shown in Table 1.

Table 1: NASA’s Technology Readiness Level Definition Summaries

TRL Summary Description

1 Basic principles observed and reported

2 Technology concept and/or application formulated

3 Analytical and experimental function study, and/or characteristic proof of concept

4 Component and/or breadboard validation in a laboratory setting

5 Component and/or brassboard (functionality and approximate physical configuration of 

operational product) validation in a relevant environment

6 System/subsystem model or prototype demonstrated in a relevant environment

7 System prototype demonstrated in an operational environment

8 Actual system completed and flight qualified through test and demonstration

9 Actual system flight proven through successful mission operations
Despite the wide utilization of TRLs in aerospace and aeronautics, there does not exist a standardized TRL 

structure or implementation. As such, the TRL for a given technology can change as it is applied to 

different agencies, environment, intended use, or even assessors within the same agency. The lack of 

clearly defined exit criteria for higher TRLs in conjunction with the existing vague (and occasionally 

conflicting) guidelines for assessing TRLs further complicates their use.

NASA’s utilization of TRLs integrates a basic threshold, where achieving a TRL of 6 is generally 

considered a prerequisite for the integration of new technologies. However, throughout the presentation it 

was stressed that the cost, scheduling, and effort required to transition from one TRL to the next is neither 

linear nor proportionate, with transitioning between TRLs within classification groups (e.g., preliminary 

research phases in TRLs 2 and 3) generally being easier than transitioning between classification groups 

(e.g., going from preliminary research to prototyping). It was also presented that the TRL structure is just

one of many factors in NASA’s decision process, with integral decision-making processes being reliant on 

Key Decision Points (KDPs) to determine the readiness of a program or project for the advancement to the 

next phase.

The presentation concluded with a discussion of the difficulties in applying technology readiness to the 

capability readiness needs of manufacturing processes. Specifically, the challenge lies not in applying the 

TRL structure to technologies used in manufacturing, but rather in the factors important to manufacturing 

that are not addressed by TRLs. These factors include:

• Throughput

• Profit margins

• Market needs

• Ease of labor and implementation

As an alternative, Manufacturing Readiness Levels (MRLs) developed by the U.S. Department of Defense 

(DOD) were presented. MRLs are used to quantitatively assess the maturity of technology components 

from a manufacturing perspective, and are utilized by decision makers to determine the risks associated 

with bringing products to the production phase.

II.2 Bin Picking

The second presentation (slides given in Appendix B.2) was entitled, “Opportunities in Bin Picking,” and 

was presented by Jeremy Marvel from NIST’s Intelligent Systems Division. This talk gave a generalized 

overview of the bin picking problem and the various metrics by which bin picking applications and their 

respective solutions can be gauged.

The presentation started with a description of the three phases of a bin “pick”: 1) the isolation of an object 

from the background image, 2) the calculation of the object’s pose relative to the sensor or robot, and 3) 

generating a path trajectory through which the robot is moved toward the object in order to grasp it. 

Illustrative examples of bin picking were then presented, as well as a description of many expected benefits 

for successfully integrating robot bin picking on a manufacturing line. Some of the benefits achieved with 

such automated solutions include:

• Increased safety and reduced injury potential of workers

• Around the clock production

• Improved production throughput

• Increased flexibility in automation due to scalable and modular components

• Reduced impact of labor shortages and costs

• Work cell size reduction

• Improved quality control

The talk then proceeded to discuss the various metrics by which bin picking solutions can be assessed, and 

the problem domains ranked in terms of difficulty. Three common performance metrics for bin picking are 

speed, efficiency, and accuracy. Speed may refer to the time required to locate or acquire individual parts 

from a bin separately, or the combined metric in the form of bandwidth (or the number of picks per a given 

time frame). Efficiency is measured in terms of time utility (e.g., the time spent searching versus the time 
spent picking up the object), grasping quality and success rates, and trajectory optimization. Accuracy 

refers to the quantifiable measurement error in object recognition and pose estimation.

The difficulty of a given bin picking task has classically been defined by the uncertainties in the appearance 

and position of objects presented to a sensor, though a growing trend has been to apply perceived levels of 

expertise and experience (similar to the maturity levels discussed in the first presentation) of integrated 

solutions in industry. As the library of bin picking literature grows, however, so does the number of 

different criteria for scaling bin picking difficulty. These criteria include, but are not limited to:

• Scene or image complexity

• Degrees of freedom of object pose/presentation

• Part location or orientation relative to the robot configuration

• Part variations such as flashings or raw material handling

• Ease by which image features can be uniquely identified

• Part rigidity

• Part occlusions, overlap, or interlock

The presentation concluded with a discussion of the anticipated complexities involved with integrating 

solutions for bin picking into the manufacturing process. Outside of the level of effort a user should expect 

to commit to integrating solutions (which varies according to the source of the solutions, e.g., whether they 

are internally developed or purchased from a 3rd party vendor), there are three primary challenge domains 

that may complicate the integration of bin picking solutions. The first domain, sensing, includes 

difficulties in both sensing and algorithm development, and also includes aspects of process and cell 

optimization. The various sub-challenges include:

• Lighting variations due to surface reflectivity, shadows, and transparency

• Weak, nonexistent, or inconsistent image features

• Shape and surface variations incurred during the manufacturing process

• Bin position uncertainty and appearance variations caused by damage

The second challenge domain reflects hardware issues, including the robot’s capabilities, gripping 

limitations, and part considerations for acquisition (such as weight, fragility, and any number of ways in 

which the parts could be connected or interlocked). The third, and arguably most difficult, challenge 

domain to overcome when implementing bin picking covers the costs of implementation of a solution in the 

real world. This domain’s challenges include the practical issues of cost (both in terms of money and time 

when introducing a new part or process), problem and solution uniqueness (specifically, what solution 

components can be reused and how well a given solution fits the specific bin picking problem), and user 

understanding and awareness. This last element is frequently characterized by users either not knowing 

what solutions are available or having unrealistic expectations of the ease by which a robot can perform a 

given task.

III. Summary of the Panel Discussion

A special panel of experts from government, industry, and academia was convened to assess the challenges 

and discuss technologies for bin picking and to determine if the development of a method such as TRLs for 

bin picking would be helpful to the user community. Alphabetically, the panel members were:

• Bob Bollinger, Procter & Gamble (P&G)

• Paul Evans, Southwest Research Institute (SwRI)

• Joyce Guthrie, United States Postal Service (USPS)

• Eric Hershberger, Cognex

• Carlos Martinez, ASEA Brown Boveri (ABB)

• Karen McNamara, National Aeronautics and Space Administration (NASA)

• James Wells, General Motors (GM)

Frank Maslar from Ford was also invited to the panel, but was not able to attend the special session. His 

input into the topics of discussion is included in Appendix C.
Roger Eastman from Loyola University, Maryland, moderated the two-hour panel discussion and prompted 

discussion based on topics relating to the development, utilization, and assessment of bin picking solutions.

The discussion began with an effort to determine categorical classification from a user’s perspective of bin 

picking. From a manufacturing perspective, there are three distinct and readily identifiable phases for 

which bin picking will be employed based on the stage of production in which the objects are being picked. 

As the manufacturing process nears a finished product, the level of care required to prevent damage 

increases. Early stages, for example, typically require the acquisition of raw (unfinished) materials 

frequently presented in randomized bins. In contrast, in-process and finished components require 

increasing levels of fixturing to prevent damage that would affect the functionality or aesthetics of the 

parts. The bin picking process varies accordingly based on the shipping or presentation method.

Improved inter-process component transfer is an impetus for production optimization, and the ability to 

handle material in a lean fashion is what is driving bin picking. One panelist, for instance, described the 

production process as a series of transformations in which the components are transferred between robots, 

hoppers, bins, conveyor belts, dunnage, and so on. Intermediate transformation steps, e.g., moving parts 

from a hopper to a conveyor belt to be acquired by delta robots, add cost and complexity to the 

manufacturing process. The need to handle a diversity of objects where the changes occur at frequent 

intervals adds complexity to the task. Therefore, the ability to handle several product types (e.g., different 

size objects, objects made of different materials) with little to no downtime (e.g., change gripper/end 

effector, re-programming) is desirable. Additionally, the ability to handle parts as they would naturally be 

presented in an unstructured form would improve process efficiency.

The panel members considered structured bin picking to be a solved problem which can be addressed by 

simple matrix handling (i.e., position-controlled moves to a grid of pre-determined coordinates for the 

acquisition of parts). However, the “Holy Grail" of bin picking, i.e., picking a particular object from a bin 

containing various objects in random poses, is a very difficult and complex problem and is not solved. 

Some solution providers have enacted policies to decline requests for unstructured bin picking. Despite 

many years of research in algorithms, robotics, and sensor systems, no unstructured bin picking solution 

has been developed that is reliable, cost effective, or widely applicable. Even within classes of parts (e.g., 

plastic container caps), the required flexibility of bin picking solutions has not materialized, and the 

capacity to compensate for product line changes requires hard automation (i.e., large, highly-fixtured, partspecific feeder and handler systems). The issue is further complicated by cases where such hard 

automation is impossible due to large variances in part shape and size. The panel concluded that, although 

solutions do not yet exist, incremental solutions were acceptable and solutions for the various bin picking 

problems could be more tractable if it were sub-categorized (e.g., bin picking for small versus large parts). 

In contrast with the hard automation solutions, the cost for robot bin picking solutions is not driven by the 

cost of the robot. Rather, it is the cost of integrating the robot into the manufacturing process that presents 

the largest hurdle. Specifically, handling safety and process-specific ancillary assembly line system 

requirements contribute the most to the overall cost of the system, and thus hinder cost efficiency and 

flexibility. The actual cost of the robot is comparatively small, as is the impact of the robot on the 

complexity of the bin picking solution. Though different bin picking classifications may require different 

robots, the control, repeatability, and reliability of robots in general are considered largely solved. 

Similarly, the gripping of the objects for process utilization, an independent component given the parts 

being acquired and subsequent utilization, is also considered addressable given current technologies.

If the physical aspects of the bin picking problem is considered a solved aspect, then what is the greatest 

hindrance? Perception and the associated sensing technologies of the various components in the 

manufacturing setup are widely seen as the limiting factor in the improvement of bin picking. For the 

USPS, perception of the material as it comes in is their biggest problem. Additionally, the bin itself 

provides a challenge in a number of ways. Identifying variations of the bin in terms of placement, shape, 

and condition (e.g., due to damage) adds complexity to both the part location and trajectory generation. 

The ability to recover from collisions, e.g., with the bin itself, is a desirable feature. For some parts, 

solutions may exist for general gripping or grasping, but perception is necessary if the objective is to mate 

one part with another. Perception for diagnostic or maintenance is also an important issue. For example, 
the system needs the ability to sense when a bin is almost empty so that the operator can bring in the next 

bin. Also, perception is needed to reliably determine if a bin is completely empty - a letter left behind in a 

bin is not a desired outcome for the USPS (or their clients), and for a manufacturer, missed components 

adds to waste and increases costs. The panelists also stated a desire for the capability of robots to work 

among or alongside humans – a capability which requires perception for the safety of the humans.

Some suggested solutions to the perception problem were the integration of software from multiple vendors 

and integration or fusion of hardware from multiple sensors. Another suggestion was the design of a bin 

that is robot friendly instead of human friendly.

In the second part of the panel discussion, the panelists were asked if the development of TRLs or some 

other evaluation method for bin picking would be helpful. A question raised by one of the panelists was, 

“Is there another standard [process] besides TRL?” Some of the larger manufacturers have an internal 

process similar to the TRL to evaluate technologies. GM has internal processes for evaluating technology 

readiness which involves management. They have two different types of technologies that undergo such 

review: technologies that are required (e.g., things that have to go into a new car design), and technologies 

that improve an existing process (e.g., a process that may be replaced by a technology). However, these 

internal processes are usually proprietary. Someone in the audience suggested that standard test methods 

could be used, instead of TRLs, to evaluate the technologies.

Karen McNamara reiterated that TRLs are only one input into the decision process for NASA. There could 

be separate or individual TRLs developed for the technologies or sub-systems (e.g., vision system, robot, or 

gripper). Then Manufacturing Readiness Levels (MRLs) could be used to evaluate a capability. In this 

case, the capability would be bin picking which requires the integration of the vision system, robot, and 

gripper. In the review process, management is involved and cost estimation is taken into account. The 

review process can include a de-scoping process, e.g., “the process can only do X and Y and not Z – is

there value in continuing?”

The panelists also liked the idea of challenges as an alternate method to evolve and evaluate bin picking

solutions. The challenge, however, should not be too specific so as to preclude potential solutions. For 

example, there could be challenges open to companies only, or open only to student researchers.

Potential metrics for bin picking include: speed, throughput, robustness, and flexibility. Flexibility, or 

agility, is the ability to re-task a robot within a short period of time. That is, a flexible robot working with 

Product A can, within a short period of time, be re-tasked to work with Product B. Flexibility could also 

include re-purposing, which evaluates how long it takes to have the robot perform a different task, and what 

level of skill is required to make this transition. It was suggested that trade organizations may have data 

performance on aggregated solutions.

At the conclusion of the panel session, the panelists and contributing audience members determined that the 

logical next steps were to develop test methods and to document available technologies and their 

capabilities.

IV. Workshop Summary and Action Items

Overall, the Special Session presentations and panel discussion concentrated on the definition of terms both 

for TRLs and for bin picking. The general term “bin picking,” when used for manufacturing logistics and 

assembly, covers a great number of applications and conditions. To make progress in advancing this 

automation capability will require carefully defining useful subcategories of the larger problem, identifying 

the technology limitations that hold back success in each subcategory, and undertaking efforts to address 

these limitations. 

A primary question of the session was whether a joint effort between NIST and the relevant community to 

advance bin picking would be worthwhile. This question was put to the panel and audience, and the 

consensus was yes, the effort is worthwhile. There is no universal, flexible solution to bin picking, and a 
community effort to identify and advance technologies would have commercial benefits as stated during 

the panel discussion. The action item from this conclusion was to draft this report and distribute it for 

continued discussion. Following the meeting, panelists were asked to provide brief synopses about their 

perspectives on the current state of the art and challenges of bin picking. Their responses are shared in 

Appendix C.

Action Item #1: Draft and distribute an initial report outlining the technological and pragmatic 

challenges of bin picking.

A second key question was whether the TRL framework is useful for manufacturing applications. No clear 

conclusion was reached here. The general idea of a TRL framework is clearly useful for identifying when 

technologies are ready for use in manufacturing applications, but the specific terminology of the 

NASA/DOD TRL documents may not be directly useful. A key point made by Karen McNamara was to 

distinguish between technologies and capabilities. Bin picking may be better labeled as industrial capability 

that can be implemented by a number of technologies, rather than a single technology itself. A single 

solution to bin picking will have a number of subcomponent technologies, each of which will have 

readiness levels as applied to different industries and stages of manufacturing. This leads to a complex 

matrix of TRL levels and no conclusion was made on whether this is practical or useful. 

Another issue in applying the NASA TRL framework is that NASA is most concerned about “one-off” 

technologies that enable a mission – once that level is reached, the technology has reached its needed 

maturity. For manufacturing applications that are continuing, not one-off, technologies will continue to 

evolve and cost optimization is more important. Being mission-proven does not mean the technology will 

be commercially cost-effective nor competitive, or continue to be so. TRLs are only part of the decision 

process.

TRLs do give clear labels to communicate how far along a technology is, and whether it is ready for the 

factory floor. TRLs state commonly understood concepts, such as whether a technology has been 

demonstrated in prototype, whether it has been demonstrated successfully as a subcomponent, whether it 

has be demonstrated as part of a complete, successful system, and also whether these demonstrations are in 

artificial lab environments or in mission environments. These concepts are useful in discussing bin picking 

both to tease apart the critical subcomponents and the complexity of the conditions under which they have 

been tested. An action item was to look further into other variations of TRLs, such as Manufacturing 

Readiness Levels and pending ISO standards for TRLs, to see whether they better applied or gave insight 

into a new form of TRL for complex, intelligent manufacturing systems.

Action Item #2: Explore and document TRL variants, alternatives, and standards efforts, focusing on 

their applicability to next-generation manufacturing systems.

The discussion of TRL terminology led into a consideration of the test methods that would enable the 

labeling of technology as meeting a readiness level. During the discussion of alternatives to TRLs as a 

process, the use of Stage-Gates was mentioned – processes with fixed evaluation procedures that 

determined if a technology or project met a set of tests and deserved to be continued, or in the context of 

this panel discussion, labeled as reaching a particular readiness level. A sense of the discussion was that 

whether or not formal TRL labels were useful for bin picking, or were still under consideration, the 

supporting test methods would be essential and worth developing. Test methods could be developed as 

standards or as challenges. There was general consensus that developing tests and challenges would be a 

good way to proceed.

In developing test methods, one element seemed straightforward, and one element seemed more 

ambiguous. There was general agreement on the requirements of bin picking, and these requirements could 

be turned into test metrics. Bin picking solutions need to be fast in operation, flexible in setup, low in 

damage and residual, adaptable in environmental conditions, and cost-effective. The output variables were 

not difficult to identify, and the related action item is to further identify and define these. 
The input variables for test methods proved harder to define. As mentioned, the term bin picking covers a 

broad set of applications and conditions, and dividing that broad spectrum into useful subcategories is not 

easy. To be useful, a subcategory of bin picking must be manageable in scope, so the conditions for a test 

can be well-defined, and simultaneously general enough to develop a flexible and adaptable solution. There 

was agreement during the discussion on aspects of these input variables – how structured parts are in 

presentation, the variation in part shape and condition, the condition of bins and other background objects, 

lighting, the geometry of parts, and others. But, it was not clear how to use these variables to divide the 

space of subcategories or how finely to refine each variable. An action item was to consider how to 

construct natural and useful subgroupings under these variables.

Action Item #3: Draft potential new readiness level assessment structures for manufacturing systems, and 

perform initial assessment of bin picking subcomponents within these new structures.

The technology maturity assessment shown in Table 2 is informal, and is intended to demonstrate one 

possible way to structure the conversation on the readiness of bin picking technologies. Here, we give 

sample maturity assessments of the state-of-the-art for perception (the capacity for application-specific 

sensing and modeling the environment and parts), grasping (the ability to physically acquire objects 

reliably), movement (the ability to execute trajectories of the robot gripper and grasped parts throughout the 

work cell as instructed), and planning (the ability to generate path and grasp trajectories for part acquisition 

and utilization). The table is not intended to be accurate in its conclusions, only representative of one way 

to apply the TRL concepts to the topic. Here we informally define four specialized Readiness Levels RL1 

through RL4. The actual values in the grid are:

RL1 – Current solutions are working in prototype but highly fragile

RL2 – Current solutions are working consistently but not efficiently 

RL3 – Solved problem for limited commercial applications

RL4 – Solved problem robust for many commercial applications

Table 2: Example readiness level assessment for various subcomponents of the bin picking problem

Informal RL levels for 

subcomponent 

technologies

Uniform rigid parts 

that are compact or of 

limited configurations

Rigid parts with greater 

variation and with more

complex configurations

Flexible parts with 

varying size and 

configuration

Highly structured, fixtured 

arrays of parts

Perception – RL4

Grasping – RL4

Movement – RL4

Planning – RL4

Perception – RL3

Grasping – RL4

Movement – RL4

Planning – RL4

Perception – RL2

Grasping – RL3

Movement – RL3

Planning – RL3

Semi-structured arrays in 

which part placement has 

limited variation

Perception – RL3

Grasping – RL4

Movement – RL4

Planning – RL4

Perception – RL3

Grasping – RL4

Movement – RL4

Planning – RL3

Perception – RL2

Grasping – RL3

Movement – RL3

Planning – RL3

Semi-structure arrays

with greater variation of 

part placement and 

orientation

Perception – RL3

Grasping – RL4

Movement – RL4

Planning – RL4

Perception – RL3

Grasping – RL4

Movement – RL4

Planning – RL3

Perception – RL3

Grasping – RL2

Movement – RL3

Planning – RL2

Fully random bins of 

parts with no constraints 

on part placement aside 

from part configurations

Perception – RL2

Grasping – RL3

Movement – RL3

Planning – RL3

Perception – RL1

Grasping – RL2

Movement – RL3

Planning – RL3

Perception – RL1

Grasping – RL2

Movement – RL2

Planning – RL1

Fully random bins of 

multiple part types (or 

support for multiple bins 

of different parts) with no 

constraints on part 

placement

Perception – RL1

Grasping – RL2

Movement – RL3

Planning – RL2

Perception – RL1

Grasping – RL2

Movement – RL3

Planning – RL2

Perception – RL1

Grasping – RL2

Movement – RL3

Planning – RL2
Appendices

Appendix A. Session Materials

A.1. Session Proposal from the PerMIS Call for Papers

Session Title: Technology Readiness for Randomized Bin Picking Solutions

Session Organizers: Jeremy Marvel (NIST, University of Maryland), Tsai Hong (NIST), Gerry 

Cheok (NIST), and Roger Eastman (Loyola)

Session Description: Although there has been considerable progress in 3D pose estimation systems 

and vision-guided robotics, one of the greatest challenges to manufacturing automation is the task of 

component acquisition from a randomized bin of parts. This special session focuses on the state of the art 

and metrics of technological readiness levels (TRLs) for bin picking solutions robust against random pose 

and part variations. We will address the indications of maturity of approaches for overcoming shape 

variation, pose and orientation uncertainty, weak or no distinguishing image features, and limited grasping 

options. Presenters will discuss the needs and challenges from both users’ and vendors’ perspectives 

regarding bin picking for manufacturing automation. We hope to gain a better understanding of how to 

map the advanced perception needs of manufacturers against the current capabilities of vendors systems, 

and to establish an advanced perception TRL framework. Following the presentations, we will host an 

expert panel discussion consisting of the presenters and session chair.

A.2. Session Agenda

Session Title: Technology Readiness Levels for Randomized Bin Picking Solutions

Session Organizers: Jeremy Marvel (U. of Md/NIST), Tsai Hong (NIST), Gerry Cheok (NIST), Roger 

Eastman (Loyola), and Elena Messina (NIST)

Session Moderator: Roger Eastman

Agenda: Time: 2 1⁄2 hours (150 minutes)

I. Introduction (5 min) – Tsai Hong

II. Presentation (15 min) – Karen McNamara – NASA-Developed TRL Methodology

III. Presentation (15 min) – Jeremy Marvel – Opportunities in Bin Picking

IV. Panel Session I (50 min) – Requirements for successful applications of bin picking

Focus: End user requirements for current and near‐term bin picking applications

Primary participants: End users/manufacturers, integrators

Discussion:

a) What are the current uses of bin picking?

b) What are near-term uses of bin picking that would be useful?

c) What needs to improve in bin picking for wider use?

d) Would bin picking be good for a community investment to advance?

Exercises:

• List and prioritize applications of bin picking 

• List and prioritize challenges holding back greater use of bin picking 

• List ways to categorize bin picking applications
 V. Panel Session II (55 min) – TRL and other processes for advancing bin picking

Focus: Processes to best advance bin picking technology

Primary participants: TRL experts, sensor/perception vendors

Discussion:

a) Does a public TRL process make sense to advance bin picking?

b) What other processes/standards would make sense?

c) What are good metrics for evaluating bin picking solutions?

Exercises:

• Create TRL for specific bin picking technologies

• List alternative processes that could be used

• Create action items for next steps

Previously distributed questions

1. Does your company currently use bin picking, and if so, under what conditions? If you do not use bin 

picking in cases when it would be useful, why not?

2. How would more robust bin picking be useful to your company? Is organizing a way to advance this 

technology a high priority task worthy of the investment of effort by NIST and the industrial 

community?

3. What are the challenges in bin picking? What needs to improve in bin picking in order to take the 

technology to the next level, i.e., make your company more productive or make bin picking more 

effective?

4. What would you need to “accept” a vendor’s claim that their solution works for your bin picking 

requirements? Since bin picking could be a big capital expense, what would make you feel more 

confident that the proposed solution would work for your application? Would TRLs be of use to help 

you decide?

5. Do you think a TRL scale for bin picking would be a good way to judge the readiness of bin picking, 

and to advance the technology? Are there better ways you might suggest or have used as an alternative?

6. Does your company currently use a way to categorize bin picking cases, so as to indicate which are 

reliably solvable and which are not? Would you be willing or able to share it? We have considered 

using characteristics like the shape of the part, the quality of imaging features, the variation and 

randomness of part positioning in the bin among others.

Appendix B. Background Presentation Slides

B.1. NASA Technology Readiness Levels: Relevance to Manufacturing

Presented by Karen McNamara

B.2. Opportunities in Bin Picking

Presented by Jeremy Marvel


Appendix C. State of the Art in Bin Picking Challenges and/or Uses in Current Facilities

As a follow-up to the PerMIS special session, the invited panel members were asked to give an assessment 

of their views on the state of the art in bin picking. The following sections are the panelists’ own opinions 

regarding the current capabilities and challenges of bin picking components and solutions.

C.1. Respondent One:

I think that a lot has been accomplished relative to structured bin picking solutions. There are many 

installed all over the world with highly successful results to date. I have also seen random bin picking 

solutions implemented, and I do not believe they are 100% reliable. These solutions typically can pick up 

90-95% of parts out of a bin. Those 5-10% left in a bin add up over time and make for a lot of scrap and a 

bad bin picking solution. There are many factors that contribute to this problem, the vision portion is 

typically complicated, there are always issues with getting the robot to the correct position without crashing 

into the sides of the bin, and then picking a single part correctly. The vision portion of random bin picking 

is becoming easier with better vision tools, faster processing and simplifying the ways to correlate between 

multiple parts in the field of view. There has been some progress on new robotic grippers that simulate a 

human hand, but they can be very complex and expensive. The problem in my opinion is that we are trying 

to replicate a task that is relatively simple for humans, with a machine that can become a complex solution. 

The simple solution of re-designing the bins or part presentation to get rid of the mechanical constraints and 

make those applications machine friendly, random bin picking could then become a 100% solution.

C.2. Respondent Two:

The following sections list some general observations regarding the state of technology and software that 

contribute toward solutions for randomized bin picking.

Robot:

Industrial robots are robust and proven with an emphasis on safety, speed, accuracy, reach, and payload. 

Recently there has been renewed interest in 7 degree-of-freedom (DOF) manipulators. These designs 

permit higher flexibility for positioning the arm for a given task. This capability becomes more important 

for dealing with unstructured and constrained environments such as bins; it facilitates collision-free path 

planning because of additional flexibility. The industrial robots manufacturers currently offer many 

platforms to choose from that should be usable without any modifications to hardware or software.

Gripper:

Historically, industrial robot grippers have been task-specific; the fingers are engineered to manipulate a 

particular object of interest. Recently, there has been a drive towards general-purpose grippers, capable of 

manipulating many objects. Adaptive grippers for industrial applications are emerging on the market, and 

will offer more options for gripping a variety of parts.

Sensing:

Traditionally, adaptive industrial robotic applications have relied on 2D machine vision to locate 

workpieces. This approach is limited in that structure must be added to the environment in the form of 

fixturing or mono-layering on a conveyor to reduce the pose estimation problem from 6 DOF to 3 DOF (X 

and Y location and one rotation). True 3D sensing is required to avoid this constraint. Various approaches 

are available including structured light, stereo-vision, and time-of-flight sensors, but each of these methods 

has cost, speed, or performance limitations. Work is being conducted to be able to better align the sensing 

technology with the application which will leverage ASTM E57.02 Task Group on Performance Evaluation 

for 6-DOF Pose Measurement. 

Development Framework:

The open-source Robot Operating System (ROS), maintained by Willow Garage, has revolutionized 

research robotics. Its adoption in the four years of its existence has seen exponential growth with currently 

over 2000 open-source software libraries written for the framework. ROS provides a standardized 
messaging protocol, common libraries needed for robotics, specialized development tools and an active 

development community. [Our] recent work to bring the ROS environment to industrial applications offers 

the opportunity to apply some of the research work established in ROS into the industrial application space.

Perception:

The fundamental perception problem is to reliably identify and estimate the 6 DOF pose of the objects of 

interest. Although simple in concept, the implementation requires a pipeline of processing algorithms that 

take the raw 3D point cloud and generate object maps. The pipeline can require noise-filtering, selffiltering, generation of compact octree voxel representations, segmentation of support structures, object 

template matching, and finally pose hypothesizing. New methods are being developed that utilize the rich 

data from RGB-D sensors to provide robust and fast techniques for identifying objects in cluttered scenes. 

Motion Planning and Grasp Planning:

Motion planning consists of both grasp planning and arm trajectory planning. The latter is a mostly solved 

problem on research platforms with regards to developing collision and singularity free paths on the fly. 

Most industrial manipulators do not offer capability. The challenge of grasp planning is still an area of 

active research, especially in the case of complex geometries and general-purpose grippers. If an a priori

knowledge of the parts to be manipulated is available, acceptable grasps may be calculated off line. In some 

cases heuristics may be used for uncertain objects. 

C.3. Respondent Three:

State of the art “bin picking” applications and future challenges:

Robot suppliers and system integrators have been making steady progress in advancing certain classes of 

bin picking applications over the past few years. This is important since many types of processes and 

material handling methods rely on packaging and presenting parts in a random, or semi random oriented 

pile in a container. Frequently, this is the most dense and lowest cost way to ship or transfer material from 

plant to plant or between steps in different processes within a plant. Most of today’s successful

applications exploit the symmetries and similarities of a narrow class of parts that facilitate the use of 

robust vision techniques, workable end effector tooling designs and industrial robot programming that all 

come together for a robust commercial solution. One good resource to learn more about bin picking and 

other robot applications is the Robotics Industries Association (RIA) web site. A good article on today’s 

bin picking solutions can be found on the RIA site at this link:

http://www.robotics.org/content-detail.cfm/Industrial-Robotics-News/The-Pervasive-Relevance-of-BinPicking-in-Nature-and-Business;-2011-Technical-Trends-and-Market-Progression/content_id/3080

Bin picking application solutions rely on machine vision techniques and frequently other types of sensors to 

assess the next, most likely next candidate part to pick out of the top layer of parts. This can range from 

strong 2D vision capability teamed with a separate range sensor to more sophisticated 3D vision techniques 

using multiple sensors. This data can be used to identify and guide the robot’s end effector to engage the 

physical features of the part that allow it to be handled. Most of the current systems are implemented in 

applications that deal with parts that lend themselves to fairly straightforward pick up strategies. Usually 

this enables the part to be handled with suction cups, magnets, or cleverly designed end effectors that 

provide the widest compliance window for grappling the part. There may be other steps involved in the 

picking sequence that prepares the part for orientation and final placement (sometimes precision placement) 

to complete the process. These applications, while successful, tend to be highly engineered and only 

capable under a narrow set of enablers, such as favorable part geometry, strong features that lend 

themselves to existing vision techniques and fairly simple, and workable strategies that allow the robot to 

empty the entire bin. These represent just the “tip of the iceberg” of potential applications that are not 

currently feasible using existing techniques. There are a wide set of conditions under which parts are 

shipped and presented in boxes, totes, bins and baskets where a picking system needs to operate over a 

wider set of conditions. These future applications present a range of challenges and will drive the 

development of new capabilities. This includes vision and part pose identification techniques that operate 

over a wider range of expected sensor to object relationships, end of arm tooling that is more flexible 
(and/or utilizing tool changers, if that is feasible and cycle time allows) and more sophisticated path 

planning techniques to avoid collisions and deal with a variety of container types. 

C.4. Respondent Four:

Random bin picking is a potentially enabling technology for a number of industrial applications. In the 

consumer and packaged goods industry, components are routinely introduced into production lines and 

packaging equipment. The traditional technologies that are available to unload, unscramble, orient and 

present these components have limited flexibility and often constrain systems. The introduction of true high 

speed random bin packing could provide substantial productivity improvements to numerous 

manufacturing processes.

Utilization of bin picking in industry is currently limited to rather low speed and low accuracy applications. 

There are a number of challenges that constrain current state of the art systems. The most significant 

challenge appears to be perception technology. Further development of higher speed and higher accuracy 

systems, which can provide position and pose data to picking equipment, will be required to enable broader 

applications in the randomized component picking field.

C.5. Respondent Five:

We have tried random bin picking with little success because of the shape and sizes of the product. It is not 

the same at any one time. We are restricted on the container design due to the massive amounts of 

equipment and plants that use them. It is not easy for us to make that type of change so we must work 

within those means. The size of our product goes from the size of a small padded envelope to anything that 

weights 70 pounds. It can have all different shapes to it and that is why we can't seem to find a tool or 

implementation that works for us. We also need to have it process sacks and do this within about 20 items 

per minute. 

C.6. Respondent Six:

There are different challenges on a truly random bin picking application. If we compare a robot with a 

human, each one has a vision, an arm and a hand that interact within a picking operation. The vision system 

(or eyes) identifies the position and angle of a single part located within a pile of other parts. The vision 

also detects the surrounding and constraints of the identified part, such as grasp points to handle the part, 

and the walls or other parts that could interfere with the picking operation. The arm provides the path 

planning and motion to extract the part from a pile. The hand securely grasps the part so it can be handled 

with a known position and orientation. Each of these systems provides unique capabilities for the whole 

operation, and all of them are coordinated by a brain or task coordinator. [We] developed a technology 

demonstrator in which [company one] was providing the arm and hand, and [company two] was providing 

the active vision system, an intelligent sensing strategy based on multiple images. The system has a 3D 

stereo camera held by the robot, so the vision system commands the robot to take pictures from different 

angles. In a regular flow, the vision system is responsible of identifying a part that could be picked from a 

unique grasp point. When a part is identified, the vision system checks with the mechanical unit (the arm 

and the hand) if that part is pickable. A part is defined as pickable if the mechanical unit can be placed at 

the grasp point without any collision between the mechanical unit and the environment. If a part passes the 

pickable check, then the mechanical unit will try to generate a path to extract the part that is within the 

range of the arm and it is also collision free. If the part is pickable and an extraction path is verified, the 

mechanical unit extracts the path. Even though the flow looks simple, there are many variables that the 

system must account, for example: different search strategies for the walls or to avoid picking from the 

same area, how to react when an unexpected collision happens, different lighting conditions, and static vs. 

active vision systems. Also, each part has unique characteristics, which affect the optimal design of the 

hand, and possibly a change on design on the bin. 

Even though this technology has been already shown, there are still great technical challenges in the vision 

systems to make this a reliable and reusable solution that could meet the customer expectations.
C.7. Respondent Seven:

Vision Guided Robots for picking parts:

[My company] currently has over 100 applications within [...] operations where 3D vision systems are 

used to guide robot to pick up parts. All the current applications required the parts to be singulated or 

separated. The vision system determines the 6DOF position of the part and sends the part location 

information to the robot. The robot can therefore automatically accommodate slight shifts in part location 

within shipping dunnage usually caused by vibrations during transportation. There are many cost 

advantages to this approach, including: 1) Lower cost shipping containers - In general, accuracy is 

expensive, and costs for shipping containers increase with accuracy requirements. Lower accuracy 

requirements result in low costs for the containers. Vision systems are used to automatically compensate 

for variations in shipping containers. 2) Lower costs for part loading systems - Most operations where a 

robot acquires parts require that the parts are accurately located. Using vision to determine the actual 

positions of the parts eliminates the need for expensive systems to accurately position the parts for the 

robot. 3) Reduced labor costs and improved ergonomics - automatic handling of large or heavy parts 

eliminates the need for the human operator, and removes the physical hazard and opportunity for injury 

from the work place. [My company] is currently investigating additional opportunities for vision-guided 

robots to improve process flexibility by reducing the requirements for dedicated tooling and systems to feed 

parts to assembly operations.