U.S. flag
        An official website of the United States government
          Here’s how you know
        Here’s how you know**Official websites use .gov**
              
              A **.gov** website belongs to an official government organization in the United States.
        **Secure .gov websites use HTTPS**
              
              A **lock** (  A locked padlock
) or **https://** means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites. https://www.nist.gov/programs-projects/performance-assessment-framework-robotic-systems
  Search NIST
    Search
              Menu

      


Close
    Topics
        All Topics
                                                        Advanced communications
                                                        Artificial intelligence
                                                        Bioscience
                                                        Buildings and construction
                                                        Chemistry
                                                        Climate
                                                        Cybersecurity
                                                        Electronics
                                                        EnergyEnvironment
                                                        Fire
                                                        Forensic science
                                                        Health
                                                        Information technology
                                                        Infrastructure
                                                        Manufacturing
                                                        Materials
                                                        Mathematics and statisticsMetrology
                                                        Nanotechnology
                                                        Neutron research
                                                        Performance excellence
                                                        Physics
                                                        Public safety
                                                        Resilience
                                                        Standards
                                                        Transportation
            Publications
            Labs & Major Programs
        Laboratories
                                Communications Technology Laboratory
                                  Engineering Laboratory
                                  Information Technology Laboratory
                                  Material Measurement Laboratory
                                  Physical Measurement LaboratoryUser Facilities
                                NIST Center for Neutron Research
                                  CNST NanoFab
                                                        Research Test Beds
                                                        Research Projects
                                                        Tools & InstrumentsMajor Programs
                                Baldrige Performance Excellence Program
                                  CHIPS for America Initiative
                                  Manufacturing Extension Partnership (MEP)
                                  Office of Advanced Manufacturing
                                  Special Programs Office
                                  Technology Partnerships Office
            Services & Resources
        Standards and Measurements
                                Calibration Services
                                  Laboratory Accreditation (NVLAP)
                                  Quality System
                                  Standard Reference Materials (SRMs)
                                  Standards.gov
                                  Time Services
                                  Office of Weights and MeasuresSoftware
                                                        Data
                                Chemistry WebBook
                                  National Vulnerability Database
                                  Physical Reference Data
                                  Standard Reference Data (SRD)
                                                        Storefront
                                                        License & PatentsComputer Security Resource Center (CSRC)
                                                        NIST Research Library
            News & Events
        News
                                                        Events
                                                        Blogs
                                                        Feature Stories
                                                        AwardsVideo Gallery
                                                        Image Gallery
                                                        Media Contacts
            About NIST
        About Us
                                Leadership
                                  Organization Structure
                                  Budget & Planning
                                                        Contact Us
                                                        Visit
                                                        Careers
                                Student programsWork with NIST
                                                        History
                                NIST Digital Archives
                                  NIST Museum
                                  NIST and the Nobel
                                                        Educational Resources
 
 PROJECTS/PROGRAMS

Performance Assessment Framework for Robotic Systems

Share
  Facebook
                      Linkedin
                      Twitter
                      Email
    
  Summary
        Most manufacturing assembly tasks are not readily automatable today due to robots’ limited perception, mobility, and dexterity. The challenge of engineering and implementing successful robotic assembly work cells is exacerbated by the lack of tools for creating assembly-centric performance models encompassing an integrated system. Characterizing and accurately predicting robot system performance is technically hard due to the complex unmodeled interactions amongst the perception, mobility, and dexterity components of a robot system and the broad range of assembly operations. Currently, the successfully-automated assembly tasks are limited implementations that require significant fixturing and tooling, resulting in high investment and limited process flexibility. To advance the adoption of robots for assembly requires overcoming many technical hurdles, including assurance of new safety systems that enable humans and robots to work together. The new technical idea is to compose assembly task-driven measures of component performance (perception, mobility, dexterity, and safety) into robot system autonomous assembly performance models. For this project, NIST will deliver a suite of test methods for perception, mobility, dexterity, and safety components of robots derived from an assembly operation taxonomy and requirements, along with a methodology and tools for composing the resulting measurements into robot system performance models. This will provide manufacturers with the currently-missing data and tools for assessing and assuring implementations of robot systems within their smart manufacturing applications, thereby reducing the risk of adopting this key disruptive technology. Small and medium enterprises, as well as large manufacturers, will be able to use robot systems that enable them to increase the variety of products and shorten their product cycles to meet customer demands.

Description
      
      Performance Assessment Framework for Robotics Systems
  
      **Objective:**Enable manufacturers to assess and assure robot system assembly task performance by delivering a methodology and tools for characterizing and composing performance of perception, mobility, dexterity, and safety components.**Technical Idea:**The new technical idea is to compose assembly task-driven measures of component performance (perception, mobility, dexterity, and safety) into robot system autonomous assembly performance models. Performance measures will provide manufacturers and systems integrators with data for selecting the appropriate system components. The plan is to create:

Framework: A decomposition of typical assembly operations into the constituent robotic subsystem capabilities that will drive the research plans for the perception, mobility, dexterity, and safety thrusts.
	Capability Models: A representational framework for modeling performance within each component area that will enable analyses and studies to increase understanding of performance of the overall system within industrial settings.
	Test Methods: A suite of test methods for perception, mobility, dexterity, and safety components of robots derived from an assembly operation taxonomy and requirements, along with a methodology and tools for composing the resulting measurements into robot system performance models.**Research Plan:**

Framework: The research plan entails development of a performance modeling and analysis framework that utilizes the results from parallel efforts to define key component metrics and test methods for robotic assembly operations. Central to the research plan is a testbed integrating robot arms, mobile robot bases, hands and grippers, and a range of sensor systems. The reconfigurable testbed will support experiments in advanced perception, mobility, dexterity, and safety for robotic assembly. The testbed will be designed to support increasing levels of experimentation complexity in the assembly domain, starting with picking and placing parts for kit-building in static environments and ending, after 5 years, with fastening and joining operations in dynamic unstructured environments.
	Capability Models: The project will produce a robot systems capability model detailing a taxonomy of assembly tasks, decomposed into sub-tasks to define performance metrics and measurement methods for the perception, mobility, dexterity, and safety components of the robot system. This capability model will guide and focus the development of robot systems for use in assembly-centric manufacturing. Test method suites of increasing complexity will be developed:
		measuring robot performance in identifying, localizing, and tracking parts and other entities in its surroundings and purposefully grasping parts. This test method suite will provide manufacturers and robot developers with the means of assessing and assuring robot performance in assembly tasks using general-purpose grippers and hands.
			measuring robot performance in joining and fastening operations, including use of assembly tools. This test method suite will provide manufacturers and robot developers the ability to assess and assure a robot systems’ performance in assembly operations using tools.
			measuring the performance of mobile robots and mobile manipulators in safely navigating unstructured and dynamic environments. Through this test suite, NIST will provide industry with the means of assessing and assuring the performance and safety of mobile robots and manipulators in support of assembly-centric operations.
	Test Methods: At the end of the project lifetime, the results will be consolidated into a general methodology and tool set for composing perception, mobility, dexterity, and safety performance into robot system autonomous assembly performance models. This will enable manufacturers to assess and assure robot system task performance in smart manufacturing assembly-centric applications.
  Manufacturing and Robotics in manufacturing
      
            Organizations
      
      NIST HeadquartersLaboratory ProgramsEngineering LaboratoryIntelligent Systems DivisionManipulation and Mobility Systems Group

NIST Staff
      
                  Roger V. Bostelman
                                        Joseph A. Falco
                                        Marek Franaszek
                                        Kamel S. Saidi
Contact

Project Status
      
                          Completed
  Related NIST Projects
      
                  Robotic Systems for Smart Manufacturing Program
                                        Agility Performance of Robotic Systems
                                        ASTM Committee F45 on Driverless Automatic Guided Industrial Vehicles
                                        Performance of Collaborative Robot Systems
                                        ASTM Committee F45 on Driverless Automatic Guided Industrial Vehicles
                                        Performance of Collaborative Robot Systems
  Related Publications
      
                  A Roadmap to Advance Measurement Science in Robot Dexterity and Manipulation
                                        Dexterous Manipulation for Manufacturing Applications Workshop
                                        Towards a Classification System of Manufacturing Objects for the Evaluation of Perception Systems
                                        Measuring and Representing the Performance of Manufacturing Assembly Robots
                                        Grasping the Performance: Facilitating Replicable Performance Measures via Benchmarking and Standardized Methodologies
                                        Performance Evaluation of Human Detection Systems for Robot Safety
                                        Intelligence Level Performance Standards Research for Autonomous Vehicles

Other Projects
      
                          Agile Robotics for Industrial Automation Competition

Mobile Manipulator Performance Measurement Data

Performance Data Analytics

Robotic Grasping and Manipulation Competition: Manufacturing Track

3D Data for the Evaluation of Point-Based, Rigid Body Registration Error

Data and Results from Three Peg-in-Hole Experiments for Improving Insertion Tasks

Program videos

    
    Created March 21, 2014, Updated December 17, 2018
 HEADQUARTERS
            100 Bureau Drive
              Gaithersburg, MD 20899301-975-2000
            Webmaster | Contact Us | Our Other Offices
        Twitter
            
Facebook
            
LinkedIn
            
Instagram
            
YouTube
            
Giphy
            
RSS Feed
            
Mailing List
                              How are we doing? Feedback

  
  Site Privacy
                      Accessibility
                      Privacy Program
                      Copyrights
                      Vulnerability Disclosure
                      No Fear Act Policy
                      FOIA
                      Environmental Policy
                      Scientific Integrity
                      Information Quality Standards
                      Commerce.gov
                      Science.gov
                      USA.gov
                      Vote.gov