SM

A

MART

AT T

STA

EN

AN AS

T MAN

THE NA

ANDAR

NGINE

FI

Panel on R

Nation

Com

Divisi

A 

SSESS

NUFAC

ATION

RDS AN

EERIN

SCAL

Review of the

nal Institute o

mmittee on N

Laboratory

on on Engine

Consensu

SMENT

CTURI

NAL IN

ND TE

NG LAB

L YEAR

e Engineering

of Standards a

NIST Technic

 Assessments

eering and Phy

s Study R

T OF T

ING A

NSTIT

ECHN

BORA

R 2017

g Laboratory 

and Technolog

cal Programs 

s Board 

hysical Scienc

Report of 

THE 

ACTIV

TUTE 

NOLOG

ATORY

7

at the 

gy 

ces 

VITIES

OF 

GY 

Y

S 
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001 

This study was supported by Contract No. SB1341-12-CQ-0036 with the National Institute of Standards and 

Technology. Any opinions, findings, conclusions, or recommendations expressed in this publication do not 

necessarily reflect the views of any agency or organization that provided support for the project. 

Digital Object Identifier: https://doi.org.10.17226/24976 

Copies of this publication are available from 

Laboratory Assessments Board 

National Academies of Sciences, Engineering, and Medicine 

500 Fifth Street, NW #928 

Washington, DC 20001 

Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, NW, 

Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu. 

Copyright 2017 by the National Academies of Sciences, Engineering, and Medicine. All rights reserved. 

Printed in the United States of America 

Suggested Citation: National Academies of Sciences, Engineering, and Medicine. 2017. An Assessment of the 

Smart Manufacturing Activities at the National Institute of Standards and Technology Engineering 

Laboratory: Fiscal Year 2017. Washington, DC: The National Academies Press. 

https://doi.org.10.17226/24976.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by 

President Lincoln, as a private, nongovernmental institution to advise the nation on issues 

related to science and technology. Members are elected by their peers for outstanding 

contributions to research. Dr. Marcia McNutt is president. 

The National Academy of Engineering was established in 1964 under the charter of the 

National Academy of Sciences to bring the practices of engineering to advising the nation. 

Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. 

Mote, Jr., is president. 

The National Academy of Medicine (formerly the Institute of Medicine) was established in 

1970 under the charter of the National Academy of Sciences to advise the nation on medical 

and health issues. Members are elected by their peers for distinguished contributions to 

medicine and health. Dr. Victor J. Dzau is president. 

The three Academies work together as the National Academies of Sciences, Engineering, 

and Medicine to provide independent, objective analysis and advice to the nation and 

conduct other activities to solve complex problems and inform public policy decisions. The 

National Academies also encourage education and research, recognize outstanding 

contributions to knowledge, and increase public understanding in matters of science, 

engineering, and medicine. 

Learn more about the National Academies of Sciences, Engineering, and Medicine at 

www.nationalacademies.org.
Consensus Study Reports published by the National Academies of Sciences, Engineering, and 

Medicine document the evidence-based consensus on the study’s statement of task by an 

authoring committee of experts. Reports typically include findings, conclusions, and 

recommendations based on information gathered by the committee and the committee’s 

deliberations. Each report has been subjected to a rigorous and independent peer-review 

process and it represents the position of the National Academies on the statement of task. 

Proceedings published by the National Academies of Sciences, Engineering, and 

Medicine chronicle the presentations and discussions at a workshop, symposium, or 

other event convened by the National Academies. The statements and opinions contained in 

proceedings are those of the participants and are not endorsed by other participants, the 

planning committee, or the National Academies. 

For information about other products and activities of the National Academies, please 

visit www.nationalacademies.org/about/whatwedo. 
v 

PANEL ON REVIEW OF THE ENGINEERING LABORATORY AT THE NATIONAL 

INSTITUTE OF STANDARDS AND TECHNOLOGY 

 

NEIL A. DUFFIE, University of Wisconsin, Madison, Chair

HADI ABU-AKEEL, Brachium, Inc. 

JOSEPH J. BEAMAN, JR., University of Texas, Austin 

DAVID L. BOURELL, University of Texas, Austin 

DIANNE CHONG, Boeing Company 

DEAN R. COLLINS, DRC Consulting, LLC 

DAVID E. CROW, University of Connecticut 

RAINER HEBERT, University of Connecticut 

S. JACK HU, University of Michigan 

MOHSEN A. JAFARI, Rutgers University 

ZHENYU (JAMES) KONG, Virginia Polytechnic Institute and State University 

JAY LEE, University of Cincinnati 

EUGENE MEIERAN, Intel Corporation (retired) 

JOHN ONEY, PostProcess Technologies 

LYNNE E. PARKER, University of Tennessee, Knoxville 

WILLIAM F. POWERS, Ford Motor Company (retired) 

H. DONALD RATLIFF, Georgia Institute of Technology 

EMANUEL M. SACHS, Massachusetts Institute of Technology 

LAURENCE C. SEIFERT, AT&T Corporation (retired) 

F. STAN SETTLES, University of Southern California 

VENKATA VEDULA, United Technologies Corporation 

JOHN WEN, Rensselaer Polytechnic Institute 

KATE S. WHITEFOOT, Carnegie Mellon University 

DAVID D. YAO, Columbia University 

Staff 

AZEB GETACHEW, Senior Program Assistant 

LIZA HAMILTON, Associate Program Officer 

EVA LABRE, Administrative Coordinator 

JAMES P. MCGEE, Director 
vi 

COMMITTEE ON NIST TECHNICAL PROGRAMS 

ELSA REICHMANIS, Georgia Institute of Technology, Chair 

MICHAEL I. BASKES, Mississippi State University 

LEWIS BRANSCOMB, University of California, San Diego 

MARTIN E. GLICKSMAN, Florida Institute of Technology 

JENNIE S. HWANG, H-Technologies Group 

CHRISTOPHER W. MACOSKO, University of Minnesota 

C. KUMAR PATEL, Pranalytica, Inc. 

BHAKTA B. RATH, Naval Research Laboratory 

ALICE WHITE, Boston University 

Staff

AZEB GETACHEW, Senior Program Assistant 

LIZA HAMILTON, Associate Program Officer 

EVA LABRE, Administrative Coordinator 

JAMES P. MCGEE, Director 
vii 

LABORATORY ASSESSMENTS BOARD 

ROSS B. COROTIS, University of Colorado, Boulder, Chair

C. WILLIAM GEAR, Princeton University 

WESLEY L. HARRIS, Massachusetts Institute of Technology 

JENNIE S. HWANG, H-Technologies Group 

W. CARL LINEBERGER, University of Colorado, Boulder 

C. KUMAR N. PATEL, Pranalytica, Inc. 

ELSA REICHMANIS, Georgia Institute of Technology 

LYLE H. SCHWARTZ, U.S. Air Force Office of Scientific Research (retired) 

Staff

AZEB GETACHEW, Senior Program Assistant 

LIZA HAMILTON, Associate Program Officer 

EVA LABRE, Administrative Coordinator 

JAMES P. MCGEE, Director 

ARUL MOZHI, Senior Program Officer 

ix 

Acknowledgment of Reviewers 

This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse 

perspectives and technical expertise. The purpose of this independent review is to provide candid and 

critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in 

making each published report as sound as possible and to ensure that it meets the institutional standards 

for quality, objectivity, evidence, and responsiveness to the study charge. The review comments and draft 

manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the 

following individuals for their review of this report: 

Agnes Chau Klucha, UTC Aerospace Systems, 

Wallace J. Hopp, University of Michigan, 

Yoram Koren, University of Michigan, 

Hau L. Lee, Stanford University, 

Deborah J. Nightingale, Massachusetts Institute of Technology, 

Robert E. Schafrik, General Electric Aircraft Engines, and 

Matthew J. Zaluzec, Ford Motor Company. 

Although the reviewers listed above provided many constructive comments and suggestions, they 

were not asked to endorse the conclusions or recommendations of this report nor did they see the final 

draft before its release. The review of this report was overseen by Michael I. Baskes, Mississippi State 

University. He was responsible for making certain that an independent examination of this report was 

carried out in accordance with the standards of the National Academies and that all review comments 

were carefully considered. Responsibility for the final content rests entirely with the authoring panel and 

the National Academies. 

xi 

Contents 

SUMMARY 1 

1 THE CHARGE TO THE PANEL AND THE ASSESSMENT PROCESS 9 

2 MEASUREMENT SCIENCE FOR ADDITIVE MANUFACTURING 11 

3 ROBOTIC SYSTEMS FOR SMART MANUFACTURING 19 

4 SMART MANUFACTURING OPERATIONS PLANNING AND CONTROL 25 

5 SMART MANUFACTURING SYSTEMS DESIGN AND ANALYSIS 30 

6 CONCLUSIONS AND RECOMMENDATIONS 36 

ACRONYMS 45 

1 

Summary 

The mission of the Engineering Laboratory (EL) of the National Institute of Standards and 

Technology (NIST) is to “promote U.S. innovation and industrial competitiveness by advancing 

measurement science, standards, and technology for engineered systems in ways that enhance economic 

security and improve quality of life.”1

 To support this mission the EL has developed thrusts in smart 

manufacturing, construction, and cyberphysical systems; in sustainable and energy-efficient 

manufacturing materials and infrastructure; and in disaster-resilient buildings, infrastructure, and 

communities. The technical work of the EL is performed in five divisions—Intelligent Systems, Materials 

and Structural Systems, Energy and Environment, Systems Integration, and Fire Research—and in two 

offices—the Applied Economics Office and the Smart Grid Program Office. 

In 2017, at the request of the acting director of NIST, the National Academies of Sciences, 

Engineering, and Medicine formed the Panel on Review of the Engineering Laboratory at the National 

Institute of Standards and Technology (referred to in this report as “the panel”) and established the 

following statement of task for the panel: 

The Panel on Review of the Engineering Laboratory at the National Institute of Standards and 

Technology will assess the scientific and technical work performed by the National Institute of 

Standards and Technology (NIST) Engineering Laboratory (EL). The panel will review technical 

reports and technical program descriptions prepared by NIST staff, and will visit the facilities of the 

NIST laboratory. The visit will include technical presentations by NIST staff, demonstrations of NIST 

projects, tours of NIST facilities, and discussions with NIST staff. The panel will deliberate findings in 

a closed session panel meeting and will prepare a report summarizing its assessment findings. 

The acting director of NIST requested that in 2017 the panel confine its assessment to the 

following smart manufacturing programs conducted at the Engineering Laboratory: Measurement 

Science for Additive Manufacturing (MSAM), Robotic Systems for Smart Manufacturing 

(RSSM), Smart Manufacturing Operations Planning and Control (SMOPAC), and Smart 

Manufacturing Systems Design and Analysis (SMSDA). 

CONCLUSIONS AND RECOMMENDATIONS 

General Conclusions and Recommendations 

Smart manufacturing programs in the EL at NIST have significantly evolved in the last five years, 

matching increased national recognition of the importance of manufacturing in the economy, as well as 

international recognition of the importance of integration of digital computing and information 

technologies into manufacturing systems. For example, the additive manufacturing program area was 

 1

 National Institute of Standards and Technology, “About EL,” https://www.nist.gov/el/about-el, accessed May 

22, 2017. 
2 

established approximately three and one-half years ago and is now an impressive effort in an area of rapid 

technology growth, staffed by competent early-career researchers, and facilitated by a measurement 

testbed of their own design with unique and powerful capabilities. Standards are being given a high 

priority, and there is excellent collaboration with other organizations and leadership in these efforts. There 

is significant focus on small and medium enterprises (SMEs), which is a valuable objective that is 

difficult to achieve. There are good research collaborations with industry, but perhaps an overreliance on 

(hoped for) trickle down from work with larger industries; more extensive outreach to SMEs, especially 

young SMEs, would be beneficial. 

The EL’s smart manufacturing program appears to lack a roadmap and measurable milestones, as do 

the individual programs and projects. Without roadmaps and measurable milestones, the quality and value 

of NIST’s smart manufacturing programs and their management were more difficult to assess, and 

therefore the value, quality, and results of individual projects were more difficult to measure. Although 

some excellent research outputs, evinced by strong publication records in peer-reviewed journals, are 

being obtained and significant progress in standards is being made, many project plans did not show 

intermediate project milestones, and so it was difficult to determine in some cases whether projects were 

making the expected amount of progress and achieving the desired results. Roadmaps and measurable 

milestones would be invaluable tools for NIST management. 

There is also, in some cases, a lack of crosscutting activities that can take advantage of the synergistic 

objectives of individual projects. Examining the project’s plans, together to find opportunities for crossfertilization and integrating the testing and evaluation work across the overall smart manufacturing 

program, could provide more synergy across the projects, as well as an opportunity to achieve broader 

impact of the program work as a whole, rather than only at the project level. This is not helped by the 

current distance between laboratory spaces, and it would be better to have a larger and more contiguous 

space so that multiple projects and programs could be integrated into a larger, more complex testbed that 

more resembles some manufacturing environments. 

The new program areas are well covered by both junior and senior staff, who have significantly and 

successfully adapted their efforts to new technologies. Some areas appear to be short on permanent staff 

and expertise, but staffing appears to be generally sufficient to support the research that is under way. The 

journal publication record is excellent. There has been significant external recognition of the staff and its 

research, but additional efforts need to be made to promote and apply for external awards for the staff. 

This also would promote recognition of the smart manufacturing research programs at NIST. 

Measurement Science for Additive Manufacturing 

The majority of national laboratories within the federal government currently pursue additive 

manufacturing (AM), and the significant progress that has been made within the Measurement Science for 

Additive Manufacturing (MSAM) program could benefit these other laboratories that, in turn, could likely 

help the MSAM program. Additionally, closer engagement and alignment with other U.S. government 

laboratories could ensure that the MSAM program is effectively leveraging the investments being made in 

this area. 

RECOMMENDATION 1: The Engineering Laboratory should consider closer engagements 

with other national laboratories’ additive manufacturing groups. 

A dialogue needs to be intensified with the Material Measurement Laboratory (MML) and its 

Polymers Processing Group in order to exploit common strengths and overlapping research interests. This 

dialogue could build on existing meetings with the MML and the Polymers Processing Group and evolve 

toward common research projects, such as feedstock characterization or the design of additive 

manufacturing machines. 
3 

RECOMMENDATION 2: The Engineering Laboratory should consider beginning an 

intensified dialogue with the Material Measurement Laboratory and the Polymers Processing 

Group. 

There are several highly industry relevant topics within AM that are not currently pursued with 

sufficient emphasis by the MSAM program. These topics include support structures and their design, the 

effects of different build orientations, design limitations for AM, and the post-processing side of AM. 

Broadening the focus into post- or secondary processing will help researchers understand what limitations 

within the technologies cannot be overcome with post-processing methods. Expanding research veins into 

support structures and post-processing, and defining new procedures in this field, would allow an 

opportunity to develop new standards, which could help industry find common ground. 

RECOMMENDATION 3: The Engineering Laboratory should consider expanding its research 

into the measurements and standards needed to assist industry in the following areas: support 

structures and their design, the effects of different build orientations, design limitations for 

additive manufacturing, and the post-processing side of additive manufacturing. 

Robotic Systems for Smart Manufacturing 

Overall, the laboratory facilities, equipment, and human resources are top-notch and sufficient for 

impactful standards development activities in the Robotic Systems for Smart Manufacturing (RSSM) 

program. The program staff has done an excellent job of identifying the top brands of robotic and sensor 

equipment that will likely be used by SMEs and has acquired and activated the equipment for its research. 

The staff has been successful in operating the equipment to develop best practice methods for using the 

hardware and related software and to conduct a wide variety of experiments and demonstrations to 

evaluate potential metrics, test artifacts, and methods for multi-robot collaboration, agility, 

interoperability, and integration. 

The RSSM program is providing important contributions to the development of standards, metrics, 

and applicable technologies in the areas of performance assessment, collaborative robotics, agility, 

interoperability, and integration. The focus on the development of standards and performance metrics and 

tests in these areas aligns well with the mission of the EL. In particular, the program has made important 

advances in identifying weaknesses with existing metrics and standards in robotic mobility and agility, 

anticipating unmet needs for future industry challenges, and developing new metrics and standards to 

address these issues. 

The RSSM program staff has an opportunity to lead in the development of a more systematic 

methodology for generating test and use cases. Such a methodology could include needs assessment, gap 

identification, research, metrics development, test methods and artifacts development, standards 

development, evaluation, and dissemination. 

RECOMMENDATION 4: The Engineering Laboratory should consider leading the 

development of methodological approaches for standards development in all component areas 

of its robotic systems for smart manufacturing research, including the areas of needs 

assessment, gap identification, research, metrics development, test methods and artifacts 

development, standards development, evaluation, and dissemination. 

The RSSM program could broaden its industrial impact through stronger interactions with the 

Manufacturing Extension Partnership (MEP) centers or other industrial consortia. Through these 

deepened interactions, the RSSM program could develop a set of use cases and obtain more in-depth 

feedback on standards development. 
4 

RECOMMENDATION 5: The Engineering Laboratory should consider interacting on a 

regular basis (e.g., twice annually) with the Manufacturing Extension Partnership (MEP) 

centers or other industry consortia to source use cases and receive feedback on the effectiveness 

of their development of metrics and standards. 

Although vision sensors are used in several RSSM testbeds, the EL has an unexplored opportunity to 

investigate how machine vision can be used to enhance the flexibility and user-friendliness of robots for 

SMEs. The exploration of how to take advantage of machine vision to simplify robot programming, 

develop self-teaching techniques, ensure safe human-robot interaction, and adapt to the varied application 

environments at SME facilities is a good fit with the EL’s objectives for the RSSM program. 

RECOMMENDATION 6: The Engineering Laboratory should consider exploring the use of 

machine vision to enhance the value of robots for small and medium enterprises (SMEs) by 

reducing programming complexity for short-run manufacturing applications. 

There is an increasing utilization of cloud and edge resources for robotics and industrial automation. 

The Industry 4.0, Microsoft Azure, and Amazon Web Services (AWS) efforts are trying to integrate 

resources in a consistent and efficient framework, and the RSSM program has the opportunity to 

contribute to and lead this nascent field. 

RECOMMENDATION 7: The Engineering Laboratory should consider exploring 

methodological approaches and standards development related to cloud robotics by 

contributing to activities in this area (e.g., Industry 4.0), and taking a leadership role in this 

field. 

Given the level of expertise and contribution of the RSSM research team, additional peer recognition 

is possible in the form of merit-based robotics awards, elevation in professional societies, invitations to 

delivering opening and keynote presentations at conferences, and early-career investigator awards. This 

type of external recognition can serve not only to award deserving individuals but also to raise the 

visibility of EL in the field. 

RECOMMENDATION 8: The Engineering Laboratory should be more proactive in seeking 

individual recognition for accomplished robotics staff personnel, rather than waiting to be 

discovered. 

Human-machine collaboration is an important challenge in robotics for manufacturing. The RSSM 

program appears to have an expertise gap in human robot interaction due to lack of permanent staff in this 

area. Active recruiting of such researchers into NIST permanent staff is important to establish this topic as 

an active research area. 

RECOMMENDATION 9: The Engineering Laboratory should establish permanent staff 

expertise in its human-robot interaction area to avoid a future expertise gap. 

The RSSM could more systematically leverage and seek out expertise from other organizations, 

including universities and industries, and through an extended use of guest researcher programs. A 

consortium of industrial companies, both large and small, as well as universities that meet with the RSSM 

on an annual or a semi-annual basis, could serve as additional sources of expertise for needs assessment, 

gap identification, and the testing and validation of standards. 

RECOMMENDATION 10: The Engineering Laboratory should consider systematically 

leveraging and seeking out expertise from other organizations, including university, industry, 
5 

and Manufacturing USA Institutes who can serve as additional sources of expertise for needs 

assessment, gap identification, and the testing and validation of standards. 

The RSSM program has researched the capabilities of a wide variety of robotic grippers. They have 

developed many test methods, artifacts, and metrics to guide in the design and use of grippers for 

dexterous grasping. They have an opportunity to leverage this research to create community-wide test sets 

that can assist the research community in developing new approaches to dexterous grasping. For example, 

the Yale-CMU-Berkeley (YCB) Object and Model Set data set is a related contribution to the research 

community that helps advance robotic manipulation, and a similar contribution based on the RSSM 

research would be highly beneficial to the research community. 

RECOMMENDATION 11: The Engineering Laboratory should consider coordinating with 

other organizations to develop community-wide test sets for dexterous grasping. 

Although the research facilities are adequate, they are also distributed across the NIST campus and 

are physically disconnected across multiple buildings on the campus. Given that the RSSM research 

projects share many common technologies, science, and resource needs, and can benefit from knowledge 

exchanges, such separations between the project teams might contribute to less cohesiveness and overall 

impact than might be possible through co-located facilities. The lack of crosscutting activities across all 

the projects of the RSSM program is not helped by the distance between laboratory spaces. 

RECOMMENDATION 12: The Engineering Laboratory should consider co-locating test 

facilities to enhance integration of Robotic Systems for Smart Manufacturing (RSSM) program 

activities across the entire program. 

There is an opportunity to develop software infrastructure resources to facilitate the interoperability of 

testbeds developed in individual projects. Several of the testbeds developed for individual projects have 

overlapping capabilities but cannot currently be integrated because they do not have interoperable 

software. 

RECOMMENDATION 13: The Engineering Laboratory should strive for interoperability of 

software infrastructure for their testbeds. 

The ability of RSSM staff to share videos, images, and other information through social media is 

hindered by the lengthy approval process at NIST and the lack of staff with expertise in social media 

outreach. As a result, the program has very little visibility on social media, which is a missed opportunity 

to improve the impact of the RSSM program’s activities. 

RECOMMENDATION 14: The Engineering Laboratory should consider adding staff who can 

improve dissemination of technical accomplishments through social media, trade publications, 

and/or popular press and should work with NIST management to identify ways to streamline 

approval processes for applying these methods. 

Smart Manufacturing Operations Planning and Control 

The Smart Manufacturing Operations Planning and Control (SMOPAC) program needs to develop an 

overarching, integrated roadmap with measurable milestones that shows the relationships of the various 

markets (automotive, aerospace, etc.) and requirements to the technologies, as well as the relationships 

between the technical areas themselves. The roadmap needs to be an integrated, high-level plan, not a set 

of semi-related programs. 
6 

RECOMMENDATION 15: The Engineering Laboratory should consider developing an 

overarching roadmap with measurable milestones for the Smart Manufacturing Operations 

Planning and Control (SMOPAC) program that integrates market needs, product 

requirements, and technologies. 

The EL needs to examine its activities within the context of national and global activities at other 

organizations to identify competitive and duplicative areas. 

RECOMMENDATION 16: The Engineering Laboratory should consider benchmarking its 

work in smart manufacturing operations planning and control against other manufacturing 

programs, such as those in EU Horizon 2020, to identify competitive and duplicative areas. 

In order to ensure alignment with manufacturers’ needs, and to identify early adopters of advanced 

technologies, programs, tools, and processes, the SMOPAC program needs to engage additional 

manufacturers, including new entrants to the manufacturing sector, such as start-up manufacturing 

companies and suppliers. They could partner, for example, with Manufacturing USA to facilitate 

information exchange. 

RECOMMENDATION 17: The Engineering Laboratory should consider engaging new 

manufacturers and suppliers to ensure alignment with manufacturers’ needs and to identify 

early adopters of advanced technologies, programs, tools, and processes. 

RECOMMENDATION 18: The Engineering Laboratory should consider partnering with 

Manufacturing USA to establish Standard for the Exchange of Product Data (STEP)-like 

computer-aided design (CAD), computer-aided manufacturing (CAM), and computerintegrated manufacturing (CIM) product information exchange standards. 

The SMOPAC program needs to develop and show project plans with measurable milestones to 

describe their 5-year projects. Such project plans can be used as communication tools to inform 

management, the staff working on the project, users, emerging companies, and the public about the value 

of the project’s goals and when the results will be made available. 

RECOMMENDATION 19: The Engineering Laboratory should develop project plans with 

measurable milestones that describe their 5-year projects and use these plans as communication 

tools. 

The definition of “life cycle” that the Digital Thread for Smart Manufacturing project is using shows 

the end of the cycle as being when the initial product is delivered to the customer. This is different from 

the Department of Defense (DoD) ManTech projects, which include the Army, Navy, and Air Force 

projects on digital thread and digital twin, where life cycle also includes the life of the aircraft. In 

aerospace and automotive industries (as well as with health care devices and others), the manufacturer 

needs to have the ability to track the vehicles and devices in the field for potential part defects as long as 

they are in service; and so life cycle is from cradle to grave. In all fields, products have increasingly 

tremendous sensory and computational capabilities, and in-the-field capabilities for diagnostics and 

prognostics need to be considered as part of the life cycle in the digital thread. 

RECOMMENDATION 20: The Engineering Laboratory should improve and clarify 

the definition of “life cycle” to be more comprehensive and include manufacturing activities 

after the product is delivered. 
7 

The concept of the digital thread can be applied to multiple industries of various sizes. The Digital 

Thread for Smart Manufacturing project is targeting medium-size manufacturers. While smaller 

manufacturers may either already have or be able to adopt these tools and standards, the team needs to 

address how large-scale industries (e.g., aerospace, automotive) will adopt NIST’s unique tools. 

RECOMMENDATION 21: The Engineering Laboratory should put together a business case 

for small-, medium-, and large-scale industries to adopt their digital thread tools. 

The Prognostics, Health Management, and Control (PHMC) team has developed an efficient and 

effective linear axis error detection methodology based on data collected from an inertial measurement 

unit (IMU), which has been developed and verified on the linear axis testbed and machine tools within the 

PHMC project at NIST. Validation efforts are ongoing both internally within NIST and externally with 

several manufacturing collaborators. The developed IMU sensor is very practical to assess machine health 

degradation rapidly. The team has demonstrated platform-based technologies to allow diversified users to 

connect to the machine and share their data. The team is planning to promote its research to system levels; 

however, the research scope needs to be more clearly defined (e.g., type of applications, sensors, and 

analytics) before this happens. Collaboration with the Clean Energy Smart Manufacturing Innovation 

Institute (CESMII) will help to define, identify, and satisfy industry needs and requirements as well as 

provide efficiencies in areas of overlapping research. 

RECOMMENDATION 22: The Engineering Laboratory should consider collaborating with 

the Clean Energy Smart Manufacturing Innovation Institute to help to identify, prioritize, and 

satisfy industry needs and requirements and to provide efficiencies in areas of overlapping 

research. 

The Cybersecurity for Smart Manufacturing project has established a testbed to validate the 

cybersecurity framework manufacturing profile. Data obtained from this testbed will be used to develop 

guidance for implementation of this framework. The testbed needs to include a simulation system that can 

generate virtual cyber threats to evaluate the resilience of the system. 

RECOMMENDATION 23: The Engineering Laboratory’s testbed for the Cybersecurity for 

Smart Manufacturing project should include a simulation system that can generate virtual 

cyber threats to evaluate the resilience of the system. 

The Wireless Systems for Industrial Environments project research is just beginning, and it needs to 

be considered more broadly. The team needs to develop testbeds that include both cybersecurity and 

wireless technology and consider integrating with a cloud-based or edge-based environment to support 

more tether-free monitoring system applications. 

RECOMMENDATION 24: The Engineering Laboratory should consider developing a testbed 

that includes both cybersecurity and wireless technology. 

RECOMMENDATION 25: The Engineering Laboratory should consider integrating with a 

cloud-based or edge-based environment to support more tether-free monitoring system 

applications. 

In some of the testbeds, there are arrangements being made to integrate with other programs, such as 

robotics. This is a good start; however, the integration is small. It would be better to have a larger space 

so that multiple projects and programs could be integrated into a bigger, more complex testbed that more 

resembles an actual manufacturing environment. Validation of models under these conditions would 

provide meaningful data. 
8 

RECOMMENDATION 26: The Engineering Laboratory should consider dedicating an area or 

a facility for the integration of multiple projects and programs to simulate an industrial 

environment. 

Smart Manufacturing Systems Design and Analysis 

The NIST Smart Manufacturing Systems Design and Analysis (SMSDA) program would greatly 

benefit from and could contribute to an overarching roadmap of smart manufacturing for the nation. 

RECOMMENDATION 27: The Engineering Laboratory should consider collaborating with 

the Smart Manufacturing Leadership Coalition (SMLC) and the newly formed Clean Energy 

and Smart Manufacturing Innovation Institute (CESMII) to develop for the National Institute 

of Standards and Technology and the United States an overarching roadmap with measurable 

milestones for smart manufacturing. 

Where best practices for smart manufacturing capabilities do not exist, the task is to develop 

structures, databases, prototypes, and techniques to define best practices. During execution of tasks of this 

type, the inclusion of a domain expert on the team would greatly improve the output. 

RECOMMENDATION 28: Where a smart manufacturing best practice does not exist, the 

Engineering Laboratory should consider embedding domain experts on the project teams to 

ensure the quality of their output. 

The SMSDA program needs to ensure that the output of projects where a smart manufacturing best 

practice does not exist is tested in at least two SMEs. These SMEs need to become an integral part of the 

project teams to ensure that the developed structures, databases, and techniques make it easier for SMEs 

to utilize smart manufacturing. 

RECOMMENDATION 29: The Engineering Laboratory should ensure that where a smart 

manufacturing best practice does not exist, the output of projects is tested in at least two small 

and medium enterprises (SMEs). 

Where there is a lack of existing standards or guidelines, the SMSDA program needs to broaden its 

scope to include identifying smart manufacturing best practices in a select set of industry sectors. Given 

that the program has been actively collaborating with various standards communities and has access to a 

wide range of industries, this broadening of scope appears both feasible and practical. 

RECOMMENDATION 30: Where there is a lack of existing standard or guidelines, the 

Engineering Laboratory should consider broadening its scope to include identifying smart 

manufacturing best practices in a select set of industry sectors. 

Project planning for multiyear projects needs to include measurable intermediate milestones so that 

progress and midcourse correction can easily be understood and executed. The projects need to have a 

clear definition of deliverables and measurable milestones. Additionally, deliverables need to include 

SME demonstrations and testing. 

RECOMMENDATION 31: The Engineering Laboratory should consider planning multiyear 

projects with measurable intermediate milestones so that progress and midcourse correction 

needs are readily visible.
9 

1 

The Charge to the Panel and the Assessment Process 

At the request of the National Institute of Standards and Technology (NIST), the National Academies 

of Sciences, Engineering, and Medicine has, since 1959, annually assembled panels of experts from 

academia, industry, medicine, and other scientific and engineering communities to assess the quality and 

effectiveness of NIST measurements and standards laboratories, of which there are now seven,1

 as well as 

the adequacy of the laboratories’ resources. The context of this technical assessment is the mission of 

NIST, which is to promote U.S. innovation and industrial competitiveness by advancing measurement 

science, standards, and technology in ways that enhance economic security and improve the quality of 

life. NIST laboratories conduct research to anticipate future metrology and standards needs, to enable 

scientific and technological advances, and to improve and refine existing measurement methods and 

services. 

At the request of the acting director of NIST, in 2017 the National Academies formed the Panel on 

Review of the Engineering Laboratory at the National Institute of Standards and Technology and 

established the following statement of task for the panel: 

The Panel on Review of the Engineering Laboratory at the National Institute of Standards and 

Technology will assess the scientific and technical work performed by the National Institute of 

Standards and Technology (NIST) Engineering Laboratory (EL). The panel will review technical 

reports and technical program descriptions prepared by NIST staff, and will visit the facilities of the 

NIST laboratory. The visit will include technical presentations by NIST staff, demonstrations of NIST 

projects, tours of NIST facilities, and discussions with NIST staff. The panel will deliberate findings in 

a closed session panel meeting and will prepare a report summarizing its assessment findings. 

The acting director of NIST requested that in 2017 the panel confine its assessment to the 

following smart manufacturing programs conducted at the Engineering Laboratory, which 

conducts activities in other areas as well: Measurement Science for Additive Manufacturing 

(MSAM), Robotic Systems for Smart Manufacturing (RSSM), Smart Manufacturing Operations 

Planning and Control (SMOPAC), and Smart Manufacturing Systems Design and Analysis 

(SMSDA). 

The acting director of NIST also suggested that the panel consider during its assessment the following 

factors: 

1. Assess the organization’s technical programs. 

 1 The seven National Institute of Standards and Technology (NIST) laboratories are the Engineering 

Laboratory, the Physical Measurement Laboratory, the Information Technology Laboratory, the Material 

Measurement Laboratory, the Communication Technology Laboratory, the Center for Nanoscale Science and 

Technology, and the NIST Center for Neutron Research.
10 

 How does the quality of the research compare to similar world-class research in the technical 

program areas? 

 Is the quality of the technical programs adequate for the organization to reach its stated 

technical objectives? How could it be improved? 

2. Assess the portfolio of scientific expertise within the organization. 

 Does the organization have world-class scientific expertise in the areas of the organization’s 

mission and program objectives? If not, what areas should be improved? 

 How well does the organization’s scientific expertise support the organization’s technical 

programs and the organization’s ability to achieve its stated objectives? 

3. Assess the adequacy of the organization’s facilities, equipment, and human resources. 

 How well do the facilities, equipment, and human resources support the organization’s 

technical programs and its ability to achieve its stated objectives? How could they be 

improved? 

To accomplish the assessment, the National Academies assembled a panel of 24 volunteers 

whose expertise matched that of the work performed by the EL staff.2

 

On March 28-30, 2017, the panel assembled at the NIST facility in Gaithersburg, Maryland, for a two 

and a half day assessment, during which it received welcoming remarks from the EL director, heard 

overview presentations by EL management and presentations by researchers at the EL, toured portions of 

the EL facility, and attended an interactive session with EL management. The panel also met in a closed 

session to deliberate on its findings and to define the contents of this assessment report. 

The panel’s approach to the assessment relied on the experience, technical knowledge, and expertise 

of its members. The panel did not attempt to report an exhaustive assessment of every project reviewed. 

Rather, the panel’s goal was to identify and report accomplishments and opportunities for further 

improvement with respect to the following: the quality of the technical programs at the EL; the portfolio 

of scientific expertise within the laboratory; and the adequacy of the laboratory’s facilities, equipment, 

and human resources. The panel illustrated its conclusions with salient examples of programs and projects 

that are intended collectively to portray an overall impression of the laboratory, while preserving useful 

suggestions specific to projects and programs. 

To accomplish its mission the panel reviewed the material provided by the EL prior to and during the 

review meeting. The choice of projects to be reviewed was made by the EL. The panel applied a largely 

qualitative approach to the assessment. Given the nonexhaustive nature of the review, the omission in this 

report of any particular EL project should not be interpreted as a negative reflection on the omitted 

project. 

 2

 See the NIST Engineering Laboratory website, at https://www.nist.gov/el, for information on the Engineering 

Laboratory organization and programs (accessed April 21, 2017). 
11 

2 

Measurement Science for Additive Manufacturing 

INTRODUCTION 

The Intelligent Systems Division within the Engineering Laboratory (EL) focuses on the key areas of 

smart manufacturing. The smart manufacturing program areas within the Intelligent Systems Division 

include Measurement Science for Additive Manufacturing (MSAM), Robotic Systems for Smart 

Manufacturing (RSSM), and Smart Manufacturing Operations Planning and Control (SMOPAC). 

Additive manufacturing (AM), also known as three-dimensional (3D) printing, continues to be 

dubbed as the “next best thing” in advanced manufacturing owing to its promise to fundamentally 

revolutionize part designs and manufacturing methods. By printing parts directly from a 3D computeraided design (CAD) digital file, the technology enables unique organic shapes that cannot be 

manufactured through conventional manufacturing methods. Additionally, additive technologies can 

reduce the manufacturing lead time, part cost, and materials use. In spite of these benefits, the technology 

has not had widespread adoption within the U.S. manufacturing industry—except for niche applications. 

The key barriers for adoption are the following: material characterization, including types of defects and 

impact on mechanical properties; lack of design tools and allowables; investment costs of additive 

equipment; added cost and time for making components; qualification and certification requirements; 

process controls and variability; and lack of industry standards. 

The AM work in the MSAM program is aimed at developing and deploying measurement science 

that will enable rapid design-to-product transformation through advances in four projects: (1) 

Characterization of Additive Manufacturing Materials; (2) Qualification for Additive Manufacturing 

Materials, Processes, and Parts; (3) Real-Time Monitoring and Control of Additive Manufacturing 

Processes; and (4) Systems Integration for Additive Manufacturing. The MSAM program is 

approximately three and one half years into a five-year program duration. The current total budget is 

$7.05 million, with a total staff of 21 federal employees, 4 NIST associates, and 4 students. 

ASSESSMENT OF TECHNICAL PROGRAMS 

Accomplishments 

The individual projects within the MSAM program are well aligned with the overall program 

objective to 

Develop and deploy measurement science that will enable rapid design to product transformation 

through advances in: material characterization, in-process sensing, monitoring and model based 
12 

optimal control; performance qualification of materials, processes and parts, and end-to-end digital 

implementation of metal additive manufacturing processes and systems.1

The MSAM projects, which are discussed below, are designed to address some of these barriers for metal 

laser powder bed machines. The MSAM staff have been leading and driving the development of additive 

standards that are required to establish the metal additive manufacturing industry. 

The Characterization of Additive Manufacturing Materials project addresses current needs and 

challenges for AM practitioners in the areas of powder characterization and handling. These two areas 

apply not only to powder bed AM technologies but also to other AM techniques. The strategy pursued in 

this project is to determine the limits and errors of major measurement techniques for AM powders in 

order to quantify uncertainties in the characterization of powders and along the processing chain. The 

project has resulted in excellent contributions that are helping to advance the state of the art in powder 

and powder bed characterization. 

The project excellence is displayed, for example, in the approach to powder characterization where 

several different characterization techniques have been set up, including powder size distribution and 

shape analysis, a custom powder spreading device, different flow and density measurement devices, as 

well as a suite of X-ray tomography instruments for defect analysis in additively manufactured parts. 

The project has generated, and continues to generate, new standards on the use of this equipment to 

characterize powders and parts. These new standards will help in comparing results from different groups 

on powder characterization. Another factor contributing to the excellence in the Characterization of 

Additive Manufacturing Materials project is the cross-project availability of the AM testbed. This testbed 

system is crucial to creating variables in the hardware and software of additive production systems. The 

experiments this testbed enables have great possibilities to advance system capabilities to produce higher 

quality parts. 

Another example of the excellence in the Characterization of Additive Manufacturing Materials 

project is the work on powder recycling. Powder recycling is of great value to industry, and a round robin 

study has been initiated on the effects of powder recycling that will lead to guidelines on measurement 

techniques and uncertainties, and information on how these effects influence the results of recycling 

studies. 

The Characterization of Additive Manufacturing Materials project aims at developing an AM 

database. Features of the web-based AM database include a viewable and navigable AM schema and 

currently accessible data and attributes (which include microstructure information on properties and 

dimensional information). The database is currently composed of NIST-funded round robin data collected 

in 2012 and 2014. Progress, so far, has been made to develop the framework of the database, and, if 

successful, this AM open database effort could significantly help U.S. industry. 

The Qualification for Additive Manufacturing Materials, Processes, and Parts project includes three 

MSAM research areas. The first is generation and interpretation of reference data to serve model-based 

qualification efforts. The second is the development of preproduction run and post-processing test 

methods to assist in equivalence efforts. The third is the establishment of minimum requirements for 

testing to reduce the cost for empirical approaches. Through research in qualification, the EL will reduce 

the cost of empirical approaches and provide foundational information, which will support equivalence 

and model-based qualification approaches. 

Work in the MSAM program contributes to model-based qualification through high-fidelity 

temperature measurements for metal laser-based powder bed fusion, which are used to validate multi-

 1

 National Institute of Standards and Technology (NIST), “Measurement Science for Additive Manufacturing 

Program,” updated December 08, 2016, https://www.nist.gov/programs-projects/measurement-science-additivemanufacturing-program. 
13 

physics models simulating individual melt tracks in an additive manufacturing process and low-fidelity 

temperature measurements typically made with an infrared (IR) camera. For example, reflections on the 

powder bed surface from spatter may be erroneously interpreted to be hot spots on the surface. Machine 

qualification studies have also led to the development of AM XY scanner test patterns. Additionally, an 

MSAM AM test artifact has been developed and is being distributed to assist with equivalence-based 

qualification and system performance characterization. The EL developed a CAD file for a metal plate 

with specific geometrical features, the test artifact. This plate and its features are designed for additive 

manufacturing, and different laboratories could use this particular CAD design to build the part and 

compare geometrical features or microstructures. The test artifact thus represents a key standard for 

additive manufacturing, and similar standards need to be developed for other aspects of additive 

manufacturing. The artifact is, essentially, a square plate containing features or assessing process 

accuracy and repeatability. From surface roughness measurement studies, MSAM researchers have 

concluded that standard RA roughness measures do not fully capture important surface morphology. Use 

of Rpc and RSm has given better surface discrimination than the RA measurement. Research is under way to 

explore relationships found between surface texture and microstructure. X-ray computed tomography is a 

powerful tool for assessing internal defects. MSAM researchers are analyzing the process to establish 

detection limits and to properly interpret images. 

To support empirically based qualification, a methodology for round robin testing has been 

articulated by MSAM staff. The plan is to publish the information as an International Organization for 

Standardization (ISO) American Society for Testing and Materials (ASTM) guide standard. If properly 

conducted, round robin studies distribute the cost burden of testing and provide statistically valuable 

information. 

The Real-Time Monitoring and Control of Additive Manufacturing Processes project team is working 

on visual monitoring of the melt pool in metal laser powder bed machines. They are motivated by the 

need to monitor the process so that correlations between the data and visually detectable flaws in parts 

can be investigated, to use the data as a quality control signature for the part, and to use the data for realtime control. 

Selective laser sintering (SLS) is the dominant method for AM of metal parts, and control of this 

process is a significant issue. The intent of the MSAM work—to improve the process control of selective 

laser melting (SLM)—is sound. Visual monitoring of the melt pool is a good target, and, in fact, visual 

monitoring of the melt pool in SLS has been under investigation by various groups for approximately 20 

years. In principle, NIST’s prowess in measurement technology can form the basis of useful contributions 

to this field. 

This work has been pursued on two hardware platforms: the testbed and the EOS M270 machine. A 

considerable portion of the testbed effort is allocated to an in situ measurement of the emissivity of the 

powder bed. The expectation is that the increased accuracy of the emissivity measurement will allow for 

better process monitoring and, hopefully, better real-time control. A thermal imaging camera has been 

adapted to an EOS M270 machine by fabricating a custom door and a camera mounting system. This 

camera is not concentric with the beam, but views the powder bed at an angle, perhaps in the range of 30 

to 45 degrees from normal to the surface of the powder bed. In an effort to measure emissivity the team 

has done experiments with powder on hot plates at temperatures up to approximately 500oC. The team 

reported experiments on emissivity measurements during the 27th International Solid Freeform 

Fabrication Symposium at the University of Texas, Austin, in 2016. They have explained that 

measurements at higher temperatures are exceedingly difficult—due to the fact that there is a challenge in 

uniformly heating. 

Another MSAM program focus is the Systems Integration for Additive Manufacturing project, the 

objective of which is to deliver an information systems architecture, including metrics, information 
14 

models, and validation methods, to shorten the design-to-product cycle time in AM.2

 The major 

deliverables for this project are separated into six thrust areas: (1) Additive Manufacturing Product Life 

Cycle Management and Digital Support in what is commonly called the “digital thread;” (2) Product 

Definition for Additive Manufacturing; (3) Design Allowable Database for Additive Manufacturing 

Materials (which focuses on statistically based mechanical property data and represents a subset of the 

Characterization of Additive Materials AM database); (4) Design Rules for Additive Manufacturing Parts; 

(5) Characterization and Uncertainty Quantification of Physics-Based Models for Additive 

Manufacturing; and (6) Process Planning Guidelines for Additive Manufacturing. The Systems 

Integration for Additive Manufacturing team has designed the architecture of each of these databases 

using state-of-the-art database methods and software project design tools. They have initially focused on 

just AM metal laser powder bed methods. In particular, they have planned for product definition and 

tolerance representations; AM design rules for metal parts; the characterization of powder bed fusion 

metal physics-based models; data structures for AM metal parts, processes, and materials; and path and 

process planning related to their home-grown powder bed fusion metal machine. 

Opportunities and Challenges 

A key challenge and opportunity for the MSAM program is to identify the limit of NIST’s territory 

within the U.S. science and technology landscape of AM. Some of the activities in the Characterization of 

Additive Manufacturing Materials project—for example, the database development or the powder 

recycling study—overlap with similar activities in the private sector and in academia. The MSAM 

projects seem to be on a good track; however, there is a need to hone in on the measurement and 

standards aspect throughout the diverse range of AM topics currently pursued in the program, and the 

Characterization of Additive Manufacturing Materials project in particular. 

The round robin studies in the Characterizations of Additive Manufacturing Materials project were 

conducted with 

outside partners. The many challenges in controlling AM machines render round robin studies rather 

difficult, and a gradual approach might be better than the full-scale study that was conducted. The inhouse real-time monitoring capabilities could be used even more than they currently are to start with a 

study of repeatability and variations for parts built on the same machine, as well as post-processing. The 

round robin study may also be too ambitious and could benefit from narrowing the parameter space or 

even from using polymers and polymer machines first. The primary goal of the round robin study might 

be modified to focus on the development of round robin testing procedures, and applying these testing 

procedures over multiple runs to understand the variance on a machine (with the constant parameter 

settings as a baseline), rather than determining differences in the properties from different sources per se. 

There are many different materials, machine brands, and build themes—and a quantification of variations 

for all combinations seems to be too broad. 

In connection with powder recycling, more information is needed on material discontinuities or 

consistency within new powders. The Characterization of Additive Manufacturing Materials project team 

has the opportunity to work with material manufacturers (powder producers) on better consistency of 

powders, and to identify critical powder properties for the AM process. This could also result in better 

density in the mechanical powder spreading process. There are many variables: geography, systems, 

calibrations, powders, method of powder shipment, shelf life of the powder, operator or technician 

experience, and the system surroundings or atmosphere. Findings cannot be trusted unless all of these 

variables can be controlled. The testing could possibly involve the same parts, build orientation, and 

 2

 See NIST, “Systems Integration for Additive Manufacturing,” updated July 13, 2017, 

https://www.nist.gov/programs-projects/systems-integration-additive-manufacturing. 
15 

support structure. Recycling could have many variables involving differences within the new powder 

added, including physical chemical properties, chemical composition, and the size and shape of powder, 

and these variables could play into the testing if they vary throughout different orders or lot numbers from 

the manufacturer. These factors need to be studied before recycling testing. 

The MSAM AM database competes with commercial codes and it is not clear what the pathway is to 

populate the database. It may be better to focus this endeavor on a database structure and format, which 

could be very useful as a framework to be made openly available to the AM community—specifically, 

schema. 

Another opportunity would be to broaden the manufacturing focus within AM, or consider closed 

loop manufacturing for additive technologies. In almost every manufacturing process or method there are 

challenges to produce high integrity or quality parts or components that do not need secondary processes 

or mechanical treatment. Within the limitations of AM to produce high-quality parts, secondary processes 

will be needed to produce suitable parts or components for many industries. Broadening the focus into 

post- or secondary processing will help researchers understand what limitations within the technologies 

cannot be overcome with post-processing methods. Expanding research veins into support structures and 

post-processing, and defining new procedures in this field, would create an opportunity to develop new 

standards, which could help industry find common ground. 

The use of software to analyze images utilizing algorithms to understand the laser effects within the 

powder bed would help to improve the technology itself. Another opportunity within the Characterization 

of Additive Manufacturing Materials project is to examine more closely (with the inclusion of a 

measurement of the degree of planarity in powder spreading) the mechanical powder spreading methods, 

being blade or rollers. 

The project could advance AM if software, in situ measurement data and images, and hardware could 

be correlated. While progress in that direction is under way for laser effects on powder, the integration of 

software and hardware needs to be continued and expanded into additive printing systems. 

For the Qualification for Additive Manufacturing Materials, Processes, and Parts project, the team is 

addressing key elements of precompetitive AM part and process qualification. They are overall doing a 

very good job of advancing the primary research tasks. The results will be helpful to industry in 

qualifying parts for service; however, for other areas within AM, it would be helpful to expand the study 

to include AM processes other than metal laser-based powder bed fusion. 

Real-time monitoring imposes a significant computational burden and for the Real-Time Monitoring 

and Control of Additive Manufacturing Processes project, the overriding concern is that there is not 

enough evidence of having laid the proper groundwork before embarking on a costly and time-consuming 

research path. It would be good to see more evidence of an informed effort to measure emissivity off-line. 

The team might also want to consider experiments designed to bracket the possible impact of uncertainty 

in emissivity values on the SLS process. For the temperature measurements a case could be made for why 

two-color pyrometry—a standard approach to simultaneous measurement of temperature and 

emissivity—could not be used. 

On a more fundamental level it is noted that the measurement of emissivity is actually a proxy for a 

measurement of absorptivity. That is, the optical property of most interest in SLS is the absorptivity of the 

powder bed. At steady state it is known from instantaneous power balance that absorptivity = emissivity + 

transmissivity. For an opaque body, transmissivity = 0, leading to the familiar result that absorptivity = 

emissivity. In this case measuring emissivity will also give absorptivity. A powder bed, however, is 

opaque only over some depth. Measuring emissivity indicates only what the absorptivity is through a 

depth of powder. It is possible for two powder beds to have the same emissivity (and absorptivity), one 

absorbing essentially all energy within the first 100 microns of powder, and the other, in an extreme case, 

requiring 5 millimeter (mm), or 50 times the depth to fully absorb the power. The latter will require far 

more power to achieve sintering and melting than the former. And so, knowledge of emissivity alone is 

not sufficient. The proposition that it is difficult to uniformly heat a small target of metal powder up to 

temperatures above 500oC is simply not correct. In fact, there are several types of relatively low-cost 

electric furnaces that will do this job admirably. Up to temperatures of 1200oC, Kanthal wire may be used 
16 

as a heating element. Silicon carbide heating elements can be used to temperatures of 1500oC. Moly disilicide elements (a bit more expensive) can be used up to 1800oC. All three types are readily available as 

off-the-shelf units, and there are many vendors who will build custom units. The combination of a wellinsulated furnace—which would act like an integrating sphere—and a light chopping wheel for the 

incident light- and phase-sensitive, wavelength-dependent detection could provide accurate data. 

It is highly desirable to see a series of experiments where the powder is deliberately modified to vary 

the emissivity. These powders used in an SLS machine, in order to get a sense of the magnitude of 

emissivity difference, need to have an impact on process control. For example, the emissivity of the 

powder could be increased by etching the powder so as to create surface texture, and the impact on the 

process could then be assessed. 

It is also desirable to see results where the robustness of SLS against changes in laser power or 

scanning speed are assessed for impact on part quality. For example, a 5 percent increase in laser power 

could be used to simulate a 5 percent relative increase in emissivity. If this makes a noticeable difference 

in the sintering process, then that is an indication that measurement of emissivity differences of 5 percent 

is useful. Several such experiments would bracket the accuracy of emissivity measurement that is desired. 

At the temperatures of a typical melt pool (1400oC and higher), two-color pyrometry is very widely 

used. In this application, the light emitted at two wavelengths is measured. Under the assumption that the 

emissivity is the same at both wavelengths, both temperature and emissivity can be extracted from the 

data. At these temperatures the two wavelengths can be relatively close together, making emissivity 

dependence on wavelength less likely. For example, silicon detectors with filters have been used to select 

approximately 550 nanometers (nm) and 950 nm. A comparison of emissivity measurement using an offthe-shelf two-color pyrometer with off-line measurements would be a good starting point. 

Furthermore, it would be useful to see experiments on light penetration into powder beds as a 

function of depth—even at room temperature. The depth of penetration would be compared with the 

depth of a melt pool. If the depth of penetration is larger than the thickness of a melt pool, this factor 

would need to be taken into account. 

For the Systems Integration for Additive Manufacturing project, the focus on metal laser-based 

powder bed fusion systems is a good starting point; however, planning needs to include processes that are 

used more frequently today for manufacturing, such as powder bed fusion polymer systems and extrusion 

systems. Archiving better design rules and methods for AM is a very useful and important mission for 

NIST. Also, developing new and improved dimensioning and tolerance methods for this new technology 

is a task that is very appropriate for this organization. Both of these project elements could lead to new 

standards in the field. 

Although the architecture of these databases is very comprehensive, populating these databases with 

useful data represents a real challenge. These data are often held very close by commercial entities and 

they will be reluctant to share this information with others. The EL might want to consider licensing or 

giving out the architecture as open source and letting commercial database companies use it. These 

companies could buy the data as required and sell the results. 

Although the desire to focus on carefully selected areas to start is understood, one item that seems 

glaringly absent is the design and use of support structures in metal parts. The design of these supports 

and their removal represents a substantial amount of the manufacturing time to deliver a metal AM part. 

Support structures may already be addressed in the project; however, they were not clearly discussed 

during the review. 

The vast majority of National Laboratories currently pursue AM, and the significant progress that has 

been made within the MSAM program could benefit other laboratories that, in turn, could likely help the 

MSAM program. Additionally, a dialogue could be intensified with MML and its Polymers Processing 

Group in order to exploit common strengths and overlapping research interests to evolve toward common 

research projects, such as feedstock characterization or the design of additive manufacturing machines. 

Such a dialogue could build on existing meetings with the MML and the Polymers Processing Group. 

Several highly industry relevant topics within AM are not currently pursued with sufficient emphasis. 

These topics include support structures and their design, the effects of different build orientations, design 
17 

limitations for AM, and, as previously mentioned, the post-processing side of AM. Expanding research 

veins into support structures and post-processing, and defining new procedures in this field, would create 

an opportunity to develop new standards, which could help industry find common ground. 

Last, for the MSAM program, the dynamic and fast-changing nature of metal laser powder bed 

machines requires continual project reviews, but also decisions to end projects early if project goals and 

elements become obsolete. 

PORTFOLIO OF SCIENTIFIC EXPERTISE 

Accomplishments 

MSAM staff are very qualified with good trajectories for achieving broad internationally recognized 

technical leadership. They are a new group that was formed about 3 years ago. The MSAM is composed 

of 21 Federal employees, 4 NIST associates, and 4 students—62% of their staff have Ph.Ds, and 17% 

have masters degrees. The group is technically qualified to perform the experimental investigations and 

accomplish the project objectives. The overall team is comprised of diverse engineering backgrounds that 

undoubtedly help with the overall program goals. 

The MSAM has published 14 journal papers since the last review, and the key personnel involved 

with the Qualification for Additive Manufacturing Materials, Processes, and Parts project efforts have 

contributed to this number through the publishing of archival journal articles as well as standards. Several 

project researchers have been recognized for outstanding contributions and collaborations, including the 

2016 America Makes Distinguished Collaborator Award. Additionally, staff within the Characterization 

of Additive Manufacturing Materials project have won the 2015 Best Paper/Presentation Award at the 

26th International Solid Freeform Fabrication Symposium and the 2016 Editor’s Choice Article in the 

Journal of Materials Engineering and Performance . Overall, these teams are developing into a strong 

talent pool of AM expertise.

The Real-Time Monitoring and Control of Additive Manufacturing Processes project brings together 

several NIST researchers with different backgrounds. For the tasks the team has set out to achieve, this 

broad array of scientific expertise is particularly well-suited. 

The two key personnel involved with the Systems Integration for Additive Manufacturing project 

efforts have been recognized for outstanding contributions and collaboration and are both publishing 

archival journal articles and standards at an acceptable rate. Although the team is small, its size appears 

to be appropriate with respect to the early stage of many elements of the design rules, commercial 

processes, and machine development, as well as other aspects of metal laser powder bed machines. 

The team is quite capable and the architecture it has developed is very comprehensive. 

Opportunities and Challenges 

The Characterization of Additive Manufacturing Materials project currently uses several different 

commercial equipment items for powder characterization and also for mechanical testing and for postprocessing. Given the goal not only to use the equipment but also to understand its limitations and 

measurement uncertainties, it is quite a challenge for the team to master all equipment items and 

techniques at the necessary level. The MSAM could consider convening a meeting to obtain user input on 

the round robin study in order to define ways to narrow the scope to focus activities on NIST’s mission in 

supporting industry. 

Like other areas within the EL AM effort, the Qualification for Additive Manufacturing Materials, 

Processes, and Parts team consists of junior researchers who are, for the most part, recent graduates. Their 

trajectory to outstanding experienced researchers is impressive, and as they continue doing excellent, 

relevant research, they are likely to become better known and will be suitably recognized. 
18 

The Real-Time Monitoring and Control of Additive Manufacturing Processes project appears to touch 

upon several different scientific and engineering topics, and the current and future monitoring projects 

might benefit from additional expertise—for example, in control theory or solid state physics—within 

NIST. If the MSAM was to move into support structures, it could benefit from additional expertise in 

component designs. 

ADEQUACY OF FACILITIES, EQUIPMENT, AND HUMAN RESOURCES 

Accomplishments 

State-of-the-art powder characterization equipment was set up to support the Characterization of 

Additive Manufacturing Materials project and the cross-project availability of the AM testbed. The 

equipment includes a custom raking device to study the motion of powder particles during powder 

spreading, as well as a range of commercial equipment. As previously mentioned, several different 

characterization techniques have been set up. These techniques include the following: powder size 

distribution and shape analysis; a custom powder spreading device; different flow and density 

measurement devices; and a suite of X-ray tomography instruments for defect analysis in additively 

manufactured parts. Collectively, the equipment is highly adequate to meet the project objectives. The 

database part of the project draws from experts on database development within NIST. 

For the Real-Time Monitoring and Control of Additive Manufacturing Processes project, the AM 

testbed facility is a major investment both for the actual testbed and for the added equipment—for 

example, the EOS M270 machine and testbed and the temperature measurement equipment that is 

coupled to the beam parameter values. Enabling full control over the laser and galvo system is invaluable 

for the team in setting up the testbed for calibrations of secondary equipment. 

The Qualification for Additive Manufacturing Materials, Processes, and Parts project and the Systems 

Integration for Additive Manufacturing project both appear to have adequate facilities, equipment, and 

human resources. 

Opportunities and Challenges 

For the Qualification for Additive Manufacturing Materials, Processes, and Parts project expansion to 

other metal-based AM processes will necessitate purchase of new equipment. 

A major challenge is that the Real-Time Monitoring and Control of Additive Manufacturing 

Processes project testbed represents one particular AM technology—powder bed AM. Other metal AM 

technologies exist, and new technologies could emerge very quickly that could render the powder-bed 

technology less relevant than it currently is. The testbed and other real-time monitoring equipment need to 

be devised with the greatest possible flexibility in mind. A larger vacuum chamber in the testbed, for 

example, could possibly accommodate a small robot for electron-beam or plasma gun AM. 

The Systems Integration for Additive Manufacturing project needs to develop a plan for population of 

its database that can bring in statistically significant information. 
19 

3 

Robotic Systems for Smart Manufacturing 

INTRODUCTION 

The mission of the Robotic Systems for Smart Manufacturing (RSSM) program is to 

Develop and deploy advances in measurement science that enhance the U.S. innovation and industrial 

competitiveness by improving robotic system performance, collaboration, agility, and ease of 

integration into the enterprise to achieve dynamic production for assembly-centric manufacturing.1

This is achieved through a variety of technical activities, including the following: discussions with 

stakeholders to assess needs; identifying gaps in standards; researching in relevant hardware to define 

useful metrics; developing technological best practices; defining test methods and artifacts; disseminating 

results that contribute to standards development; forming working groups of relevant stakeholders; and 

leading in standards development. This work is conducted primarily by personnel from the EL Intelligent 

Systems Division. The scientific staff is organized into five groups: (1) Cognition and Collaboration 

Systems; (2) Manipulation and Mobility Systems; (3) Networked Control Systems; (4) Production 

Systems; and (5) Sensing and Perception Systems. The program has a budget of $7 million. There are 16 

full-time equivalent (FTE) NIST employees in this area, along with 3 guest researchers and one 

contractor. 

ASSESSMENT OF TECHNICAL PROGRAMS 

Accomplishments 

The technical activities of the RSSM program seek to address several current challenges experienced 

by SMEs in their use of robotics. In particular, the vision is for SMEs to have access to robots that are 

easy to install and integrate into their industrial enterprise; that are reconfigurable, adaptable, and agile; 

and that can partner with humans to amplify productivity and quality. Technical work is organized around 

five projects that address these open challenges: (1) Performance Assessment Framework for Robotic 

Systems; (2) Performance of Collaborative Robotic Systems; (3) Tools for Collaborative Robots within 

SME Workcells; (4) Agility Performance of Robotic Systems; and (5) Robotic Systems Interoperability 

and Integration. The RSSM program has acquired state-of-the-art equipment and developed appropriate 

testbeds for the respective projects. The RSSM vision and projects are well aligned with the current state 

 1

 National Institute of Standards and Technology, “Robotic Systems for Smart Manufacturing Program,” 

updated July 20, 2017, https://www.nist.gov/programs-projects/robotic-systems-smart-manufacturing-program. 
20 

of the field, and adeptly address the key challenges faced by SMEs that currently limit the use of robotics 

in industry. 

The RSSM staff has demonstrated good alignment with NIST’s mission through leadership and active 

participation in standards and metrics developments. The researchers have gathered input and identified 

needs from stakeholders to formulate project objectives and directions. To engage the broader community 

and ensure impact, the staff has participated in leadership roles in standards committees, professional 

organizations, and trade organizations. Results have been disseminated through publications, workshops 

with professional societies, and inputs to consortia. The staff needs to be particularly commended for 

using competitions at major robotics conferences (e.g., Intelligent Robots and Systems [IROS] and 

International Conference on Robotics and Automation [ICRA]) to engage the community and generate 

inputs and requirements. The five technical projects within the RSSM program have all made notable 

technical contributions. 

Significant accomplishments of the RSSM program involve substantial contributions to important 

new standards. The Agility Performance of Robotics Systems project has led to the development of a new 

standard for knowledge representation for robot systems, Institute for Electrical and Electronics 

Engineering (IEEE) 1872-2015 Standard Ontologies for Robotics and Automation, which has been 

adopted and used by other organizations. 

The Performance of Collaborative Robot Systems project has provided critical contributions to the 

development of multiple standards for safe operations of collaborative robots, including the first 

international technical specification for safe operation of collaborative industrial robot systems, ISO 

Technical Specification 15066 Robot and Robotic Devices—Collaborative Robots. 

The RSSM Performance Assessment Framework for Robotic Systems project team founded the 

Committee on Performance Standards for Industrial Vehicles, and developed the ANSI/ITSDG B56.5 

Safety Standard for Driverless, Automatic Guided Industrial Vehicles and Automated Functions of 

Manned Industrial Vehicles. It also led the standards development through ASTM E57 3D Imaging 

Systems for image-based test systems, including test methods for six-dimensional pose measurement. 

The RSSM staff has also served in a leadership role in several activities that will drive future 

standards. These include the development of evaluation metrics for robotics, led by the Performance 

Assessment Framework for Robotic Systems project, which resulted in the formation of the IEEE Grasp 

Metrics Working Group for grasping metrics. The Agility Performance of Robotic Systems project and 

the Robotic Systems Interoperability and Integration project created a canonical robot command language 

and tools that have been adopted by other organizations. The agility project also conceived, developed, 

and conducted competitions in conjunction with the IEEE Conference on Automation Science and 

Engineering (CASE) conference to drive future standards on robot agility with participation within the 

United States and internationally, including both industry and academia. These projects also developed 

test methods to identify inadequacies in existing tools, such as Robot Operating System (ROS) Industrial, 

that impede interoperability and agility. 

The RSSM staff have also successfully built a set of relevant testbeds for generation of technical 

input to standards, including the mobile manipulation, collaborative robotics, multifinger grasp, and robot 

agility testbeds. These testbeds offer the opportunity to explore and evaluate interoperability and 

integration. These testbeds are critically important for the RSSM research that leads toward standards 

development. 

All of the RSSM projects have actively disseminated their results through publications, data sharing, 

and invited talks. In particular, the RSSM staff have established a healthy and strong publication record in 

leading robotics journals and conferences such as IEEE Transactions on Automation Science and 

Engineering; IEEE Transactions on Systems, Man, and Cybernetics; and others. The Performance 

Assessment Framework for Robotic Systems project has shared some of its test data with the research 

community, including its grasp test data for commercial multifinger hands. They also regularly speak at 

relevant technical venues to discuss their activities in standards development. 

The RSSM staff also actively engage with the user community through workshops and other 

meetings. For example, the team conducted the 2015 Collaborative Robotics Workshop and reached out 
21 

to Manufacturing Extension Partnership (MEP) centers to develop a deeper understanding of the key 

obstacles to robotics adoption, as well as the technology needs of SMEs. They have also actively 

participated in various consortiums, including the ROS-Industrial Consortium, Robotic Industries 

Association, and the Advanced Robotics for Manufacturing (ARM) Institute to discuss standards 

development. They regularly collaborate with other organizations, including universities, companies, and 

consortiums, to conduct joint tests and leverage results. 

Opportunities and Challenges 

The robotics technical area is rapidly changing, especially in software and data-driven methods, such 

as deep learning, and this creates a number of technical challenges in keeping up with the state of the field 

and the practical impact of the approaches being developed in the RSSM program. However, the RSSM 

program also has unique opportunities to leverage NIST’s traditional leadership in standards and 

technology, its in-house expertise, and its broad array of coalitions to forge consensus, break new paths, 

and deliver lasting impact. One example of an opportunity would be for EL to broaden its industrial 

impact through stronger interactions with MEP centers and through other industrial consortia. EL can 

develop a set of use cases and obtain more in-depth feedback on standards development through these 

deepened interactions. EL can also ensure that the selected projects achieve the broadest impact on 

industry. 

Furthermore, the RSSM staff could lead in the development of a more systematic methodology for 

generating test and use cases. This methodology could include gap identification, research, developing 

metrics, developing test methods and artifacts, standard development, evaluation, and dissemination. 

The RSSM program can further ensure relevance by working more closely with industry (including 

newer companies) in testing and evaluation. The RSSM staff could also aim more of their dissemination 

activities to industry by publishing more articles in trade magazines and by holding workshops in industry 

tradeshows (e.g., Automate). 

The RSSM program has a lack of crosscutting activities that take advantage of synergistic objectives 

of the individual projects. Integrating the testing and evaluation work across the overall program would 

provide more synergy across the projects, as well as an opportunity to achieve broader impact of the 

program work as a whole, rather than only at the project level. The RSSM staff also has potential 

opportunities to leverage other fast-developing robotics domains, such as driverless cars, for relevant 

knowledge of standards development in smart manufacturing and industrial automation. 

While vision sensors are used in several testbeds, the RSSM program has an unexplored opportunity 

to investigate how machine vision, and associated algorithms such as deep learning, can be used to 

enhance the flexibility, user-friendliness, and safety of robots for SMEs. The exploration of how to take 

advantage of machine vision to simplify robot programming, develop self-teaching techniques, ensure 

safe human-robot interaction, and adapt to the varied application environments at SME facilities conforms 

well to the EL objectives for the RSSM program. There is also an increasing utilization of cloud and edge 

resources for robotics and industrial automation. The Industry 4.0, Microsoft Azure, and Amazon Web 

Services (AWS) efforts are trying to integrate resources in a consistent and efficient framework, and the 

RSSM program has the opportunity to contribute to and take a leadership role in this emerging field. 

The RSSM has developed many test methods, artifacts, and metrics to guide in the design and use of 

grippers for dexterous grasping. They could leverage this research to create community-wide test sets that 

can assist the research community in developing new approaches to dexterous grasping. They could make 

a contribution to the research community similar to that of the Yale-CMU-Berkeley (YCB) Object and 

Model Set data set, which helps advance robotic manipulation. 
22 

PORTFOLIO OF SCIENTIFIC EXPERTISE 

Accomplishments 

NIST has a long history in robotics research. The RSSM program has assembled a team of 

researchers with expertise in computer vision and perception, human-machine interaction, collaborative 

robots, machining learning, mathematics and algorithms, performance metrics, and testing. Their 

educational backgrounds include computer science, computer engineering, electrical engineering, and 

mechanical engineering, as well as engineering sciences and mathematics. Across the team, 52 percent of 

the research staff hold Ph.D.s, 36 percent hold master’s degrees, and the remainder have B.S. degrees. 

The RSSM program’s scientific staff is strong and intellectually diverse, with broad expertise and 

educational preparation in the many technical areas that are necessary for achieving the program goals. 

This excellent mix of research interests and professional preparation enables the staff to accomplish the 

research necessary to support the development, testing, and evaluation of performance standards for 

robotics in manufacturing. 

Several RSSM staff members are well established in their fields, as evidenced by their high number 

of top-quality peer-reviewed publications; their leadership roles in standards development, professional 

societies, and conferences; as well as their many invited presentations. They have achieved a high level of 

productivity comparable to highly performing faculty in top research universities. RSSM researchers are 

also recipients of several external awards, including the IEEE Standards Association Emerging 

Technology Award and the ASTM International Robert J. Painter Memorial Award for meritorious 

service. 

Because the RSSM researchers are experts in their respective fields of research and have many 

collective years of experience in the profession, they understand the state of the art in robotics for 

manufacturing. They are also active in technical meetings and have established strong connections with 

relevant professional organizations and collaborators. Members of the staff serve in leadership roles in 

technical committees (TCs), including the IEEE Robotics and Automation Society (RAS) TC on 

Performance Evaluation and Benchmarking, and the IEEE RAS TC on Robotic Hands Grasping and 

Manipulation. The RSSM staff are also active as organizers of professional conferences, workshops, 

competitions, and summer schools. The RSSM program has established collaborations with a number of 

universities and companies. Four guest researchers currently on the RSSM staff are from the collaborative 

partner universities. All of these engagement activities are important for enabling the RSSM staff to 

understand the needs of the industrial user community and draw on the collective manufacturing robotics 

expertise across the technical community. 

Opportunities and Challenges 

Given the strong records of performance of the RSSM researchers, greater recognition for the 

program (and for NIST) could be achieved through external merit-based robotics awards, early-career 

investigator awards, elevation in professional societies, as well as invitations to deliver opening and 

keynote presentations at conferences. NIST researchers are eligible for many prestigious awards, 

depending on their current career stage. Examples of relevant awards include fellows of professional 

societies, the Robotics Industry Association (RIA) Engelberger Robotics Award, invitations to Frontiers 

of Engineering meetings, the Society of Manufacturing Engineers Outstanding Young Manufacturing 

Engineer Award, the International Symposium for Flexible Automation Young Investigator Award, and 

the IEEE RAS Early Career Award. This type of external recognition can serve not only to award 

deserving individuals but also to raise the visibility of EL in the field. 

The RSSM program could more systematically leverage (and seek out) expertise from other 

organizations, including university and industry, and through an extended use of the guest researcher 

programs. A consortium of industrial companies, both large and small, as well as universities that meet 
23 

with the RSSM staff on an annual or semi-annual basis, could serve as additional sources of expertise for 

needs assessment, gap identification, and the testing and validation of standards. The RSSM program 

could also leverage regional programs and facilities affiliated with MEPs. In the interoperability area, the 

RSSM program could coordinate with broader efforts in industrial automation, including Industrial 

Internet, Industry 4.0, ROS 2.0, and commercial activities such as Microsoft Azure and Amazon AWS. 

RSSM researchers publish their work in highly technical journals and professional conferences. 

While such publications are encouraged, the RSSM could also expand its scientific network and the 

impact of its work by enhancing translation of its technical work through publication of critical use cases 

in trade magazines. They could also conduct a road show to discuss important standards-relevant 

challenges, coordinating with ARM and other organizations such as ROS-Industrial in the roll-out of 

standards and performance testing and evaluation methods. 

Human-machine collaboration is an important challenge in robotics for manufacturing. The RSSM 

program appears to have an expertise gap in human robot interaction due to lack of permanent staff in this 

area. It is important to actively recruit researchers in human-robot interaction into the EL permanent staff 

in order to establish this topic as an active research area. 

ADEQUACY OF FACILITIES, EQUIPMENT, AND HUMAN RESOURCES 

Accomplishments 

The RSSM researchers have several well-equipped laboratories for conducting their research projects. 

These laboratories include testbeds and facilities for safety validation systems, multirobot collaboration, 

human-robot interaction, dexterous grasping performance measurement, task-oriented performance 

measurement, mobile industrial vehicle testing, mobile manipulator performance, agility testing, and 

sensor/robot calibration and registration. These facilities are well equipped with many state-of-the-art 

industrial and collaborative robots from top robotics vendors (e.g., Kuka, Rethink Robotics, Universal 

Robots, ABB Robotics, Fanuc, and Motoman); state-of-the-art versatile grippers and high-degree-offreedom hands with touch sensors (e.g., Schunk, Robotiq, Allegro Robotics LDT, SoftRobotics, and 

Empire Robotics); mobile industrial vehicles with manipulation (e.g., America in Motion Automatic 

Guided Vehicles [AGVs], Adept Lynx); many relevant sensor systems (e.g., monocular and stereo vision 

and structured light); devices for human-robot interaction (e.g., consumer wearables, and devices for 

virtual and augmented reality); and high-accuracy ground truth position tracking measurement systems. 

These facilities are comparable to the best-equipped robotics laboratories in the world, and so provide an 

excellent environment in which to conduct standards development research in the RSSM. 

In summary, the RSSM program has adequate facilities and equipment supportive of the type and 

requirements of their projects. The laboratory facilities are well equipped with machines, robots, 

instruments, equipment, and related materials that are necessary to conduct high-quality research. For the 

type and number of projects pursued, the human resources are also qualified and adequate. These 

resources appear to be appropriate for the capacity of the assigned space, without observable waste or 

redundancy. 

Opportunities and Challenges 

The RSSM program has two main opportunities to better accomplish its objectives through enhanced 

facilities, equipment, and human resources. The first is to improve the visibility of the program’s 

activities and output by increasing human resources for media outreach and information dissemination. 

The ability of RSSM staff to share videos, images, and other information through social media is hindered 

by the lengthy approval process at NIST and the lack of staff with expertise in social media outreach. As a 

result, the RSSM has very little visibility on social media, which is a missed opportunity to improve the 
24 

impact of the program’s activities. The RSSM needs to consider adding staff who can improve 

dissemination of technical accomplishments through social media, trade publications, and/or popular 

press. While the RSSM program does have some presence in scientific and trade associations, such as the 

IEEE and the American Society of Mechanical Engineers (ASME), it could increase its visibility in trade 

publications and popular press that are read by industry practitioners. For example, increasing media 

coverage of the limitations of current standards for robotic safety and agility and the work the EL has 

done to address these limitations would help disseminate the work of the EL to more practitioners. 

The second area of opportunity is to arrange the equipment and facilities to allow for closer 

integration of activities across the program. While research facilities are adequate, they are distributed 

across the NIST campus and are physically disconnected across multiple buildings on the campus. The 

lack of crosscutting activities across all the projects of the RSSM program is not helped by the distance 

between laboratory spaces. Given that the RSSM research projects share many common technologies, 

science, and resource needs, and can benefit from knowledge exchange, these separations between the 

project teams might contribute to less cohesiveness and overall impact than might be possible through colocated facilities. 

Several of the testbeds developed for individual projects have overlapping capabilities, but cannot 

currently be integrated because they do not have interoperable software. There is an opportunity to 

develop software infrastructure resources to facilitate the interoperability of testbeds developed in 

individual projects. 
25 

4 

Smart Manufacturing Operations Planning and Control 

INTRODUCTION 

Smart Manufacturing Operations Planning and Control (SMOPAC) is one of the programs under the 

EL’s smart manufacturing research focus. The mission of this program is to 

Develop and deploy advances in measurement science that enable performance, quality, 

interoperability, wireless, and cybersecurity standards for real-time prognostics and health 

management, control, and optimization of smart manufacturing systems.1

This program currently has a budget of $9.7 million. There are 24 full-time equivalent (FTE) employees 

that include 22 federal staff, 3 guest researchers, and 12 new hires. 

ASSESSMENT OF TECHNICAL PROGRAMS 

Accomplishments 

During the review five SMOPAC projects were presented: (1) Digital Thread for Smart 

Manufacturing; (2) Prognostics, Health Management, and Control; (3) Wireless Systems for Industrial 

Environments; (4) Cybersecurity for Smart Manufacturing Systems; and (5) Systems Analysis Integration 

for Smart Manufacturing Operations. In all of these projects, the teams have demonstrated multiple 

accomplishments through the standards that have been produced and the awards and recognition that have 

been received. All five projects exhibit sound approaches; each project has captured the state of the 

industry, identified current practices and standards, defined the challenges, and presented proposals for 

obtaining solutions. The projects all have inputs or collaborations with industry, academia, and/or 

government agencies and laboratories. 

Over the past three decades certain industries (e.g., integrated circuits) have developed tools that 

allow the development of a product from concept to tracking in the field utilizing near-perfect models at 

each stage in the process. Industries associated with mechatronic products (i.e., a blend of mechanical and 

electronic systems), have attempted to develop a similar concept to the field tracking model-based 

approach as well, and the phrase “digital thread” has been employed for this concept. A great deal of 

resources have been devoted to such approaches, especially in the aerospace and automotive industries. 

These industries have validated approaches for all the key individual processes involving product 

development, manufacturing, and field diagnostics. However, it is desired that basic information be 

 1

 National Institute of Standards and Technology (NIST), “Smart Manufacturing Operations Planning and 

Control Program,” updated March 17, 2017, https://www.nist.gov/programs-projects/smart-manufacturingoperations-planning-and-control-program. 
26 

utilized throughout the total process, as opposed to inefficient starting and stopping as the product moves 

from initial concept to placement in the field. 

The EL is well aware of this problem area and was instrumental in developing one of the major 

standards involved in this process, namely the Standard for the Exchange of Product Data (STEP). The 

development of standards beyond STEP has been a very difficult process, and remains a major challenge 

for numerous reasons (e.g., industries protecting proprietary information and total systems models 

involving multiple time and spatial scales). 

The goal of the Digital Thread for Smart Manufacturing project is to reduce the time required for part 

production by ensuring that the correct part requirements are provided. The team is developing a sound 

toolkit in order to support industry in managing their integrated design, manufacturing, and product 

lifecycle data based on a number of standards. It has been well-received to date, and information has been 

disseminated widely. There are similarities between the Digital Thread for Smart Manufacturing project 

and the DoD ManTech projects, which include the Army, Navy, and Air Force projects on digital thread 

and digital twin. It appears that there is communication between these groups. This is to be encouraged, as 

it allows NIST to have a government customer for collaboration. 

The goal of the Prognostics, Health Management, and Control (PHMC) project is to deliver methods, 

protocols, and tools for robust sensing, diagnostics, prognostics, and control that enable manufacturers to 

respond to planned (e.g., scheduled change-overs, new productivity targets) and unplanned (e.g., faults, 

failures) performance changes, thereby enhancing the efficiency of smart manufacturing systems. This is 

done through the identification and analysis of key data that would support decision making. This project 

is also focused on providing vendor-neutral approaches and plug-and-play solutions. The work covers all 

levels—component, work cell, and systems, and is an area of great opportunity. This is clearly 

demonstrated by the automotive and aerospace industries’ use of diagnostics (e.g., determining on-board 

emissions sensor deterioration on the automobile in the field and in aircraft monitoring) that allow users 

to assess the health of the platform. Analysis of the data from the systems’ health monitoring is used to 

develop prognostics used for vehicle maintenance and sustainment. 

An efficient and effective linear axis error detection methodology based on data collected from an 

inertial measurement unit (IMU) has been developed and verified on the linear axis testbed and machine 

tools within the PHMC project. Validation efforts are ongoing both internally within NIST and externally 

with several manufacturing collaborators. The team has developed a very unique program and testbed for 

prognostics and health management, and it has the proper hardware to work on machining projects for the 

manufacturing sector. The developed IMU sensor is a practical means for rapidly assessing machine 

health degradation. The team has demonstrated platform-based technologies to allow diversified users to 

connect machines and share Prognostics, Health Management, and Control project data. There is an 

expectation that this will be deployed in actual industrial settings. The plan is to develop an updated 

testbed with more integrated sensors to diagnose some typical problems in machining. 

The Cybersecurity for Smart Manufacturing Systems project will establish a cybersecurity risk 

management program and validate this in real and virtual manufacturing systems. The roadmapping 

activity has established concerns of manufacturers relative to cybersecurity. The problem that was 

identified was that manufacturers are reluctant to adopt cybersecurity technologies because they are 

concerned about potential negative impacts to their manufacturing systems. The team has also identified 

the different cybersecurity standards that are currently in use. Tools and guidelines that the SMOPAC 

program has had a part in developing are well disseminated and are being tested on industrial control 

systems. A manufacturing profile is being published that will be applied to industry practices. A testbed 

has been established to validate the cybersecurity framework manufacturing profile. Data obtained from 

this research will be used to develop guidance for implementation of the framework. 

The Wireless Systems for Industrial Environments project is designed to provide radio frequency 

(RF) measurements in factory environments that will be used to develop tools for measuring wireless and 

factory performance. The challenges that this project faces are in understanding how solutions may differ 

with the various systems, understanding the relationship between performance and the data being 

measured, and understanding how this information is used for prediction in future systems. The team has 
27 

invested in an RF channel emulator and has created a testbed that simulates industrial environments. This 

will provide significant capabilities for measurement and prediction. The approach recognizes the 

variability and resulting complexity of different manufacturing processes. Advanced data analytics 

methods will need to be demonstrated to manage interpretation of this data. 

Opportunities and Challenges 

The Digital Thread for Smart Manufacturing project is using a definition of “life cycle” that shows 

the end of the cycle as being when the initial product is delivered to the customer. This end of the life 

cycle is different from the DoD ManTech projects, which include the Army, Navy, and Air Force projects 

on digital thread and digital twin, where life cycle also includes the life of the aircraft. In all fields 

products have increasingly tremendous sensory and computational capabilities, and “in the field” 

capabilities for diagnostics and prognostics need to be considered as part of the life cycle in the digital 

thread. In aerospace and automotive industries (as well as with health care devices and others), the 

manufacturer needs to have the ability to track the vehicles and devices in the field for potential part 

defects as long as they are in service, and so life cycle is from cradle to grave. The SMOPAC program 

needs to extend the definition of life cycle to be more comprehensive and include manufacturing activities 

after the product is delivered. 

While some of the tools and standards that are currently being developed may apply to post-delivery 

manufacturing, the manufacturing that occurs in maintenance, repair, and overhaul (MRO) and in depots 

may be different due to limitations on materials and equipment. These limitations may make the 

SMOPAC tools not useful in depots and in MROs. Modifications are constantly being made to many 

platforms for a variety of reasons that include upgrades of equipment, redesigns to improve performance, 

new materials, and incorporation of new technologies for manufacturing. The system needs to be flexible 

enough to accommodate these changes so that the goal can be achieved. The Digital Thread for Smart 

Manufacturing team needs to demonstrate clearly how modifications to the products and advanced 

technologies will be handled in the system that is being developed. The team can further consider sensorrich and sensorless environments to integrate process and usage sensor data into closed-loop product 

design support. 

The concept of digital thread can be applied to multiple industries of various sizes. This project is 

targeting medium-size manufacturers. While smaller manufacturers may either already have, or be able to 

adopt, these tools and standards, there is a question as to how large integrators (e.g., aerospace, 

automotive) will adopt NIST’s unique tools. What are the implications and challenges, technically and 

economically, for the range of companies that could use these tools? The SMOPAC program needs to put 

together a business case for small-, medium-, and large-scale industries to use these tools. 

As previously mentioned, the Prognostics, Health Management, and Control team has developed an 

effective and efficient linear axis error detection methodology based on data collected from an inertial 

measurement unit (IMU), which has been developed and verified on the linear axis testbed and machine 

tools within the PHMC project. Validation efforts are ongoing both internally within NIST and externally 

with several manufacturing collaborators. The developed IMU sensor is very practical to assess machine 

health degradation rapidly. The team has demonstrated platform-based technologies to allow diversified 

users to connect to the machine and share their data, and it is planning to promote its research to system 

levels; however, the research scope needs to be more clearly defined (e.g., type of applications, sensors, 

analytics, etc.) before this happens. This is a relatively well studied and crowded research area 

investigated by both academia and industry. Collaboration with the Clean Energy Smart Manufacturing 

Innovation Institute (CESMII) will help in defining, identifying, and satisfying industry needs and 

requirements, as well as provide efficiencies in areas of overlapping research. The team also needs to 

address the need to apply this technology to other materials and processes such as those used for 

manufacturing polymeric materials or composites. Additionally, while this project is directed at machine 
28 

shops, the team needs to define how the technology would work at the larger systems levels that deal with 

integration and assembly of complex products.

The goal of the Systems Analysis Integration for Smart Manufacturing Operations project is to 

develop information models and transformations for submission to standards organizations to enable a 

more efficient integration of systems analysis. This is being done in collaboration with a number of 

industry and academic institutions. Currently, there are software packages that are employed in the design 

and manufacturing of products, and some of these are integrated to some degree. While the approach and 

areas for integration were well described during the review, it was not clear what had already been 

accomplished. There needed to be a clear linking of this project to the other projects and a description of 

the interdependence for data gathering and analytic development to maintain an integrated modeling 

environment. The relationship of this analysis and the physical findings of the testbed then needs to be 

demonstrated as validation of the data analytics. This project needs to be integrated with the others to 

show how it supports digital thread and prognostics. 

The Cybersecurity for Smart Manufacturing Systems project has established a testbed to validate the 

cybersecurity framework manufacturing profile. The data obtained from this research will be used to 

develop guidance for implementation of this framework. In order to evaluate the resilience of the system, 

the testbed needs to include a simulation system that can generate virtual cyber threats. 

A very sound approach has been established to address the impacts of cybersecurity systems on 

manufacturing systems. However, the team also needs to take into consideration companies that are 

reluctant to move to smart manufacturing because of the fear of being hacked. The frameworks and 

profiles that are being developed could also be enablers for companies that are considering the 

implementation of digital or smart manufacturing. There may be opportunities to incorporate the use of 

the standards at the initial setup of smart manufacturing for these companies. The team is also encouraged 

to include manufacturers that are not currently digital in the discussion of cybersecurity needs. 

Wireless technology is important to all the other SMOPAC projects as well as the other smart 

manufacturing programs. There needs to be more integration in the testbeds of the Wireless Systems for 

Industrial Environments project with other projects and programs to determine the best wireless 

technologies. While the project research is just beginning, it needs to be considered more broadly. 

Additionally, the team needs to demonstrate the approach to data analytics for the wireless systems. It 

also needs to develop testbeds that include both cybersecurity and wireless technology, and it needs to 

consider integrating with a cloud-based or edge-based environment to support a more tether-free 

monitoring system. 

Several overarching suggestions for the SMOPAC program were identified during the review. First, 

the SMOPAC program needs to develop a high-level, integrated roadmap with measurable milestones. 

Such a roadmap would show the relationships of various markets (automotive, aerospace, etc.) and 

requirements to the technologies and between the technical areas themselves. This roadmap could help to 

define strategies for technology planning, resources needs, and the transition to manufacturing systems. 

This roadmap needs to be an integrated, high-level plan, not a set of semi-related programs. They can also 

develop and show project plans with measurable milestones to describe these 5-year projects. These 

project plans could be used as communication tools to inform management, the staff working on the 

project, users, emerging companies, and the public about the value of the project’s goals and when the 

results will be made available. 

The impact to, and the role of, the manufacturers’ supply chains also needs to be defined for the 

projects. The SMOPAC program can also engage additional manufacturers, including new entrants to the 

manufacturing sector, such as start-up manufacturing companies and suppliers to ensure alignment and 

identify early adopters of advanced technologies, programs, tools, and processes. They could also partner 

with Manufacturing USA to establish STEP-like computer-aided design (CAD), computer-aided 

manufacturing (CAM), and computer-integrated manufacturing (CIM) Product Information Exchange 

Standards. They could also benchmark other smart manufacturing programs, such as those in EU Horizon 

2020, to identify duplicative areas. 
29 

PORTFOLIO OF SCIENTIFIC EXPERTISE 

Accomplishments 

All of the SMOPAC projects had a good mix of technical experts with varying levels of experience. 

With the Cybersecurity for Smart Manufacturing Systems and Wireless Systems for Industrial 

Environments projects, technical experts from other NIST laboratories (the Information Technology 

Laboratory for Cybersecurity and Wireless and the Communication Technology Laboratory for Wireless) 

were part of the teams. The credentials that were found in all the biographies showed achievements and 

recognition in their areas of expertise, such as PDES, Inc., Technical Excellence and Technical 

Management Awards and the DOC Bronze Medal. The technologists were very knowledgeable about the 

projects they had managed. The SMOPAC personnel collaborate well with each other and others outside 

the EL and NIST. Additionally, they have actively disseminated the results of their work in 19 journal 

articles and 29 conference papers. 

Opportunities and Challenges 

As mentioned, SMOPAC technologists were very knowledgeable about the projects they had 

managed. They are well-published and their publications are cited; still, more publications need to be 

submitted in juried technical journals. 

Some of the technical personnel had industry experience that allows them to understand smart 

manufacturing and the challenges of an industrial environment. To improve this industry understanding, 

the SMOPAC program could solicit visiting scientists from manufacturing companies. 

ADEQUACY OF FACILITIES, EQUIPMENT, AND HUMAN RESOURCES 

Accomplishments 

Investments have been made in equipment that is required to gather data and verify the SMOPAC 

models in virtual environments. All of the testbeds that were toured during the review were well arranged 

to accomplish the work of the various SMOPAC projects; these testbeds included the Cybersecurity for 

Manufacturing Systems Testbed; Linear Axes Testbed; and the Industrial Wireless Systems Testbed. 

Opportunities and Challenges 

In some of the SMOPAC testbeds, arrangements are being made to integrate with other EL programs, 

such as robotics. While this is a good start to integrating the projects, they are very localized, and the level 

of integration is low. It would be better to have a larger space so that multiple projects and programs 

could be integrated into a larger, more complex testbed that more resembles some manufacturing 

environments. Validation of models under these conditions would provide meaningful data. 

In conclusion, the SMOPAC program possesses much of the expertise, facilities, and equipment to 

accomplish the goals for each of its projects. While an overall perspective may still be needed, the 

individual projects themselves are well managed and incorporate manufacturing needs with technical 

challenges. The program area needs to consider related areas outside the boundaries of the work that they 

have defined to ensure that the technologies being measured and standardized are comprehensive. 

Publishing in juried publications and making presentations at higher level conferences would help the 

scientists and technologists to develop their careers at a more international level. 
30 

5 

Smart Manufacturing Systems Design and Analysis 

INTRODUCTION 

The Smart Manufacturing Systems Design and Analysis (SMSDA) program has the objective of 

delivering measurement science, standards, and tools needed to design and analyze smart manufacturing 

systems. Accomplishment of this objective would enable manufacturers to improve productivity, agility, 

sustainability, and quality of their products. The SMSDA program consists of six major projects, staffed 

by 19 NIST personnel, 2 contractors, and 10 guest researchers. These projects are in the following areas: 

(1) Agri-Food Manufacturing System and Supply Chain Integration; (2) Data Analytics for Smart 

Manufacturing Systems; (3) New Technology Adoption and Industry Operations Analysis for Smart 

Manufacturing; (4) Modeling Methodology for Smart Manufacturing Systems; (5) Operations-Driven 

Performance Measurement for Smart Manufacturing Systems; and (6) Service-Oriented Architectures for 

Smart Manufacturing Systems. 

The SMSDA budget is approximately $7 million per year, and the SMSDA is envisioned to be a 5-

year program. With the exception of the Agri-Food Manufacturing System and Supply Chain Integration

project, which has been under way for approximately one year, all SMSDA projects have been ongoing 

for approximately three and one half years. During the assessment, four of the six SMSDA projects were 

reviewed. The two projects that were not reviewed in depth were the Agri-Food Manufacturing System 

and Supply Chain Integration project and the New Technology Adoption and Industry Operations 

Analysis for Smart Manufacturing project. 

SMSDA personnel stated that their work was in two different categories. The end objective of both of 

these categories was to produce standards; however, in the first category, best practices were known or 

could be determined. Therefore, the job was to encode the best practice. This definition of standards as 

encoding best practices was very useful during the review of the projects. The Service-Oriented 

Architectures for Smart Manufacturing Systems project is discussed below and is an example of this first 

category of work. The second category of work is one where the best practice is not known. In this case 

the job of the researchers is to produce the structure, databases, prototypes, or techniques that will form 

the basis of a best practice. Three of the projects reviewed are in this second category. The OperationsDriven Performance Measurement for Smart Manufacturing project will be utilized as an example of this 

type of work and is discussed below. 

ASSESSMENT OF TECHNICAL PROGRAMS 

Accomplishments 

During the review it became evident that all the activities of the reviewed projects where aligned 

with, and supportive of, the stated project objectives. To begin, the Service-Oriented Architectures for 

Smart Manufacturing project remediates the problem of moving syntax information (verbal or written 
31 

orders) across an organization with different enterprise platforms. For example, an engineering change 

from the parent company’s engineering organization needs to be passed to the scheduling organization 

(material organization), which in turn will send the change to manufacturing floor and vendors. Vendors 

may, in turn, send it to subtier vendors. Inside the parent company the engineering change is also sent to 

the customer support organization, which in turn needs to order spare parts and put warrantee reserves in 

place. The change is then sent to the finance department, where the engineering change is billed to 

customers. As engineering changes pass through each of the organizations, the attributes of the 

engineering change affecting that organization need to be entered into the organization’s computer system 

by hand. It is not unusual for engineering change attribute data to be input by hand into 20 to 30 different 

computer programs and several different computer languages. The cost of people entering the data is very 

large, the reentry of data by hand into different operating systems over and over slows the process, and 

increases the possibility of errors. The Service-Oriented Architectures for Smart Manufacturing Systems 

project remediates the inefficiency of this problem by developing a smart manufacturing solution. The 

solution consists of developing a large database of standard syntax terms (e.g., engineering change and 

purchase order) together with the attributes of each syntax term. The database allows individual users to 

add attributes if their application requires them. The database of integration message standards allows 

translation into the commonly used language for systems integration. Three translations are in 

development, and, as requirements for new languages emerge, new translators can be developed. The 

team has transferred the first version of syntax-independent database and computer language translators to 

the not-for-profit Open Applications Group Integration Specification (OAGIS) organization for 

maintenance and dissemination to the public. 

This project is a major success, and it is likely to have a profound impact on smart manufacturing. 

Providing improvements in company speed, cost, quality, and supply chain integration. This project is 

best in class and could be a benchmark for other NIST projects. Regardless of what a particular 

company’s goal is—speed, efficiency, cost, flexibility, competitiveness, and so on—the great 

simplification of standard syntax database and translation systems that the SMSDA program has 

developed and made available to the public domain will be of great value. This database will continue to 

grow, and will benefit companies who are deploying Enterprise Resource Planning (ERP) systems, 

integrating supply chains, and integrating new companies or products into their portfolio. Similarly, this 

standard is also beneficial to large factories producing and consuming large amounts of data and 

information, which have the requisite staff to help manage all of this data, as well as small and medium 

enterprises (SMEs), which usually do not have sufficiently talented staff to deal with the data explosion. 

The SMSDA program staff recognizes the impact that information technology has had. The staff has 

identified the Internet of Things, cloud services, service-based integration, and data analytics as 

technologies with specific potential for impact on smart manufacturing. It has as one of its primary goals 

the enablement of SMEs to participate in smart manufacturing, and it has defined agri-food 

manufacturing as a new section for its program, through its Agri-Food Manufacturing System and Supply 

Chain Integration project. 

The identified technologies are transformative for smart manufacturing, and agri-food manufacturing 

is an ideal domain for the SMSDA program to explore the need for standards development and 

measurement science, to aid in developing the required capability to enable SMEs. Agri-food 

manufacturing is a critical industry involving hundreds of thousands of SMEs in the United States alone. 

Many of the participating entities are independent and geographically distributed. There is a flood of new 

low-cost sensors appropriate for monitoring food products from farms to consumers. There is increasing 

capability to connect these sensors to the Internet, and there are evolving technologies and platforms to 

effectively use the resulting data captured to improve food manufacturing. However, to date, the 

standards and measurement capabilities have not nearly kept pace with the technology becoming 

available. 

The Agri-Food Manufacturing System and Supply Chain Integration project could provide a unifying 

domain for all of the SMSDA program’s current projects as well as a manufacturing domain that 

everyone can understand for presenting their research accomplishments. They already have two very large 
32 

agri-food manufacturing collaborators, Land O’Lakes and General Mills, and need to add multiple SMEs 

from each segment of the agri-food manufacturing supply chain (i.e., farmers, packers, transporters, 

wholesalers, distributors, and retailers). They also need to add an international component to their 

research effort because a large quantity of U.S. food is imported. This project represents an opportunity 

for the SMSDA program to make a profound impact on a large and very important industry. Therefore, it 

needs to be pursued vigorously. 

Opportunities and Challenges 

The SMSDA program, as well as other NIST smart manufacturing programs, seems to be somewhat 

ad hoc and disconnected—they are involved with a variety of domain experts and different entities that 

produce standards. This is not a criticism restricted to NIST—even the U.S. National Smart 

Manufacturing Initiative does not have a clear and comprehensive set of objectives, and more importantly 

a time-based roadmap. As the definition of smart manufacturing is elusive and ever-changing owing to 

the fact that new technology is constantly introduced into the workplace, the difficulty NIST faces is 

understandable. 

For clarity a best-in-class example is offered to demonstrate the value of a comprehensive roadmap. 

For over 50 years the semiconductor industry has been driven by what has become a self-fulfilling 

prophecy known as Moore’s law. The goal is to double the complexity of computer chips on every device 

generation (a generation is about 18 months). This simple law has been followed despite numerous 

enormous, cultural, economic, and technical obstacles. It became the driving force for the industry when 

it was codified as the Semiconductor Industry Association (SIA) roadmap, which for 20 years has helped 

define the specific programs within the industry that are needed to remain on the Moore’s law curve. 

Although the semiconductor industry is ferociously competitive, numerous manufacturers, equipment 

makers, material suppliers, research institutions, and other support facilities have cooperated to make the 

SIA roadmap a best-in-class example of how to cooperate on a global scale. This law, or vision, has 

remained unchanged despite 50 years of progress over many dimensions and domains. It drives 

technology (feature size, wafer size, speed, capacity, reliability, etc.), economics (cost productivity, 

wealth generation, etc.), and innovation (smart phones, IPads, household devices, etc.), and it has led to 

the U.S. dominance in new technology. 

Without a roadmap such as this, the quality and value of NIST's approach to managing the smart 

manufacturing program is difficult to assess, and therefore the value and quality of individual projects are 

difficult to measure. In particular, the SMSDA program has specific projects aimed at addressing specific 

topics. However, without a general roadmap with measurable milestones that define smart manufacturing, 

it is difficult for the program, and for NIST, to determine if it is working in the more critical or most 

impactful areas to improve smart manufacturing, or even in the most important areas at a given point in 

time. Regardless of these challenges, smart manufacturing has become a U.S. national priority, and so the 

question is: How could the efforts and resources focused on smart manufacturing be prioritized to best 

achieve the initiative’s inferred intent? 

A Smart Manufacturing Leadership Coalition (SMLC) was established some years ago for this 

purpose. More recently a nonprofit organization, the Clean Energy Smart Manufacturing Innovation 

Institute (CESMII), has been established to support the coalition. CESMII was established by Congress 

and is located in Los Angeles. NIST is involved with both of these organizations. The EL or NIST could, 

in cooperation with these organizations, develop an overarching roadmap with measurable milestones for 

smart manufacturing for the nation. 

Additionally, while all the SMSDA projects had been under way for approximately three and one half 

years, the project plans did not show intermediate project milestones, and so it was not possible to 

determine whether the project was making the expected amount of progress. Project planning for 

multiyear projects need to include intermediate milestones in order for progress and midcourse correction 
33 

to be easily understood and executed. The projects need to have a clear definition of deliverables and 

measurable milestones. Deliverables need to include demonstration and testing at SMEs. 

The objectives of each of the reviewed projects focused, in part, on making smart manufacturing 

easier and more available for SMEs. Most of the company partners shown in the project demonstrations 

were large to very large companies. It appears to be an implicit assumption that the technology and 

standards will trickle down from large to small manufacturers, or be mandated by them to their small to 

medium-size suppliers. A more direct approach involving direct contact with SMEs is desirable. 

It was also noted that the SMSDA program does not have a laboratory inside NIST to test the output 

of its projects. Testing the project output is particularly important where best practices do not exist and 

the new databases, techniques, and prototypes are being aimed to help SMEs. 

The stated objective of the Operations-Driven Performance Measurement for Smart Manufacturing 

Systems project is to “develop and deploy standards, guidelines and reference data for measuring the 

performance of manufacturing systems to inform the design and aid in analysis of factory 

improvements.”1

To this end the project team has chosen to focus, rightly, on four key performance objectives as 

identified by the SMLC: agility, productivity, sustainability, and quality. This project is in the second 

category of work undertaken by the SMSDA. That is, no clear best practice exists for this project. The 

main project effort has been to develop a unit manufacturing process (UMP) depository that will become 

a resource center to house the collective community knowledge, reference data sources, and performance 

baselines and trade-offs. This particular project is in the manufacturing engineering domain, and a domain 

expert needs to be an integral part of the project team. 

Once the database is populated, these UMP models could then be utilized for several functions. For 

example, a group of UMP models could be linked to model an entire production line for a given product. 

Optimization programs could be deployed to determine the best mix and location of machines. 

A demonstration of the utilization of a UMP modeled process was shown during the review. The 

demonstration consisted of the processing of a small metal part—a heat sink for an avionics board. The 

process is divided into 11 steps of various milling and drilling operations. Input to the model is productprocess info—design specifications, the type and nature of the materials, required electrical energy 

(which is both speed and process dependent). Along with resource data (operator, machine tool, fixture, 

software, etc.), the model goes through a set of equations that govern the physics and thermodynamics of 

the product and process and generates a set of outputs that include the total heat and waste generated. The 

model is a sophisticated tool—for each of the 11 processing stages, the control variables (e.g., machine 

speed) are optimized, under constraints (e.g., on total energy consumption). 

One stated use case for the UMP is to generate life cycle inventory data. For such a case it is 

questionable whether the model is developed at the right granularity and with the right proportion. 

Specifically, it is questionable whether the effort of detailed modeling and optimization of the processing 

of a tiny part of what could be a complex product (that may involve hundreds of similar parts) is 

commensurate with the life cycle analysis of the product. Other factors such as choice of raw material and 

end-of-life recycling provisions can easily wipe out any gains teased out of optimizing the machining 

speed and British thermal units (BTUs) saved, not to mention what can be achieved via a redesign and 

overhaul of the entire manufacturing process (as opposed to fine-tuning the tool speed), as embodied in 

the green manufacturing or remanufacturing movement in recent years. 

Overall, in a case like this, where there is a lack of existing standards or guidelines, the SMSDA 

program needs to broaden its scope to include identifying best practices in a select set of industry sectors. 

This broadening of scope appears both feasible and practical given that the team is currently collaborating 

 1

 Simon Frechette, Smart Manufacturing Systems Design and Analysis Program Manager, “Smart 

Manufacturing Systems Design and Analysis Program, Project Descriptions,” delivered to the panel on March 28, 

2017. 
34 

with the American Society for Testing and Materials (ASTM), an international standards organization, on 

this project. 

The Agri-Food Manufacturing System and Supply Chain Integration project intends to leverage other 

work being done to improve information flow among participants in the food chain, from growers to 

consumers. The U.S. agricultural industry is very large and diverse, and smart manufacturing technology 

can be and needs to be adapted to and deployed in agriculture applications. However, food and agriculture 

have major sectors with vastly different operations and needs. For example, harvesting operations for 

major crops are highly automated and dominated by very large corporations, while fresh foods, 

encompassing hundreds of products from vegetables to tree fruits and nuts to potatoes, are labor intensive. 

The latter is in dire need of automation, as labor shortages cause a significant portion of crops to rot on 

trees and in fields. Furthermore, fresh foods have short shelf lives and must be brought to the market 

quickly by 200,000 U.S. enterprises. The application of the outputs of the Service-Oriented Architectures 

for Smart Manufacturing Systems project to this project could have a profound effect on all aspects of the 

information processes food chain; however, they would have to meet the different needs of the different 

sectors, and need to engage different domain experts. One advantage the agricultural industry has is its 

strong network of commodity associations that can be used to provide both domain expertise and solution 

delivery channels. 

A big issue that was missing from discussions during the review was the value generated by these 

programs and projects; there was an underlying assumption that more technology is good. In particular, 

SMEs will require some proof that smart manufacturing will be able to help their bottom lines. While the 

SMSDA is the right place to develop tools and standards for gap analysis and valuation of new 

technologies, it is questionable whether the program will be able to execute the gap analysis and valuation 

of technologies and standards to obtain low-hanging fruit without a smart manufacturing roadmap with 

measurable milestones. 

PORTFOLIO OF SCIENTIFIC EXPERTISE 

Accomplishments 

The SMSDA staff are very involved in the standards and technical committees of the professional 

engineering societies, the academic community and partner companies. Over approximately the last three 

and one half years the SMSDA program has produced approximately 38 journal papers, created and 

organized approximately 30 workshops and conference sessions, and received honors, such as the ASTM 

International President’s Leadership Award and OAGi Outstanding Contributor Awards for its 

outstanding work. These personnel are also active in the correct standards committees of the U.S. 

professional societies. The staff were found to be, with a few exceptions that will be discussed below, 

very capable of executing its assigned tasks. 

Opportunities and Challenges 

The SMSDA personnel are very well trained and qualified to do the work necessary for the first 

category of work, which is to produce the basis and structure for a standard and to aid getting it into the 

public domain where it can be accessed by people and companies wishing to do smart manufacturing. 

Additionally, the personnel are capable of performing the tasks defined by the project objectives. They 

appear to work best and are most effective in problem areas where there exists an active standards 

community. In areas where there is a lack of existing standards or best practices, it is not clear that the 

SMSDA research is focused on the right problems or solutions to serve the SMEs. 

In projects where a best practice does not exist (that is, the second category of work), the SMSDA 

needs to include a domain expert on the team for areas where NIST personnel may not have technical 
35 

expertise. Having a domain expert as an integral part of this category of projects is a necessity. For 

example, if the project is to produce the structure, databases, prototypes, or techniques that will form the 

basis of best-practice production control, then a production control domain expert needs to be an integral 

part of the team working on this project. The SMSDA could acquire the domain expert either by hiring 

one, or by getting a partner company to provide one. Having a domain expert is required to ensure that 

developed structures, databases, prototypes, or techniques are truly needed, and will provide the best 

practice needed by the domain. 

The SMSDA also needs to utilize real SMEs as its laboratory to test its output. Making at least two 

SMEs an integral part of each project team would give the team the means to test its outputs in a real-life 

environment. 

The SMSDA program consists of a number of projects; however, it was unclear from the project 

presentations shown during the review how the different projects relate to the overarching objectives of 

the program. Further, it was unclear from the presentations what the main importance, goals, and progress 

to date are for each project, and how each project contributes to the overall goals of the program. While 

there are many enablers for smart manufacturing, it was not apparent how the project contributes to these 

enablers. 

ADEQUACY OF FACILITIES, EQUIPMENT, AND HUMAN RESOURCES 

Opportunities and Challenges 

The SMSDA program does not have a laboratory inside NIST to test the output of their projects. 

Testing the project output is particularly important where best practices do not exist and the new 

databases, techniques, and prototypes are being aimed at helping SMEs. 
36 

6 

Conclusions and Recommendations 

General Conclusions and Recommendations 

Smart manufacturing programs in the EL at NIST have significantly evolved in the last five years, 

matching increased national recognition of the importance of manufacturing in the economy, as well as 

international recognition of the importance of integration of digital computing and information 

technologies into manufacturing systems. For example, the additive manufacturing program area was 

established approximately three and one-half years ago and is now an impressive effort in an area of rapid 

technology growth, staffed by competent early-career researchers, and facilitated by a measurement 

testbed of their own design with unique and powerful capabilities. Standards are being given a high 

priority, and there is excellent collaboration with other organizations and leadership in these efforts. There 

is significant focus on small and medium enterprises (SMEs), which is a valuable objective that is 

difficult to achieve. There are good research collaborations with industry, but perhaps an overreliance on 

(hoped for) trickle down from work with larger industries; more extensive outreach to SMEs, especially 

young SMEs, would be beneficial. 

The EL’s smart manufacturing program appears to lack a roadmap and measurable milestones, as do 

the individual programs and projects. Without roadmaps and measurable milestones, the quality and value 

of NIST’s smart manufacturing programs and their management were more difficult to assess, and 

therefore the value, quality, and results of individual projects were more difficult to measure. Although 

some excellent research outputs, evinced by strong publication records in peer-reviewed journals, are 

being obtained and significant progress in standards is being made, many project plans did not show 

intermediate project milestones, and so it was difficult to determine in some cases whether projects were 

making the expected amount of progress and achieving the desired results. Roadmaps and measurable 

milestones would be invaluable tools for NIST management. 

There is also, in some cases, a lack of crosscutting activities that can take advantage of the synergistic 

objectives of individual projects. Examining the project’s plans, together to find opportunities for crossfertilization and integrating the testing and evaluation work across the overall smart manufacturing 

program, could provide more synergy across the projects, as well as an opportunity to achieve broader 

impact of the program work as a whole, rather than only at the project level. This is not helped by the 

current distance between laboratory spaces, and it would be better to have a larger and more contiguous 

space so that multiple projects and programs could be integrated into a larger, more complex testbed that 

more resembles some manufacturing environments. 

The new program areas are well covered by both junior and senior staff, who have significantly and 

successfully adapted their efforts to new technologies. Some areas appear to be short on permanent staff 

and expertise, but staffing appears to be generally sufficient to support the research that is under way. The 

journal publication record is excellent. There has been significant external recognition of the staff and its 

research, but additional efforts need to be made to promote and apply for external awards for the staff. 

This also would promote recognition of the smart manufacturing research programs at NIST. 
37 

Measurement Science for Additive Manufacturing 

The majority of national laboratories within the federal government currently pursue additive 

manufacturing (AM), and the significant progress that has been made within the Measurement Science for 

Additive Manufacturing (MSAM) program could benefit these other laboratories that, in turn, could likely 

help the MSAM program. Additionally, closer engagement and alignment with other U.S. government 

laboratories could ensure that the MSAM program is effectively leveraging the investments being made in 

this area. 

RECOMMENDATION 1: The Engineering Laboratory should consider closer engagements 

with other national laboratories’ additive manufacturing groups. 

A dialogue needs to be intensified with the Material Measurement Laboratory (MML) and its 

Polymers Processing Group in order to exploit common strengths and overlapping research interests. This 

dialogue could build on existing meetings with the MML and the Polymers Processing Group and evolve 

toward common research projects, such as feedstock characterization or the design of additive 

manufacturing machines. 

RECOMMENDATION 2: The Engineering Laboratory should consider beginning an 

intensified dialogue with the Material Measurement Laboratory and the Polymers Processing 

Group. 

There are several highly industry relevant topics within AM that are not currently pursued with 

sufficient emphasis by the MSAM program. These topics include support structures and their design, the 

effects of different build orientations, design limitations for AM, and the post-processing side of AM. 

Broadening the focus into post- or secondary processing will help researchers understand what limitations 

within the technologies cannot be overcome with post-processing methods. Expanding research veins into 

support structures and post-processing, and defining new procedures in this field, would allow an 

opportunity to develop new standards, which could help industry find common ground. 

RECOMMENDATION 3: The Engineering Laboratory should consider expanding its research 

into the measurements and standards needed to assist industry in the following areas: support 

structures and their design, the effects of different build orientations, design limitations for 

additive manufacturing, and the post-processing side of additive manufacturing. 

Robotic Systems for Smart Manufacturing 

Overall, the laboratory facilities, equipment, and human resources are top-notch and sufficient for 

impactful standards development activities in the Robotic Systems for Smart Manufacturing (RSSM) 

program. The program staff has done an excellent job of identifying the top brands of robotic and sensor 

equipment that will likely be used by SMEs and has acquired and activated the equipment for its research. 

The staff has been successful in operating the equipment to develop best practice methods for using the 

hardware and related software and to conduct a wide variety of experiments and demonstrations to 

evaluate potential metrics, test artifacts, and methods for multi-robot collaboration, agility, 

interoperability, and integration. 

The RSSM program is providing important contributions to the development of standards, metrics, 

and applicable technologies in the areas of performance assessment, collaborative robotics, agility, 

interoperability, and integration. The focus on the development of standards and performance metrics and 

tests in these areas aligns well with the mission of the EL. In particular, the program has made important 

advances in identifying weaknesses with existing metrics and standards in robotic mobility and agility, 
38 

anticipating unmet needs for future industry challenges, and developing new metrics and standards to 

address these issues. 

The RSSM program staff has an opportunity to lead in the development of a more systematic 

methodology for generating test and use cases. Such a methodology could include needs assessment, gap 

identification, research, metrics development, test methods and artifacts development, standards 

development, evaluation, and dissemination. 

RECOMMENDATION 4: The Engineering Laboratory should consider leading the 

development of methodological approaches for standards development in all component areas 

of its robotic systems for smart manufacturing research, including the areas of needs 

assessment, gap identification, research, metrics development, test methods and artifacts 

development, standards development, evaluation, and dissemination. 

The RSSM program could broaden its industrial impact through stronger interactions with the 

Manufacturing Extension Partnership (MEP) centers or other industrial consortia. Through these 

deepened interactions, the RSSM program could develop a set of use cases and obtain more in-depth 

feedback on standards development. 

RECOMMENDATION 5: The Engineering Laboratory should consider interacting on a 

regular basis (e.g., twice annually) with the Manufacturing Extension Partnership (MEP) 

centers or other industry consortia to source use cases and receive feedback on the effectiveness 

of their development of metrics and standards. 

Although vision sensors are used in several RSSM testbeds, the EL has an unexplored opportunity to 

investigate how machine vision can be used to enhance the flexibility and user-friendliness of robots for 

SMEs. The exploration of how to take advantage of machine vision to simplify robot programming, 

develop self-teaching techniques, ensure safe human-robot interaction, and adapt to the varied application 

environments at SME facilities is a good fit with the EL’s objectives for the RSSM program. 

RECOMMENDATION 6: The Engineering Laboratory should consider exploring the use of 

machine vision to enhance the value of robots for small and medium enterprises (SMEs) by 

reducing programming complexity for short-run manufacturing applications. 

There is an increasing utilization of cloud and edge resources for robotics and industrial automation. 

The Industry 4.0, Microsoft Azure, and Amazon Web Services (AWS) efforts are trying to integrate 

resources in a consistent and efficient framework, and the RSSM program has the opportunity to 

contribute to and lead this nascent field. 

RECOMMENDATION 7: The Engineering Laboratory should consider exploring 

methodological approaches and standards development related to cloud robotics by 

contributing to activities in this area (e.g., Industry 4.0), and taking a leadership role in this 

field. 

Given the level of expertise and contribution of the RSSM research team, additional peer recognition 

is possible in the form of merit-based robotics awards, elevation in professional societies, invitations to 

delivering opening and keynote presentations at conferences, and early-career investigator awards. This 

type of external recognition can serve not only to award deserving individuals but also to raise the 

visibility of EL in the field. 
39 

RECOMMENDATION 8: The Engineering Laboratory should be more proactive in seeking 

individual recognition for accomplished robotics staff personnel, rather than waiting to be 

discovered. 

Human-machine collaboration is an important challenge in robotics for manufacturing. The RSSM 

program appears to have an expertise gap in human robot interaction due to lack of permanent staff in this 

area. Active recruiting of such researchers into NIST permanent staff is important to establish this topic as 

an active research area. 

RECOMMENDATION 9: The Engineering Laboratory should establish permanent staff 

expertise in its human-robot interaction area to avoid a future expertise gap. 

The RSSM could more systematically leverage and seek out expertise from other organizations, 

including universities and industries, and through an extended use of guest researcher programs. A 

consortium of industrial companies, both large and small, as well as universities that meet with the RSSM 

on an annual or a semi-annual basis, could serve as additional sources of expertise for needs assessment, 

gap identification, and the testing and validation of standards. 

RECOMMENDATION 10: The Engineering Laboratory should consider systematically 

leveraging and seeking out expertise from other organizations, including university, industry, 

and Manufacturing USA Institutes who can serve as additional sources of expertise for needs 

assessment, gap identification, and the testing and validation of standards. 

The RSSM program has researched the capabilities of a wide variety of robotic grippers. They have 

developed many test methods, artifacts, and metrics to guide in the design and use of grippers for 

dexterous grasping. They have an opportunity to leverage this research to create community-wide test sets 

that can assist the research community in developing new approaches to dexterous grasping. For example, 

the Yale-CMU-Berkeley (YCB) Object and Model Set data set is a related contribution to the research 

community that helps advance robotic manipulation, and a similar contribution based on the RSSM 

research would be highly beneficial to the research community. 

RECOMMENDATION 11: The Engineering Laboratory should consider coordinating with 

other organizations to develop community-wide test sets for dexterous grasping. 

Although the research facilities are adequate, they are also distributed across the NIST campus and 

are physically disconnected across multiple buildings on the campus. Given that the RSSM research 

projects share many common technologies, science, and resource needs, and can benefit from knowledge 

exchanges, such separations between the project teams might contribute to less cohesiveness and overall 

impact than might be possible through co-located facilities. The lack of crosscutting activities across all 

the projects of the RSSM program is not helped by the distance between laboratory spaces. 

RECOMMENDATION 12: The Engineering Laboratory should consider co-locating test 

facilities to enhance integration of Robotic Systems for Smart Manufacturing (RSSM) program 

activities across the entire program. 

There is an opportunity to develop software infrastructure resources to facilitate the interoperability of 

testbeds developed in individual projects. Several of the testbeds developed for individual projects have 

overlapping capabilities but cannot currently be integrated because they do not have interoperable 

software. 
40 

RECOMMENDATION 13: The Engineering Laboratory should strive for interoperability of 

software infrastructure for their testbeds. 

The ability of RSSM staff to share videos, images, and other information through social media is 

hindered by the lengthy approval process at NIST and the lack of staff with expertise in social media 

outreach. As a result, the program has very little visibility on social media, which is a missed opportunity 

to improve the impact of the RSSM program’s activities. 

RECOMMENDATION 14: The Engineering Laboratory should consider adding staff who can 

improve dissemination of technical accomplishments through social media, trade publications, 

and/or popular press and should work with NIST management to identify ways to streamline 

approval processes for applying these methods. 

Smart Manufacturing Operations Planning and Control 

The Smart Manufacturing Operations Planning and Control (SMOPAC) program needs to develop an 

overarching, integrated roadmap with measurable milestones that shows the relationships of the various 

markets (automotive, aerospace, etc.) and requirements to the technologies, as well as the relationships 

between the technical areas themselves. The roadmap needs to be an integrated, high-level plan, not a set 

of semi-related programs. 

RECOMMENDATION 15: The Engineering Laboratory should consider developing an 

overarching roadmap with measurable milestones for the Smart Manufacturing Operations 

Planning and Control (SMOPAC) program that integrates market needs, product 

requirements, and technologies. 

The EL needs to examine its activities within the context of national and global activities at other 

organizations to identify competitive and duplicative areas. 

RECOMMENDATION 16: The Engineering Laboratory should consider benchmarking its 

work in smart manufacturing operations planning and control against other manufacturing 

programs, such as those in EU Horizon 2020, to identify competitive and duplicative areas. 

In order to ensure alignment with manufacturers’ needs, and to identify early adopters of advanced 

technologies, programs, tools, and processes, the SMOPAC program needs to engage additional 

manufacturers, including new entrants to the manufacturing sector, such as start-up manufacturing 

companies and suppliers. They could partner, for example, with Manufacturing USA to facilitate 

information exchange. 

RECOMMENDATION 17: The Engineering Laboratory should consider engaging new 

manufacturers and suppliers to ensure alignment with manufacturers’ needs and to identify 

early adopters of advanced technologies, programs, tools, and processes. 

RECOMMENDATION 18: The Engineering Laboratory should consider partnering with 

Manufacturing USA to establish Standard for the Exchange of Product Data (STEP)-like 

computer-aided design (CAD), computer-aided manufacturing (CAM), and computerintegrated manufacturing (CIM) product information exchange standards. 
41 

The SMOPAC program needs to develop and show project plans with measurable milestones to 

describe their 5-year projects. Such project plans can be used as communication tools to inform 

management, the staff working on the project, users, emerging companies, and the public about the value 

of the project’s goals and when the results will be made available. 

RECOMMENDATION 19: The Engineering Laboratory should develop project plans with 

measurable milestones that describe their 5-year projects and use these plans as communication 

tools. 

The definition of “life cycle” that the Digital Thread for Smart Manufacturing project is using shows 

the end of the cycle as being when the initial product is delivered to the customer. This is different from 

the Department of Defense (DoD) ManTech projects, which include the Army, Navy, and Air Force 

projects on digital thread and digital twin, where life cycle also includes the life of the aircraft. In 

aerospace and automotive industries (as well as with health care devices and others), the manufacturer 

needs to have the ability to track the vehicles and devices in the field for potential part defects as long as 

they are in service; and so life cycle is from cradle to grave. In all fields, products have increasingly 

tremendous sensory and computational capabilities, and in-the-field capabilities for diagnostics and 

prognostics need to be considered as part of the life cycle in the digital thread. 

RECOMMENDATION 20: The Engineering Laboratory should improve and clarify 

the definition of “life cycle” to be more comprehensive and include manufacturing activities 

after the product is delivered. 

The concept of the digital thread can be applied to multiple industries of various sizes. The Digital 

Thread for Smart Manufacturing project is targeting medium-size manufacturers. While smaller 

manufacturers may either already have or be able to adopt these tools and standards, the team needs to 

address how large-scale industries (e.g., aerospace, automotive) will adopt NIST’s unique tools. 

RECOMMENDATION 21: The Engineering Laboratory should put together a business case 

for small-, medium-, and large-scale industries to adopt their digital thread tools. 

The Prognostics, Health Management, and Control (PHMC) team has developed an efficient and 

effective linear axis error detection methodology based on data collected from an inertial measurement 

unit (IMU), which has been developed and verified on the linear axis testbed and machine tools within the 

PHMC project at NIST. Validation efforts are ongoing both internally within NIST and externally with 

several manufacturing collaborators. The developed IMU sensor is very practical to assess machine health 

degradation rapidly. The team has demonstrated platform-based technologies to allow diversified users to 

connect to the machine and share their data. The team is planning to promote its research to system levels; 

however, the research scope needs to be more clearly defined (e.g., type of applications, sensors, and 

analytics) before this happens. Collaboration with the Clean Energy Smart Manufacturing Innovation 

Institute (CESMII) will help to define, identify, and satisfy industry needs and requirements as well as 

provide efficiencies in areas of overlapping research. 

RECOMMENDATION 22: The Engineering Laboratory should consider collaborating with 

the Clean Energy Smart Manufacturing Innovation Institute to help to identify, prioritize, and 

satisfy industry needs and requirements and to provide efficiencies in areas of overlapping 

research. 

The Cybersecurity for Smart Manufacturing project has established a testbed to validate the 

cybersecurity framework manufacturing profile. Data obtained from this testbed will be used to develop 
42 

guidance for implementation of this framework. The testbed needs to include a simulation system that can 

generate virtual cyber threats to evaluate the resilience of the system. 

RECOMMENDATION 23: The Engineering Laboratory’s testbed for the Cybersecurity for 

Smart Manufacturing project should include a simulation system that can generate virtual 

cyber threats to evaluate the resilience of the system. 

The Wireless Systems for Industrial Environments project research is just beginning, and it needs to 

be considered more broadly. The team needs to develop testbeds that include both cybersecurity and 

wireless technology and consider integrating with a cloud-based or edge-based environment to support 

more tether-free monitoring system applications. 

RECOMMENDATION 24: The Engineering Laboratory should consider developing a testbed 

that includes both cybersecurity and wireless technology. 

RECOMMENDATION 25: The Engineering Laboratory should consider integrating with a 

cloud-based or edge-based environment to support more tether-free monitoring system 

applications. 

In some of the testbeds, there are arrangements being made to integrate with other programs, such as 

robotics. This is a good start; however, the integration is small. It would be better to have a larger space 

so that multiple projects and programs could be integrated into a bigger, more complex testbed that more 

resembles an actual manufacturing environment. Validation of models under these conditions would 

provide meaningful data. 

RECOMMENDATION 26: The Engineering Laboratory should consider dedicating an area or 

a facility for the integration of multiple projects and programs to simulate an industrial 

environment. 

Smart Manufacturing Systems Design and Analysis 

The NIST Smart Manufacturing Systems Design and Analysis (SMSDA) program would greatly 

benefit from and could contribute to an overarching roadmap of smart manufacturing for the nation. 

RECOMMENDATION 27: The Engineering Laboratory should consider collaborating with 

the Smart Manufacturing Leadership Coalition (SMLC) and the newly formed Clean Energy 

and Smart Manufacturing Innovation Institute (CESMII) to develop for the National Institute 

of Standards and Technology and the United States an overarching roadmap with measurable 

milestones for smart manufacturing. 

Where best practices for smart manufacturing capabilities do not exist, the task is to develop 

structures, databases, prototypes, and techniques to define best practices. During execution of tasks of this 

type, the inclusion of a domain expert on the team would greatly improve the output. 

RECOMMENDATION 28: Where a smart manufacturing best practice does not exist, the 

Engineering Laboratory should consider embedding domain experts on the project teams to 

ensure the quality of their output. 

 
43 

The SMSDA program needs to ensure that the output of projects where a smart manufacturing best 

practice does not exist is tested in at least two SMEs. These SMEs need to become an integral part of the 

project teams to ensure that the developed structures, databases, and techniques make it easier for SMEs 

to utilize smart manufacturing. 

RECOMMENDATION 29: The Engineering Laboratory should ensure that where a smart 

manufacturing best practice does not exist, the output of projects is tested in at least two small 

and medium enterprises (SMEs). 

Where there is a lack of existing standards or guidelines, the SMSDA program needs to broaden its 

scope to include identifying smart manufacturing best practices in a select set of industry sectors. Given 

that the program has been actively collaborating with various standards communities and has access to a 

wide range of industries, this broadening of scope appears both feasible and practical. 

RECOMMENDATION 30: Where there is a lack of existing standard or guidelines, the 

Engineering Laboratory should consider broadening its scope to include identifying smart 

manufacturing best practices in a select set of industry sectors. 

Project planning for multiyear projects needs to include measurable intermediate milestones so that 

progress and midcourse correction can easily be understood and executed. The projects need to have a 

clear definition of deliverables and measurable milestones. Additionally, deliverables need to include 

SME demonstrations and testing. 

RECOMMENDATION 31: The Engineering Laboratory should consider planning multiyear 

projects with measurable intermediate milestones so that progress and midcourse correction 

needs are readily visible. 

45 

Acronyms 

3D Three-dimensional 

AGVs Automatic Guided Vehicles 

AM additive manufacturing 

ARM Advanced Robotics for Manufacturing 

ASME American Society of Mechanical Engineers 

ASTM American Society for Testing and Materials 

AWS Amazon Web Services 

BTU British thermal unit 

CAD computer-aided design 

CAM computer-aided manufacturing 

CASE Conference on Automation Science and Engineering 

CESMII Clean Energy Smart Manufacturing Innovation Institute 

CIM computer-integrated manufacturing 

DoD Department of Defense 

EL Engineering Laboratory 

ERP Enterprise Resource Planning 

FTE full-time equivalent 

ICRA International Conference on Robotics and Automation 

IEEE Institute for Electrical and Electronics Engineering 

IMU inertial measurement unit 

IR infrared 

IROS International Robotics and Systems 

ISO International Organization for Standardization 

MEP Manufacturing Extension Partnership 

MML Material Measurement Laboratory 

MRO maintenance, repair, and overhaul 

MSAM Measurement Science for Additive Manufacturing 

NIST National Institute of Standards and Technology 
46 

OAGIS Open Applications Group Integration Specification 

PHMC Prognostics, Health Management, and Control 

RAS Robotics and Automation Society 

RF radio frequency 

RIA Robotics Industry Association 

ROS Robot Operating System 

RSSM Robotic Systems for Smart Manufacturing 

SIA Semiconductor Industry Association 

SLM selective laser melting 

SLS selective laser sintering 

SME small and medium enterprise 

SMLC Smart Manufacturing Leadership Coalition 

SMOPAC Smart Manufacturing Operations Planning and Control 

SMSDA Smart Manufacturing System Design and Analysis 

STEP Standard for the Exchange of Product Data 

TC technical committee 

UMP unit manufacturing process 

YCB Yale-CMU-Berkeley