An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

AN ASSESSMENT OF THE COMMUNICATIONS 

TECHNOLOGY LABORATORY AT THE NATIONAL 

INSTITUTE OF STANDARDS AND TECHNOLOGY 

FISCAL YEAR 2019

Panel on Review of the National Institute of Standards and Technology’s 

Communications Technology Laboratory 

Laboratory Assessments Board 

Division on Engineering and Physical Sciences 

A Consensus Study Report of 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001 

This activity was supported by Contract No. SB134117CQ0017 with the National Institute of Standards 

and Technology. Any opinions, findings, conclusions, or recommendations expressed in this publication 

do not necessarily reflect the views of any organization or agency that provided support for the project. 

International Standard Book Number-13: 978-0-309-49895-1 

International Standard Book Number-10: 0-309-49895-3 

Digital Object Identifier: https://doi.org/10.17226/25602 

Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, 

NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu. 

Copyright 2019 by the National Academy of Sciences. All rights reserved. 

Printed in the United States of America 

Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2019. An Assessment of 

the Communications Technology Laboratory at the National Institute of Standards and Technology: 

Fiscal Year 2019. Washington, DC: The National Academies Press. https://doi.org/10.17226/25602. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

The National Academy of Sciences was established in 1863 by an Act of Congress, signed by 

President Lincoln, as a private, nongovernmental institution to advise the nation on issues 

related to science and technology. Members are elected by their peers for outstanding 

contributions to research. Dr. Marcia McNutt is president. 

The National Academy of Engineering was established in 1964 under the charter of the 

National Academy of Sciences to bring the practices of engineering to advising the nation. 

Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. 

Anderson, is president. 

The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 

under the charter of the National Academy of Sciences to advise the nation on medical and 

health issues. Members are elected by their peers for distinguished contributions to medicine 

and health. Dr. Victor J. Dzau is president. 

The three Academies work together as the National Academies of Sciences, Engineering, and 

Medicine to provide independent, objective analysis and advice to the nation and conduct other 

activities to solve complex problems and inform public policy decisions. The National Academies 

also encourage education and research, recognize outstanding contributions to knowledge, and 

increase public understanding in matters of science, engineering, and medicine. 

Learn more about the National Academies of Sciences, Engineering, and Medicine at 

www.nationalacademies.org.
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

Consensus Study Reports published by the National Academies of Sciences, Engineering, and 

Medicine document the evidence-based consensus on the study’s statement of task by an 

authoring committee of experts. Reports typically include findings, conclusions, and 

recommendations based on information gathered by the committee and the committee’s 

deliberations. Each report has been subjected to a rigorous and independent peer-review 

process and it represents the position of the National Academies on the statement of task. 

Proceedings published by the National Academies of Sciences, Engineering, and Medicine 

chronicle the presentations and discussions at a workshop, symposium, or other event convened 

by the National Academies. The statements and opinions contained in proceedings are those of 

the participants and are not endorsed by other participants, the planning committee, or the 

National Academies. 

For information about other products and activities of the National Academies, please visit 

www.nationalacademies.org/about/whatwedo.
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

v 

PANEL ON REVIEW OF THE NATIONAL INSTITUTE OF STANDARDS AND 

TECHNOLOGY’S COMMUNICATIONS TECHNOLOGY LABORATORY 

DOUGLAS SICKER, Carnegie Mellon University, Chair

MILIND BUDDHIKOT, Nokia Bell Labs 

SAMIR DAS, State University of New York, Stony Brook 

MONISHA GHOSH, University of Chicago 

DAVID GOODMAN, NAE,1 Tandon School of Engineering, New York University 

ISMAIL GUVENC, North Carolina State University 

CYNTHIA HOOD, Illinois Institute of Technology 

JOSEP JORNET, Northeastern University 

PAUL KOLODZY, Kolodzy Consulting, LLC 

LAWRENCE LARSON, Brown University 

DAVID MATOLAK, University of South Carolina 

JEFFREY REED, Virginia Polytechnic Institute and State University 

AKBAR SAYEED, University of Wisconsin, Madison 

HAROLD ZHENG, Johns Hopkins University Applied Physics Laboratory 

Staff 

AZEB GETACHEW, Senior Program Assistant 

EVA LABRE, Administrative Coordinator 

JAMES P. MCGEE, Director 

MARTIN OFFUTT, Senior Program Officer 

 1

 Member, National Academy of Engineering. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

vi 

COMMITTEE ON NIST TECHNICAL PROGRAMS 

ELSA REICHMANIS, NAE,1

 Georgia Institute of Technology, Chair

MICHAEL I. BASKES, NAE, Mississippi State University 

LEWIS M. BRANSCOMB, NAS2

/NAE/NAM,3 University of California, San Diego 

MARTIN E. GLICKSMAN, NAE, Florida Institute of Technology 

JENNIE S. HWANG, NAE, H-Technologies Group, Inc. 

CHRISTOPHER W. MACOSKO, NAE, University of Minnesota, Minneapolis 

C. KUMAR N. PATEL, NAS/NAE, Pranalytica, Inc. 

BHAKTA B. RATH, NAE, Naval Research Laboratory 

ALICE WHITE, Boston University 

Staff 

AZEB GETACHEW, Senior Program Assistant 

EVA LABRE, Administrative Coordinator 

JAMES P. MCGEE, Director 

MARTIN OFFUTT, Senior Program Officer 

 1

 Member, National Academy of Engineering, 

2

 Member, National Academy of Sciences. 

3

 Member, National Academy of Medicine. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

vii 

LABORATORY ASSESSMENTS BOARD 

ROSS B. COROTIS, NAE,1

 University of Colorado, Boulder, Chair 

WESLEY L. HARRIS, NAE, Massachusetts Institute of Technology 

JENNIE S. HWANG, NAE, H-Technologies Group 

W. CARL LINEBERGER, NAS,2

 University of Colorado, Boulder 

C. KUMAR N. PATEL, NAS/NAE, Pranalytica, Inc. 

ELSA REICHMANIS, NAE, Georgia Institute of Technology 

LYLE H. SCHWARTZ, NAE, U.S. Air Force Office of Scientific Research (retired) 

Staff 

AZEB GETACHEW, Senior Program Assistant 

EVA LABRE, Administrative Coordinator 

JAMES P. MCGEE, Director 

ARUL MOZHI, Senior Program Officer 

MARTIN OFFUTT, Senior Program Officer 

 1

 Member, National Academy of Engineering. 

2

 Member, National Academy of Sciences. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

ix 

Preface 

A White House advisory council has noted: “If the Nation instead expands its options for 

managing federal spectrum, we can transform the availability of a precious national resource —

spectrum—from scarcity to abundance.”1

 The use of radio-frequency technology for communications, 

radio location, entertainment, and more continues to experience tremendous growth, and with this it plays 

an increasingly critical role in our daily lives. Given the challenge of accessing additional RF spectrum 

for these technologies, it is essential to effectively and efficiently use these spectrum resources. The 

Department of Commerce operates the Communications Technology Laboratory (CTL), which provides 

the nation with a laboratory dedicated to improving the use of radio frequency spectrum. CTL was 

established in 2014 by the National Institute of Standards and Technology (NIST) by merging several 

components of existing NIST laboratories into a single entity focused on promoting standards and 

metrology in the area of communications technologies. In the last 5 years, CTL has grown to establish 

significant efforts in the areas of measurement and standards relating to spectrum use and other wireless 

technologies. The lab also has a prominent role in public safety communications, where it serves as the 

future technology lead for the Department of Commerce’s First Responder Network Authority (FirstNet), 

the emerging public safety broadband network in the United States. CTL has also recently launched the 

National Advanced Spectrum and Communications Technology Network, designed to help create a 

trusted spectrum testing and measurement organization to aid in spectrum sharing efforts. 

In 2015, the National Academies of Sciences, Engineering, and Medicine report on the Boulder, 

Colorado, communications technology laboratory of the Department of Commerce2

 described many of 

the critical uses of radio communications and highlighted important research priorities for this laboratory. 

It also guided CTL to continue to refine and extend its efforts in wireless metrology, spectrum sharing, 

and public safety communications. In 2019, a second panel was engaged to assess efforts within CTL. 

The Director of NIST requested that the panel focus its assessment on the following factors: (1) the 

organization’s technical programs; (2) the portfolio of scientific expertise within the organization; (3) the 

adequacy of the organization’s facilities, equipment, and human resources; and (4) the effectiveness by 

which the organization disseminates its program outputs.3

 The panel was directed to focus on two main 

categories: public safety communications, and metrology for advanced communications. These include 

 1

 President’s Council of Advisors on Science and Technology, 2012, Realizing the Full Potential of 

Government-Held Spectrum to Spur Economic Growth, Executive Office of the President, Washington, D.C., July 

20. 2

 National Academies of Sciences, Engineering, and Medicine, 2015, Telecommunications Research and 

Engineering at the Communications Technology Laboratory of the Department of Commerce: Meeting the Nation’s 

Telecommunications Needs, The National Academies Press, Washington, D.C. 3

 W.G. Copan, Ph.D., Director, NIST and Undersecretary of Commerce for Standards and Technology, 

“Memorandum for Panel of the National Research Council Committee on National Institute of Standards and 

Technology Technical Programs,” March 14, 2019. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

x 

the four CTL priority areas: (1) public safety communications, (2) trusted spectrum testing, (3) 

fundamental metrology for communications, and (4) Next Generation Wireless (5G and Beyond). This 

report provides the National Academies’ assessment of CTL. The panel also assessed the extent to which 

CTL followed the recommendations made in the 2015 National Academies’ report.1

 The panel visited the 

Boulder telecommunications laboratories on June 25-27, 2019, meeting with staff from CTL to 

understand the current activities of the laboratory, its strengths and weaknesses as an organization, and its 

plans for the near future. The assessment included in this report stems from these visits and discussions, 

and the committee’s own expertise. 

 

Douglas Sicker, Chair

Panel on Review of the National Institute of Standards and 

Technology’s Communications Technology Laboratory 

 

 1

 National Academies of Sciences, Engineering, and Medicine, 2015, Telecommunications Research and 

Engineering at the Communications Technology Laboratory of the Department of Commerce: Meeting the Nation’s 

Telecommunications Needs, The National Academies Press, Washington, D.C. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

xi 

Acknowledgment of Reviewers 

This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse 

perspectives and technical expertise. The purpose of this independent review is to provide candid and 

critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in 

making each published report as sound as possible and to ensure that it meets the institutional standards 

for quality, objectivity, evidence, and responsiveness to the study charge. The review comments and draft 

manuscript remain confidential to protect the integrity of the deliberative process. 

We thank the following individuals for their review of this report: 

Ian Akyildiz, Georgia Institute of Technology, 

Danijela Cabric, University of California, Los Angeles, 

Edward Frank,1

 NAE, Cloud Parity, Inc., 

Alan Kaplan, Princeton University, 

Rob Leonard, Fire Department of the City of New York, 

Dennis Roberson, Illinois Institute of Technology, and 

Murat Yuksel, University of Central Florida. 

Although the reviewers listed above provided many constructive comments and suggestions, they 

were not asked to endorse the conclusions or recommendations of this report nor did they see the final 

draft before its release. The review of this report was overseen by Neil G. Siegel, NAE, University of 

Southern California. He was responsible for making certain that an independent examination of this report 

was carried out in accordance with the standards of the National Academies and that all review comments 

were carefully considered. Responsibility for the final content rests entirely with the authoring committee 

and the National Academies.

 1

 Member, National Academy of Engineering. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

xiii 

Contents 

SUMMARY 1 

1 INTRODUCTION 6

2 OVERVIEW OF CTL 8

3 PUBLIC SAFETY COMMUNICATIONS RESEARCH 11

4 METROLOGY FOR ADVANCED COMMUNICATIONS 24

5 LOOK-BACK AT PREVIOUS REVIEW 36

6 LIST OF KEY RECOMMENDATIONS 40 

ACRONYMS 43 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

1

Summary 

In 2019, at the request of the Director of the National Institute of Standards and Technology 

(NIST), the National Academies of Sciences, Engineering, and Medicine formed the Panel on Review of 

the Communications Technology Laboratory (CTL) of the National Institute of Standards and 

Technology (the “panel”), having earlier established the following statement of work: 

The NASEM shall form a panel of experts who shall perform an independent technical assessment 

of the quality of the National Institute of Standards and Technology’s (NIST’s) Communications 

Technology Laboratory (CTL). The panel shall prepare a final report that reflects the expert 

consensus opinion of the panelists based on materials provided by NIST before and during a site 

visit to NIST laboratories in Boulder, CO. During this site visit, the panel shall conduct an indepth technical assessment and draft a consensus report addressing the topics requested by the 

NIST Director in his or her charge to the panel. Subsequent to the site visit, the panel shall finalize 

the report draft and the NASEM shall conduct any necessary reviews to ensure that the final report 

is of high quality and is impartial and objective in its assessment of the NIST CTL. 

The assessment shall be responsive to the charge from the NIST Director. The following draft 

criteria for the assessment are proposed by the NIST sponsor and are expected to be formalized in 

the charge of the NIST Director that will be provided at contract award, expected November 1, 

2018. 

1. The technical merit of the current laboratory program relative to current state-of-the-art 

programs worldwide; 

2. The portfolio of scientific expertise as it supports the ability of the organization to 

achieve its stated objectives; 

3. The adequacy of the laboratory budget, facilities, equipment, and human resources, as 

they affect the quality of the laboratory’s technical programs; and 

4. The effectiveness by which the laboratory disseminates its program outputs. 

The Director of NIST requested that the panel focus its assessment on the following factors: (1) 

the organization’s technical programs; (2) the portfolio of scientific expertise within the organization; (3) 

the adequacy of the organization’s facilities, equipment, and human resources; and (4) the effectiveness 

by which the organization disseminates its program outputs.1

 

One of the six laboratories of NIST, the CTL was created in 2014 and “promotes the development 

and deployment of advanced communications technologies through the dissemination of high quality 

 1

 W.G. Copan, Ph.D., Director, NIST and Undersecretary of Commerce for Standards and Technology, 

“Memorandum for Panel of the National Research Council Committee on National Institute of Standards and 

Technology Technical Programs,” March 14, 2019. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

2

measurements, data, and research supporting U.S. innovation, industrial competitiveness, and public 

safety.”2

 Of CTL’s four organizational units, three are located at NIST’s Boulder, Colorado, campus and 

a fourth in Gaithersburg, Maryland. As of June 2019, there were slightly fewer than 200 personnel in the 

CTL. There are in addition 29 term appointments. 

The national importance of wireless communications provides an opportunity for CTL to play an 

important role in the economic vitality of the country. CTL can play an even greater role in the evolution 

of the connected world the public has come to expect through research advances. The anticipated 

deployment of a variety of new, connected “smart” devices will require greater access to spectrum and 

advanced networked communication technologies. Not only the devices, but new applications such as 

telehealth, machine-to-machine communications, and augmented reality will stimulate demand for 

wireless communications. 

CTL’s role in communications research and engineering includes key areas such as: spectrum 

measurement and propagation modeling; applied research on wireless network access technologies; and 

applied research, testing, and evaluation of newly developed communication technologies. CTL also 

provides technical support to other federal agencies and the private sector, principally for spectrum 

measurement and analysis of spectrum sharing and service coexistence. 

For this review, the CTL director asked that the panel focus on two main categories: public safety 

communications and metrology for advanced communications.3

 Public safety communications addresses 

the CTL priority area of the same name. Metrology for advanced communications addresses three of the 

four CTL priority areas: (1) trusted spectrum testing, (2) fundamental metrology for communications, and 

(3) Next Generation Wireless (5G and Beyond). The following paragraphs describe the accomplishments, 

opportunities, and challenges of the technical programs, as well as the expertise, adequacy of resources, 

and dissemination of outputs for work of the CTL divisions in support of these two main categories. 

Included as well are key recommendations the panel considered especially worthy of attention. 

The panel visited the Boulder CTL laboratories on June 25-27, 2019, to receive briefings from 

and hold discussions with CTL staff—including staff who traveled from Gaithersburg, Maryland—to 

learn about current activities in CTL’s laboratories, strengths and weaknesses, and plans for the near 

future. 

The first main category of CTL’s work addressed by the panel was Public Safety 

Communications. CTL’s Public Safety Communication Program is charged with important research tasks 

for enabling future broadband wireless capabilities for the public safety community. Through the MiddleClass Tax Relief and Job Creation Act of 2012, the Public Safety Communications Research (PSCR) 

Division was appropriated $300 million4

 to conduct research focused on improving future broadband 

public safety wireless communications. The PSCR Division further defined this research through a 

successful and broad stakeholder outreach, to include the following: mission-critical voice, location-based 

services, analytics, enhanced user interface/user experience (UI/UX), security, and resilient systems. 

While the PSCR has been very successful in outreach, initial research, and engagement, there remain 

challenges for the group, including budget and human resources and the need to articulate a long term 

research plan that incorporates research from other industries and other parts of the federal government. 

PSCR has made use of open innovation tools such as prize challenges.5

 Key recommendations include: 

 2

 National Institute of Standards and Technology (NIST), Undated, “This is CTL,” U.S. Department of 

Commerce, Boulder, Colo., p. 1. 3

 Ibid. 

4

 The first portion of the allocated funds ($100 million) became available to PSCR at the start of FY 2016, the 

second allocation ($186.4 million) became available July 2016, and the final allocation ($13.6 million) became 

available FY 2017. 5

 Prize challenges are financial award or incentive-based activities aimed at a diverse array of non-federal 

contributors working on discrete, well-defined challenges. These were authorized by the America COMPETES 

Reauthorization Act of 2010. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

3

Key Recommendation: The PSCR Division should develop an research and development 

roadmap for mission-critical voice, considering how the various activities it includes therein can 

be used as integral elements. CTL should consult on the roadmap’s development with other 

organizations, both government and commercial, to determine overlap of technology 

development. CTL should conduct its own critical technology assessment to inform its roadmap. 

(Chapter 3) 

Key Recommendation: CTL should update its roadmap for public safety analytics, taking into 

consideration projected future areas of interest. (Chapter 3) 

Key Recommendation: The PSCR Division should consider integration of the UI/UX (user 

interface/user experience) research and prize challenges with the other PSCR portfolios as 

appropriate provided efficiencies can be gained by such integration. The division should consider 

developing a methodology and process for studying UI/UX along with the study of new 

technologies. (Chapter 3) 

Although the Division’s reliance on term employees makes sense given the short timeframe of the 

funding and it appears to be sufficient to make progress in each of the research areas, the limited growth 

of PSCR internal expertise is a concern. PSCR is addressing key issues in public safety, and it is likely 

that the research mission will continue to be of importance beyond the 2022 end date of the appropriated 

funds from the spectrum auction. 

Key Recommendation: CTL should evaluate the possibility and pros and cons of a strategic 

expansion of PSCR’s internal research staff aimed at ensuring continuity of research in key 

priority areas in particular after fiscal year 2022 when the spectrum auction funds will have been 

spent or no be longer available. In addition, CTL should develop a plan for leveraging the 

expertise developed through the prize challenges. (Chapter 3) 

The activities that fall under the second main category under review, metrology for advanced 

communications, address fundamental and near-term application measurement problems for emerging 

communication standards. The CTL is poised to capitalize on a number of timely and impactful 

opportunities, including: quantum-enabled metrology with applications in quantum information science 

and engineering (QISE); a new framework for channel metrology and modeling and innovative hardware 

and platforms for next generation wireless networks; and machine learning, data science, and statistical 

techniques. This group appears to be well-resourced for the tasks they are conducting. The staff are 

consistently and diligently engaged in dissemination of their scientific and technical findings. A key overarching recommendation is: 

Key Recommendation: CTL should develop a 3-year or 5-year strategic plan for its activities in 

metrology for advanced communication, to include: identifying and evaluating new research 

directions and opportunities for growth; developing strategic partnerships with other NIST 

laboratories for pursuing new areas; identifying resource needs (equipment, facilities, staff) for 

pursuing strategic growth areas of research; identifying and pursuing internal and external sources 

of funding to support the plan; and developing measurable criteria and metrics for annually 

evaluating progress toward 3-year and 5-year goals. The strategic plan should explain how its 

execution will support the successful attainment of the CTL priorities. (Chapter 4) 

The physical infrastructure is mostly adequate for the current metrology programs. CTL’s 

measurement facilities—currently available on a supervised basis to outside users through mechanisms 

such as cooperative research and development agreements (CRADA)—can be monetized as a “paid 

service” to be provided to users in the research and business communities in the United States and 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

4

worldwide. For this to succeed, it is essential that the facilities be able to handle latent demand for such 

services. 

Key Recommendation: CTL should undertake planning and resource allocation for renewal and 

renovation of its measurement facilities with a degree of urgency to support their functioning as a 

“paid service” to the research and business communities. (Chapter 4) 

Three organizational units contribute to the area of metrology for advanced communications: the 

Radio Frequency Technology (RFT) Division, the Wireless Networks (WN) Division, and the National 

Advanced Spectrum and Communications Test Network (NASCTN) program. 

The RFT Division engages in fundamental radio frequency (RF) metrology research and 

standards to characterize both integrated circuits and systems, wired and wireless. The RFT team has 

delivered significant results, and, for example, the Rydberg atom electric field sensor appears to have 

breakthrough potential. The CTL has proactively identified optical communications and QISE as 

promising new areas for research. 

Key Recommendation: The Radio Frequency Technology Division should broaden its research 

portfolio into the areas of optical communications technology and quantum information science 

and engineering—both of which it has identified already— while leveraging strategic 

collaborative partnerships with other NIST laboratories, including the Physical Measurement 

Laboratory and the Information Technology Laboratory. (Chapter 4) 

The Wireless Networks Division has been heavily focused in 5G (fifth-generation wireless) 

technology, metrology, and applications and is the steward for the successful 5G mmWave Channel 

Model Alliance. There are a number of opportunities and challenges for this division as their work 

progresses toward the millimeter-wave efforts, as well as outdoor measurements in novel applications, 

such as communication channels for unmanned aerial vehicles (UAVs). 

Key Recommendation: To develop closer partnerships with industry, academia and 

governments and to explore new sources of revenue the Wireless Networks Division and Radio 

Frequency Technology Division should integrate and build on the recent accomplishments in 5G 

millimeter-wave channel modeling, millimeter-wave propagation channel sounding and 

measurements, on-wafer measurements, over-the-air measurements, and a new design framework 

for future vector network analyzers. (Chapter 4)

CTL jointly established the NASCTN program in 2015 with the Department of Defense (DoD) 

and the National Telecommunications and Information Administration’s (NTIA’s) Institute for 

Telecommunication Sciences (ITS) to organize a network of test facilities to support spectrum-related 

testing, modeling, and analysis.6

 The work done by the NASCTN program in CTL, covering the broad 

topic of shared use of federal spectrum by commercial systems, has contributed to several high-value, 

high-impact spectrum reallocation efforts in the past few years and is positioned to have a continuing role 

in helping coordinate this very important national resource. Such efforts address coexistence metrics for 

shared-spectrum environments, test methods for spectrum management methods, and waveform 

metrology and calibration. The work has been addressing, in a necessarily reactive manner, problems in 

spectrum as they are identified. A more strategic and proactive approach toward spectrum sensing and 

sharing would enhance the impact of the team in regulatory actions and industry standardization 

activities. 

 6

 An additional three federal agencies have since joined: the National Oceanographic and Atmospheric 

Administration, the National Science Foundation, and the National Aeronautics and Space Administration. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

5

The current operating model in NASCTN is to respond to requests for spectrum sensing and 

characterization from industry or other federal agencies. Given the level of current staffing, this is 

probably appropriate. In the future, the program could have a much broader impact if it would 

independently select “hot-button” spectrum issues on which to work and increase the number of staff 

appropriately. 

Key Recommendation: Consistent with its future staffing levels, NASCTN should take a more 

proactive role in advising on future spectrum allocation decisions. NASCTN should engage 

impartially with all sides of the debates on emerging and urgent issues. (Chapter 4) 

Key Recommendation: CTL should continue vigorous support for the spectrum sensing and 

sharing activity, which has delivered impactful results. (Chapter 4) 

Since its inception in 2015, CTL has established significant efforts in measurement and standards 

relating to spectrum use and other wireless technologies. The four organizational units can all claim 

impressive achievements. The CTL has attained a prominent role in public safety communications, 

including FirstNet—the emerging public safety broadband network in the United States. CTL has also 

recently launched the multi-agency NASCTN, designed to help create a trusted spectrum testing and 

measurement organization to aid in spectrum sharing efforts. RTF and WN divisions have made novel 

findings for measurement of SI-derived quantities and for elaboration and testing of next-generation 

wireless concepts. There are opportunities and challenges to these portfolios created by the changing 

application space and stakeholder needs. The anticipated end in 2022 of the auction proceeds is a further 

challenge to mission and staffing. The recent upgrades to its facilities present opportunities to articulate 

with non-federal groups wishing to access these test beds, which, furthermore, could be monetized.
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

6 

1 

Introduction 

The National Academies of Sciences, Engineering, and Medicine has, starting in 1959, annually 

assembled panels of experts—from academia, industry, medicine, and other scientific and engineering 

communities of practice—to assess the quality and effectiveness of the NIST measurements and standards 

laboratories, of which there are now six,1

 as well as the adequacy of the laboratories’ resources. These 

reviews are conducted under contract at the request of the NIST. 

In 2019, at the request of the Director of NIST, the National Academies formed the Panel on 

Review of the Communications Technology Laboratory of the National Institute of Standards and 

Technology (the “panel”), having earlier established the following statement of work:

The NASEM shall form a panel of experts who shall perform an independent technical assessment 

of the quality of the National Institute of Standards and Technology’s (NIST’s) Communications 

Technology Laboratory (CTL). The panel shall prepare a final report that reflects the expert 

consensus opinion of the panelists based on materials provided by NIST before and during a site 

visit to NIST laboratories in Boulder, CO. During this site visit, the panel shall conduct an indepth technical assessment and draft a consensus report addressing the topics requested by the 

NIST Director in his or her charge to the panel. Subsequent to the site visit, the panel shall finalize 

the report draft and the NASEM shall conduct any necessary reviews to ensure that the final report 

is of high quality and is impartial and objective in its assessment of the NIST CTL. 

The assessment shall be responsive to the charge from the NIST Director. The following draft 

criteria for the assessment are proposed by the NIST sponsor and are expected to be formalized in 

the charge of the NIST Director that will be provided at contract award, expected November 1, 

2018. 

1. The technical merit of the current laboratory program relative to current state-of-the-art 

programs worldwide; 

2. The portfolio of scientific expertise as it supports the ability of the organization to 

achieve its stated objectives; 

3. The adequacy of the laboratory budget, facilities, equipment, and human resources, as 

they affect the quality of the laboratory’s technical programs; and 

4. The effectiveness by which the laboratory disseminates its program outputs. 

For this review, the CTL director asked that the panel focus on two main categories: public safety 

communications and metrology for advanced communications.2

 Public safety communications addresses 

the CTL priority area of the same name. Metrology for advanced communications addresses three of the 

 1

 The six National Institute of Standards and Technology (NIST) laboratories are the Communications 

Technology Laboratory, the Engineering Laboratory, the Information Technology Laboratory, the Material 

Measurement Laboratory, the Physical Measurement Laboratory, and the Center for Neutron Research. 2

 NIST, Undated, “This is CTL,” U.S. Department of Commerce, Boulder, Colo., p. 1. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

7 

four CTL priority areas: (1) trusted spectrum testing, (2) fundamental metrology for communications, and 

(3) Next Generation Wireless (5G and Beyond).3

The four organizational units of CTL are located in Boulder, Colorado, and in Gaithersburg, 

Maryland. The Boulder facilities were visited by the panel on June 25-27, 2019. 

The panel’s approach to the assessment relied on the experience, technical knowledge, and 

expertise of its members. Time constraints did not allow the panel to explore all aspects of CTL. Rather, 

the panel focused on the research that the leadership of the CTL chose to present to it and on a number of 

issues related to laboratory development that the panel identified as requiring particular attention. The 

panel’s report includes recommendations that specify “who should do what” to address any 

determinations as to what might warrant action. The panel underpins the Recommendations with salient 

examples of programs and projects that are intended collectively to portray an overall impression of the 

laboratory, while preserving useful suggestions specific to projects and programs. Key recommendations 

are those that the panel considers especially worthy of attention. 

To accomplish its mission, the panel reviewed the material provided by the CTL prior to and 

during the review meeting. The choice of projects to be reviewed was made by the CTL. The panel 

applied a largely qualitative approach to the assessment. Given the nonexhaustive nature of the review, 

the omission in this report of any particular CTL project should not be interpreted as a negative reflection 

on the omitted project.

 3

 Marla Dowell, Ph.D., Director CTL, “Communications Technology Laboratory Overview,” presentation to the 

panel on June 25, 2019. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

8 

2 

Overview of CTL 

Created in 2014, the CTL of NIST “promotes the development and deployment of advanced 

communications technologies through the dissemination of high quality measurements, data, and research 

supporting U.S. innovation, industrial competitiveness, and public safety.”1

 There are four organizational 

units: (1) Public Safety Communications Research Division; (2) Radio Frequency Technology Division; 

(3) National Advanced Spectrum and Communications Test Network; and (4) Wireless Networks 

Division. With the exception of the Wireless Network Division, situated in Gaithersburg, Maryland, all 

the above are located at the Boulder, Colorado, campus of the Department of Commerce (DOC) located at 

325 Broadway. 

The budget for CTL was $74.4 million in fiscal year (FY) 2019, the majority of which was 

funded through the Public Safety Trust Fund.2

 The next-largest portion is from annual congressional 

appropriations, followed by other agency funds and fees for calibrations—with these last two being 

variable year-to-year. The progression with time is shown in Figure 2.1. The percentage allocated to each 

of the four priority areas is shown in Figure 2.2. 

The activities of the CTL consist of research and development, testing, fundamental 

measurements, calibration services, and convening and outreach activities. As of June 2019, there are 

slightly fewer than 200 personnel in the CTL. Their allocation to the four organizational units and to 

headquarters is shown in Figure 2.3. There are in addition 29 term appointments. Four current and one 

retired CTL staff members are fellows of the Institute of Electrical and Electronics Engineers. 

Each of the four CTL priority areas is addressed by the work of one or more of the four 

organizational units. These relationships are outlined in Table 2.1. 

CTL conducts an annual research planning process focused on its three program areas of public 

safety, trusted spectrum testing, and metrology for advanced communications. Group plans filter up to 

Divisions for coordination and approval, while the Division and CTL leadership set strategic direction 

and long-term planning based on NIST priority areas. The present report identifies opportunities in 

specific program areas where performing strategic planning exercises and road mapping would improve 

stakeholder engagement. These would have the further benefit of ensuring that CTL programs continue to 

address stakeholder needs. The annual reviews noted above will allow NIST to monitor progress. 

 1

 National Institute of Standards and Technology (NIST), Undated, “This is CTL,” U.S. Department of 

Commerce, Boulder, Colo., p. 1. 2

 The Public Safety Trust Fund resulted from spectrum auction. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

9 

FIGURE 2.1 Communications Technology Laboratory budget and source of funds over the past 4 fiscal 

years. SOURCE: NIST, Undated, “This is CTL,” U.S. Department of Commerce, Boulder, Colo. 

FIGURE 2.2 Allocation of Communications Technology Laboratory funds across its four priorities. 

SOURCE: NIST, Undated, “This is CTL,” U.S. Department of Commerce, Boulder, Colo. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

10 

FIGURE 2.3 Communications Technology Laboratory staffing levels as of June 2019. SOURCE: NIST, 

Undated, “This is CTL,” U.S. Department of Commerce, Boulder, Colo. 

TABLE 2.1 Communications Technology Laboratory (CTL) Priority Areas Addressed by Each 

Organizational Unit 

 CTL Priority Areas 

Public Safety 

Communications 

Trusted Spectrum 

Testing 

Next Generation 

Wireless 

Fundamental Metrology 

for Communications 

PSCR X 

WN X X X 

RFT X X X 

NASCTN X 

NOTE: PSCR = Public Safety Communication Research Division; WN = Wireless Networks Division; RFT = Radio 

Frequency Technology Division; NASCTN = National Advanced Spectrum and Communications Test Network. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

11 

3 

Public Safety Communications Research 

INTRODUCTION AND OVERALL COMMENTS 

The first main category of CTL’s work addressed by the panel, Public Safety Communications, is 

discussed in this chapter. The PSCR Division of CTL evolved from a program that was established in 

2002 to provide research and development for public safety innovations and technologies.1

 Since its 

inception, PSCR has played a unique role supporting research with a focus on the distinctive requirements 

of future public safety communications “so that the public safety community can more effectively carry 

out their mission to protect lives and property during day-to-day operations, large-scale events, and 

emergencies.”2

 PSCR undertakes quarterly portfolio reviews to “monitor program effectiveness, mitigate 

risk [and] improve outcomes generated by PSCR.”3

The WN Division of CTL also works on the Public Safety Communications program. Funded in 

this effort by PSCR, WN focuses on “development and performance aspects of Mission-Critical 

Communications.”4

 This includes in particular the Mission-Critical Voice (MCV) portfolio, discussed 

below. (Separately, WN also works on the Metrology of Advanced Communications as discussed in 

Chapter 4.) 

In 2012, PSCR was called to play a significant part in the future of broadband public safety 

communication; the Middle Class Tax Relief and Job Creation Act of 20125

 (the “Act”) appropriated 

$300 million from spectrum auction proceeds to NIST to be used as specified in Section 6303 of the Act 

as shown in Box 3.1. 

Although the funds were appropriated in 2012, NIST was not granted borrowing authority and 

thus had to wait for the auction proceeds to become available before it could carry out the activities 

specified in the Act. The first portion of the allocated funds ($100 million) became available to PSCR at 

the start of FY 2016, the second allocation ($186.4 million) became available July 2016, and the final 

allocation ($13.6 million) became available FY 2017. The Act specifies that at the end of FY 2022, all 

appropriated funds that are unused will be returned to the Treasury. 

 1

 The PSCR effort was originally a joint effort between the National Institute of Standards and Technology 

(NIST) Office of Law Enforcement Standards (now the NIST PSCR Division within CTL) and NTIA’s Institute for 

Telecommunication Sciences. 2

 NIST, Undated, “This is CTL,” U.S. Department of Commerce, Boulder, Colo., p. 10. 

3

 Dereck Orr, NIST, “Pulling the Future Forward: Accelerating Public Safety Communications Research,” 

presentation to the panel on June 25, 2019. 4

 NIST, Undated, “This is CTL,” U.S. Department of Commerce, Boulder, Colo., p. 16. 

5

 P.L. 112-96, signed into law on February 22, 2012. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

12 

In 2013, PSCR began a process of engaging stakeholders to identify the needs and technical 

requirements for future broadband public safety wireless communications. Stakeholders including public 

safety practitioners (police, fire, emergency medical service [EMS]), industry, academia and local, state 

and federal agencies provided input to identify and prioritize research needs that were specific to public 

safety and were not being met by efforts in industry or academia. This process resulted in the 

development of research portfolios in the following areas: 

 MCV 

 Location-Based Services (LBS) 

 Public Safety Analytics 

 Enhanced UI/UX 

Along with two crosscutting efforts: 

 Security 

 Resilient Systems 

BOX 3.1 Statutory Authority for Use of Auction Proceeds 

47 USC 1443 SEC. 6303—PUBLIC SAFETY WIRELESS COMMUNICATIONS RESEARCH AND 

DEVELOPMENT. 

(a) NIST Directed Research and Development Program. 

From amounts made available from the Public Safety Trust Fund, the Director of NIST, in 

consultation with the Commission, the Secretary of Homeland Security, and the National Institute of 

Justice of the Department of Justice, as appropriate, shall conduct research and assist with the 

development of standards, technologies, and applications to advance wireless public safety 

communications. 

(b) Required Activities. In carrying out the requirement under subsection (a), the Director of 

NIST, in consultation with the First Responder Network Authority and the public safety advisory 

committee established under section 6205(a), shall- 

(1) document public safety wireless communications technical requirements 

(2) accelerate the development of the capability for communications between currently 

deployed public safety narrowband systems and the nationwide public safety broadband network; 

(3) establish a research plan, and direct research, that addresses the wireless communications 

needs of public safety entities beyond what can be provided by the current generation of 

broadband technology; 

(4) accelerate the development of mission critical voice, including device-to-device 

“talkaround” capability over broadband networks, public safety prioritization, authentication 

capabilities, and standard application programming interfaces for the nationwide public safety 

broadband network, if necessary and practical; 

(5) accelerate the development of communications technology and equipment that can 

facilitate the eventual migration of public safety narrowband communications to the nationwide 

public safety broadband network; and 

(6) convene working groups of relevant government and commercial parties to achieve the 

requirements in paragraphs (1) through (5). 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

13 

The crosscutting efforts are focused on use cases that were identified as being important to PSCR 

stakeholders. 

PSCR is using a variety of different mechanisms to achieve the requirements set forth by the Act. 

These instruments include internal research, research collaborations with government partners, external 

research grants and cooperative agreements, and prize challenges. Each research portfolio listed above 

includes a mix of these different instruments. Starting in FY 2017, PSCR began awarding grants for 

research and development projects focused on priority areas within the research portfolios that were not 

being addressed through internal research. To date, PSCR has funded 46 grants and cooperative 

agreements. 

Prize challenges were devised as a mechanism to cost-effectively foster innovation in areas where 

the other research instruments were unlikely to achieve the desired outcome. They seek to raise awareness 

of public safety-specific technical issues, engage innovators around the world,6

 and facilitate 

multidisciplinary collaboration between participants, public safety, industry and government. PSCR prize 

challenges began in 2018. An example is the 2018 Unmanned Aerial Systems Flight and Payload 

Challenge, which addresses the trade-off of flight time versus payload that proves critical in the public 

safety domain. The 10 teams that participated developed drones at low cost that optimized energy 

efficiency and flexibility. Incentive prizes such as this are awarded throughout different phases of the 

challenges to encourage participants with the best ideas. Prize winners are also incentivized to make their 

software open source to increase the potential impact on the public safety ecosystem.7

As of June 2019, PSCR has 30 technical staff, 5 administrative staff and 2 support staff for a total 

of 37 employees. The workforce includes 16 permanent employees and 21 term employees. The 

permanent staff all have bachelor’s degrees and 7 have master’s degrees. There are no Ph.D.-level staff 

working in PSCR although some of the research portfolios include Ph.D.-level staff from other 

organizations in NIST. Unique among the four CTL divisions, PSCR has a greater number of term 

appointees than permanent ones, owing to the anticipated sunsetting of the funding from the spectrum 

auction in FY 2022, at which point the term appointees would no longer be working in CTL under current 

assumptions. 

Overall Public Safety Communication Discussion 

The PSCR has adequate and stable near-term funding to carry out its mission. The technical 

quality of the team is excellent and the communication among team members is highly supportive of a 

collaborative, highly productive and interdisciplinary culture. The PSCR is a critical part of the public 

safety community and provides the leadership on communications research as well as on requirement 

gathering for the community. The importance of PSCR’s mission will likely extend past the anticipated 

end of funding after 2022. 

CTL staff has identified an appropriate set of projects to support future public safety 

communications needs. This work is centered on requirements gathering, research across six key 

technology areas, multi-stakeholder events, and standards work. 

The process for the future strategic selection of new projects—to continue PSCR impact—was 

less clear from the standpoint of this review. 

The PSCR staff is consistently and diligently engaged in the important task of disseminating its 

findings to the broader community of public safety professionals and stakeholders. 

The prize challenges in general have the beneficial short-term impact of making the community 

aware of what’s possible and to highlight those groups that are doing great work in relevant areas. 

 6

 Only teams with at least one U.S. citizen are eligible for the cash prizes. 

7

 In some cases, it is too soon to measure the impact of the prize winners—for example, the 2018 Differential 

Privacy Synthetic Data Challenge was completed in May 2019. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

14 

The PSCR is a unique national resource, and its scientific output and research need to be 

continued and made broadly available in order to ensure the highest impact of the work for the first 

responder community. 

The CTL physical infrastructure is adequate for the current programs in public safety 

communications. 

There is substantial evidence that PSCR has accomplished a significant amount in multiple areas 

of interest to its stakeholders in a short period of time. 

ASSESSMENT OF TECHNICAL PROGRAMS 

Across the six portfolios identified in the above section “Introduction,” PSCR has established a 

strong set of research components critical to future public safety communications. While significant work 

remains to be done, to date the PSCR has provided the community a number of accomplishments, 

outlined below. These contributions would not have been possible without the significant funding 

provided through the Act. 

Mission Critical Voice 

Several goals related to MCV were explicitly stated in the Act. These goals include the following: 

(1) accelerate technical and systems research to develop of mission critical voice communications, (2) 

enable public safety prioritization, (3) enable authentication capabilities, and (4) develop standard 

application programming interfaces.8

 Two divisions of CTL, PSCR and WN, contribute to this portfolio. 

The current activities in PSCR for MCV include: active participation (including contributions) to the 

3GPP (3rd Generation Partnership Project) standards body;9

 maintenance of the list10 of certified devices 

that meet appropriate protocols and standards for access to, use of, or compatibility with the nationwide 

public safety broadband network that FirstNet and their network partner, AT&T, build and maintain; 

research in critical open source tools that lower the barrier to entry for researchers and companies to 

perform research in the area of MCV; and development and improvement of tools, datasets, and models 

for public safety, commercial broadband operators, and outside researchers to simulate networks from the 

radio layer down through payload. (The authentication activities are part of the Security portfolio 

discussed below.) WN conducts work such as Device-To-Device (D2D) Communication, Mission Critical 

Push to Talk (MCPTT), and other areas.11

Future activities include: producing an inexpensive, trusted, and scientific measurement system 

that can be used by public safety practitioners and industry as a fair baseline and comparison of MCV 

capabilities; developing generic call models for public safety MCV that describe public safety’s existing 

Land Mobile Radio (LMR) use; creating a prototype system and release open-source software and 

capabilities that can be used by industry to create inexpensive solutions for public safety that bridge 

existing, non-Inter-RF-Subsystem Interface (non-ISSI) capable LMR systems with 3GPP standard 

interfaces (interworking function/ application server) on broadband networks; and implementing direct 

comparisons of LMR and broadband capabilities for MCV. 

 8

 From The Middle Class Tax Relief and Job Creation Act of 2012 (P.L. 112-96). 

9

 Specifically in the 3GPP TS 36.579 family of specifications and the 3GPP Core Network and Terminals 

Working Group 1 for Mission Critical Services. 10 The Middle Class Tax Relief and Job Creation Act of 2012 (P.L. 112-96) directed NIST to ensure the 

development of a list of certified devices. PSCR authored the process document for this requirement and maintains 

the list (J.D. Kahn, “Process Document for the NIST List of Certified Devices,” October 24, 2017, 

https://www.nist.gov/communications-technology-laboratory/pscr/process-document-nist-list-certified-devices). 11 See NIST, Undated, “This is CTL,” U.S. Department of Commerce, Boulder, Colo., p. 21. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

15 

Accomplishments 

Through FY 2019, PSCR has expended approximately $41 million for MCV activities. The three 

primary accomplishments presented to the review team included the following: 

 PSCR maintains the list of certified devices that meet appropriate protocols, and that list has 

been updated on a monthly basis with the most recent update being of August 6, 2019. 

 PSCR has undertaken the role of representative for U.S. public safety at key standards 

organizations. Albeit not a research and development role, the NIST role at 3GPP (who 

formulates all of the worldwide cellular standards) has provided key value for FirstNet 

implementations and future cellular implementations. For example, 

— Recently, PSCR informed a delegate from FirstNet to the 3GPP Core Network and 

Terminals Working Group 1 (CT1) that a configuration table in 3GPP TS 24.483 

needed many parameter changes. 

— PSCR interacts with delegates from various companies and organizations while 

informing 3GPP CT1 regarding required fixes in the reference specifications for 

Mission Critical Services. 

 The internal research within NIST has focused on testing methodology for determining voice 

intelligibility. It is expected that this work will continue and provide additional results and 

impact. Primary research accomplishments to date include the following: 

— Studies of quality of experience (QoE) and subsequent metrics aimed at providing a 

high QoE including: end-to-end access time, speech intelligibility, voice quality and 

intelligibility, and probability of access and retention. 

— Initial M2E (mouth-to-ear) latency measurements comparison for Project 25 (P25) 

trunk and direct radio systems. This work can be used as a comparison when using 

3GPP defined PTT user access time. 

— Voice quality and intelligibility analysis using Modified Rhyme Test (MRT) 

developed by the NTIA’s ITS. 

Challenges and Opportunities 

The review team focused on two of the three focus areas as discussed above: (1) involvement in 

the standards bodies and (2) MCV internal research. 

Although not a research and development (R&D) function, the NIST role with the standards 

bodies and maintaining the certified device list has been valuable both from near-term as well as longterm perspectives. It is critical that the public safety community be represented by technically informed 

personnel in order to address highly technical issues within the standards bodies. 

Recommendation: CTL’s participation in standards bodies concerning Mission Critical Voice 

should be driven toward specific goals that should be explicitly agreed upon by the user as well as 

the research and development communities. Those goals should be internally and externally 

vetted. 

The MCV Internal Research, representing a substantial amount of the PSCR expenditures, 

appears to be in its initial stages. This research is building on a great deal of work by NTIA, and it is 

interesting to note that there are not many cooperative interactions between the organizations. The 

specific role of this work versus the work done by other organizations (i.e., ITS [part of NTIA]; the 

Department of Homeland Security’s Advanced Research Projects Agency; and DoD) was not made 

available to the panel but would be a useful input for planning research and standards. Although the radio 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

16 

systems are unique, the problems that are to be addressed are very similar to those being addressed by 

these other organizations. Such interactions with other researchers could quickly allow bigger strides in 

resolving public safety MCV challenges. 

There appears to be no roadmap of technology development for MCV and thus no knowledge of 

how the projected future development of technologies might be used to focus the various R&D efforts. 

Such a roadmap could assist CTL in coordinating with other federal entities and would justify the 

relatively small expenditure of resources needed to develop it. 

Key Recommendation: The PSCR Division should develop a research and development 

roadmap for mission-critical voice, considering how the various activities it includes therein can 

be used as integral elements. CTL should consult on the roadmap’s development with other 

organizations, both government and commercial, to determine overlap of technology 

development. CTL should conduct its own critical technology assessment to inform its roadmap. 

Location-Based Services 

PSCR aims to demonstrate capabilities that first responders can reply upon in three major 

research areas in LBS: (1) indoor mapping, (2) tracking, and (3) navigation. A major effort has been in 

pre-incident planning, by developing a point cloud database of major non-residential indoor 

environments, as well as sensor data fusion (i.e., fusing sensor data to participants). PSCR has engaged 

extensively with external organizations to build research capacity in the area of indoor LBS.

Accomplishments 

Some of the key accomplishments of the PSCR’s LBS program include the following: 

 32 scientific and engineering articles have been published in the field of indoor localization 

and specific applications to public safety with. 15 projects have been launched (12 grants, 1 

contract, 3 NIST projects) that have engaged 140 R&D personnel, including 90 students and 

postdocs. 

 PSCR has demonstrated that point clouds could be transformed to IndoorGML (Indoor 

Geographic Markup Language)12 networks for use in indoor navigation. 

 The Point Cloud City projects have already proven critical for two different LBS initiatives: 

(1) The Indoor Mapping and Navigation Pilot, and (2) internal PSCR research focused on 

object anonymization and scene classification—highlighting the importance, but lack of 

available, point cloud data sets for research. 

 Work in the Indoor Mapping and Navigation Pilot has also made several key contributions to 

geospatial standards: (1) A public safety application domain extension (ADE) was created for 

the CityGML13 (City Geographic Markup Language) standard and (2) the concept of an ADE 

was added to the IndoorGML standard, along with a new public safety specific ADE. The 

new CityGML and IndoorGML Public Safety ADEs provide a mechanism to enrich building 

models with new features and attributes required for public safety use cases. Interestingly, 

prior to this pilot, the concept of an ADE did not exist for IndoorGML. However, early on in 

the pilot it became clear that in order to serve effectively as a navigation model for the public 

 12 Further information available on the IndoorGML website at http://indoorgml.net/. 

13 Further information available on the CityGML website at http://www.citygml.org/. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

17 

safety use case, or any specific use cases for that matter, the concept of an ADE would need 

to be extended to the IndoorGML standard. 

Challenges and Opportunities 

In addition to ongoing efforts in mapping and navigation, the PSCR Division is preparing to 

launch a multi-year, multi-phase large-scale prize challenge that will engage innovators across industry 

and academia to meet or exceed the public safety responder community’s desire for sub-meter indoor 

localization accuracy without the need for pre-deployed dedicated infrastructure. The challenge will occur 

in at least three different building environments and will mirror other challenges by seeking direct 

FirstNet14 involvement and partnerships. The first track will focus on identifying the highest performing 

sensors in each of five categories broadly defined as: inertial-based, imagers and scanners, signals of 

opportunity, external anchors, and cooperative ranging. In this track, teams will be evaluated based on the 

performance of their sensor using test scenarios designed to stress that sensor category. The second track 

will focus on identifying the highest performing multi-sensor fusion algorithms. In this track, teams will 

be evaluated based on their ability to generate accurate positions of first responders by fusing the 

individual sensor data from the first track just mentioned as it is streamed in “real-time” to participants. 

This prize challenge, if executed successfully, will offer a great opportunity to engage the broader indoor 

LBS and data analytics community to introduce innovative solutions for the PSCR program. 

The generated data from the Point Cloud City project is valuable for a wide range of other PSCR 

research, and will bring new opportunities in localization, object anonymization, and scene classification. 

Recommendation: The PSCR Division should explore some of the recent developments in the 

3GPP standardization group related to indoor localization enhancements using cellular networks 

and consider adding such to its own research agenda. 

Public Safety Analytics 

The CTL analytics group staff is composed of five key personnel who are on loan to PSCR. Two 

of these individuals hold Ph.D.s, and all are experienced researchers. They are supplemented by several 

contractors who perform project research, principally at universities, under the guidance of the internal 

staff. The skill sets of the staff members are varied and complementary, which is a positive characteristic. 

Accomplishments 

The principle accomplishments have been achieved by the external contractors monitored by the 

internal personnel. CTL stood up an external research program as of June 2017, with contracts that were 

fully engaged sometime after that date. The research portfolio includes research projects in emergency 

medical services analytics, firefighting analytics, crosscutting research in video analytics, differential 

privacy prize challenge, social media incident streams challenge (first-generation and a coming secondgeneration challenge), analytics container environment framework to support scalable streaming 

analytics, collaboration with Baltimore CitiWatch,15 and a planned challenge in automated streams 

analytics for public safety. The output of this research has principally been conference papers and opensource software. 

 14 The First Responder Network Authority, or FirstNet, was created by the Middle Class Taxcut and Job 

Creation Act of 2012. Situated within the Department of Commerce, FirstNet is given the mission of developing and 

operating a nationwide broadband network for first responders. 15 Further information available on the CityGML website at http://www.citygml.org. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

18 

While it is still early in the process to judge the outcomes from these research projects, there are 

some projects that stand out as potentially very useful. These include the work in: developing privacy 

analysis, developing metrics to assess video quality for analytics processing, and collecting relevant data 

sets for doing video analytics comparisons. The privacy issue in video analytics is of growing public 

concern, and it is timely and appropriate that CTL addresses this area in their long-term research 

portfolio. 

In 2016 PSCR published the Public Safety Analytics R&D Roadmap16 following an analytics 

summit. PSCR has undertaken many activities— both intramural and extramural—since that time that 

might need to be considered in its long-term planning. 

Key Recommendation: CTL should update its roadmap for public safety analytics, taking into 

consideration projected future areas of interest. 

Challenges and Opportunities 

Key challenges for the public safety analytics program include defining metrics to ensure that the 

overall portfolio is on track to produce (in the long term) results that will be useful to the public safety 

community through instilling into the performers the importance of quantifying impact or potential 

impact of their research. The metrics should allow for comparison of their work against similar work 

being done worldwide. These metrics could be very useful for the internal NIST researchers to gauge the 

performance of their contractors. The video analytics work is a start to understand the metrics that can be 

used to develop tangible operational metrics, and the work between CTL and Baltimore CitiWatch looks 

very promising for supplying validation data for those metrics. The video metrics work and the creation 

of useful data sets has the potential to be magnified in future research efforts. 

The prize challenges in public safety have beneficial short-term impact of making the community 

aware of what’s possible and to highlight those groups that are doing great work in relevant areas. The 

work of the PSCR Division might benefit from giving consideration to how this effort will produce longlasting infrastructure, talent, and tools for public safety beyond the end date of the competition. 

Enhanced User Interface/User Experience 

First responders face many unique communication challenges. The challenges vary based on 

factors such as task, environment and equipment constraints. In addition, broadband communication 

capabilities along with new sources of information have the potential to dramatically increase the amount 

of data available to first responders. Innovative tools and techniques that enable first responders to 

interact with these new capabilities and systems are needed. To facilitate this, the UI/UX portfolio focuses 

on UI research, virtual reality (VR) and augmented reality (AR)-based research and UX research. This 

portfolio includes internal and external research along with prize challenges. Seven external grants and 

cooperative agreements were awarded in 2018. 

Accomplishments 

The major accomplishments of the PSCR UI/UX portfolio include the following: 

 16 NIST, 2016, Public Safety Analytics R&D Roadmap, NIST Technical Note 1917, U.S. Department of 

Commerce, Washington, D.C. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

19 

 PSCR has built a UI/UX research and development team to demonstrate the importance and 

utility of UI/UX design for public safety. The team has significant collaboration with the 

interdisciplinary Atlas Institute at the University of Colorado, Boulder. 

 Two surveys of the public safety community were conducted to identify the required 

functionality for future communications systems. These surveys included 7,000 responses 

from public safety professionals and 200 in-person interviews. The findings from the 

interviews have been published as NIST reports.17,18 Additionally, PSCR has a publicly 

available search tool for identifying quotes from the interviews based on a variety of criteria. 

This information could be useful in influencing the design and development of public safety 

technology. 

 Three unique VR environments with scenarios for fire, police, and EMS were developed to 

serve as test platforms for potential UIs. These environments are instrumented to facilitate 

qualitative and quantitative measurements on the performance of UI prototypes. Notably, this 

set of tools is open source and available to the public. 

 PSCR has completed a prize challenge to create a heads-up display (HUD) with unimpeded 

visual aids and has an ongoing prize challenge focused on haptic interfaces. Both of these 

challenges utilize the VR test environments. The interest of external partners in the prize 

challenges is increasing. Although FirstNet participated in judging the HUD challenge, there 

were no external partners. Currently PSCR has a number of official partners for the haptic 

challenge including the FirstNet Authority, AT&T, MSA, and West Metro Fire Protection 

District (CO). 

Challenges and Opportunities 

 Effective UI/UX design is one of the keys to the public safety community taking advantage of 

the new technologies enabled by broadband communication capabilities. By providing resources such as 

the first responder survey findings and open source VR tools, PSCR has the opportunity to facilitate 

expanded UI/UX research and development. In addition to the prize challenges, PSCR might consider 

other ways to raise awareness and understanding of public safety research problems to engage a broader 

community in the research and development. One potential avenue for reaching a broader academic 

audience would be use of the resources in education. 

 PSCR plans to launch several new UI/UX prize challenges and do continued development on 

new VR environments. UI/UX emerged as a priority for public safety stakeholders and it is critical to 

successful integration of new technologies into public safety operation. It is necessary to think about how 

to keep UI/UX in the forefront, particularly after 2022. 

Key Recommendation: The PSCR Division should consider integration of the UI/UX (user 

interface/user experience) research and prize challenges with the other PSCR portfolios as 

appropriate provided efficiencies can be gained by such integration. The division should consider 

developing a methodology and process for studying UI/UX along with the study of new 

technologies. 

 17 NIST, 2018, Voices of First Responders—Identifying Public Safety Communication Problems: Findings from 

User-Centered Interviews. Phase 1, Volume 1, NISTIR 8216, U.S. Department of Commerce, Washington, D.C. 18 NIST, 2018, Voices of First Responders—Examining Public Safety Communication Problems and Requested 

Functionality Findings from User-Centered Interviews Phase 1, Volume 2.1, NISTIR 8245, U.S. Department of 

Commerce, Washington, D.C. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

20 

Security

Though CTL considers PSCR security as a primary responsibility of FirstNet and its partners, 

CTL still identified use cases for security authentication that would be valuable to FirstNet. CTL has 

dedicated its efforts in this focus area and has built technology solutions around it. 

A resource of about 10 personnel from CTL and other organizations within NIST have been 

dedicated to supporting PSCR security. 

Accomplishments 

CTL invested heavily in PSCR security areas with funding levels of $2.2 million in FY 2018 and 

$2.6 million in FY 2019. Highlighted accomplishments from CTL’s PSCR security program are the 

following: 

 CTL supported 3GPP standardization efforts, including mandating elliptic curve 

implementation, analyzing drivers with 256-bit algorithms in 5G (5th Generation Wireless 

Network), and enabling applications to understand security postures with cellular 

connections. 

 CTL PSCR program has built Mobile Single Sign-On (SSO) capabilities for improving 

authentication for public safety first responders. 

 CTL has developed and published a NIST Interagency or Internal Report (NISTIR), Security 

Analysis of First Responder Mobile and Wearable Devices.

19 This document reviews the 

current and potential use cases of mobile and wearable devices by first responders and then 

analyzes these devices from a cybersecurity perspective. Ultimately, the goal of the document 

is to identify security objectives for mobile and wearable devices to assist jurisdictions with 

selecting secure devices and to enable industry to design and produce secure public safety 

devices. 

 CTL launched and expanded the “SIM Card Use for Public Safety” challenge to investigate 

the possibility of using SIM card as a storage container for user credentials. PSCR will award 

up to $100,000 to the winning teams that successfully propose and demonstrate their idea on 

how they will securely store a file on a SIM card; create a mobile application that accesses a 

credential stored on a SIM card; and authenticates to a FIDO220 (Fast IDentitiy Online) 

service provider. CTL has partnered directly with FirstNet, AT&T, IBM, and Nok Labs for 

this challenge. 

 CTL developed a unified secure information access framework, called ICAM (Identity, 

Credential and Access Management), to meet the first responder community needs. 

 CTL provided a test bed where industry, FirstNet, PSCR, and other research organizations 

can evaluate and incorporate public safety communication research objectives. 

Challenges and Opportunities 

Besides performing research in mobile access authentication, the PSCR Division has 

opportunities for protecting user applications and user data in the PSCR mobile environment. User 

 19 NIST, 2018, Security Analysis of First Responder Mobile and Wearable Devices, Draft NISTIR 8196, U.S. 

Department of Commerce, Washington, D.C., January. 20 A joint effort between the Fast Identity Online Alliance and the World Wide Web Consortium (W3C) to 

create authentication solutions. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

21 

applications need to be certified before installation. User data can only be accessed from authorized 

applications and with right privileges. 

Recommendation: CTL should proactively seek best practices from industry and other 

government organizations for both mobile application and user data protection. 

CTL has heavily leveraged collaboration partners from industry and other NIST organizations in 

the cyber security studies. This practice could result in lack of core competencies for its own team. CTL 

may consider devoting more resources for building up an internal research team in better support of PSCR 

stakeholders and development of its own core technologies. 

Cyber security is essential for PSCR stakeholders. New cyber threats, new operational 

requirements, and technology advances present continuous challenges to the CTL PSCR program, 

especially when the 5G cellular system is in service. Providing proper support after year 2022 will be a 

challenge and an opportunity for the CTL PSCR program. 

Resilient Systems 

The PSCR resilient systems portfolio is limited to specific use cases around rapidly deployable 

and fault tolerant networks that are valuable to FirstNet. There is significant resilient systems research 

across multiple sectors that can be directly leveraged by FirstNet and public safety. The resilient systems 

portfolio is comprised of prize challenges and external research. 

Accomplishments 

 In 2018, PSCR designed and launched the UAS Flight and Payload Challenge. The 

competition was designed by NIST to support field operations for first responders. UAS 

operations in a public safety realm entail a tradeoff between payload versus flight time. There 

are different missions capabilities enabled by Vertical takeoff and landing (VTOL) of a UAS 

but their flight time is nonetheless limited. The parameters of payload capacity, energy source 

and flight time imply design trade-offs in the quest to optimize the vehicle for efficiency and 

flexibility. This prize challenge was thus designed to help public safety operations by keeping 

a UAS and its payload airborne for the longest time possible with vertical and hovering 

accuracy. Additionally, at a cost of less than $20,000 per UAS, this challenge shows first 

responders that there may someday be an affordable drone in their toolkit to carry wireless 

networks for emergency operations.21 

 In 2017, PSCR funded six grants to evaluate or enhance the resilience of public safety 

mission critical systems in the face of connectivity challenges. This includes traditional 

research, evaluation prototypes, and enhancements to existing systems. All six grants should 

be completed by the end of 2019. 

Challenges and Opportunities 

Building on the results and lessons learned from the first UAS challenge, PSCR is designing a 

second Unmanned Aerial Systems Flight and Payload Challenge. The goal is to push the current limits of 

 21 Adapted from Office of Science and Technology Policy, 2019, Implementation of Federal Prize and Citizen 

Science Authority: Fiscal Years 2017-18, Executive Office of the President, Washington, D.C., June. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

22 

designing affordable UAS that can be used to carry wireless networks into areas that are not accessible to 

typical deployable units. 

Recommendation: CTL should continue to work closely with FirstNet stakeholders to identify 

research needs relevant to the resilient systems use cases of interest to FirstNet. 

PORTFOLIO OF SCIENTIFIC EXPERTISE 

To build the scientific expertise necessary to carry out the mission as mandated in the Act, PSCR 

has established research, testing and development expertise that aligns with the needs of their stakeholder 

population. PSCR has grown its permanent technical staff to 15, utilized experts from other organizations 

in NIST, funded 46 grants and contracts with teams from academia, industry, and public safety, and 

created a series of prize challenges open to innovators around the world. This approach to developing 

scientific capabilities is driven by the limited term of the $300 million funding from the Act. It has 

resulted in significant accomplishments in each of the four research portfolios as well as the two 

crosscutting areas. (See “Introduction” to this chapter, above.) 

ADEQUACY OF FACILITIES, EQUIPMENT AND HUMAN RESOURCES 

PSCR facilities and equipment are adequate to the mission. The PSCR team moved into a newly 

renovated building in early 2018. Their laboratories are well-equipped, and they also have a mobile 

research vehicle that provides a mobile research laboratory capability. 

PSCR is comprised of a mix of permanent and term employees. Less than half of the staff (43 

percent) are permanent employees. Term employees generally come from other government organizations 

as well as universities. The expiration of funding in 2022 makes it challenging to attract the best talent 

due to PSCR’s limited ability to hire permanent employees. As the 2022 end date approaches, PSCR will 

likely find it increasingly difficult to attract highly talented term employees as well.

PSCR is addressing key issues in public safety, and it is likely that the research mission will 

continue to be of importance beyond the 2022 end date of the appropriated funds from the spectrum 

auction. 

Key Recommendation: CTL should evaluate the possibility and pros and cons of a strategic 

expansion of PSCR’s internal research staff aimed at ensuring continuity of research in key 

priority areas in particular after fiscal year 2022 when the spectrum auction funds will have been 

spent or no longer be available. In addition, CTL should develop a plan for leveraging the 

expertise developed through the prize challenges. 

DISSEMINATION OF OUTPUTS 

Dissemination is a key component of the PSCR goal to create a public safety ecosystem and this 

is an area where PSCR has established a strong and successful effort. PSCR has many different avenues 

for disseminating results. Every year since 2016, PSCR has held a multi-day stakeholder meeting with 

attendees representing the many different sectors interested in public safety communications. The 2019 

stakeholder’s meeting held in Chicago had 486 attendees. The breakdown of attendees was as shown in 

Figure 3.1. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

23 

FIGURE 3.1 Attendees at PSCR Stakeholder Meeting, July 2019, by sector. 

During the stakeholder’s meeting, results from each of the portfolios and crosscutting areas were 

presented along with talks and panels from stakeholders. In addition there were demonstrations of 

capabilities developed. These meetings give stakeholders the opportunity to learn about the latest 

developments, provide input and interact with each other. 

PSCR has made a point of requiring deliverables such as data sets, application programming 

interfaces and open-source tools—all of which encourage the growth of the public safety ecosystem and 

can be utilized by others and built upon. These results are freely available via the PSCR website and 

many have been integrated into prize challenges to facilitate new efforts. In addition, prize challenge 

winners are incentivized to make their winning systems open source. Results of PSCR activities are also 

published in appropriate scientific conferences and journals as well as in NIST reports.
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

24 

4 

Metrology for Advanced Communications 

The second main category of CTL’s work addressed by the panel, Metrology for Advanced 

Communications, is discussed in this chapter. This category is implemented through three of the four 

CTL priority areas: (1) Trusted Spectrum Testing, (2) Fundamental Metrology for Communications, and 

(3) Next Generation Wireless (5G and Beyond). As may be seen from Table 4.1, three of the 

organizational units in CTL address priority areas that comprise the main category of CTL work treated in 

this chapter. In the discussion below, the assessment of the work of the aforementioned three 

organizational units is subsumed within the discussion of each priority. 

OVERALL METROLOGY DISCUSSION AND RECOMMENDATIONS 

CTL has adequate and stable funding to carry out its metrology mission. The scientific and 

technical quality of the team is excellent and the communication between team members is highly 

supportive of a collaborative, highly productive and interdisciplinary culture. 

CTL staff has identified an excellent set of metrology projects to support future communications 

needs. This work is centered on solving fundamental and near-term applicable measurement challenges 

for emerging communications standards. Some of the projects clearly emerge from fundamental 

metrology scientific advances of clear practical import, and others seem to be (appropriately) driven by 

short term needs resulting from emerging national policy initiatives. The process for the future strategic 

selection of new projects—to maximize CTL impact—is unclear. 

 

TABLE 4.1 Metrology for Advanced Communications and Its Implementation across 

Communications Technology Laboratory (CTL) Priorities and Organizational Units 

 CTL Priority Areas within Metrology for Advanced Communications 

 Trusted Spectrum Testing 

Next Generation 

Wireless 

Fundamental Metrology for 

Communications 

PSCR 

WN X X 

RFT X X X 

NASCTN X 

NOTE: PSCR = Public Safety Communication Research Division; WN = Wireless Networks Division; RFT = 

Radio Frequency Technology Division; NASCTN = National Advanced Spectrum and Communications Test 

Network. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

25 

The CTL has capacity to grow in important metrology areas in order to address emerging 

communication needs in new technology and application areas, and the nation would be well served by 

this growth. 

The CTL metrology staff are consistently and diligently engaged in the important task of 

dissemination of their scientific and technical findings to the broader community of technical 

professionals and stakeholders. 

The CTL is a unique national resource, and its scientific output and test infrastructure need to be 

better publicized and made clearly available in order to have the highest impact. 

The CTL physical infrastructure is mostly adequate for the current programs in its highly 

ambitious metrology mission. 

There is substantial evidence that CTL has accomplished a significant amount in multiple areas of 

interest to its stakeholders on a relatively limited and uncertain budget in a short period of time. 

Based on its significant accomplishments in a diverse portfolio of technical areas in a short period 

of time, the CTL is poised to capitalize on a number of timely and impactful opportunities, including: 

quantum-enabled metrology with applications in QISE; a new framework for channel metrology and 

modeling with the capability of predictive performance analysis to inform the design and deployment of 

next generation wireless networks; innovative hardware and platforms for testing and metrology of next 

generation wireless technology; continuing the growth and development of the NASCTN testbed and 

operational processes; leveraging machine learning, data science and statistical techniques to fully utilize 

the range of metrology capabilities at CTL and NIST to significantly increase, enhance and broaden the 

scope of its contributions and output to its stakeholders. 

The stakeholders of CTL represent diverse communities (industry, government, academia and 

research communities in the various technical areas reflected in CTL’s projects) and have different needs 

ranging from basic science and engineering to applied research for design and technology development to 

offering its test and measurement facilities for stakeholder projects. A range of metrics are needed to 

evaluate NIST CTL. 

CTL—with the diverse expertise of its world-class staff—has significant latent capacity that it 

can leverage to capitalize on timely and impactful opportunities in collaboration with other NIST 

laboratories (e.g., the Physical Measurement Laboratory [PML], the Information Technology Laboratory 

[ITL], and the Engineering Laboratory). 

Key Recommendation: CTL should develop a 3-year or 5-year strategic plan for its activities in 

metrology for advanced communications, to include: identifying and evaluating new research 

directions and opportunities for growth; developing strategic partnerships with other NIST 

laboratories for pursuing new areas; identifying resource needs (equipment, facilities, staff) for 

pursuing strategic growth areas of research; identifying and pursuing internal and external sources 

of funding to support the plan; and developing measurable criteria and metrics for annually 

evaluating progress toward 3-year and 5-year goals. The strategic plan should explain how its 

execution will support the successful attainment of the CTL priorities. 

FUNDAMENTAL METROLOGY FOR COMMUNICATIONS 

Assessment of Technical Programs 

Nearly a decade ago, NIST CTL researchers began investigations on better, more accurate ways 

to measure electric fields. This was done with an aim toward making such measurements more 

“traceable” to fundamental SI quantities.1

 Subsequent years of effort have borne impressive results. 

 1

 The International System of Units (SI) is made up of 7 base units: kilogram (kg) mass, kelvin (K) temperature, 

candela (cd) luminous intensity, ampere (A) electric current, meter (m) length, second (s) time, and mole (mol) 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

26 

The primary result is a practical, demonstrated, Rydberg atom-based, SI-traceable measurement 

method (and prototype devices) for precision electric field (E-field) and power measurements. This result 

extends the amplitude range, frequency range, and accuracy of E-field measurements. Through the use of 

lasers and microwave signals to induce transparency in a closed cell containing a vapor of particular 

atoms (rubidium or cesium), one or two electrons are excited to states causing their host atoms to have 

large dipole moments, allowing them to be used as very sensitive E-field detectors. Initial results indicate 

multiple orders of magnitude improvement in the measurement sensitivity compared to the state of the art. 

This invention has been shown to enable several other applications, including measurement of 

phase and modulation of fields for communications transmitters and receivers, quantum-enabled imaging 

for medical and near-field applications, power calibrations and more. 

The measurement capabilities developed by CTL using Rydberg atoms also offer a promising 

bridge to future research directions mentioned by the CTL director: quantum computing and optical 

communication networks. Rydberg atoms are being explored as promising candidates for physical 

realization of qubits (quantum bits) and CTL is well-positioned to build on their recent successes to 

initiate a research program in QISE in collaboration with other NIST laboratories, such as the PML, 

which are already engaged in QISE-related research. This is a timely opportunity given the recent 

announcement of the National Quantum Initiative (NQI) in December 2018 and the worldwide attention 

QISE is receiving. In this endeavor, CTL is well-positioned to identify strategic collaborative 

opportunities with PML (and other laboratories, such as ITL) that draw on its unique strengths in 

communication metrology and technology to advance the science, engineering and technology frontiers of 

quantum computing, quantum communication and quantum sensing. 

The technical area of optical communication technology also represents a synergistic direction 

that CTL is interested in adding to its research portfolio in the future. On the one hand, the need for higher 

bandwidths, which is driving emerging wireless technologies (5G and beyond), is also driving new 

technologies and innovations in fiber-optic and free-space optical communications to which CTL can 

fruitfully contribute. On the other hand, optical and microwave communication techniques and 

technologies are expected to play a key role in the emerging QISE technologies, and this provides further 

impetus for CTL to develop core expertise in this area, possibly by hiring new scientists and engineers 

with the requisite expertise. 

The RFT Division continues to perform fundamental metrology research and development of 

techniques for high-frequency measurements applicable to a wide array of industry and government 

stakeholders. This high impact work has significant impact on 5G integrated circuit development, a field 

of high national priority. Related work is in advanced vector network analysis and calibration. NIST is 

viewed as a best in the world resource in this area. They have also developed a “microwave uncertainty 

framework.” This framework will have much broader application than only the microwave frequency 

band. 

The technological trend towards fully integrated communication devices, in which the digital 

processing, analog-digital converters, and RF hardware and antennas are all integrated onto a single board 

with multiple chips, is radically changing the paradigm for testing and characterizing the performance of 

these new devices. NIST CTL is at the forefront of pioneering a new approach to testing next generation 

wireless devices and systems that would combine on-wafer measurements of individual components with 

over-the-air (OTA) testing of integrated devices and complement them with new approaches for making 

near-field measurements of various components in an integrated board. 

 

amount of substance. SOURCE: NIST, 2017, SI Measurement System, NIST SP 304A, U.S. Department of 

Commerce, Washington, D.C., July. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

27 

Accomplishments 

The quality of the quantum SI-traceable measurements and application research is outstanding. 

The team is defining the state of the art in E-field amplitude, phase and polarization metrology. Among 

others, the developed Rydberg-atom-based methods are able to measure E-fields with amplitudes as low 

as 10 millivolt per meter (mV/m) at room temperature, over frequency bands ranging from tens of 

megahertz (MHz) to hundreds of gigahertz (GHz) or even terahertz (THz), with an instantaneous 

bandwidth of tens of megahertz. The team has also demonstrated the ability to accurately measure the 

phase of the E-field over a wide frequency range relying on the same fundamental principles. Several 

proof-of-concept demonstrators have been implemented, including analog (stereo AM/FM) as well as 

digital (up to 64 quadrature amplitude modulation [QAM]) modulation and demodulation. 

The developed technology is also an enabler for high frequency (>110 GHz) calibration 

procedures. This is a field in which NIST is playing a key role in different ways, including the 

development of high-fidelity on-wafer measurement techniques and integrating these with on-board and 

over-the-air measurements to invent the future of vector network analyzers. The need for these new 

classes of measurements and network analyzers is driven by the advances in integrated circuit (IC) 

technology for communication applications in which the IC chips for various subsystems (e.g., RF 

transceivers, analog-digital convertors, and digital processors) are all integrated onto a single printed 

circuit board (PCB) with no connectors available for testing the individual systems. This is a stark 

departure from current testing methodologies that are based on conductive measurements of individual 

components. This area also ties in well with the increasing use of machine learning and data science 

techniques at CTL as evidenced by the newly-established laboratory space on the third floor of Building 

3 that included a dedicated server for machine learning and high-intensity data analytics. 

Overall, this program is an excellent example of how timely investment of resources for the 

exploration of very fundamental physics principles (Rydberg atoms) can lead to transformative practical 

solutions (SI traceable E-field measurements) for industry/societal problems (calibration of equipment). 

This is also an excellent example of how such investment of resources can also lead to serendipitous 

dividends in terms of new unanticipated opportunities; new programs in QISE and optical communication 

technologies, in this case. CTL’s success in winning multiple Institute of Electrical and Electronics 

Engineers Instrumentation and Measurement Society (IMS) awards in these areas, in collaboration with 

other laboratories (e.g., PML and ITL), is commendable and reflective of the excellence of technical 

programs and the expertise of the staff. 

A significant accomplishment has been the development of three different types of sounders, 

operating in 28 GHz, 60 GHz, and 83 GHz bands, with directional measurement capabilities using a 

variety of antenna architectures, including switched arrays and phased arrays. The measurements taken by 

these sounders have led to a set of “ground truth” propagation data that can be used for evaluating the 

performance of different sounders.2

 These measurement technologies, facilities, and advanced data 

processing capabilities are at the forefront of advanced metrology for next generation wireless 

communications. 

Challenges and Opportunities 

Several technical challenges must be overcome before the Rydberg-atom based devices can 

become widely useable. These include further miniaturization and eventual embedding of the system in a 

compact probe, widening of modulation bandwidths (currently in the order of 20 MHz, set by the 

electron/atom relaxation time), application of filtering, etc. The team is well aware of these challenges, 

and is currently at work on addressing them. 

 2

 Data are available at NIST, “5G mmWave Channel Model Alliance,” https://5gmm.nist.gov/. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

28 

The accomplishments in quantum-enabled measurements also offer new opportunities in the 

growing and important area of QISE, in addition to new avenues in quantum-enabled sensing that team is 

already pursuing—for example, building on the measurement work to create higher-fidelity Rydberg 

atom-based qubits with longer coherence times. CTL’s expertise in optical and microwave metrology 

techniques would also be valuable in realizing high-fidelity signals for controlling the initialization and 

evolution of qubit states. A related opportunity is in the development of optical quantum communication 

techniques for exchanging qubits across multiple quantum processors (with tens to hundreds of qubits) to 

create a distributed quantum computer with larger number of qubits. NIST and CTL have made 

fundamental contributions to optical communications in the past and these potential opportunities align 

well with the CTL’s plan to re-establish its program in the area of optical communications. An important 

related research challenge is transduction of quantum states across multiple modalities—for example, 

between Rydberg atom-based qubits (for use within a quantum processor) and photonic qubits (for 

transporting qubits across quantum processors). 

Another opportunity for the CTL within the quantum realm relates to new applications in 

quantum information science, supported by optical communications, the latter being a field in which 

NIST has made many fundamental contributions in the past. As the need for communication between 

quantum computing increases, developing technologies to support the reliable exchange of quantum bits 

is needed. 

The CTL team has delivered exemplary results noted above. The Rydberg atom E-field sensor 

appears to have breakthrough potential. 

Recommendation: CTL should continue to support work on the Rydberg atom E-field sensor to 

maintain development and extend applications. 

The CTL team has proactively identified Optical Communications and Quantum Information 

Science and Engineering as promising new areas for research that synergistically build on the quantumenabled sensing work. 

Key Recommendation: The Radio Frequency Technology Division should broaden its research 

portfolio into the areas of optical communications technology and quantum information science 

and engineering—both of which it has identified already—while leveraging strategic 

collaborative partnerships with other NIST laboratories, including the Physical Measurement 

Laboratory and the Information Technology Laboratory. 

Portfolio of Scientific Expertise 

The RFT Division’s scientific expertise in SI-traceable measurements is closely matched to the 

program’s technical needs and strongly positions CTL at the forefront of the field, as evinced by the 

visibility and citations on this work. 

Challenges and Opportunities 

While expanding CTL’s capabilities into the synergistic areas of QISE and optical 

communications is timely and potentially impactful, it would likely require hiring new scientists and 

engineers with requisite expertise that can provide a bridge between CTL’s capabilities in communication 

technology and metrology and the capabilities of the collaborating laboratories, such as PML, in quantum 

information science. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

29 

Future technical staff hiring may require expertise that can provide a bridge between CTL’s 

capabilities in communication technology and metrology and the capabilities of the collaborating 

laboratories, such as PML, in QISE. 

Recommendation: The CTL should evaluate the resource and staff needs for future efforts in 

quantum information science and optical communications, in consultation with collaborating 

NIST laboratories (e.g., PML and ITL) and develop a plan for meeting those needs through 

internal and external sources of funding. 

Adequacy of Facilities, Equipment, and Human Resources 

Accomplishments 

CTL has made significant progress in developing in-house capabilities for fundamental metrology 

in the context of next generation wireless technologies, including those for channel modeling and 

measurement. In the work led by the RFT Division; a number of new measurement and testing facilities 

have been developed, including anechoic chambers with robotic arms for radiation pattern measurements 

in static and mobile scenarios, a robotic aperture scanning testbed for making measurements appropriate 

for antenna arrays, reverberation chambers for creating different multipath propagation environments, and 

new methodologies and testbeds for characterizing the performance of channel sounders, including 

angular measurements. (The complementary work by the WN Division on channel modeling is discussed 

below in the section, Next Generation Wireless.) 

Challenges and Opportunities 

Sustainability of adequate personnel and facilities could present a programmatic challenge for 

building on recent accomplishments and expanding into new areas in quantum information science and 

optical communications. This effort will require strategic collaborations with other laboratories at NIST, 

such as PML and ITL. It is not clear whether the collective capabilities of CTL and other NIST 

laboratories, in terms of facilities, equipment and human resources will be adequate to successfully 

execute the potential new research activities. There appear to be multiple commercial opportunities for 

quantum-enabled metrology and potentially many future ones in the broader area of quantum information 

science. However, CTL will have to act in a thoughtful and strategic fashion to evaluate the resource 

needs and address them in a timely fashion. 

The available resources in terms of facilities and staff seem to be adequate to sustain the current 

programs. However, growing into the new synergistic areas of optical communications and quantum 

information science will require strategic collaborations with other laboratories, such as the PML and the 

ITL, and likely require additional resources. The facilities and equipment being used for the quantum SItraceable measurements are excellent. Overall RFT Division facilities are excellent. 

Effective Dissemination of Outputs 

The publications and patents from the quantum SI-traceable measurement research are first-rate 

in terms of venue, quantity, and visibility. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

30 

Accomplishments 

This work has led to 17 journal publications in top-tier journals in the applied physics society, 

including Physical Review Applied, Applied Physics Letters, and Journal of Applied Physics, and 10 

conference papers.3

 These works have had, and are having, a high impact in the community, as shown by 

the very high number of downloads in the short term as well as citations in the long term. 

Challenges and Opportunities 

Opportunities exist in commercialization and development of applications. The novel modulation 

and musical demonstrations possible using the Rydberg atom could be used for outreach to K-12 student 

communities. Another opportunity may exist if the quantum SI-traceable measurement research could be 

connected with future applications in quantum information science and engineering. 

NEXT GENERATION WIRELESS 

Assessment of Technical Programs 

CTL is tackling a diverse range of timely and significant problems in the area of 5G wireless 

technology, fundamental metrology, and applications, particularly on the topics of 5G Channel Modeling 

and Measurements. In addition, CTL is providing outstanding stewardship of the 5G mmWave Channel 

Model Alliance. This organization is a worldwide community of researchers engaged in research and 

development for new approaches to measurement and modeling of propagation channels at millimeterwave frequencies. CTL also conducts work on other technologies that support 5G including for example, 

on massive multiple-input, multiple-output (MIMO) antennas on frequencies up to and including those of 

millimeter wave. Measurements of MIMO are enabled in the reverberation chambers of the National 

Broadband Interoperability Testbed on the Boulder campus as are over-the-air (OTA) measurements. 

The CTL work on millimeter-wave channel measurements and modeling began in 2011, and CTL 

initiated the 5G mmWave Channel Model Alliance (the “Alliance”) in 2015. The Alliance has grown to 

include over 80 organizations from industry, academia, and government. 

The Alliance has worked to develop multiple publications for the broader community, 

methodologies for channel measurements and model development, and has brought numerous individuals 

together for collaborations, who otherwise would likely not have interacted. The CTL team members have 

continued in a leading role in the Alliance, in direct response to a recommendation from the 2015 

National Academies review.4

The CTL team has also taken several of their own measurement-based models and employed 

them in higher level network simulators, provided a commercial software vendor with code to enable the 

use of an 802.11 model by the wider community, and has also published widely. In addition to journal 

and conference papers, the Alliance has sponsored several workshops at major conferences. 

 3

 Christopher Holloway, Matt Simons, and Joshua Gordon, NIST CTL, “Quantum SI Traceable Measurements 

and Calibrations: Radio Frequency Electric Fields and Power,” presentation to the panel, June 26, 2019. 4

 National Academies of Sciences, Engineering, and Medicine, 2015, Telecommunications Research and 

Engineering at the Communications Technology Laboratory of the Department of Commerce: Meeting the Nation’s 

Telecommunications Needs, The National Academies Press, Washington, D.C. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

31 

Accomplishments 

The 5G mmWave Channel Model Alliance has achieved significant accomplishments since its 

inception in July 2015. The membership of the alliance has grown significantly to over 80 members from 

industry, academia and U.S. government agencies. Two comprehensive white papers have been 

developed by the two working groups—one on channel modeling techniques, and one of channel 

measurement and verification—capturing the diverse expertise of the Alliance members.5

 NIST has also 

created a repository for sharing of data between the Alliance members and the wider research community. 

CTL has also made significant progress in developing in-house capabilities in channel modeling 

and measurement. In the work led by the WN Division, channel models are being incorporated into 

network simulation software, and a framework for comparing the performance of different sounders is 

being developed. The measurement and verification work, led by the RFT Division, utilizes the testing 

facilities described above6

—including anechoic chambers, a robotic aperture scanning testbed, and 

reverberation chambers. 

A significant accomplishment has been the development of three different types of sounders, 

operating in 28 GHz, 60 GHz, and 83 GHz bands, with directional measurement capabilities using a 

variety of antenna architectures, including switched arrays and phased arrays. The measurements taken by 

these sounders, and extension of propagation parameters from those measurements, has led to a set of 

“ground truth” propagation data that can be used in the models for numerically evaluating the 

performance of different sounders 

Challenges and Opportunities 

There are a number of opportunities in the area of 5G channel modeling and measurements and 

next generation wireless networks that the CTL is poised to pursue, including: expanding the work to 

higher THz frequencies; expanding the work to UAV network channels; measurement campaigns for 

outdoor scenarios; quantifying the effects of human occupancy on indoor and outdoor channels and 

developing accurate channel models and an appropriate network modeling framework that has the 

capability to predict network performance in a given deployment area. 

The microwave uncertainty framework (MUF) is a powerful approach for studying and 

quantifying the impact of different sources of uncertainty in measurements of future wireless devices and 

systems—not inherently limited to the microwave band. New capabilities of future wireless systems, 

including higher frequencies, larger bandwidths, and directional antennas, coupled with new approaches 

to analysis and processing of data, including machine learning techniques, present exciting new 

opportunities and challenges for further developing this uncertainty framework. 

CTL is well positioned to grow the effort in next generation wireless technologies by building on 

their accomplishments at millimeter-wave frequencies. Additional measurements and models will be 

needed for other (higher) frequency bands and multiple outdoor settings. Planning for the future—for 

example, 6G (6th Generation Wireless)—is beginning in the wider community. Another challenge—for 

which NIST may be well suited in collaboration with the National Institutes of Health—is to address 

human health impacts of the use of these frequency bands. 

The approaches developed by CTL for advancing the state of the art of modeling and 

measurement of millimeter-wave channels reflects a high level of creativity and ingenuity to complement 

commercially available hardware/software with internal development of antennas, hardware, automated 

measurement capability, and software for data processing and visualization. 

 5

 The white papers were undergoing peer review and final publication at the time of the panel’s review. Further 

information may be found at NIST, “5G mmWave Channel Model Alliance,” https://www.nist.gov/ctl/5g-mmwavechannel-model-alliance. 6

 See the subsection “Accomplishments” in the section “Fundamental Metrology for Communications.” 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

32 

Data science and machine learning offer a powerful new approach that CTL is beginning to 

leverage in its work on metrology of future highly integrated wireless devices, propagation channel 

modeling and measurement, and creating traceable measurements and associated uncertainty analysis. 

Key Recommendation: To develop closer partnerships with industry, academia and governments 

and to explore new sources of revenue the Wireless Networks Division and Radio Frequency 

Technology Division should integrate and build on the recent accomplishments in 5G millimeterwave channel modeling, millimeter-wave propagation channel sounding and measurements, onwafer measurements, over-the-air measurements, and a new design framework for future vector 

network analyzers. 

Portfolio of Scientific Expertise 

The CTL staff engaged in Next Generation Wireless are highly qualified and represent a diverse 

range of expertise (including in channel modeling and measurement) that is essential for this crossdisciplinary effort, including RF hardware design, testing and measurement expertise, channel and 

network modeling, and data processing and analysis. In FY 2019, 22 percent of CTL’s budget authority 

went to Next Generation Wireless priority area. 

Adequacy of Facilities, Equipment, and Human Resources 

The facilities and equipment being used for the next generation wireless metrology, including 

millimeter-wave and higher frequencies, are state of the art. This includes components and capabilities 

developed locally by CTL staff to augment commercial measurement equipment such as vector network 

analyzers. The CTL facilities fall into two categories: (1) very modern, well equipped, well maintained 

new infrastructure, and (2) one rather old building that houses valuable instrumentation and capabilities 

but suffers from lack of infrastructure such as high speed network connectivity, and modern HVAC. The 

panel did not make a site visit to Gaithersburg, Maryland, where the WN Division’s facilities are located 

so cannot comment on facilities located there. 

CTL allows users outside of NIST to have access, on a supervised basis, through formal 

mechanisms, such as CRADAs, and informal collaborations. In addition CTL can also serve as a testing 

location for programs that need a neutral third party to serve as the technical evaluator, for example, the 

Defense Advanced Research Projects Agency. The precise rates of utilization through such agreements 

and arrangements are not readily available. 

CTL’s measurement facilities can be of great use to users in the research and business 

communities in the United States and worldwide. For this to succeed, it is essential that the facilities be 

able to handle the data and infrastructure requirements of modern communications challenges. 

Key Recommendation: CTL should undertake planning and resource allocation for renewal and 

renovation of its measurement facilities with a degree of urgency to support latent demand for 

time at such facilities in the research and business communities. CTL should aim to provide the 

usage of these facilities as a “paid service” to these communities. 

Effective Dissemination of Outputs 

The outputs of the millimeter-wave channel measurement and modeling work are numerous, and 

are of relevance and use to the community. These include the two white papers on modeling and 

measurement developed by the 5G mmWave Channel Model Alliance, contributions to standards bodies, 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

33 

and hosting of a series of workshops on the topic over the last few years. In addition, various members of 

CTL have also disseminated this work through invited presentations and engagement in other 

organizations such as the National Science Foundation (NSF)-sponsored research coordination network 

on millimeter-wave wireless. 

Accomplishments 

The growth and outputs of the 5G Millimeter Wave Channel Model Alliance are substantial 

outcomes. At least one book is being published, datasets are being archived, and software and hardware 

instrument makers are likely to use some of the Alliance outputs. The three millimeter-wave channel 

sounders that CTL has developed are also a community asset. The new capabilities for testing and 

measurement being developed for next generation wireless system at the CTL would further build on 

these assets. 

TRUSTED SPECTRUM TESTING 

Assessment of Technical Programs 

The CTL team’s work in the broad topic of shared use of federal spectrum by commercial 

systems has been enormously important. This work consists of (1) two NASCTN projects in the context 

of advanced wireless services (AWS)-3 spectrum sharing (1755-1780 MHz paired with 2155-2180 MHz), 

and (2) work in the WN Division on technologies for sharing of the 3550-3700 MHz band. The RFT 

Division provides expertise and collaboration in shared spectrum metrology on NASCTN projects using 

the National Broadband Interoperability Test Bed (NBIT) on the Boulder campus. Additionally, RFT 

works on spectrum sharing through contributions to the ANSI 63.27 standard and through research on 

techniques for quantifying the ability of wireless systems to coexist. The NASCTN projects also include 

independent measurements on the MITRE testbed and there are contributions from numerous other 

groups including NASA, NTIA and Johns Hopkins Applied Physics Laboratory.7

The projects in the context of AWS-3 sharing are very important as they aim to produce tools and 

methodology for characterizing interference of commercial LTE (long-term evolution [an early 4G 

protocol]) user equipment (UE) to Aeronautical Mobile Telemetry (AMT) systems and provide 

confidence to DoD and Commercial Service Providers (CSP) that their radio systems effectively co-exist 

in the band. This activity has two good projects: (1) Characterizing LTE User Equipment Emissions: 

Develop method for modeling LTE UE emissions, a component of the DoD aggregate interference model 

(proposed by the Defense Spectrum Organization [DSO]); and (2) LTE Impacts on AMT (proposed by 

Edwards Air Force Base). 

The newly promulgated rules found in Part 96 of Title 47 of the Code of Federal Regulations (47 

CFR Part 96) apply to use of 150 MHz of the 3550-3700 MHz band termed as Citizens Broadband Radio 

Service (CBRS). This represents significant regulatory policy innovation promoting shared use of 

valuable radio spectrum. As per these rules, incumbent systems such as naval radars, Fixed Earth Satellite 

Stations (FSS), government radio sites, and wireless Internet service providers (ISPs) share the spectrum 

with commercial radio networks. The rules require 3-tier spectrum access priority structure wherein (1) 

incumbent systems occupy the top tier and have highest priority access, (2) the bottom most tier called 

Generalized Authorized Access (GAA) provides lowest-priority access without interference protection, 

whereas (3) a middle tier called Priority Access License (PAL) provides protection from GAA devices. 

The rules also require a Spectrum Access System (SAS)—a network resident server to manage the 

 7

 Jason Coder, Adam Wunderlich, and Melissa Midzor, NIST CTL, “AWS-3 Spectrum Sharing,” presentation 

to the panel on June 25, 2019. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

34 

sharing in this tiered framework, and Environmental Sensing Capability (ESC)—a network of sensors to 

detect activity of dynamic incumbent and provide information to SAS for successful incumbent 

protection. 

The development and advancement of CBRS spectrum sharing technologies based on this threetier architecture has been undertaken in the Spectrum Sharing Committee—a multi-stake holder group 

consisting of industry and government members in the Wireless Innovation Forum (WinnForum) 

standards organization. The consensus standards developed in this forum are expected to drive rapid 

adoption of CBRS spectrum driving new business models and new use cases such as private wireless 

networks for diverse verticals and enterprises. 

Accomplishments 

The CTL team members have been valued participants in the WinnForum standardization 

activities and have made contributions that have had significant impact on standards and component 

technologies. Their key contributions are in the following areas: (1) development of SAS test procedures 

for incumbent protection, (2) a “Move List Computation” methodology for assessing and limiting the 

impact of commercial systems to prospective locations of naval radars in the dynamic protection area 

(DPA) and development of SAS test procedures—this methodology will be adopted in all commercially 

operating SAS systems; (3) development of algorithms for selection of installation sites for ESC sensors 

to provide coverage of DPAs; (4) development of machine learning and statistical processing based 

algorithms for detection of naval radars in ESC sensors; and (5) extensive collection and analysis of naval 

radar data under a NASCTN project for characterization of incumbent signals. This work has provided 

valuable insights to community toward development of ESC sensors and availability of CBRS spectrum 

when radar is active. 

Challenges and Opportunities 

With demand for more sub-6 GHz spectrum for 5G networks in the United States continuing 

unabated, the Federal Communications Commission is considering opening up new bands such as 3.1-3.5 

GHz and several new U-NII (unlicensed national information infrastructure) bands in 5.925 to 7.125 GHz 

range. Many of these spectrum bands have incumbent systems, and simplified spectrum sharing may be 

employed to accelerate availability of these bands for commercial use. The NIST CTL team has an 

opportunity to leverage its expertise and experience with CBRS to contribute to the development of 

solutions for these new bands, balancing the need for incumbent protection and the need for guaranteed 

spectrum access for commercial networks. 

The CTL team might also make contributions to the design of solutions to the problem of coexistence of 5G new radio8

 in unlicensed bands with other systems such as 802.11ax and WiFi-6. 

The CTL team in the NASCTN program and WN Division have delivered exemplary results 

noted above, with a small staff. However, their activities have been opportunistic and reactive. A more 

strategic and proactive approach towards spectrum sensing and sharing would enhance the impact of the 

team in both regulatory actions and industry standardization activities. 

The current operating model in NASCTN is to respond to requests for spectrum sensing and 

characterization from industry or other federal agencies. Given the level of current staffing, this is 

probably appropriate. In the future, the program could have a much broader impact if it would 

independently select “hot-button” spectrum issues on which to work and scale-up the number of staff 

appropriately. 

 8

 “5G New Radio (NR) is the global standard for a unified, more capable 5G wireless air interface” (Qualcomm, 

2019, “We’re leading the charge to 5G NR,” https://www.qualcomm.com/invention/5g/5g-nr). 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

35 

Recommendation: NASCTN should develop a roadmap on research needs for spectrum sensing 

characterization, including areas where NIST was primary and areas in which industry and other 

federal agencies were primary. This roadmap would feed into the 3- to 5-year research plan the 

panel recommends. 

Key Recommendation: Consistent with its future staffing levels, NASCTN should take a more 

proactive role in advising on future spectrum allocation decisions. NASCTN should engage 

impartially with all sides of the debates on emerging and urgent issues. 

Key Recommendation: CTL should continue vigorous support for the spectrum sensing and 

sharing activity, which has delivered impactful results. 

Portfolio of Scientific Expertise 

The CTL staff engaged in NASCTN are highly qualified and represent a diverse range of 

expertise that is essential for this cross-disciplinary effort, including RF hardware design, testing and 

measurement expertise, spectrum measurements, and data processing and analysis. Their stated role as a 

neutral and unbiased testing source is adhered to carefully, and they have become a trusted source of 

information and expertise on spectrum management issues. The accomplishments of the group are 

especially impressive given their relatively small size—in FY 2019, 3 percent of CTL’s budget authority 

went to trusted spectrum testing. 

Adequacy of Facilities, Equipment, and Human Resources 

The review panel toured the anechoic chambers and reverberation chamber utilized by NASCTN, 

RFT, PSCR, and other NIST laboratories and located in Building 24. Together, these chambers comprise 

the aforementioned NBIT (see Figure 4.1) and the NIST Antenna and Communication Metrology 

Laboratory (ACML), which includes the Large Antenna Positioning System and the Configurable 

Robotic MilliMeter-wave Antenna Facility. The anechoic and reverberation chambers themselves are 

impressive and can be configured to provide a range of electro-magnetic environments by configuring 

absorbent and reflective materials as appropriate to the required testing. Staff noted the lack of high-speed 

data connections from Building 24 to other parts of the NIST-Boulder campus, necessitating that the large 

data sets generated be carried by hand on portable drives between buildings. In addition, the HVAC 

system has not been upgraded in some time and does not offer ideal temperature and humidity control. 

Dissemination of Outputs 

NASCTN has had a significant impact with its report on the effect of LTE deployments on GPS 

(Global Positioning System) receivers. The team also developed a curated set of LTE uplink waveforms, 

and made these widely available for use by the community. NASCTN led the effort to characterize the 

spectrum in the 3.5 GHz band and published the results of these tests widely. Overall, NASCTN has had a 

significant impact on the community thanks to the wide dissemination of its test results. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

36 

FIGURE 4.1 The anechoic chamber of the National Broadband Interoperability Testbed located on the 

Boulder, Colorado, campus of NIST. SOURCE: NIST, available at https://www.nist.gov/sites/default/ 

files/styles/480_x_480_limit/public/images/2017/02/14/nbitanechoicchamber.jpg?itok=tTjw56mZ. 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

37 

5 

Look-Back at Previous Review 

This chapter assesses the extent to which CTL followed the recommendations made in the 

National Academies’ 2015 report, Telecommunications Research and Engineering at the 

Communications Technology Laboratory of the Department of Commerce: Meeting the Nation’s 

Telecommunications Needs. 

Recommendation: CTL should maintain a position of leadership in the 5G mmWave [millimeter 

wave] Channel Model Alliance, seek to expand the membership of the alliance, and engage in 

mm-wave work with other standard and industry bodies. 

CTL has maintained and even enhanced its leadership in the 5G mmWave Channel Model 

Alliance. Launched in July 2015, the NIST 5G mmWave Channel Model Alliance helps accelerate the 

development and use of measurements and models for next-generation communications technologies. By 

bringing together researchers from multiple stakeholder groups, including communications technology 

companies, wireless service providers, academia, and government, CTL has developed consensus 

technical findings that benefit standards and best practices for 5G deployment. Currently, the Alliance 

includes participants from over 75 organizations, and this effort is a great success within CTL. 

Recommendation: CTL should develop a more defined research agenda that outlines in detail its 

research goals and future plans. 

CTL conducts an annual research planning process focused on its three program areas of public 

safety, trusted spectrum testing, and metrology for advanced communications. Group plans filter up to 

divisions for coordination and approval, while the division and CTL leadership set strategic direction and 

long-term planning based on NIST priority areas. CTL’s process consists of three major steps: (1) process 

review, (2) program review, and (3) resource allocation. CTL focuses its resources on meeting its 

priorities by asking, “What do you need to accomplish task x?,” rather than simply status quo funding of 

projects. 

This approach to research planning appears sound, but CTL did not sufficiently explain how these 

plans might align with the needs of industry or how well the Divisions were able to meet these plans. 

Therefore, while CTL has defined a general research agenda setting process, the goals and future plans 

might be more readily shared with future National Academies panels. 

Recommendation: CTL should quickly hire and train personnel to establish a leading-edge skill 

set in areas associated with their research goals and upgrade aging facilities and instrumentation. 

CTL has done very well in terms of hiring junior personnel across its various research areas. CTL 

has also been successful in upgrading its buildings, facilitates, and laboratory instrumentation. These new 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

38 

facilities include the NIST Large Antenna Positioning System, the NIST Configurable Robotic 

Millimeter-Wave Antenna Facility, the NIST Broadband Interoperability Testbed, as well as renovating 

an entire building to house the PSCR Division. Beyond the substantial federal hiring, CTL has also 

engaged in innovative use of contractors and detailees on loan from other federal agencies to fill hiring 

needs. The CTL has been using the NIST Professional Research Experience Program to attract staff. 

Recommendation: CTL should further develop opportunities to quickly and frequently engage 

outside stakeholders and obtain frequent outside technical reviews as it moves its research plan 

forward. 

Parts of CTL have been very successful in engaging outside stakeholders (e.g., NASTCN, PSCR, 

and NIST 5G mmWave Channel Model Alliance), but CTL might benefit from looking at how other 

groups can better engage outside stakeholders for technical review and input. Of course, there is the 

Visiting Committee on Advanced Technology (a FACA committee) that provides input, but this is limited 

in scope and not directly mapping to each of the research domains/divisions within CTL. There is 

considerable activity by the CTL staff in presenting research at conferences and in journals, but this may 

not directly engage the stakeholders most important to the industry-focused mission of CTL. CTL might 

consider establishing a broader structure for stakeholder engagement. 

Recommendation: ITS and CTL leadership should work to build an environment of trust and 

collaboration across both laboratories. 

There does not appear to be the amount of collaboration that had been envisioned in the 2015 

National Academies report. While CTL does support ITS staff in a limited manner, the hope was that the 

Center for Advanced Communications (CAC) would serve as a vehicle for uniting CTL and ITS on 

common mission research planning and development. With the shutdown of the CAC, there appears to be 

a new opportunity to consider how best to enable engagement between these valuable research groups 

within the DOC. 

Recommendation: The Public Safety Communications Research Program should be considered 

as a template for collaboration across the laboratories. 

The PSCR continues to thrive. The nature of its outreach mission and the significant funding 

allows PSCR to engage both within DOC and with other parts of the government (including state and 

local government). PSCR continues to be an example of a successful collaboration template. 

Recommendation: The National Advanced Spectrum and Communications Test Network should 

be made fully functional as soon as possible to be able to handle the important mission that it has 

been assigned. This includes the recruitment of customers and additional government, academic, 

and industrial organizations to utilize the skills in the various affiliated laboratories. 

NASCTN has successfully ramped up operations and has completed several projects in the last 

few years. Its membership now includes the National Oceanic and Atmospheric Administration, the 

NASA, NSF, the NTIA and DoD. NASCTN has brought together federal agencies and commercial 

operators to develop common measurement framework methods and has enabled technical input for 

sharing of valuable spectrum resources. NASCTN is encouraged to grow its membership and to engage in 

projects where multiple sides of a given debate might fund the research output. NASCTN will need to 

remain as a neutral body from the perspective of its stakeholders for conducting these important technical 

sharing efforts. 

Recommendation: The Department of Commerce (DOC) should develop short- and long-term 

application and basic research plans that would provide the country with the necessary knowledge 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

39 

base in spectrum areas and enhance the capability for spectrum sharing and repurposing analysis. 

The DOC plans should include opportunities for various users of spectrum to identify their needs 

and long-term objectives. A research agenda should consider the most efficient use of DOC’s—

and the relevant laboratories’—resources and develop an effective organizational structure and 

funding strategies to ensure that research goals are met and resources are effectively used. 

While CTL did provide evidence of long-term research planning, they did not provide input 

specific to this recommendation as it relates to spectrum sharing and repurposing analysis. CTL would 

benefit from further implementing this recommendation. 

Recommendation: The Boulder telecommunications laboratories should expand their visible 

leadership roles by providing technical expertise for agencies and policy makers and providing 

objective scientific expertise. 

The CTL has taken an active role in engaging with agencies on a variety of research 

engagements, leading advisory roles across federal agencies, standards development organizations, as 

well as industry consortium. These organizations include the NSF Millimeter-Wave Research 

Coordination Network Steering Committee, the NIST 5G mmWave Channel Model Alliance, the 

Telecom Infra Project, and other efforts. However, there remains a bigger opportunity for CTL to provide 

more input to policymakers outside of the DOC on a wide set of wireless and public safety issues. 

Recommendation: The Boulder telecommunications laboratories should fully engage in the 

current and emerging work in Institute of Electrical and Electronics Engineers (IEEE) 802 

LAN/MAN Standards Committee, the 3rd Generation Partnership Project, and the Internet 

Engineering Task Force. This must be a long-term commitment, because the time constant for 

standards evolution is on the order of 3 to 10 years. 

CTL has engaged with a number of standards efforts, including IEEE 802, 3GPP, Telecom Infra 

Project, and the Internet Engineering Task Force. PSCR won a NIST Bronze Medal for their contributions 

to 3GPP standards for mission critical voice for public safety. PSCR has worked to create dozens of 

standards that were ultimately accepted and met all public safety mission critical voice requirements. 

Nonetheless, there is still room for growth of CTL in terms of broader engagement in standards efforts, 

particularly beyond the impressive work related to public safety.
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

40 

6 

List of Key Recommendations 

PUBLIC SAFETY COMMUNICATION 

Key Recommendation: The PSCR Division should develop a research and development 

roadmap for mission-critical voice, considering how the various activities it includes therein can 

be used as integral elements. CTL should consult on the roadmap’s development with other 

organizations, both government and commercial, to determine overlap of technology 

development. CTL should conduct its own critical technology assessment to inform its roadmap. 

(Chapter 3) 

Key Recommendation: CTL should develop a roadmap for public safety analytics, taking into 

consideration projected future areas of interest. (Chapter 3) 

Key Recommendation: The PSCR Division should consider integration of the UI/UX (user 

interface/user experience) research and prize challenges with the other PSCR portfolios as 

appropriate provided efficiencies can be gained by such integration. The division should consider 

developing a methodology and process for studying UI/UX along with study of new technologies. 

(Chapter 3) 

Key Recommendation: CTL should evaluate the possibility of a strategic expansion of PSCR’s 

internal research staff aimed at ensuring continuity of research in key priority areas in particular 

after fiscal year 2022 when the spectrum auction funds will have been spent or no longer be 

available. In addition, CTL should develop a plan for leveraging the expertise developed through 

the prize challenges. (Chapter 3) 

METROLOGY OF ADVANCED COMMUNICATION 

Key Recommendation: CTL should develop a 3-year or 5-year strategic plan for its activities in 

metrology for advanced communication, to include: identifying and evaluating new research 

directions and opportunities for growth; developing strategic partnerships with other NIST 

laboratories for pursuing new areas; identifying resource needs (equipment, facilities, staff) for 

pursuing strategic growth areas of research; identifying and pursuing internal and external sources 

of funding to support the plan; and developing measurable criteria and metrics for annually 

evaluating progress toward 3-year and 5-year goals. The strategic plan should explain how its 

execution will support the successful attainment of the CTL priorities. (Chapter 4) 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

41 

Key Recommendation: CTL should undertake planning and resource allocation for renewal and 

renovation of its measurement facilities with a degree of urgency to support their functioning as a 

“paid service” to the research and business communities. 

Key Recommendation: The Radio Frequency Technology Division should broaden its research 

portfolio into the areas of optical communications technology and quantum information science 

and engineering—both of which it has identified already—while leveraging strategic 

collaborative partnerships with other NIST laboratories, including the Physical Measurement 

Laboratory and the Information Technology Laboratory. (Chapter 4) 

Key Recommendation: To develop closer partnerships with industry, academia and 

governments and to explore new sources of revenue the Wireless Networks Division and Radio 

Frequency Technology Division should integrate and build on the recent accomplishments in 5G 

millimeter-wave channel modeling, millimeter-wave propagation channel sounding and 

measurements, on-wafer measurements, over-the-air measurements, and a new design framework 

for future vector network analyzers. (Chapter 4)

Key Recommendation: Consistent with its future staffing levels, NASCTN should take a more 

proactive role in advising on future spectrum allocation decisions. NASCTN should engage 

impartially with all sides of the debates on emerging and urgent issues. (Chapter 4) 

Key Recommendation: CTL should continue vigorous support for the spectrum sensing and 

sharing activity, which has delivered impactful results. (Chapter 4)
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

43 

Acronyms 

3GPP 3rd Generation Partnership Project 

5G Fifth-generation Wireless/Cellular Network 

 

AR augmented reality 

AWS advanced wireless services 

 

CAC Center for Advanced Communications 

CBRS Citizens Broadband Radio Service 

CRADA cooperative research and development agreement 

CTL Communications Technology Laboratory 

 

DOC Department of Commerce 

DoD Department of Defense 

DPA dynamic protection area 

 

EMS emergency medical service 

 

FY fiscal year 

FirstNet First Responder Network Authority 

 

IEEE Institute of Electrical and Electronics Engineers 

ISO International Standards Organization 

ISSI Inter-RF Subsystem Interface 

ITL Information Technology Laboratory 

ITS Institute for Telecommunication Sciences 

 

LAN local area network 

LBS Location-Based Services 

LMR Land Mobile Radio 

LTE long-term evolution 

 

MAN metropolitan area network 

MCV Mission-Critical Voice 

MIMO multiple input, multiple output 
An Assessment of the Communications Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2019

Copyright National Academy of Sciences. All rights reserved.

44 

 

NASCTN National Advanced Spectrum and Communications Test Network 

NIST National Institute of Standards and Technology 

NTIA National Telecommunications and Information Administration 

 

PML Physical Measurement Laboratory 

PSCR Public Safety Communications Research Program 

 

QAM quadrature amplitude modulation 

QISE quantum information science and engineering 

 

R&D research and development 

RF radio frequency 

RFT Radio Frequency Technology Division 

 

SAS Spectrum Access System 

 

UAV unmanned aerial vehicle 

UE user equipment 

UI user interface 

UX user experience 

U-NII unlicensed national information infrastructure 

 

VR virtual reality 

 

WN Wireless Networks Division