Skip to main content    

An official website of the United States government

Here’s how you know

**Official websites use .gov**              A **.gov** website belongs to an official government organization in the United States.            

**Secure .gov websites use HTTPS**              A **lock** (  LockA locked padlock) or **https://** means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.            

https://www.nist.gov/itl/math/acmd-seminar-efficient-neural-network-training-streaming-hyperspherical-rotation-circuits

            -  Topics  -  All Topics 
-  Advanced communications 
-  Artificial intelligence 
-  Bioscience 
-  Buildings and construction 
-  Chemistry 
-  Climate 
-  Cybersecurity 
-  Electronics 
-  Energy 
-  Environment 
-  Fire 
-  Forensic science 
-  Health 
-  Information technology 
-  Infrastructure 
-  Manufacturing 
-  Materials 
-  Mathematics and statistics 
-  Metrology 
-  Nanotechnology 
-  Neutron research 
-  Performance excellence 
-  Physics 
-  Public safety 
-  Resilience 
-  Standards 
-  Transportation 
 
            -  Publications 
            -  Labs & Major Programs  -  Laboratories    -  Communications Technology Laboratory 
  -  Engineering Laboratory 
  -  Information Technology Laboratory 
  -  Material Measurement Laboratory 
  -  Physical Measurement Laboratory 
-  User Facilities    -  NIST Center for Neutron Research 
  -  CNST NanoFab 
-  Research Test Beds 
-  Research Projects 
-  Tools & Instruments 
-  Major Programs    -  Baldrige Performance Excellence Program 
  -  CHIPS for America Initiative 
  -  Manufacturing Extension Partnership (MEP) 
  -  Office of Advanced Manufacturing 
  -  Special Programs Office 
  -  Technology Partnerships Office 
 
            -  Services & Resources  -  Standards and Measurements    -  Calibration Services 
  -  Laboratory Accreditation (NVLAP) 
  -  Quality System 
  -  Standard Reference Materials (SRMs) 
  -  Standards.gov 
  -  Time Services 
  -  Office of Weights and Measures 
-  Software 
-  Data    -  Chemistry WebBook 
  -  National Vulnerability Database 
  -  Physical Reference Data 
  -  Standard Reference Data (SRD) 
-  Storefront 
-  License & Patents 
-  Computer Security Resource Center (CSRC) 
-  NIST Research Library 
 
            -  News & Events  -  News 
-  Events 
-  Blogs 
-  Feature Stories 
-  Awards 
-  Video Gallery 
-  Image Gallery 
-  Media Contacts 
 
            -  About NIST  -  About Us    -  Leadership 
  -  Organization Structure 
  -  Budget & Planning 
-  Contact Us 
-  Visit 
-  Careers    -  Student programs 
-  Work with NIST 
-  History    -  NIST Digital Archives 
  -  NIST Museum 
  -  NIST and the Nobel 
-  Educational Resources 
 

Information Technology Laboratory

## Applied and Computational Mathematics Division

-  Projects/Programs 
-  How to Work With Us 
-  NIST Publications Searchable Database 
-  Staff Directory 
-  News & Updates 
-  Resources/Services 

# ACMD Seminar: Efficient neural network training: streaming hyperspherical rotation circuits for neural network gradient compression

## Share

Facebook

Linkedin

Twitter

Email

Matthew Daniels*NIST Physical Measurement Laboratory*

Tuesday, June 15, 2021, 3:00 EDT (1:00 MDT)

**A video of this talk is available to NIST staff in the Math channel on NISTube, which is accessible from the NIST internal home page.**

**Abstract**: In training ever larger deep neural networks, the communication of gradients of the synaptic weights between computational nodes in the data center is emerging as a bottleneck in energy and latency. Recent theoretical work has demonstrated that most of the important information in these matrices is contained in just a small part of the gradient’s eigenspectrum. Proposed methods of using low-rank approximations of the gradient suggest that this compression could save energy and time in network training but the known algorithms for doing so are essentially software based, and far slower than the hardware accelerated inference engines they are attempting to guide.

In this talk, I describe our efforts at building an efficient machine that can learn low-rank approximations of neural network gradients at roughly the same speed as the associated hardware-accelerated inference engine. Inspired at a basic level by neuron models where synapses compete for finite resources, the machine is comprised of interlocking, irregular trees of planar rotators, each associated with a unique rotation angle. These hardware-efficient rotators not only process the streaming neural network data but also manage a distributed representation of the local geometry in gradient space. We describe a simple algorithmic model of the machine, but also discuss issues introduced by the distributed non-local architecture and the problems that arise in the exploration of a non-Euclidean space. Finally, we discuss our results using models of this device to train a simple two-layer perceptron and contemplate future work, applications, and generalizations.

**Bio**: Matthew W. Daniels is a postdoctoral physicist in the Alternative Computing Group at the National Institute for Standards and Technology (NIST). He received a B.S. in Physics with a minor in Mathematics from Clemson University, and M.S. and Ph.D. degrees in Physics from Carnegie Mellon University. His dissertation work was on antiferromagnetic magnon physics. His postdoctoral work touches multiple topics in neuromorphic computing, with a focus on discovering and utilizing new information encodings that bridge the gap between traditional CMOS engineering and the energy efficient, native computing capabilities of magnetic and memristive nanotechnology.

**Note:**This talk will be recorded to provide access to NIST staff and associates who could not be present to the time of the seminar. The recording will be made available in the Math channel on NISTube, which is accessible only on the NIST internal network. This recording could be released to the public through a Freedom of Information Act (FOIA) request. Do not discuss or visually present any sensitive (CUI/PII/BII) material. Ensure that no inappropriate material or any minors are contained within the background of any recording. (To facilitate this, we request that cameras of attendees are muted except when asking questions.)

Information technologyArtificial intelligenceMathematics and statisticsPhysics

## Contacts

                -  Stephen Langer  stephen.langer@nist.gov      (301) 975-5423   

### HEADQUARTERS

301-975-2000

Webmaster | Contact Us | Our Other Offices

TwitterFacebookLinkedInInstagramYouTubeGiphyRSS FeedMailing List

Feedback

            -  Site Privacy 
            -  Accessibility 
            -  Privacy Program 
            -  Copyrights 
            -  Vulnerability Disclosure 
            -  No Fear Act Policy 
            -  FOIA 
            -  Environmental Policy 
            -  Scientific Integrity 
            -  Information Quality Standards 
            -  Commerce.gov 
            -  Science.gov 
            -  USA.gov 
            -  Vote.gov