Query: In-memory hyperdimensional computing

Candidate papers:

abstract: Hyperdimensional computing is an emerging computational framework that takes inspiration from attributes of neuronal circuits including hyperdimensionality, fully distributed holographic representation and (pseudo)randomness. When employed for machine learning tasks, such as learning and classification, the framework involves manipulation and comparison of large patterns within memory. A key attribute of hyperdimensional computing is its robustness to the imperfections associated with the computational substrates on which it is implemented. It is therefore particularly amenable to emerging non von Neumann approaches such as in memory computing, where the physical attributes of nanoscale memristive devices are exploited to perform computation. Here, we report a complete in memory hyperdimensional computing system in which all operations are implemented on two memristive crossbar engines together with peripheral digital complementary metal oxide semiconductor (CMOS) circuits. Our approach can achieve a near optimum trade off between design complexity and classification accuracy based on three prototypical hyperdimensional computing related learning tasks: language classification, news classification and hand gesture recognition from electromyography signals. Experiments using 760,000 phase change memory devices performing analog in memory computing achieve comparable accuracies to software implementations. A complete in memory hyperdimensional computing system, which uses 760,000 phase change memory devices, can efficiently perform machine learning related tasks including language classification, news classification and hand gesture recognition from electromyography signals.
author_names: Geethan Karunaratne

Manuel Le Gallo

Giovanni Cherubini

Luca Benini

Abbas Rahimi

Abu Sebastian
title: In memory hyperdimensional computing
venue: ArXiv
year: 2019

abstract: "Our work was initiated by the natural fit between the two concepts of in memory computing and hyperdimensional computing," Abu Sebastian and Abbas Rahimi, the two lead researchers behind the study, told TechXplore. "At IBM ResearchZurich, we have been developing in memory computing platforms based on phase change memory (PCM) while at ETH Zurich, we have been exploring a braininspired computing paradigm called hyperdimensional computing."
author_names: Ingrid Fadelli
title: A hyperdimensional computing system that performs all core computations in memory
year: 2020

abstract: Abstract One viable solution for continuous reduction in energy per operation is to rethink functionality to cope with uncertainty by adopting computational approaches that are inherently robust to uncertainty. It requires a novel look at data representations, associated operations, and circuits, and at materials and substrates that enable them. 3D integrated nanotechnologies combined with novel brain inspired computational paradigms that support fast learning and fault tolerance could lead the way. Recognizing the very size of the brain's circuits, hyperdimensional (HD) computing can model neural activity patterns with points in a HD space, that is, with hypervectors as large randomly generated patterns. At its very core, HD computing is about manipulating and comparing these patterns inside memory. Emerging nanotechnologies such as carbon nanotube field effect transistors (CNFETs) and resistive RAM (RRAM) and their monolithic 3D integration offer opportunities for hardware implementations of HD computing through tight integration of logic and memory, energy efficient computation, and unique device characteristics. We experimentally demonstrate and characterize an end to end HD computing nanosystem built using monolithic 3D integration of CNFETs and RRAM. With our nanosystem, we experimentally demonstrate classification of 21 languages with measured accuracy of up to 98% on >20,000 sentences (6.4 million characters) training using one text sample (100,000 characters) per language, and resilient operation (98% accuracy) despite 78% hardware errors in HD representation (outputs stuck at 0 or 1) By exploiting the unique properties of the underlying nanotechnologies, we show that HD computing, when implemented with monolithic 3D integration, can be up to 420x more energy efficient while using 25x less area compared to traditional silicon complementary metal oxide semiconductor (CMOS) implementations.
author_names: Abbas Rahimi

Tony F Wu

Haitong Li

Jan M Rabaey

H -S Philip Wong

Max M Shulaker

Subhasish Mitra
title: Hyperdimensional computing nanosystem: in memory computing using monolithic 3D integration of RRAM and CNFET
year: 2020

abstract: Brain inspired Hyperdimensional Computing (HDC) is a fast and robust classification algorithm, which works by mapping low dimensional features to high dimensional vectors and comparing distance in a high dimensional space. However, in traditional Von Neumann architecture, HDC causes high energy consumption because of large data movements between processor and memory. In this paper, we propose HDC IM, a Hyperdimensional Computing In Memory architecture based on Resistive Random Access Memory (RRAM) to boost the energy efficiency of HDC. HDC IM puts computations in or near memory, which eliminates most of the data movements, providing a solution to reduce the energy consumption. In addition, to improve the computing parallelism, we use in crossbar RRAM based logic design to process encoding operation in HDC. The experimental results show that HDC IM provides more than 100x speedup and higher energy efficiency compared with HDC on CPU. Moreover, in comparison with existing RRAM based Neural Network accelerators, HDC IM is more fault tolerant taking into account RRAM device faults, achieving 20% higher accuracy than RRAM based DNN on ISOLET dataset when 20% RRAM devices suffer from Stuck At Faults (SAFs)
author_names: Jialong Liu

Mingyuan Ma

Zhenhua Zhu

Hua-zhong Yang
title: HDC IM: Hyperdimensional Computing In Memory Architecture based on RRAM
venue: 2019 26th IEEE International Conference on Electronics, Circuits and Systems (ICECS)
year: 2019

abstract: The ability to learn from few examples, known as one shot learning, is a hallmark of human cognition. Hyperdimensional (HD) computing is a brain inspired computational framework capable of one shot learning, using random binary vectors with high dimensionality. Device architecture co design of HD cognitive computing systems using 3D VRRAM/CMOS is presented for language recognition. Multiplication addition permutation (MAP) the central operations of HD computing, are experimentally demonstrated on 4 layer 3D VRRAM/FinFET as non volatile in memory MAP kernels. Extensive cycle to cycle (up to 1012 cycles) and wafer level device to device (256 RRAMs) experiments are performed to validate reproducibility and robustness. For 28 nm node, the 3D in memory architecture reduces total energy consumption by 52.2% with 412 times less area compared with LP digital design (using registers as memory) owing to the energy efficient VRRAM MAP kernels and dense connectivity. Meanwhile, the system trained with 21 samples texts achieves 90.4% accuracy recognizing 21 European languages on 21,000 test sentences. Hard error analysis shows the HD architecture is amazingly resilient to RRAM endurance failures, making the use of various types of RRAMs/CBRAMs (1k 10M endurance) feasible.
author_names: Haitong Li

Tony F Wu

Abbas Rahimi

Kai-Shin Li

Miles Rusch

Chang-Hsien Lin

Juo-Luen Hsu

Mohamed M Sabry

Sukru Burc Eryilmaz

Joon Sohn

Wen-Cheng Chiu

Min-Cheng Chen

Tsung-Ta Wu

Jia-Min Shieh

W K Yeh

Jan M Rabaey

Subhasish Mitra

H -S Philip Wong
title: Hyperdimensional computing with 3D VRRAM in memory kernels: Device architecture co design for energy efficient, error resilient language recognition
venue: 2016 IEEE International Electron Devices Meeting (IEDM)
year: 2016

abstract: Brain inspired hyperdimensional (HD) computing emulates cognitive tasks by computing with long binary vectors also know as hypervectors as opposed to computing with numbers. However, we observed that in order to provide acceptable classification accuracy on practical applications, HD algorithms need to be trained and tested on nonbinary hypervectors. In this article, we propose SearcHD, a fully binarized HD computing algorithm with a fully binary training. SearcHD maps every data points to a high dimensional space with binary elements. Instead of training an HD model with nonbinary elements, SearcHD implements a full binary training method which generates multiple binary hypervectors for each class. We also use the analog characteristic of nonvolatile memories (NVMs) to perform all encoding, training, and inference computations in memory. We evaluate the efficiency and accuracy of SearcHD on a wide range of classification applications. Our evaluation shows that SearcHD can provide on average <inline formula> <tex math notation="LaTeX"$31.1\times /tex math>/inline formula> higher energy efficiency and <inline formula> <tex math notation="LaTeX"$12.8\times /tex math>/inline formula> faster training as compared to the state of the art HD computing algorithms.
author_names: Mohsen Imani

Xunzhao Yin

John G Messerly

Saransh Gupta

Michael Thaddeus Niemier

Xiaobo Sharon Hu

Tajana Simunic Rosing
title: SearcHD: A Memory Centric Hyperdimensional Computing With Stochastic Training
venue: IEEE Transactions on Computer Aided Design of Integrated Circuits and Systems
year: 2020

abstract: This paper proposes evolvable hyperdimensional (HD) computing to maintain high classification accuracy as permanent faults occur in emerging non volatile memory fabrics. Our proposed HD architecture can detect, localize, and isolate faulty PCM blocks in discriminative classifiers, followed by unsupervised regeneration of new blocks to compensate accuracy loss. We demonstrate its application on a language recognition task: it is able to quickly relearn and fully recover the accuracy from 90.48% to 96.86% at fault rates as high as 42% by using solely 4. 2MB of text for regeneration. The new evolved model is still 285 more compact than state of the art fastText.
author_names: Michael Hersche

Sara Sangalli

Luca Benini

Abbas Rahimi
title: Evolvable Hyperdimensional Computing: Unsupervised Regeneration of Associative Memory to Recover Faulty Components
venue: 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS)
year: 2020

abstract: Brain inspired hyperdimensional (HD) computing models neural activity patterns of the very size of the brain's circuits with points of a hyperdimensional space, that is, with hypervectors. Hypervectors are D dimensional (pseudo)random vectors with independent and identically distributed (i.i.d. components constituting ultra wide holographic words: D=10,000 bits, for instance. At its very core, HD computing manipulates a set of seed hypervectors to build composite hypervectors representing objects of interest. It demands memory optimizations with simple operations for an efficient hardware realization. In this article, we propose hardware techniques for optimizations of HD computing, in a synthesizable open source VHDL library, to enable co located implementation of both learning and classification tasks on only a small portion of Xilinx UltraScale FPGAs: (1) We propose simple logical operations to rematerialize the hypervectors on the fly rather than loading them from memory. These operations massively reduce the memory footprint by directly computing the composite hypervectors whose individual seed hypervectors do not need to be stored in memory. (2) Bundling a series of hypervectors over time requires a multibit counter per every hypervector component. We instead propose a binarized back to back bundling without requiring any counters. This truly enables on chip learning with minimal resources as every hypervector component remains binary over the course of training to avoid otherwise multibit components. (3) For every classification event, an associative memory is in charge of finding the closest match between a set of learned hypervectors and a query hypervector by using a distance metric. This operator is proportional to hypervector dimension (D) and hence may take O(D) cycles per classification event. Accordingly, we significantly improve the throughput of classification by proposing associative memories that steadily reduce the latency of classification to the extreme of a single cycle. (4) We perform a design space exploration incorporating the proposed techniques on FPGAs for a wearable biosignal processing application as a case study. Our techniques achieve up to 2.39x area saving, or 2,337x throughput improvement. The Pareto optimal HD architecture is mapped on only 18,340 configurable logic blocks (CLBs) to learn and classify five hand gestures using four electromyography sensors.
author_names: Manuel Schmuck

Luca Benini

Abbas Rahimi
title: Hardware Optimizations of Dense Binary Hyperdimensional Computing: Rematerialization of Hypervectors, Binarized Bundling, and Combinational Associative Memory
venue: ACM J. Emerg. Technol. Comput. Syst.
year: 2019

abstract: Brain inspired hyperdimensional (HD) computing models neural activity patterns of the very size of the brain's circuits with points of a hyperdimensional space, that is, with hypervectors. Hypervectors are $D$ dimensional (pseudo)random vectors with independent and identically distributed (i.i.d. components constituting ultra wide holographic words: $D 10,000$ bits, for instance. At its very core, HD computing manipulates a set of seed hypervectors to build composite hypervectors representing objects of interest. It demands memory optimizations with simple operations for an e cient hardware realization. In this paper, we propose hardware techniques for optimizations of HD computing, in a synthesizable VHDL library, to enable co located implementation of both learning and classification tasks on only a small portion of Xilinx(R) UltraScale(TM) FPGAs: (1) We propose simple logical operations to rematerialize the hypervectors on the fly rather than loading them from memory. These operations massively reduce the memory footprint by directly computing the composite hypervectors whose individual seed hypervectors do not need to be stored in memory. (2) Bundling a series of hypervectors over time requires a multibit counter per every hypervector component. We instead propose a binarized back to back bundling without requiring any counters. This truly enables on chip learning with minimal resources as every hypervector component remains binary over the course of training to avoid otherwise multibit component. (3) For every classification event, an associative memory is in charge of finding the closest match between a set of learned hypervectors and a query hypervector by using a distance metric. This operator is proportional to
author_names: Manuel Schmuck

Luca Benini

Abbas Rahimi
title: Hardware Optimizations of Dense Binary Hyperdimensional Computing: Rematerialization of Hypervectors, Binarized Bundling, and Combinational Associative Memory
venue: ArXiv
year: 2018

abstract: DNA pattern matching is widely applied in many bioinformatics applications. The increasing volume of the DNA data exacerbates the runtime and power consumption to discover DNA patterns. In this paper, we propose a hardware software co design, called GenieHD, which efficiently parallelizes the DNA pattern matching task. We exploit brain inspired hyperdimensional (HD) computing which mimics pattern based computations in human memory. We transform inherent sequential processes of the DNA pattern matching to highly parallelizable computation tasks using HD computing. The proposed technique first encodes the whole genome sequence and target DNA pattern to high dimensional vectors. Once encoded, a light weight operation on the high dimensional vectors can identify if the target pattern exists in the whole sequence. We also design an accelerator architecture which effectively parallelizes the HD based DNA pattern matching while significantly reducing the number of memory accesses. The architecture can be implemented on various parallel computing platforms to meet target system requirements, e.g. FPGA for low power devices and ASIC for high performance systems. We evaluate GenieHD on practical large size DNA datasets such as human and Escherichia Coli genomes. Our evaluation shows that GenieHD significantly accelerates the DNA matching procedure, e.g. 44.4x speedup and 54.1x higher energy efficiency as compared to a state of the art FPGA based design.
author_names: Yeseong Kim

Mohsen Imani

Niema Moshiri

Tajana Simunic
title: GenieHD: Efficient DNA Pattern Matching Accelerator Using Hyperdimensional Computing
venue: 2020 Design, Automation Test in Europe Conference Exhibition (DATE)
year: 2020