High-speed random number generator co-processors for machine learning and AI acceleration

Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-12 07:06:26

Abstract: Random number generation is a hidden draw on runtime, power, and compute resources in machine learning (ML) and artificial intelligence (AI) applications. Random numbers are required, for instance, to initialize network weights, shuffle and partition datasets during training, and generate synthetic data. In each of these cases, the volume of random numbers required scales with the size of the model and dataset.
We argue that there is an opportunity to accelerate ML applications at system level by interfacing existing general-purpose and specialized hardware with a high-speed random number generator (RNG) co-processor. Such a co-processor would support ML workloads by independently generating random numbers from a high-speed physical entropy source and feeding them into the application, freeing up CPU or GPU resources that would otherwise be required to run software-based pseudo-random number generators.
Reducing RNG overhead will become even more critical to application performance as the industry increasingly adopts probabilistic approaches which rely heavily on random sampling. Use cases include differential privacy, uncertainty tracking, and Bayesian methods for ML, as well as synthetic data for generative AI training.

Main paper: 
 introduction: 
 Recent years have seen massive investment in accelerator hardware for machine learning (ML) and artificial intelligence applications (AI), with NVIDIA GPUs dominating the market (1) . Hardware acceleration approaches for ML have focused on matrix operations, whose inherent parallelizability and relatively simple arithmetic make them well-adapted to running on GPUs (2) .As runtimes for training and inference continue improving thanks to advances in GPUs and AIspecific ASICs, peripheral operations of ML applications such as data transfer, communication overhead, and overall availability of GPU/CPU resources will become the more significant bottlenecks. This led us to pose the question: aside from matrix multiplication, are there any other computations in ML/AI workloads which could be accelerated at hardware level? In order to produce meaningful acceleration at application level, the computation of interest would have to occur very frequently in the training/inference process; have enough speedup potential to justify implementing on a specialized processor rather than on CPU; and also be sufficiently general so that the same hardware can support a variety of applications, and will remain relevant even as methods and models rapidly change. The task of random number generation fits these preliminary requirements.In this position paper we explore how building a hardware-accelerated high-speed random number generator (RNG) could impact the speed and resource efficiency of ML applications, while supporting emerging trends towards more probabilistic approaches in ML and AI. We believe such an RNG is attainable in present-day, CMOS-integrable technology, and could have a relatively near-term impact on the field.In section 2, we review how random numbers are used in ML, where they come from, and highlight emerging probabilistic methods in ML that would especially benefit from a hardware RNG accelerator. In section 3 we survey hardware approaches to RNG and state basic design requirements for a high-speed RNG co-processor. 
random number generation in ml: 
 Random number generation is surprisingly prevalent in ML applications. Random numbers must be generated any time network weights are initialized, data is shuffled, or some subset of either data or model weights are selected. It thus underpins such fundamental methods as regularization/dropout (3), mini-batch training (4), and stochastic optimization methods like stochastic gradient descent (SGD) (5) , RMSprop , and Adam (6) .ML applications today almost exclusively use pseudo-random number generators (PRNGs): algorithms typically implemented in software which produce a deterministic stream of random numbers.These are contrasted with hardware-based true random number generators (TRNGs), which produce random numbers by measuring a physical "entropy source," typically analog electronic noise in a circuit.Like matrix operations, PRNG algorithms are inherently parallelizable and excellent performance can be achieved by distributing different streams of random numbers across many CPU/GPU cores.Older works showed acceleration of PRNG throughput from 1-10 GBps on high-performance CPUs up to 200 GBps on NVIDIA GPUs (7; 8) . A number of PRNGs are implemented in cuRAND, a library within NVIDIA's CUDA Toolkit enabling GPU-accelerated RNG, and NVIDIA now reports throughputs approaching 1 TBps for uniform RNGs on its A100 chips (9) . We suggest Ref. (7) for an introduction to both theory and implementation of parallel PRNGs, as well as an excellent discussion of tradeoffs between speed, memory, and cryptographic strength for different kinds of PRNGs.The key disadvantages of software-based PRNGs are memory usage and high GPU/CPU occupancy when generating multiple streams, which is required for good performance due to the computational complexity and sequential nature of PRNG algorithms. This memory and power investment may be reasonable for the current uses of random numbers in ML, but there is a growing trend towards probabilistic methods which require much higher rates of random sampling. These include differential privacy (DP) methods for ML training (10; 11) , synthetic data (11; 12; 13) , and Bayesian neural networks or other models designed to model and quantify uncertainty (14) . For each of these methods the volume of random sampling required is proportional to the size of the training dataset and/or number of model parameters. Given the size of today's models random number generation overhead could become a significant bottleneck for these applications, limiting their adoption. It is widely accepted that private training methods increase runtime by at least 1.5x, largely due to additional overhead of generating and adding noise to SGD gradient vectors (15) . Developing these applications supports the safe and responsible deployment of ML/AI, but we will likely need fast, more resource-efficient RNGs to enable them. Even if parallel PRNG algorithms are improved, they will still compete directly with training and inference tasks for relatively scarce GPU resources. We thus believe that integrating specialized RNG hardware into AI workloads will be an essential ingredient in continued scaling. A separate RNG co-processor would allow applications to offload random number generation from CPU/GPU, liberating them for the computations on which they perform best. tronic noise, jitter in ring oscillator circuits (16; 17) , or a quantum process such as single photon emission and detection (18) .TRNGs are already commercialized for application in cryptography, where they most often serve as random seed generators for faster cryptographically-secure PRNGs (CSPRNGs) (19) . This application requires only moderate speeds, with commercial products operating at ∼ 200 Mbps1 (20) , and random bit or integer outputs.RNGs can also be implemented in FPGAs, and these RNGs tend to have higher performance (1-10 Gbps). However the difficulty of programming FPGAs tends to deter developers from adopting these devices.These existing devices are unlikely meet the performance and integration demands of ML applications. New commercially-viable RNG co-processors will need to be designed, which integrate a significantly higher-speed entropy source with additional logic units to convert bitstream outputs to continuous distributions useful for ML: uniform, normal, and Laplacian in particular. In order to facilitate integration with this logic, the TRNG should be implemented in CMOS-compatible hardware with existing high-yield manufacturing processes. This constrains the search space for an entropy source considerably.We believe the most promising hardware platform for the TRNG core are emerging nonvolatile memory (NVM) technologies (21) . These devices are low power, often CMOS-compatible, and randomness can be harvested from the stochastic behaviour observed when switching the memory state.Because of these properties, NVMs have been studied extensively for application in neuromorphic computing and stochastic computing (22; 23) . With respect to designing a high-speed TRNG, the key advantage of NVMs is that individual memory cells can be assembled into large arrays with multiple cells accessible in parallel. The array size and degree of parallelism can be optimized during design in order to produce a device with higher random bit throughput, even if access time is fixed.To facilitate integration with other parts of the ML workflow one could deliver the RNG co-processor as an add-on card with PCI Express (PCIe) interface, the same standard interface used by GPUs, or integrate it as a chiplet in a System-on-Chip design. The latter approach is in keeping with industry trends towards chiplet architectures for more flexible adaptation to different and changing AI workloads (24) .Based on performance benchmarks for PRNG and our survey of existing TRNG technologies, we propose the following design guidelines for an RNG co-processor:1. Random bit generation at > 100 Gbps 2. Energy efficiency on the order 1-10 picojoules-per-bit 3. Modular component or chiplet connecting to system through standard interfaces (PCIe, UCIe, etc.) 4. Compliant with NIST guidelines for random number generation SP 800-90A/B/C (25; 26; 27) , and able to pass standard statistical tests for binary sequences such as NIST 800-22 (28) and TestU01 (29) . This aspirational target of 100 Gbps is set with the goal of approaching data rates achieved in highperformance PRNGs, and matching data transfer rates of standard interfaces like PCIe Gen 6.0, which sits at 121 GBps in the maximum 16-lane configuration. The planned PCIe Gen 7.0 is expected to double the maximum data transfer rate to 242 GBps by 2025. 
discussion: 
 Despite the prevalence of random numbers in ML, optimization of RNGs for ML and AI applications is a relatively under-explored area. Parallelized PRNGs implemented on GPUs can produce random samples from various probability distributions at very high rates (up to 1 TBps), but come with a high memory and compute cost, as well as certain constraints on application design.As noted in Section 2.7 of Ref. (30) , generating random numbers on GPUs is most efficient when producing large blocks of random numbers (kB-MB of data) in fewer calls, due to considerable setup time when initializing the PRNG state. Ensuring performance requires more optimization on the developer's part, and more memory to store the large stream of random numbers. Thus for applications that generate smaller blocks of random numbers on an as-needed basis and immediately feed them into the application, an efficient hardware TRNG with low setup time would have an advantage over PRNGs on GPUs. This could be the case when random sampling is performed to modify or generate relatively few training examples, for instance when implementing on-device privacy (31) or other probabilistic ML methods discussed in Section 2.Such application-specific advantages, improved power consumption, and the need to preserve GPU resources for intensive training and inference computations on large models, make the development of a high-speed RNG co-processor an attractive prospect. Addressing the RNG bottleneck will become even more critical as ML applications requiring high-volume random sampling become more mainstream-including private training, synthetic data generation, and Bayesian neural networks.Scaling these probabilistic methods will play an important role in the continued growth of differential privacy and generative AI.The path to an RNG co-processor for ML Specialized hardware TRNGs can generate random numbers with superior memory and power efficiency, as the large RNG "state space" is initialized not in memory, but in an actual physical process that the TRNG circuit implementation merely samples. The entropy source may be thermal or elec-Note the difference in unit convention for performance metrics between TRNG (bits per second) and PRNG (bytes per second). For consistency with cited sources we use the conventions from literature rather than converting. 
 
 Appendix:  



Reviews: review: title: title: This position paper proposes using TRNG co-processors to improve ML/AI accelerator performance; however, there is relatively little discussion of the current impact of PRNGs on ML/AI performance.
main_review: main_review: This position paper proposes using TRNG co-processors to improve ML/AI accelerator performance.  It is a compelling idea that could improve future AI/ML systems, particularly those that will require significant random sampling.  This position paper could help influence the design of future AI/ML accelerator decisions.

Pros:

-- Compelling idea, with good initial motivation

-- Quantifiable recommendations provided, helping to guide future work

Cons:

-- I would like to have seen more quantification of the current impact of PRNGs on AI/ML performance to help motivate this position.

-- I would like to have seen more description on how and why the guidelines were chosen.