Hyperspectral Compute-In-Memory: An Opto-Electronic Computing Architecture Enabling Compute Density Beyond PetaOPS/mm$^2$

Authors: myoung-gyun  suh
jun  park
honari  latifpour
yoshihisa  yamamoto
Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-12 09:16:23

Abstract: We present a hyperspectral compute-in-memory architecture that utilizes both frequency and spatial dimensions for single-shot matrix-matrix multiplication. This approach offers exceptional parallelism, scalability, programmability, and efficient chip area utilization, potentially enabling a compute density exceeding PetaOPS/mm$^2$. The architecture demonstrates potential for energy-efficient, three-dimensional opto-electronic computing in future data center applications.

Main paper: 
 : 
 Unlike its electronic counterparts, the Opto-Electronic Hyperspectral CIM architecture eliminates the need for physical wiring in MAC operations, enabling a 3D architectural design. By dividing the "multiply" and "accumulate" operations across two distinct chips (a synapse chip and a neuron chip), the architecture optimizes chip area utilization and is capable of achieving a compute density exceeding PetaOPS/mm 2 . the lower density limitations of space multiplexing compared to electronic systems, our architecture additionally integrates frequency multiplexing with optical frequency combs (OFCs) (16; 17) , drawing inspiration from hyperspectral imaging (18) and advanced optical fiber communications (19) .In our proof-of-concept experiments, we manipulate 2D optical input data for single-shot matrixmatrix multiplication (MMM), where each SLM pixel encodes a matrix weight across multiple wavelengths. This method allows batch processing of matrix-vector multiplication using wavelength-division multiplexing. We conducted numerous MMM tests, and the results confirmed theoretical predictions, including the multiplication of the NTT logo with the identity matrix, as shown in Figure 2d . Although hyperspectral imaging usually involves 3D data both in input and output, our computing system maintains 2D inputs and outputs, utilizing the third dimension internally. This strategy transforms the "curse of dimensionality" into a computational asset.Figure 2a illustrates the experimental setup for demonstrating the hyperspectral compute-in-memory architecture. The input source is a fiber optical frequency comb (OFC) in the C-band, featuring a 250 MHz pulse repetition rate and is coarsely filtered using line-by-line waveshaping (14) as shown in Figure 2b . The optical source, with an average power of around 1 mW, is then introduced into the system. The coarsely filtered comb lines are spatially dispersed using a grating, expanded vertically by a cylindrical lens, and then focused onto SLM 1, where the first matrix is encoded. The comb lines are then recombined and expanded horizontally by another cylindrical lens before being focused onto SLM 2 to encode the second matrix. After another vertical fanning-in by a cylindrical lens, the comb lines are sorted vertically by color via a grating to complete the hyperspectral multiplyaccumulate operation. A linear polarizer enables the phase-only SLM to modulate intensity, and system non-uniformity is calibrated by adjusting the SLM pixel phases.To demonstrate the hyperspectral operation, we conducted MMM tests with a hyperspectral factor of 5, encoding each SLM pixel with a matrix weight across five comb lines (see Figure 3a ). Minor adjustments to our system allowed for a potential increase in the hyperspectral factor to 10 or higher.We evaluated the computational accuracy by analyzing the error distribution for each possible MAC value. The matrices were encoded using non-negative weights with 4 bits. We performed 400 measurements for each MAC value, ranging from 0 to 150 (see Figure 3b ). As the target MAC values increased, the standard deviation of the error grew until reaching a saturation point. The relative error, defined as the absolute difference between the measured and target MAC values divided by the target MAC value, showed a standard deviation decreasing to below 5 percent as the target MAC value increased. These errors likely arose from intensity fluctuations in the OFC source, crosstalk between adjacent pixels, and optical alignment errors. We anticipate that the standard deviation of the relative error will stabilize at a similar level even when the system scales up in matrix size.Notably, noise up to a certain threshold may not significantly affect computational outcomes in many AI tasks, as confirmed by analyzing MNIST data classification under various noise conditions.The system currently operates in an open-loop configuration, encoding the input matrix and independently reading out MAC results using standard digital electronics. Fast external modulation and readout are vital for high-throughput computation in such setups. Conversely, in a closed-loop configuration with nonlinear operations, the system efficiently solves optimization problems without the need for rapid external modulation and readout. Most computations here are analog, with only the initial input and final output digitally managed. To enable rapid, pixel-by-pixel parallel modulation in the closed-loop system, a novel 2D opto-electronic "neuron" array is essential. This array connects each photodetector pixel directly to its corresponding modulator (or light emitter) pixel via through-silicon-via (TSV), reducing delays and energy consumption by avoiding the inefficiencies of connecting a camera to an SLM via a serial bus. Such an array would enable seamless parallel processing.In the near term, we aim to operate our MMM system in closed-loop mode (refer to Figure 4b ), primarily for its simplicity. This configuration requires just one hyperspectral MAC module, and it removes the need for parallel modulation and readout through an external electronic interface. In this setup, only one external intensity modulation at the computational clock frequency is necessary to generate the input optical pulse stream. To determine the total power consumption of this system, we calculated the power required for each pixel during N b -bit precision MAC operations, using actual parameters and factoring in the significant fixed energy costs from our current experimental setup. Given the hyperspectral factor H for multiplying matrices of sizes (H × K) and (K × K),the system executes approximately (H × K × K) MAC operations per single clock cycle, and the formula for total power consumption is as follows:EQUATIONHere, N b represents the effective bit precision, I th is the threshold current for detection in the photodetector, η P D denotes the photodetector responsivity, η L refers to the laser wall-plug efficiency,η ′ ois the efficiency of optical power utilization, and P mod , P SLM , and P T IA are the respective power consumptions for the optical modulator, the spatial light modulator (SLM), and the transimpedance amplifier.With improved alignment and wider spectral bandwidth, the closed-loop system is expected to reach 100 peta operations per second (PetaOPS), with H = 100, K = 1000, and a 1 GHz clock frequency, and an anticipated efficiency close to 2 W/PetaOPS (as shown in Figure 4b and Scenario 2 of Table I ). The 'hyperspectral factor' mitigates the need for extensive physical scaling. For instance, with a hyperspectral factor of 400 and maintaining the same clock speed, only a 500-by-500 matrix (i.e., K = 500) is required to achieve 100 PetaOPS. Further scaling in the space and frequency dimensions could push the system beyond ExaOPS while keeping the power efficiency around 2 W/PetaOPS.A multi-layered (L-layer) open-loop hyperspectral system (outlined in Figure 4a and Scenario 3 of Table I ) is expected to demonstrate comparable power efficiency, provided that the number of layers is sufficient to effectively offset the energy overhead from input electro-optic (EO) and output opto-electronic (OE) conversions. While direct comparisons of power consumption between mature digital electronic computing technologies and nascent optical computing lab demonstrations are challenging, our projections indicate a considerable boost in efficiency compared to state-of-the-art electronic GPUs. Our hyperspectral compute-in-memory architecture operates as a 3D opto-electronic computing system, processing 2D optical input data through a 2D optical memory "synapse" that conducts an O(N 3 ) hyperspectral "multiply" operation. Concurrently, the 2D opto-electronic "neuron" performs O(N 2 ) "accumulate" and nonlinear activation functions in parallel at every clock cycle, ensuring minimal latency. This architecture optimally uses chip area by directly linking the "synapse" and "neuron" chips optically, removing the need for physical wires and potentially achieving a compute density that exceeds PetaOPS/mm 2 (See Figure 1b ). Significantly, by localizing electronic operations within each pixel during computation, this setup minimizes electronic data movement, with most data communication handled optically. This efficiency substantially offsets the costs associated with electrical-to-optical (EO) and optical-to-electrical (OE) conversions.Our proposed hyperspectral in-memory computing system fully utilizes the dimensions of frequency, space, and time to enhance computational throughput and energy efficiency. It integrates space and frequency multiplexing using scalable SLM and OFC technologies, which are seeing rapid advancements through both industry and academic contributions. The modular nature of this design not only enables manufacturing by leveraging existing technologies and ecosystems but also encourages enhancements in individual component technologies, thereby driving overall system performance improvements. As scalability extends, incorporating optical element arrays and polarization multiplexing is envisaged, though large computational tasks are likely to be distributed across multiple small-scale optical computing modules, similar to traditional electronic systems. Integrating advanced optical components like metalenses (20) , chip-integrated OFCs (21) , and amplifiers (22) into a single or fewer optical elements as part of a modular assembly, suggests a trajectory towards significant system miniaturization. This advancement enables the integration of these systems into data centers as rack-mounted solutions. With ongoing improvements in component technology and the increasing importance of optics in data centers, this 3D opto-electronic computing architecture has the potential to revolutionize high-performance accelerated computing hardware in future data center applications.-Note: Most of the experimental data and figures are from our recent paper published in Optica (12) . 
 
 Appendix:  



Reviews: review: title: title: Well written, but not entirely novel matrix-matrix multiplier using frequency and spatial multiplexing
main_review: main_review: The authors present a hyperspectral compute-in-memory architecture that utilizes both frequency and spatial dimensions for matrix-matrix multiplication (MMM) in order to further improve the system throughput.
The paper is well written and it is primarily a work in progress and the only reported results are about accuracy of the MMM. I am already aware of several papers proposing space and wavelength as two possible dimensions to explore to scale out the system.
See for example the below reference:
M. Honari, B. Park, Y. Kim, and M. Suh, "Matrix-Vector Multiplication using Mixed Space-Frequency Multiplexing of Optical Frequency Combs," in Optica Nonlinear Optics Topical Meeting 2023, Technical Digest Series (Optica Publishing Group, 2023), paper Tu3B.3.

So given the fact that they report preliminary work and the lack of novelty of the idea they propose, I would say that the paper is borderline acceptable, given that what they presented is correct.