Title: Optical Transformers
Venue: NeurIPS 2023
Accepted: no
Keywords: Optics

Transformers

Accelerator

Energy Efficiency

Power Consumption

LLM

Large Language Models

Hardware

Optical Neural Networks

Scaling

Scaling Laws

Quantization
Abstract: The rapidly increasing size of deep-learning models has caused renewed and growing interest in alternatives to digital computers to dramatically reduce the energy cost of running state-of-the-art neural networks. Optical matrix-vector multipliers are best suited to performing computations with very large operands, which leads us to hypothesize that large Transformer models might achieve asymptotic energy advantages with optics over running digitally. To test this idea, we performed small-scale optical experiments with a prototype accelerator to demonstrate that Transformer operations can run on optical hardware despite noise and errors. Using experiment-calibrated simulations of our hardware, we studied the behavior of running Transformers optically, identifying scaling laws for model performance with respect to optical energy usage and estimating total system power consumption. We found that the optical energy per multiply-accumulate (MAC) scales as $\frac{1}{d}$ where $d$ is the Transformer width, an asymptotic advantage over digital systems. Should well-engineered, large-scale optical hardware be developed, it might achieve a $100 \times$ energy-efficiency advantage for running some of the largest current Transformer models, and if both the models and the optical hardware are scaled to the quadrillion-parameter regime, optical computers could have a $>8,000\times$ energy-efficiency advantage over state-of-the-art digital-electronic processors (300 fJ/MAC). We discussed how these results motivate and inform the construction of future optical accelerators and optics-amenable deep-learning approaches. With assumptions about future improvements to electronics and Transformer quantization techniques (5× cheaper memory access, double the digital–analog conversion efficiency, and 4-bit precision), we estimated that optical computers' advantage against these digital processors could grow to $>100,000\times$.

Abstract

The rapidly increasing size of deep-learning models has caused renewed and growing interest in alternatives to digital computers to dramatically reduce the energy cost of running state-of-the-art neural networks. Optical matrix-vector multipliers are best suited to performing computations with very large operands, which leads us to hypothesize that large Transformer models might achieve asymptotic energy advantages with optics over running digitally. To test this idea, we performed small-scale optical experiments with a prototype accelerator to demonstrate that Transformer operations can run on optical hardware despite noise and errors. Using experiment-calibrated simulations of our hardware, we studied the behavior of running Transformers optically, identifying scaling laws for model performance with respect to optical energy usage and estimating total system power consumption. We found that the optical energy per multiply-accumulate (MAC) scales as 1 d where d is the Transformer width, an asymptotic advantage over digital systems. Should well-engineered, large-scale optical hardware be developed, it might achieve a 100 × energy-efficiency advantage for running some of the largest current Transformer models, and if both the models and the optical hardware are scaled to the quadrillion-parameter regime, optical computers could have a > 8 , 000 × energy-efficiency advantage over state-of-the-art digital-electronic processors (300 fJ/MAC). We discussed how these results motivate and inform the construction of future optical accelerators and optics-amenable deep-learning approaches. With assumptions about future improvements to electronics and Transformer quantization techniques (5× cheaper memory access, double the digital-analog conversion efficiency, and 4-bit precision), we estimated that optical computers' advantage against these digital processors could grow to > 100 , 000 × .

1 Introduction

Deep learning models' exponentially increasing scale is both a key driver in advancing the state-of26 the-art and a cause of growing concern about their energy usage, speed, and practicality. This has led 27 to the development of hardware accelerators and model training/compression/design techniques for 28 efficient and fast inference on them. 29

While digital-electronic accelerators [47, 16, 8, 1, 17] can improve performance by some constant 30

factor, alternative analog computing platforms using optics have been proposed as a new paradigm 31

for better scalability [49, 7, 62, 41, 56, 24, 51]. Ideally, the scaling is asymptotically better than 32

digital systems in energy per MAC [18, 61, 53, 41]. But these optical neural networks (ONNs) have 33

additional complexities and limitations of their own such as low precision, noise, and analog/digital 34

data conversion overheads which depend on the access patterns of the model running (Figure 1). 35

Thus, advantageously accelerating any neural network architecture with ONNs is hard. Here, we 36

hope to answer whether Transformers' efficient data-access patterns (wide layers, parallel/batched 37

GPU

Model Affects ONN Precision and Energy

Architecture

Shot noise sets lower bound

for optical energy/MAC

Detect Optical

Output

Optically

Mach-Zehnder

Interferometer Mesh

Spatially-Multiplexed

Free-Space Propagation

(Our Example System)

Wavelength-Multiplexed

Micro-Ring Weight Banks

Optical Matrix-Vector

Multiplication Engine

Hypothesis:

O(n) energy-

efficiency

advantage for

large-scale models

Ground Truth

Systematic

Error

ONN - O(n)

Digital/GPU - O(n )

Optical

Weight/Data

Statistics

Data Access

Patterns

Serial

Batched

?

(

n

-dim Vector)

(

n

-dim Vector)

(

n x n

Matrix)

2

token processing, etc.), trends in methods for scaling them, and sufficient effort to train them for 38 ONNs afford them the asymptotic energy-efficiency advantages of running optically. 39

Here we demonstrate how the popular Transformer architecture is able to run on ONN systems, 40 and estimate the potential benefits of doing so. To first verify that Transformers may run on these 41 systems despite their imprecision, we sampled operations from a Transformer and ran them on a real 42 spatial light modulator (SLM) based experimental system, and used the results to create a calibrated 43 simulation of the optical hardware, with the systematic error, noise, and imprecision of weights/inputs 44 we observed. Transformers running on the simulated hardware could perform nearly as well as those 45 running digitally, and could be far more efficient. We summarize our key contributions as follows: 46

47

48

49

We demonstrated linear Transformer operations (the bulk of a Transformer's computation) running with sufficient accuracy on real optical hardware and in a matching simulation, despite errors and noise.

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

Via simulation, we established scaling laws for optical Transformer performance versus optical energy usage, and optical energy usage versus model size.

Based on our simulations and experiments we estimated an orders-of-magnitude energy consumption advantage of full ONN accelerators versus state-of-the-art GPUs.

We discussed Transformers' suitability for optical acceleration, and more generally how specific elements of DNN architecture affect the function of ONN systems running them.

We identified the hardware and systems design challenges that future work on building ONN accelerators should target.

While our experiments and simulations were based on specific hardware as a representative example, our scope here is more general. We are interested in understanding how uniquely optical energy scaling and noise relate to Transformer performance and architecture. As such nearly all our findings apply broadly to linear optical processors (and hopefully future ones), irrespective of their underlying hardware implementation details.

2 Background and Related Work

2.1 Transformer Models

Transformers are models for processing sequential data based on multi-head attention. Transformers 65

consist of two-layer feed-forward blocks and multi-head attention (Figure 2) operations. Multi66

head attention computes relationships between sequence elements by deriving query, key, and 67 value sequences Q,K,V and computing dot products with a softmax nonlinearity in-between [60]. 68 Transformers also leverage modern design elements such as additive residual skip connections [20] 69 and normalization layers [3]. A defining feature of Transformers is that entire sequences may be 70 processed in matrix-matrix products in parallel (instead of one token/input at a time). 71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

2.2 Large-Scale Deep Learning

In the past few years, it has been found in particular that Transformer [60] architectures significantly improve when sized up to billions or even trillions of parameters [6, 28, 10, 22, 59, 66], causing an exponential growth of deep learning compute usage [48, 50]. These large-scale Transformers achieve ever more impressive results in not only natural language processing, but also in other domains such as computer vision [14, 36], graphs [30], and in multi-modal settings [27, 26, 44, 45, 65, 46], making them a popular but expensive solution for many tasks-digital hardware's energy efficiency (ie. per-flop or per-inference cost) has not kept up with the growing FLOP requirements of state-of-the-art deep learning models [50]. They also have transfer learning capabilities [42, 13, 43, 6, 37, 14], allowing them to easily generalize to specific tasks, in some cases in a zero-shot setting where no further training is necessary [6, 45, 33].

2.3 Optical Accelerators

Researchers have explored a wide variety of controllable optical systems which manipulate different types of optical modes to effectively implement arbitrary matrix-vector multiplications, vector-vector dot products [52, 2, 18, 55, 4, 61, 19, 39, 57], or convolutions [63, 15, 40, 64]. In this work, we adopt the free-space multiplier [61, 55, 19] (Figure 2, top left) to demonstrate Transformer operations in optical experiments and for our simulations. We selected this system because it has many of the same behaviors as other ONN implementations, and aim to draw conclusions that could generally be useful for those working with other ONN designs. Many ONN systems, including ours, share the following typical traits:

Device Imprecision and Optical Shot Noise Optical systems are subject to errors in both the actual hardware and from photon detection. Detection of optical intensity in particular is subject to a phenomenon known as shot noise where the detected value is Poisson distributed: given vectors x and w , with the elements of x encoded as optical intensity, the output Y is distributed as:

For other encoding schemes such as amplitude or phase encoding, equation 1 should be modified, but the detection is still subject to shot noise.

Efficient Photon Usage Shot noise, and therefore an optical dot product's signal-to-noise ratio (SNR, which serves as an effective bit precision) is related to the mean number of photons at the output . The efficiency of photon usage can therefore grow with increasing multiply-accumulate operations (MACs): the SNR for the product w · x is

which explains this behavior; if the desired output precision does not change, constant photons are required regardless of dot product size. Work on ONNs has studied this behavior in a variety of scenarios [18, 41, 61, 53]. This efficient scaling is not a guarantee-the required number of photons may be influenced by a model architecture's activation/weight distributions, encoding schemes, precision requirements, etc.

Optical Neural Network Energy Costs The energy cost of optical neural networks is broken down into the optical costs of performing MACs and the electrical costs of loading/detecting data, which are usually dominant. Consider a product between two matrices, A ∈ R n × d , B ∈ R d × k . Such a product results in loading (detecting) nd + dk ( nk ) scalars, and performing ndk MACs. If the energy

to electrically load (detect) a scalar is E load ( E det ), and to perform a MAC optically is E optical , then 111 the total energy is: 112

This illustrates how ONNs may have asymptotic energy advantages over digital computers. Notice that regardless of the number of reuses, all data is only loaded once in Equation 3. This is because copying a vector's data and transporting it is free optically. Meanwhile, E optical ideally scales as 1 /d . These properties make energy cost disproportional to the number of MACs, ndk . In other words, E digital E ONN ∼ min ( n, d ) .

Streaming Weights Versus Weights-In-Place There are two approaches for loading weights. Weights-in-place schemes involve loading them once, and re-using them for many inputs. Alternatively, systems can employ streaming weights where at every computation the required weight matrix is loaded. Our experimental system is a weights-in-place scheme. For weights-in-place operations, the energy advantage scales as just E digital E ONN ∼ d .

2.4 Previous Optical Neural Network Architectures

Previous work has considered deep learning models such as MLPs and convolutional networks on benchmark tasks like MNIST [40, 61], and simulations of larger convolutional models such as AlexNet [32] on more difficult datasets such as ImageNet [18]. This begs the question of how well newer, larger models perform on optical systems.

2.5 Scalable Compression and Quantization of Large Language Models (LLMs)

Optical hardware's low precision raises the question of whether scaled-up models could be quantized sufficiently to run. Thankfully, continual research in LLM compression has progressively shown that larger models do not have increasing precision requirements. For example, [34] found that larger Transformers can be compressed more easily, to the degree that it is more worthwhile to train large ones and compress them over training smaller ones of the target size. Furthermore, [5] and [12] demonstrated running Transformers at scale with int8 precision, and the recent work of [11] proposes that 4-bit is optimal for nearly all model scales, except for the largest tested (175B parameters) where 3-bit was sometimes found to work better.

3 Optical Transformers

We designed models that are intentionally similar to other Transformers, with the goal of simulating their behavior (informed by some experimental measurements) and energy consumption on optical hardware. A summary of our approach and model is in Figure 2.

3.1 Architecture and Task

We created optical Transformer models with a GPT2-like [43] architecture that replaces the GELU [21] activation with ReLU6, which is known to improve low-precision model performance [31, 23, 29]. For language modelling, we used the raw Wikitext-103 dataset [38]. The models we simulated have 12 layers (consisting of multi-head attention and feed-forward blocks), operate on a context length of 1024 tokens, use 12 attention heads, and have embedding dimension d varying from 192 to 1536. The full details of the training technique, architecture, and hyperparameters are in Appendix A.

3.2 Transformer Computations on Optical Hardware

We ran experiments using a real Transformer's (we used the base-sized model with d = 768 ) weights in order to characterize the behavior of an ONN system. We adopted as a representative example of an optical accelerator a spatial light modulator (SLM) based system which computes vector-vector dot products [61]. Vectors are encoded on a display, and copies are shone through the SLM which has varying transmission corresponding to some data (ie. a weight matrix). The outputs of this

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

Linear

Linear

Linear

LayerNorm

Input (

n

)

Output (

n

x

v

)

Embedding

(

)

Positional

Encoding

Softmax

MatMul

Add

...

MatMul

Linear

...

Linear

ReLU6

Linear

Add

Linear

Softmax

LayerNorm

Multi-Head Attention (MHA) Layer

Feed-Forward (MLP) Layer

Quantize

or LUT

Sample

Noise

Output

Input

Experiment-Informed Accelerator Simulation

Weights

Light Encoding

Detection

Optical Vector-Vector/Matrix-Vector Multiplication

Experimental Accelerator

ONN System Energy Model

Optical Computation

RAM

DAC

MOD

RAM

ADC

AMP

GPT2

MT-NLG

Future Model

Incoming Light

Camera

Digital Operation

Run Experimentally

x

h

Attention Heads

x

L

Layers

operation-element-wise products-are collected at detectors as the resultant dot products (Figure 2, 154 top left). We collected lookup tables (LUTs)-mappings of the available discrete levels in both the 155 display and SLM devices-and used them to train a 'LUT-aware' optical Transformer model to run 156 on the setup. We then collected calibration curves, mappings from the detected output light intensity 157 to the actual neuron floating-point values. To do this, we ran many random dot products on the 158 hardware and collected pairs of detected values and digitally-computed ground-truth values. We then 159 fit the relationship linearly. We used high photon counts to eliminate shot noise, so deviation from 160 the linear fit was considered the hardware's systematic error . Full details of experimental procedures 161 and calibration are in Appendix B. 162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

3.3 Simulation of Optical Hardware

Informed by our experiments, we constructed a simulation of the optical hardware. By simulating the hardware behavior directly we model how any arbitrary operation would behave if run on the physical setup. This allows us to avoid the computationally demanding task of simulating much larger Transformers to verify that our simulation method works. We aimed to emulate the noise, error, and preci-

sion that we observed in order to understand how well full Transformers would perform when running on optical hardware. The configurations for different scenarios are summarized in Table 1. We also evaluated the digital, 8-bit-QAT-trained model for comparison purposes.

Hybrid Scheme Pure optical systems cannot easily compute activation or normalization functions. Thus we assumed LayerNorm, ReLU activations, and residual skip connections are performed digitally at full precision. Thankfully, even in smaller models, linear computations are the overwhelming majority (Section 4.3).

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

Non-Negative Weights and Inputs ('4-Pass' Multiplication) An important limitation is that our display and SLM only support non-negative values. The constraint of having all-positive data is present in many but not all optical neural network systems.We worked around this by decomposing products into sums/differences of products with non-negative operands. Consider a product between matrices W and X . If we let W + ( X + ) and W -( X -) be matrices with only the positive and negative elements of W ( X ) respectively, then:

Data Scaling On the real system, we define a maximum activation/weight value as 1.0 and minimum as 0.0. To simulate operation, the inputs and weights of every simulated NN layer are scaled to this range, and then rescaled back afterwards.

Device Quantization Real hardware may only have certain number of representable levels. To emulate this behavior, we fine-tuned pretrained models using quantization-aware training [25](QAT) and applied the following in simulation (hyperparameters in Appendix A):

For optics-simulated layers, we emulated quantization to int8 (256 levels). Then, instead of dequantizing, we used the integer values directly as indices into the LUTs that we gathered from experiment.

We also quantized weights, but with the SLM LUT. We clamped smaller values to 0.02 in the simulation, as our SLM does not have a high extinction ratio, and the smallest transmission is 0.02.

Accumulation can be high precision, but we used int8 quantization for outputs, since analog-digital conversion (ADC) is expensive in practice.

We used both deterministic and stochastic rounding when quantizing, with similar results.

Systematic Errors Issues like cross-talk, misalignment, defects in ONNs give rise to systematic errors. We simulated such a constraint by adding Gaussian noise to simulated model outputs (Figure 2), scaled relative to the mean sizes of the outputs, as this was the noise behavior we observed experimentally (it is related to the rescaling of data between 0 and 1).

Optical Encoding and Shot Noise We modeled optical encoding by subjecting layer outputs to simulated shot noise (Figure 2), which differs from the systematic error model. Outputs were scaled by a number such that the average photon number per feature (photons/MAC) was some target value. Each of these features was used as the mean of a Poisson distribution, which we sampled. These outputs were then scaled back down to represent neuron values. In the simulations for optical scaling we used vanilla 8-bit QAT (no LUTs or systematic error, which can overwhelm shot noise) to cleanly demonstrate the optical scaling properties-which are model-dependent and not hardware-dependent-of Transformers.

4 Results

4.1 Transformer Error Tolerance and Hardware-Simulation Accuracy

We determined experimentally that Transformer operations are able to run on real hardware without severely degraded performance from systematic errors. The bottom four panels of Figure 3 are histograms of the experimental differences from correct values. The simulated noise distributions (dotted lines) match well with the experimental data, which confirms that they are an accurate representation of the real systematic error behavior. Figure 3 (top) is a map of the performance of the simulated model over different configurations of the mean-relative (in percent) noise at every layer of feed-forward and attention blocks. The model performs well with significant noise (experimental noise levels marked with stars), within 1 perplexity from noise-free performance unless the noise is very high. These results show that our digital model of the system is a plausible approximation of how a real one might behave.

While 8-bit precision was used for 229 QAT, the optical Transformer can per230 form inference at lower precision, as 231 implied by its error tolerance. To 232 study this further we conducted a sim233 ple ablation on the input and output 234 precisions used at inference, on the 8235 bit-QAT base-sized model with LUT 236 in Appendix C. 237

238

4.2 Optical Scaling Laws

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

Optical Transformers achieve language modelling performance close to their digital counterparts' when shot-noise-limited at modest photon budgets. The perplexities on the Wikitext-103 validation set of various optical Transformer models simulated with different total photon usage (amount used for input data) are shown in Figure 4 (left). The curves illustrate a tradeoff: larger models need larger photon totals to function well, and there are different optimal model choices based on the photon budget. We define photons/MAC as the total photon budget (amount at input) divided by total MACs. The percentage difference from the performance at 10K photons/MAC (Figure 4, middle)-chosen to represent an ideal high-precision scenario-is roughly power-law scaled in photons/MAC for all models with truncation near 10K; better performance can be had with more photons, but with diminishing returns, and the performance matches or exceeds that of the 8-bit digital models' when the photon budget is not too low ( ∼ 10 2 ).

The models use fewer photons/MAC

as they scale, achieving the theoretical efficient scaling where the total per-dot-product photons needed is constant. To study how photon usage scales, we determined how many photons it takes to reach the performance of 8-bit digital models. These values, in Figure 4 (right), decrease nearly as 1 d -the total photons needed per dot product is constant (bottom dashed line). The Transformer architecture clearly takes advantage of efficient optical scaling with larger model sizes. In fact, smaller per-dot-product totals are required for the largest model, suggesting that larger Transformers may require less output precision. This is consistent with other work which found that precision requirements are constant or reduced with scale [34]. Meanwhile, the already low photon usage of the largest model suggests that models larger than our simulations (>10B parameters) may use <1 photon/MAC. This sub-photon operation works in optical systems [61, 53] and is in essence no different at all from operation at higher photon counts (since the number summed at detection is still high).

These empirical scaling results are tied to our specific configurations and training strategies. Depending on the scales and dynamic ranges of inputs and weights, different amounts of photons may be transmitted to the output; the statistics of a model affect its efficiency. In Appendix H we explore a different scheme, but the effects of different methods remains an interesting topic for future work.

Experiment

Experimental

Error

Simulated

Noise

First FF

σ = 2.09%

First Attention

σ = 5.65%

20.25

Last FF

σ = 1.31%

Last Attention

σ = 5.99%

285

Constant photons/MAC

Constant dot-product total

8-bit PPL

4.3 Estimated Energy Usage

The efficient photon scaling trend we observed in Section 4.2 suggests that Transformers running 286 on optical hardware could achieve significant energy efficiency advantages over running on digital 287 hardware. To understand the efficiency of Transformers on optical hardware, we designed an ONN 288 system based on current hardware that is like our experimental setup, with our measured precision 289 and photon scaling. It is an inference system with in-place weights which are loaded once and reused 290 forever, activations read from and written to SRAM for every layer, a 10 GHz light modulator array, 291 and an optical 'core' which can perform 10M multiplications per cycle (this can be thought of as a 292 10 megapixel SLM). The photon-per-MAC scaling versus model dimension is taken to be the 1 /d 293 scaling which we found was possible in our simulations, and we assumed that the model operates 294 with 5-bit input precision, 8-bit weight precision, and 7-bit output precision, as determined by our 295 study of low precision performance in Appendix C. We then calculated according to the approach 296 in Section 2.3. For electrical energy we assumed in-place weights and did not include the energy 297 for loading them. In Appendix D we explain all assumed energy quantities based on contemporary 298 hardware. 299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

As models grow, running Transformers on optical hardware has a large and asymptotic efficiency advantage over running on digital hardware. In Figure 5 we chart estimates of the forward pass energy required for various models 1 , including a hypothetical family of large, dense Transformer models designed in a similar fashion, which we label FUTURE-* . For comparison, we also chart various digital systems [47] in different performance regimes, and a hypothetical 'next generation' GPU that can use ∼ 10 fJ/MAC. For small models, the optics-based system uses about the same energy, but eventually gains an advantage that scales asymptotically with the number of MACs. For the larger models, MT-NLG-530B and FUTURE-4q, the optics-based approach would have ∼ 140 × and ∼ 8500 × energy advantages over the current state-of-the-art GPU (NVIDIA A100) respectively.

The breakdown of compute and energy costs by source is in Appendix E. In summary we found that as models get larger the feed-forward layers require most of the computation, but that the energy of data access in attention is still very expensive due to the many heads. This is because of the parallel operation of the Transformer, where the linear layer weights can be re-used for many tokens at a time (weights-in-place is not possible for attention, and there are h n × n attention maps to store). 2

5 Discussion

The results given in Section 4.3 on optical Transformers' efficiency have implications for the design of future ONN hardware/software systems.

In Appendix G we discuss in detail the specifications for an ONN system to run large Transformers, as a target for future work in their design. In summary, we found: once matrix-matrix product operands exceed 10 4 × 10 4 in size the advantage is significant, and therefore a future ONN should implement at least this level of parallelism to achieve > 100 × efficiency improvements over current state-of-the-art GPUs (NVIDIA A100). Given the assumptions we made about weight-maintenance costs in making our estimates (5.6 µW per weight; see Appendix D), an Optical Transformer would need to operate in the regime where a single matrix-vector multiplication is performed every 0.1 nanoseconds. Current ONN prototypes either operate at low clock rate or at small scale. Thus building a full ONN system that realizes the potential benefit is still an open challenge.

Future improvements in CMOS technology will be greatly beneficial. In Appendix F we estimate that future optics-based systems might achieve energy advantages of > 100 , 000 × running models the size of FUTURE-4q (over 300 fJ/MAC).

Our studies on Transformers illustrates more broadly the relationships between model design and ONN efficiency. Transformers sought to make large models run efficiently by exploiting hardware's strengths in performing large, parallel, dense calculations, and improved in this aspect as they scaled. As a consequence, as Transformers continue to be optimized for parallel digital electronic hardware, they will continue to become even more efficient on optical hardware. More generally, architectures that perform more computations per data access (such as those focusing strongly on linear operations [58, 35]) will be most promising for optical implementation.

Conclusion We have demonstrated the ability of Transformer models to run accurately and efficiently on optical hardware through optical experiments and an experiment-informed simulation of the hardware. We examined Transformers' scaling behavior with optics and used our findings to show that optical systems could have a large and asymptotic energy advantage over digital ones that grows with the model size. For example, we showed that optical hardware may achieve an over 100 × energy advantage when running the largest Transformer models today ( ∼ 500 billion parameters) and that larger, future Transformers ( ∼ 4 quadrillion parameters) may be realized with an > 8000 × optical energy advantage. We believe our findings about the potential energy-efficiency of optical accelerator hardware strongly motivate the development of optical processors for large-scale deep learning with Transformers.

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

References

Michael Andersch, Greg Palmer, Ronny Krashinsky, Nick Stam, Vishal Mehta, Gonzalo Brito, and Sridhar Ramaswamy. NVIDIA Hopper architecture in-depth. Technical report, March 2022. URL https://developer.nvidia.com/blog/ nvidia-hopper-architecture-in-depth/ .

William Andregg, Michael Andregg, Robert T Weverka, and Lionel Clermont. Wavelength multiplexed matrix-matrix multiplier, April 19 2019. URL https://patents.google.com/ patent/US10274989B2/en . (U.S. Patent No. 10,274,989). U.S. Patent and Trademark Office.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https://arxiv.org/abs/1607.06450 .

Wim Bogaerts, Daniel Pérez, José Capmany, David A B Miller, Joyce Poon, Dirk Englund, Francesco Morichetti, and Andrea Melloni. Programmable photonic circuits. Nature , 586 (7828):207-216, 2020. URL https://doi.org/10.1038/s41586-020-2764-0 .

Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient Transformer quantization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7947-7969, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.emnlp-main.627 .

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https://arxiv.org/abs/2005.14165 .

H John Caulfield and Shlomi Dolev. Why future supercomputing requires optics. Nature Photonics , 4(5):261-263, 2010. URL https://doi.org/10.1038/nphoton.2010.94 .

Cerebras Systems. Cerebras systems: Achieving industry best AI performance through a systems approach. Technical report, Apr 2021. URL https: //8968533.fs1.hubspotusercontent-na1.net/hubfs/8968533/Whitepapers/ Cerebras-CS-2-Whitepaper.pdf .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with Pathways, 2022. URL https://arxiv.org/abs/2204.02311 .

Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George Bm Van Den Driessche, Eliza Rutherford, Tom Hennigan, Matthew J Johnson, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Marc'Aurelio Ranzato, Jack Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. Unified scaling laws for routed language models. In Kamalika Chaudhuri,

Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Pro396 ceedings of the 39th International Conference on Machine Learning , volume 162 of Pro397 ceedings of Machine Learning Research , pages 4057-4086. PMLR, 17-23 Jul 2022. URL 398 https://proceedings.mlr.press/v162/clark22a.html . 399

400

401

Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws, 2022.

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for Transformers at scale, 2022. URL https://arxiv.org/abs/2208.07339 .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423 .

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR , 2021.

Johannes Feldmann, Nathan Youngblood, Maxim Karpov, Helge Gehring, Xuan Li, Maik Stappers, Manuel Le Gallo, Xin Fu, Anton Lukashchuk, Arslan Sajid Raja, et al. Parallel convolutional processing using an integrated photonic tensor core. Nature , 589(7840):52-58, 2021.

Graphcore. The data center architecture for graphcore computing. Technical report, Apr 2021. URL https://www.graphcore.ai/hubfs/ Graphcore-Mk2-IPU-System-Architecture-GC.pdf .

Habana Labs. HABANA® GAUDI®2 white paper. Technical report, June 2022. URL https://habana.ai/wp-content/uploads/pdf/2022/gaudi2-whitepaper.pdf .

Ryan Hamerly, Liane Bernstein, Alexander Sludds, Marin Soljaˇ ci´ c, and Dirk Englund. Largescale optical neural networks based on photoelectric multiplication. Physical Review X , 9(2): 021032, 2019. URL https://doi.org/10.1103/PhysRevX.9.021032 .

Yoshio Hayasaki, Ichiro Tohyama, Toyohiko Yatagai, Masahiko Mori, and Satoshi Ishihara. Optical learning neural network using Selfoc microlens array. Japanese Journal of Applied Physics , 31(5S):1689, 1992. URL https://doi.org/10.1143/JJAP.31.1689 .

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, June 2016. doi: 10.1109/cvpr.2016.90. URL https://doi.org/10.1109/cvpr.2016. 90 .

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016. URL https: //arxiv.org/abs/1606.08415 .

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL https://arxiv.org/ abs/2203.15556 .

Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. ArXiv , abs/1704.04861, 2017.

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

Chaoran Huang, Volker J. Sorger, Mario Miscuglio, Mohammed Al-Qadasi, Avilash Mukherjee, Lutz Lampe, Mitchell Nichols, Alexander N. Tait, Thomas Ferreira de Lima, Bicky A. Marquez, Jiahui Wang, Lukas Chrostowski, Mable P. Fok, Daniel Brunner, Shanhui Fan, Sudip Shekhar, Paul R. Prucnal, and Bhavin J. Shastri. Prospects and applications of photonic neural networks. Advances in Physics: X , 7(1), October 2021. doi: 10.1080/23746149.2021.1981155. URL https://doi.org/10.1080/23746149.2021.1981155 .

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2704-2713, 2018. URL https://doi.org/10.1109/ CVPR.2018.00286 .

Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and Jo¯ ao Carreira. Perceiver IO: A general architecture for structured inputs & outputs, 2021. URL https://arxiv.org/abs/ 2107.14795 .

Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 4651-4664. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/jaegle21a.html .

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361 .

Hyungjun Kim, Jihoon Park, Changhun Lee, and Jae-Joon Kim. Improving accuracy of binary neural networks using unbalanced activation distribution. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7858-7867, 2021. doi: 10.1109/ CVPR46437.2021.00777.

Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure Transformers are powerful graph learners. arXiv , abs/2207.02505, 2022. URL https://arxiv.org/abs/2207.02505 .

Alex Krizhevsky. Convolutional deep belief networks on cifar-10. 2010. URL https: //www.cs.toronto.edu/~kriz/conv-cifar10-aug2010.pdf .

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper/2012/file/ c399862d3b9d6b76c8436e924a68c45b-Paper.pdf .

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models, 2022. URL https://arxiv.org/abs/2206.14858 .

Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers. In Proceedings of the 37th International Conference on Machine Learning , ICML'20. JMLR.org, 2020.

Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. Pay attention to mlps, 2021. URL https://arxiv.org/abs/2105.08050 .

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal 495 computation engines. arXiv preprint arXiv:2103.05247 , 2021. 496

497

498

499

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR) , 2017. URL https: //openreview.net/forum?id=Byj72udxe .

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

Charis Mesaritakis, Vassilis Papataxiarhis, and Dimitris Syvridis. Micro ring resonators as building blocks for an all-optical high-speed reservoir-computing bit-pattern-recognition system. J. Opt. Soc. Am. B , 30(11):3048-3055, Nov 2013. doi: 10.1364/JOSAB.30.003048. URL https://opg.optica.org/josab/abstract.cfm?URI=josab-30-11-3048 .

Mario Miscuglio, Zibo Hu, Shurui Li, Jonathan K George, Roberto Capanna, Hamed Dalir, Philippe M Bardet, Puneet Gupta, and Volker ˜ J. Sorger. Massively parallel amplitude-only fourier neural network. Optica , 7(12):1812-1819, 2020. URL https://doi.org/10.1364/ OPTICA.408659 .

Mitchell A Nahmias, Thomas Ferreira De Lima, Alexander N Tait, Hsuan-Tung Peng, Bhavin J Shastri, and Paul R Prucnal. Photonic multiply-accumulate operations for neural networks. IEEE Journal of Selected Topics in Quantum Electronics , 26:1-18, 2020. URL https://doi. org/10.1109/JSTQE.2019.2941485 .

Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018. URL https://openai.com/blog/language-unsupervised/ .

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. URL https://openai.com/blog/ better-language-models/ .

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/radford21a.html .

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8821-8831. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/ramesh21a.html .

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research , 2022. ISSN 2835-8856. URL https://openreview.net/ forum?id=1ikK0kHjvj . Featured Certification.

Albert Reuther, Peter Michaleas, Michael Jones, Vijay Gadepally, Siddharth Samsi, and Jeremy Kepner. Survey of machine learning accelerators. arXiv:2009.00993 , 2020. URL https: //arxiv.org/abs/2009.00993 .

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv , abs/1910.01108, 2019.

Abu Sebastian, Manuel Le Gallo, Riduan Khaddam-Aljameh, and Evangelos Eleftheriou. Memory devices and applications for in-memory computing. Nature Nanotechnology , 15(7): 529-544, March 2020. doi: 10.1038/s41565-020-0655-z. URL https://doi.org/10.1038/ s41565-020-0655-z .

Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo 543 Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint 544 Conference on Neural Networks (IJCNN) , pages 1-8, 2022. doi: 10.1109/IJCNN55064.2022. 545 9891914. 546

547

548

549

550

Bhavin J Shastri, Alexander N Tait, T Ferreira de Lima, Wolfram HP Pernice, Harish Bhaskaran, C David Wright, and Paul R Prucnal. Photonics for artificial intelligence and neuromorphic computing. Nature Photonics , 15(2):102-114, 2021. URL https://doi.org/10.1038/ s41566-020-00754-y .

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

Yichen Shen, Nicholas C Harris, Scott Skirlo, Mihika Prabhu, Tom Baehr-Jones, Michael Hochberg, Xin Sun, Shijie Zhao, Hugo Larochelle, Dirk Englund, and Marin Soljaˇ ci´ c. Deep learning with coherent nanophotonic circuits. Nature Photonics , 11(7):441, 2017. URL https://doi.org/10.1038/nphoton.2017.93 .

Alexander Sludds, Saumil Bandyopadhyay, Zaijun Chen, Zhizhen Zhong, Jared Cochrane, Liane Bernstein, Darius Bunandar, P. Ben Dixon, Scott A. Hamilton, Matthew Streshinsky, Ari Novack, Tom Baehr-Jones, Michael Hochberg, Manya Ghobadi, Ryan Hamerly, and Dirk Englund. Delocalized photonic deep learning on the internet's edge. Science , 378(6617):270276, 2022. doi: 10.1126/science.abq8271. URL https://www.science.org/doi/abs/10. 1126/science.abq8271 .

Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B, a large-scale generative language model, 2022. URL https://arxiv.org/abs/2201.11990 .

James Spall, Xianxin Guo, Thomas D Barrett, and AI Lvovsky. Fully reconfigurable coherent optical vector-matrix multiplication. Optics Letters , 45(20):5752-5755, 2020. URL https: //doi.org/10.1364/OL.401675 .

Pascal Stark, Folkert Horst, Roger Dangel, Jonas Weiss, and Bert Jan Offrein. Opportunities for integrated photonic neural networks. Nanophotonics , 9(13):4221-4232, 2020. URL https: //doi.org/10.1515/nanoph-2020-0297 .

Alexander N. Tait, John Chang, Bhavin J. Shastri, Mitchell A. Nahmias, and Paul R. Prucnal. Demonstration of WDM weighted addition for principal component analysis. Opt. Express , 23(10):12758-12765, May 2015. doi: 10.1364/OE.23.012758. URL https://opg.optica. org/oe/abstract.cfm?URI=oe-23-10-12758 .

Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP-Mixer: An all-MLP architecture for vision. arXiv preprint arXiv:2105.01601 , 2021.

Marcos Treviso, Tianchu Ji, Ji-Ung Lee, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Pedro H. Martins, André F. T. Martins, Peter Milder, Colin Raffel, Edwin Simpson, Noam Slonim, Niranjan Balasubramanian, Leon Derczynski, and Roy Schwartz. Efficient methods for natural language processing: A survey, 2022. URL https://arxiv.org/abs/2209.00099 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998-6008. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .

Tianyu Wang, Shi-Yuan Ma, Logan G. Wright, Tatsuhiro Onodera, Brian C. Richard, and Peter L. McMahon. An optical neural network using less than 1 photon per multiplication. Nature Communications , 13(1), January 2022. doi: 10.1038/s41467-021-27774-8. URL https://doi.org/10.1038/s41467-021-27774-8 .

Gordon Wetzstein, Aydogan Ozcan, Sylvain Gigan, Shanhui Fan, Dirk Englund, Marin Soljaˇ ci´ c, 595 Cornelia Denz, David A. B. Miller, and Demetri Psaltis. Inference in artificial intelligence 596 with deep optics and photonics. Nature , 588(7836):39-47, Dec 2020. ISSN 1476-4687. doi: 597 10.1038/s41586-020-2973-6. URL https://doi.org/10.1038/s41586-020-2973-6 . 598

599

600

601

Changming Wu, Heshan Yu, Seokhyeong Lee, Ruoming Peng, Ichiro Takeuchi, and Mo Li. Programmable phase-change metasurfaces on waveguides for multimode photonic convolutional neural network. arXiv preprint arXiv:2004.10651 , 2020.

602

603

604

605

606

607

608

609

610

611

612

Xingyuan Xu, Mengxi Tan, Bill Corcoran, Jiayang Wu, Andreas Boes, Thach G Nguyen, Sai T Chu, Brent E Little, Damien G Hicks, Roberto Morandotti, Arnan Mitchell, and David J Moss. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature , 589(7840): 44-51, 2021. URL https://doi.org/10.1038/s41586-020-03063-0 .

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research , 2022. ISSN 2835-8856. URL https://openreview.net/forum?id= Ee277P3AYC .

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12104-12113, June 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyze the performance, efficiency, and robustness of free-space optical dot-product engines for Transformer accelerations. Measurement results on an SLM-based optical system are demonstrated on some layers in a GPT-like model. System performance/efficiency are estimated and compared to digital computers. Scaling of optical processors are discussed to show the scalability of optical computing platforms.\n\nSTRENGTHS:\n1. Experimental results on SLM-based free-space optical system has been demonstrated for matrix multiplication.\n2. Scalability with future technologies are discussed to show the benefit of optical computing in the future.\n\nWEAKNESSES:\n1.\tThe novelty of the paper raises some concerns as no new hardware design or algorithm innovations have been shown. The SLM-based system and its experimental demonstration are not new. No customized hardware is shown for Transformer. The claimed optical hardware is designed for CNNs/MLPs. On the algorithm part, device quantization, the LUT-based training method, noise analysis, and 4-pass multiplication are standard methods for analog computing. NeurIPS community usually requires certain machine learning contributions. What is the main ML contribution? Probably other venues in the optics community are more suitable for this paper.\n2. The demonstrated system is weight-in-place which needs a large number of parallel MVM to amortize the weight programming/encoding cost. However, the dynamic attention operations in Transformer and fully-connected layers usually have low arithmetic intensity, especially GPT-like architecture with KV cache, which cannot provide enough batch dimension to amortize such cost for weight-in-place systems. More justification for the usage of the weight-in-place system needs further discussion. A weight streaming system might be the suitable architecture for Transformer.\n3.\tIn Fig. 2, only a small part of layers in the Transformer block are implemented by optics, while other operations are on all digital computers. In this hybrid case, how large are the efficiency/performance benefits, or is it worthwhile to use optics?\n4.\tFor the noise analysis, only shot noise is emphasized, which is much smaller compared to other variations in the system both on the electrical and optics sides. A simple Gaussian added to the output results might be oversimplified as system error modeling. \n5.\tIn Line 291, the system assumes a 10 GHz light modulator array. If I understand correctly, the spatial light modulator typically has high resolution but very low switching frequency. This 10 GHz modulation speed needs further justification. How fast is the switching freq for weights and input feature maps? The modulation energy cost is based on thin-film lithium niobate modulators, which are fairly large. How many such large modulators are required to modulate a million pixels?\n6.\tAlso, the light source/TIA/ADC power consumption in the camera will be very large if working at such a high frequency. The incoming data fetched from memory will also be a bottleneck, which might not be able to fast enough to feed the 10 GHz optical core. In Line 297, the memory part is completely ignored when compared with digital computers, which might not be a fair comparison even in the near future. Only multiplications are done in optics, the partial product summation is done digitally, especially when it requires 4-pass, which raises concerns about the benefit of this SLM system for Transformer acceleration. More discussion on the system performance/efficiency is recommended.\n7.\tThe paper title optical Transformer is very broad, however the current paper only focuses on SLM-based free-space optics, which suffer from bulky optical setups (low compactness) and noise/alignment/sensitivity issues in practical deployment. More discussion and comparison on other integrated photonic/diffractive hardware is important and necessary. Otherwise, the scope/title of the paper is more suitable to be narrowed down to an SLM-based Transformer acceleration platform.\n\nQUESTIONS:\nMy questions are listed in the above weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

comments: SUMMARY:
This paper propose a photonic hardware accelerator to process the inferences of large language models, i.e., transformers, using optical multiply-accumulate (MAC) operations. Optical MACs are suitable for computations with large operands, thereby leading to asymptotic energy advantages over the digital hardware accelerators.

STRENGTHS:
1. The paper is well-organized.
2. The paper works on an important problem.

WEAKNESSES:
1. The paper does NOT consider the energy consumption of analog-to-digital converters, digital-to-analog converters, and various memories such as on-chip SRAM and off-chip DRAM. I totally agree with the cornstone of this paper, which is optical MACs or matrix multiplications are super energy-efficient. However, gaining this energy advantage is not easy. Reading 530B parameters of a transformer, converting these many digital parameters to analog optical signials, and converting the analog optical result signials back to digital values may donimate the energy consumption of an inference. As a result the energy efficiency improvement may not be very large.

Please check the comparison in this paper:
W. Liu, W. Liu, Y. Ye, Q. Lou, Y. Xie and L. Jiang, "HolyLight: A Nanophotonic Accelerator for Deep Learning in Data Centers," 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE), Florence, Italy, 2019, pp. 1483-1488, doi: 10.23919/DATE.2019.8715195.

QUESTIONS:
Please comment on the energy consumption of analog-to-digital converters, digital-to-analog converters, and various memories such as on-chip SRAM and off-chip DRAM during the otpical transformer inferences.

LIMITATIONS:
No potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the feasibility and benefits of employing optical computing techniques for machine learning and specifically focusing on large language models (LLMs).  The paper builds upon earlier work on optical neural networks, primarily [61] (Wang et al., Nature, 2022), which experimentally demonstrated the feasibility of performing dot products optically in a two layer neural network applied to MNIST while achieving around 90% accuracy at about one photon per multiplication optical energy.  As LLM computations involved a large and rapidly growing number of multiply accumulate operations, the objective of the paper is to explore whether optical techniques can yield benefits over existing CMOS-based accelerators (GPUs, TPUs).   The paper tackles this by employing a simulation based methodology where the simulator attempts to model the various noise sources (systemic, shot noise) along with limited precision of a potential electro-optical system.  The evaluation shows that as LLMs continue to scale such optical systems may potentially yield many order of magnitude benefits in terms of energy efficiency over current approaches.\n\nSTRENGTHS:\nMakes a reasonably strong case for further exploration of optical computing for large language models.\n\nWEAKNESSES:\nNot enough discussion of the remaining challenges that need to be overcome to make such systems competitive in reality.\n\nSomewhat limited contributions in-so-far as earlier works have already explored using optical for ML.  \n\nUnclear how accurate the simulation methodology is.\n\nSome aspects unclear.\n\nQUESTIONS:\nHow far are current optical systems from the 10GHz operating frequency assumed in this paper (Line 291) and what would Figure 5 look like and to what extent would the main conclusions about the benefits of optical be undermined if more modest (or perhaps realistic) rates were assumed?  How much improvement is required in this direction to achieve the hoped for benefits?  What frequency do current modulators operate at?\n\nHow do you separate out shot noise from systematic noise in the measurements in Figure 3?\n\nDoes the y-axis in Figure 4 of the main paper include the electrical energy overheads like those in Figure 3 and 4 in the supplemental? Figure 3 in the supplemental seems to show optical energy is entirely negligible so in this case the accuracy of Figure 4 in the main paper would be entirely dependent upon a full accounting of all other energy sources.   What assurances can you give that all significant sources of non-optical energy are properly accounted for?\n\nThe assumption of weights-in-place seems quite unrealistic and while the supplemental discusses (Section G) a \"chunking\" technique to scale up to larger numbers of parameters it was unclear whether the assumptions used to plot Figure 6 in the supplemental section are optimistic or conservative and to what extent.  Please comment.\n\nTransformers are having a lot of impact, which makes the focus of this paper on them make sense, but can you comment on how much optical may other network architectures?   Accelerators like GPUs and TPUs try to be relatively general purpose and so were able to be quickly retargeted to transformer based models without needing much in the way of hardware changes.  Can your simulation based study approach yield insights about whether optical will help networks other than transformers and/or under which circumstances?\n\nHow accurate is the simulation methodology in Figure 2?   Can you attempt to compare your simulators predictions versus your actual hardware for a small network like the one studied in [61]?\n\nI did not follow what the lookup-table (LUT) in the simulator does.  What values do the LUTs in the simulator contain and what quantity is used to index into the LUT?\n\nLIMITATIONS:\nThere is some discussion of limitations.  If the paper is meant to rally others to work on optical techniques for machine learning it would be helpful if the authors could more systematically highlight the remaining challenges to making such systems practical.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors perform experimental analysis with a spatial light modulator to optically perform the computations of the linear components of the Transformer architecture. These measurements allow them to create a noise-model that is then used to simulate a GPT-2 like model and measure performance (validation perplexity) in function of model parameters (system noise, not including optical shot noise). These optical systems are physically constrained by \"optical shot noise\" that dictates the minimum number of photons to achieve a target precision. This optical shot noise scales favorably with larger models, and the authors then extrapolate their observations to existing large Transformer models (like PaLM), and further to even larger hypothetical future models, showing a substantial advantage of a large scale hypothetical optical system over existing electrical systems (the energy advantage scaling with the width of the model).\n\nSTRENGTHS:\n1. While there have been previous works that examine smaller scale optical neural networks and the optical shot noise scaling behavior seems to be well established (I'm inferring this from the works cited in the submission, I'm not familiar with the area), the authors highlight the importance of the scaling behavior in context of the Transformar architectures that are typically used with large language models. I think the combination of innovative research in the field of optical computation with the computation requirements of applied machine learning models at the largest scales is particularly interesting.\n\n2. The authors used small scale experiments to establish realistic system properties and then simulated an entire Transformer to check how the physical properties would influence overall (validation) performance of the model. They did this at a scale (GPT-2 like quantized model with 15M-416M parameters) that seems large enough to gather realistic predictions.\n\n3. The interpolation of the simulated data to much larger models is necessarily based on many assumptions, both in scaling from the simulation to larger models, and with respect to the hypothetical hardware that would run very large models on optical hardware. The appendix goes into some detail on the different assumptions that led to the conclusions summarized in the main part of the article.\n\n4. The overall presentation of the work seems adequate for a public that is knowledgeable in machine learning, but probably has much less experience with physical properties of optical computational hardware, which is no easy task given the relatively large gap between these domains.\n\nWEAKNESSES:\n1. All the discussion of the optical vs. electrical implementation of the computation is in the lens of energy consumption. After reading the paper I'm not sure how the proposed architecture would fare with respect to latency/speed, or other constraints (e.g. if there is a theoretical limitation to the size – or cost – of different components that scales very differently between the currently used IC technology of purely digital microchips vs. optical components). Latency is mentioned in Appendix G, but there is no mention of these other dimensions in the main text.\n\n2. The authors highlight the scaling behavior of optical shot noise, but other sources of noise (called \"systematic errors\" in the paper) are simply measured for the system at hand, and no information is given how these other sources of noise would scale when the system is scaled. For example, I assume that any realistic scaling would require miniaturization of the optical components, and I would assume that this miniaturization also causes the systematic errors to change in relative magnitude. It would be interesting to at least briefly discuss the scaling behavior of these other sources of noise.\n\nQUESTIONS:\nSmall things I noticed while reading the paper:\n\na) 155-157: not clear what this means: “We collected lookup tables (LUTs)—mappings of the available discrete levels in both the display and SLM devices—and used them to train a “LUT-aware” optical Transformer model to run on the setup.” why are the LUTs needed? what does “collected” refer to? (measurements used for downstream analysis? used for the computation itself?) … maybe referencing what is exactly meant with LUTs (is it a hardware component?) would also help here\n\nb) 171, 179: the abbreviation QAT is used without being introduced (it’s spelled out on line 194)\n\nc) Table 1 is a bit hard to read because the “Setting” column spans multiple rows, but the text neatly aligns (Hardware->QAT, Simulation->Eval, etc) with the individual rows (one could e.g. add more horizontal lines in columns 2+ to make this clearer)\n\nd) 196: when saying “int8” is it actually “uint8” ?\n\ne) Figure 5 I find it surprising to see TPU int8 use 800fJ/MAC vs. NVIDIA A1000 fp16 300 fJ/MAC – I could not readily find these numbers in [47] – how exactly were they computed?\n\nf) 329: either “study” or “illustrate” (but not “our studies illustrates”)\n\ng) Appendix, Figure 4 (left): “Digital Ops” are not visible – why? If they are zero (e.g. “not doing any compute”) then that should be mentioned.\n\nh) Appendix, Figure 4 (right): the colors are hard to tell apart (especially it would be interesting to clearly see optical vs. electrical). It’s also not very clear what are “Ele FF” and “Ele Attn FF” (maybe ReLU6, LayerNorm, and Add from Figure 2?)\n\nLIMITATIONS:\nI don't see a negative societal impact of the presented work, other than making Transformers more energy efficient would allow to scale them further than otherwise possible, and accordingly accentuate any potential opportunities and risks of models such as LLMs. I don't think it's required to point these out explicitly in the paper.\n\nThere seem to be a number of technical limitations with respect to the predictions in the sense that there is a lot of uncertainty whether the hardware to run optical neural networks could be scaled up as much as state of the art digital circuits. But the authors make it clear already in the abstract that their main interest is in establishing a scaling law and not concrete predictions about the future of the implementation of large scale optical neural networks.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}