Universal approximation capabilities of coherent diffractive systems

Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-11 17:12:15

Abstract: Coherent optical computing systems are a promising avenue for increasing computation speed and solving energy requirements for machine learning applications. These systems utilize the diffraction process of coherent waves to perform calculations in the optical domain. While the diffraction process is linear in complex space $\mathbb{C}$, it empirically has been shown, that these systems are able to outperform standard linear matrix multiplications in $\mathbb{R}$, because photo-sensors project from complex space to real space. Here we give theoretical insights, of why this is the case and show, that a system consisting of multiple phase-plates, two output photo-detectors, and the appropriate input encoding is theoretically able to learn any one-dimensional function. We furthermore use these theoretical insights to show that encoding the input information solely in the intensity of the diffractive system is never enough to make the system a universal function aproximator. These results are useful to understand the capabilities of diffractive optical systems and improve their training.

Main paper: 
 introduction: 
 In recent years large artificial neural networks have set new standards in research and industrial applications. The rise of these large models has been largely made possible by an increase in memory and computing power. Training these large neural networks requires general purpose computing devices that are used to calculate huge matrix multiplications and other operations in parallel. Usually these devices are GPUs. However, since many artificial neural network architectures show increased performance with increasing number of parameters [1] , the best models are in general those with the maximum possible number of trainable parameters, only constrained by available data and training time. This leads to huge energy and computational requirements during training and deployment.To tackle the ever increasing computing power requirements for large neural networks new computation architectures are needed to change the way a computer handles these large amount of floating point operations.One promising approach for such devices is optical computing [2] [3] [4] . Approaches can be divided into those using incoherent light [5, 6] , and those using coherent waves [7] [8] [9] [10] [11] [12] . Some of these systems rely on optical waveguides, while other systems utilize free space propagation of light [13] [14] [15] [16] [17] [18] .Systems that utilize coherent free space propagation and coherent holographic plates are also called diffractive deep neural networks (DDNN) [13] . An input signal is encoded in the amplitude and/or phase of a coherent wave that propagates through multiple plates, which are masks that change the phase or amplitude of a wave across its wavefrong. These plates are designed in such a way, that the complete network performs a desired operation directly in the analog space of the wave. These networks have been realized with optical light [19] , terahertz waves [13] and with ultrasound [20] . They are able to perform a whole array of different computations, ranging from image classification and mode conversion [18] to learning of logical functions [14, 15] . However such networks are inherently limited by the lack of a nonlinearity, since free space wave diffraction and hologram plates are linear in complex space, in which the wave resides in. It has been shown, that diffractive deep neural networks without a nonlinearity can perform a complex matrix multiplication [18] .Optical nonlinearities in optical neural networks are an active research field [21, 22] . However one of the easiest implementable nonlinearities is a photosensor. Such a photosensor maps the complex field to a voltage, that is proportional to the intensity reaching the sensor. It has numerically been shown, that such a system is able to learn certain nonlinear mappings, like an XOR logical function [15] . This raises questions about the theoretical capabilities of such systems. Here we show, that a network with a Fourier encoding is able to learn arbitrary onedimensional functions, and that information should be encoded in the phase of the coherent wave, instead of the intensity, because intensity encodings can never be universal function approximators. This is promising for the further development of coherent optical computational systems, and gives suggestions, on how to best set up such a system. 
universal function approximators: 
 A function that fulfills the universal function approximation property can approximate any continuous function f (x) ∈ C(R n ). This means that for every f (x) ∈ C(R n ) there exists an g(x; θ) and θ such that max x∈K |f (x) -g(x; θ)| < ϵ for any ϵ > 0. θ are learnable parameters. Crucially this property does not determine how to calculate these learnable parameters, just that a combination exists, for which f and g are arbitrarily close everywhere in K ⊆ R m . Most of these theorems are used to describe artificial neural networks, and estimate bounds and limitations on the depth and width of the layers of deep neural networks [23] [24] [25] . These theorems give us a framework to analyze the theoretical capability of diffractive deep neural network and similar systems.The input of a diffractive system can be described as a vector of complex numbers, that represent the amplitude and phase of a wavefield at certain input positions.I =    I 1 e iψ1. . .EQUATIONThis input propagates through the system of diffractive plates, until it hits a photodiode at which point only the intensity, which is the square of the absolute value of the complex field, is measured.To further derive the capability of DDNNs, we need to take a look at the output of the system.The first result we need, is that with enough plates diffractive deep neural networks can perform a complex matrix multiplication (see [18] ). This allows us to write the propagation of waves through the diffractive system as a matrix multiplicationY = WI (2) =    A 1,1 e iϕ1,1 . . . A 1,n e iϕ1,n . . . . . . . . . A n,1 e iϕn,1 . . . A n,n e iϕn,n      I 1 e iψ1 . . .EQUATION.Secondly, the measurement of the output wave only measure the intensity of the light. We can thus calculate the output for a single pixel on the photodiode as follows:EQUATIONEQUATION, with δ = ϕ k,j + ψ k -ϕ i,j -ψ i . This results enables us to make further statements about the universal approximation capability of diffractive plates.Equation 5 lets us make statements about the nature of the input encoding that should be used. The first input encoding is the most natural one, encoding the information in the input intensity. The following result states that these networks are never universal function approximators. To prove this we will need the following result. Neural networks with one hidden layer, that is of the form y(x) = n i c i σ(Ax -b) with A, x ∈ R n , c i ∈ REQUATION, equation 5 reduces toEQUATIONwhich is a second degree polynomial in A i x i , since cos(ϕ k,j -ϕ i,j ) is not dependent on the input encoding. Thus no diffractive deep neural network with intensity encoding and no other nonlinearity other than the output diode can be an universal function approximator. This result holds for the onedimensional case, as well as the multidimensional case.Since encoding the information only in the intensity is insufficient to make the network an universal function approximator, the phase can be included in the encoding. Using an input encoding ofI =   I 1 e iψ1 . . .EQUATION, equation 5 reduces toEQUATIONThis is equivalent to a single layer neural network with fixed weights of 1 in the first layer and offsets that can be trained.EQUATIONHere c i are the weights in the last layer, x the input variable and x i biases at different positions.According to some results, networks of this type are arbitrary function approximators [27] for single valued functions, if σ is a even periodic continuous function. Since cos fullfills these requirements, this result holds for diffractive deep neural networks with phase encoding. However in numerical experiments we have been unable to sufficiently approximate functions that contained higher order frequencies, or where different from the family of cosine functions. This is possibly due to the finite number of output pixels.Instead of using only one input for each variable the signal can be encoded with different input weights w i . This method was used by Yildrim. M et. al. [28] to improve the performance of a diffractive system in image classification tasks on multiple datasets. Instead of learning the input weights however, we opted to simply use full integer increments to introduce higher frequencies into the system. The encoding for a onedimensional variable in the optical system is displayed in equation 11I =   I 1 e iψ1 . . .I n e iψn    =       e i2πx1e i4πx1 e i6πx1 . . .EQUATIONNote that all intensities are assumed to be 1. Using this encoding in equation 5 givesEQUATIONWith this encoding higher order frequencies exist and can be used to approximate desired output functions. Due to the notable similarities to the Fourier seriesEQUATION)we call this encoding Fourier encoding. Since the Fourier series is able to approximate a function arbitrarily well on an interval P , a diffractive neural network with this encoding should be able to do the same.However, in equation 12 the coefficients A i are all ≥ 0, meaning that we can only create positive coefficients. The Fourier series coefficients on the other hand are ∈ R. To create coefficients that can be negative, two photodiodes can be used, the first acting as a measurement device for the positive coefficients, and the second one capturing the negative coefficients. The output of these photodiodes can easily be subtracted in an analog electrical system. 
experimental results: 
 The following numerical and experimental results have been performed with a diffractive deep neural network consisting of 3 layers of 112 × 112 pixel on an area with a sidelength of 0.1792 mm corresponding to a pixel size of 16 µm. A laser with a wavelength of 781 nm focused in a Gaussian beam with a radius of 0.09 mm was used as the input source. The input was encoded on a plate with the same dimensions as the trainable plates. The experimental setup used a single reflective spatial light modulator and a mirror slightly angled to reflect the beam back to the SLM. The setup is similar to the one used in [28] . This system was implemented and trained with Tensorflow [29] .Fourier encodings with 1,2,5, 8,10,20,50 and 100 components have been trained to compare the influence of more complicated encodings on the capability of the system. Results for four different nonlinear functions are displayed in 2 It can be seen that the system is able to learn all four of the . This function needs even more Fourier coefficients to be well approximated. It can be seen, that only the system that takes negative output coefficients into account is able to approximate the function well. d) 0.2 if x ∈ [0.35, 0.65]. This function is the hardest one to approximate, probably because it is not smooth. In all cases, the network that takes negative coefficients into account gives a better approximation. functions. Higher frequencies are needed to approximate all functions. This is especially clear when comparing the sinusoidal functions with increasing frequencies. While 2 frequencies are enough to train a simple sinus function, the requirements get harder with increasing frequencies. It can also be seen, that the negative output values improve the results in all cases, and are necessary to learn functions with negative values ( see plot a) of figure 2 ). Overall the system with negative output values manages to learn all functions to a high degree of accuracy.These trained phase plates where tested in a physical system, and the output values where measured.For this a mirror and a single SLM was used that influenced the phase of the incoming wave. The output was measured with a CCD sensor and the two output regions where estimated and summed up. Since the output values form the camera sensor are not normed, a multiplicative offset, that corrects for the overall energy in the system and an additive offset that corrects for the base noise level in the camera sensor where manually chosen for all measured output values. The results are displayed in figure 3 The results confirm the numerical experiments. They show that it is possible Figure 3 : Real world experimental results. The functions for the plots a) to f) are the same was in figure 2 . It can be seen, that the measured functions match the numerical ones. Some noise is introduced during measurement.to train a diffractive deep neural network to perform arbitrary nonlinear functions. 
discussion: 
 We have shown, that a coherent optical system with diffractive plates, also called deep diffractive neural network, is capable of performing arbitrary nonlinear functions in one dimension. We utilized an encoding, that embeds the input variable into multiple phase inputs, that have different factors, that match those of the Fourier series. Furthermore we have shown a mathematical proof that a DDNN with only an intensity encoding can never be considered an universal function approximator.A pure phase encoding that does not introduce higher frequencies into the system has been shown to be theoretically sufficient for the system to be an universal function approximator, as long as ref [27] holds. However we have not been able to numerically train a real world system that purely relies on phase encoding to perform arbitrary nonlinear functions. This is potentially due to the finite size of the system. Using the Fourier encoding however made it possible to train any onedimensional function. A negative output region was useful for all trained functions, but made the biggest difference for functions that had negative values for obvious reasons.Using the results mentioned above should also be helpful when training DDNNs for different tasks, that are potentially more demanding than a onedimensional function. In general the information should not be encoded solely in the intensity of the incoming wave, to ensure cosine nonlinearities in the output. Furthermore a region with negative values has been shown to be important for allSubmitted to the Second Workshop on Machine Learning with New Compute Paradigms at NeurIPS (MLNCP 2024). Do not distribute. 
 
 Appendix:  



Reviews: review: title: title: Universal approximation capabilities of coherent diffractive systems
main_review: main_review: The manuscript “Universal approximation capabilities of coherent diffractive systems” analyzes the ability of a deep diffractive neural network (DDNN) to act as a universal function approximator. The authors claim that a DDNN system containing multiple phase plates and two output photodetectors is theoretically capable of learning any one-dimensional function. However, I do not agree with the authors’ claim.

It is well known that a single hidden layer feedforward network with a non-polynomial activation function can be a universal function approximator. However, this requires that the elements in both “weight” matrices can be set as any real value. The authors state (line 93-94) that Eq. 9 "is equivalent to a single layer neural network with fixed weights of 1 in the first layer and offsets that can be trained." Thus, since the (equivalent) first layer is not trainable, I do not see how the DDNN can be a universal approximator.  The authors are also unable to "demonstrate experimentally that they can approximate functions that contained higher order frequencies, or where different from the family of cosine functions."

Additionally, the weight matrix representing the wave’s propagation in Eq. 3 with elements $Ae^{i\phi}$ cannot be set as any arbitrary value. DDNNs consist of a series of free-space propagation and element-wise multiplication operations. The authors provide no proof that they can tune a DDNN to represent an arbitrary matrix. As an example, it is clearly not possible to represent a transformation with a matrix norm that is larger than 1, because this would imply that energy is added to the system during the transformation. 

Due to the flawed reasoning in the theory and lack of rigorous analysis, I cannot recommend this submission for acceptance to the workshop. 

Minor remarks:

- There are several grammar and punctuation mistakes throughout the paper (“wavefrong “ in line 33, unnecessary comma in lines 70, 73, etc). I encourage the authors to further proof-read their manuscript before submission.