Patent Publication Number: US-2021192342-A1

Title: Training of Photonic Neural Networks Through in situ Backpropagation

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     The present application claims the benefit of and priority to U.S. Provisional Patent Application No. 62/669,899 entitled “Training of Photonic Neural Networks Through in Situ Backpropagation”, filed May 10, 2018 and to U.S. Provisional Patent Application No. 62/783,992 entitled “Training of Photonic Neural Networks Through in Situ Backpropagation”, filed Dec. 21, 2018. The disclosure of U.S. Provisional Patent Application Ser. Nos. 62/669,899 and 62/783,992 are herein incorporated by reference in its entirety. 
    
    
     STATEMENT OF FEDERALLY SPONSORED RESEARCH 
     This invention was made with Government support under contract FA9550-17-1-0002 awarded by the Air Force Office of Scientific Research. The Government has certain rights in the invention. 
    
    
     FIELD OF THE INVENTION 
     The present invention generally relates to photonic neural networks and more specifically relates to training of photonic neural networks through in situ backpropagation. 
     BACKGROUND 
     Recently, integrated optics has gained interest as a hardware platform for implementing machine learning algorithms, including artificial neural networks (ANNs), which rely heavily on matrix-vector multiplications that may be done efficiently in photonic circuits. Artificial neural networks, and machine learning in general, are becoming ubiquitous for an impressively large number of applications. This has brought ANNs into the focus of research in not only computer science, but also electrical engineering, with hardware specifically suited to perform neural network operations actively being developed. There are significant efforts in constructing artificial neural network architectures using various electronic solid-state platforms, but ever since the conception of ANNs, a hardware implementation using optical signals has also been considered. Photonic implementations benefit from the fact that, due to the non-interacting nature of photons, linear operations—like the repeated matrix multiplications found in every neural network algorithm—can be performed in parallel, and at a lower energy cost, when using light as opposed to electrons. 
     Many implementations of photonic neural networks are trained using a model of the system simulated on a regular computer, but this can be inefficient for two reasons. First, this strategy depends entirely on the accuracy of the model representation of the physical system. Second, unless one is interested in deploying a large number of identical, fixed copies of the ANN, any advantage in speed or energy associated with using the photonic circuit is lost if the training must be done on a regular computer. Alternatively, training using a brute force, in situ computation of the gradient of the objective function has been proposed. However, this strategy involves sequentially perturbing each individual parameter of the circuit, which is highly inefficient for large systems. 
     SUMMARY OF THE INVENTION 
     Systems and methods for training photonic neural networks in accordance with embodiments of the invention are illustrated. One embodiment includes a method for training a set of one or more optical interference units (OIUs) of a photonic artificial neural network (ANN), wherein the method includes calculating a loss for an original input to the photonic ANN, computing an adjoint input based on the calculated loss, measuring intensities for a set of one or more phase shifters in the set of OIUs when the computed adjoint input and the original input are interfered with each other within the set of OIUs, computing a gradient from the measured intensities, and tuning phase shifters of the OIU based on the computed gradient. 
     In a further embodiment, computing the adjoint input includes sending the calculated loss through output ports of the ANN. 
     In still another embodiment, the intensities for each phase shifter of the set of phase shifters is measured after the phase shifter. 
     In a still further embodiment, the set of OIUs includes a mesh of controllable Mach-Zehnder interferometers (MZIs) integrated in a silicon photonic circuit. 
     In yet another embodiment, computing the adjoint input includes sending the calculated loss through output ports of the ANN. 
     In a yet further embodiment, the loss is the result of a mean squared cost function. 
     In another additional embodiment, the ANN is a feed-forward ANN. 
     In a further additional embodiment, the ANN is a recurrent neural network (RNN). 
     In another embodiment again, the ANN further includes a set of one or more activation units, wherein each OIU performs a linear operation and each activation unit performs a non-linear function on the input. 
     In a further embodiment again, computing the adjoint input includes linearizing at least one activation unit of the set of activation units prior to sending an input through the ANN. 
     In still yet another embodiment, the method further includes steps for performing dropout during training by shutting off channels in the activation units. 
     In a still yet further embodiment, non-linear functions of the ANN are performed using an electronic circuit. 
     In still another additional embodiment, each OIUs of the set of OIUs includes a same number of input ports and output ports. 
     Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the invention. A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings, which forms a part of this disclosure. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The description and claims will be more fully understood with reference to the following figures and data graphs, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention. 
         FIG. 1  illustrates an example of a schematic for an artificial neural network (ANN). 
         FIG. 2  illustrates an example of operations in an ANN. 
         FIG. 3  illustrates a schematic of a process for experimental measurement of gradient information. 
         FIG. 4  conceptually illustrates a process for experimental measurement of gradient information. 
         FIG. 5  illustrates a numerical demonstration of a time-reversal procedure. 
         FIG. 6  illustrates how a time-reversal interference technique could be performed for a layer embedded in a network. 
         FIG. 7  conceptually illustrates a process for using a time-reversal inference method to measure sensitivities. 
         FIG. 8  illustrates how a time-reversal inference technique can be applied without internal coherent detection and preparation. 
         FIG. 9  illustrates a schematic of a recurrent neural network (RNN). 
     
    
    
     DETAILED DFSCRIPTION 
     Turning now to the drawings, systems and methods in accordance with certain embodiments of the invention can be used to train photonic neural networks. In some embodiments, methods can compute the gradient of the cost function of a photonic artificial neural network (ANN) by use of only in situ intensity measurements. Processes in accordance with several embodiments of the invention physically implement the adjoint variable method (AVM). Furthermore, methods in accordance with a number of embodiments of the invention scale in constant time with respect to the number of parameters, which allows for backpropagation to be efficiently implemented in a hybrid opto-electronic network. Although many of the examples described herein are described with reference to a particular hardware implementation of a photonic ANN, one skilled in the art will recognize that methods and systems can be readily applied to other photonic platforms without departing from the heart of the invention. 
     Currently, there is no efficient protocol for the training of photonic neural networks, which is a crucial step for any machine learning application, and should ideally be performed on the same platform. Methods in accordance with a number of embodiments of the invention enable highly efficient, in situ training of a photonic neural network. In a variety of embodiments, adjoint methods may be used to derive the photonic analogue of the backpropagation algorithm, which is the standard method for computing gradients of conventional neural networks. Gradients in accordance with a number of embodiments of the invention may be obtained exactly by performing intensity measurements within the device. 
     Training protocols in accordance with many embodiments of the invention can greatly simplify the implementation of backpropagation. Beyond the training of photonic machine learning implementations, methods in accordance with some embodiments of the invention may also be of broader interest to experimental sensitivity analysis of photonic systems and the optimization of reconfigurable optics platforms, among other applications. 
     Photonic Neural Networks 
     In its most general case, a feed-forward ANN maps an input vector to an output vector via an alternating sequence of linear operations and element-wise nonlinear functions of the vectors, also called ‘activations’. A cost function,  , is defined over the outputs of the ANN and the matrix elements involved in the linear operations are tuned to minimize   over a number of training examples via gradient-based optimization. The ‘backpropagation algorithm’ is typically used to compute these gradients analytically by sequentially utilizing the chain rule from the output layer backwards to the input layer. 
     A photonic hardware platform implemented in accordance with certain embodiments of the invention is illustrated in  FIG. 1 . The boxed regions correspond to optical interference units (OIUs)  105  that perform a linear operation represented by the matrix Ŵ l . Each OIU can include a number of integrated phase shifters (e.g.,  110 ) illustrated as rounded shapes within each OIU. In many embodiments, integrated phase shifters can be used to control an OIU and train a network. Photonic hardware platform  100  also includes nonlinear activations  115  represented as f l (·). 
     In this example, photonic element  100  performs linear operations using optical interference units (OIUs). OIUs in accordance with several embodiments of the invention are meshes of controllable Mach-Zehnder interferometers (MZIs) integrated in a silicon photonic circuit. By tuning the phase shifters integrated in the MZIs, any unitary N×N operation on the input can be implemented, which finds applications both in classical and quantum photonics. In photonic ANNs in accordance with some embodiments of the invention. OIUs can be used for each linear matrix-vector multiplication. In certain embodiments, nonlinear activations can be performed using an electronic circuit, which involves measuring the optical state before activation, performing the nonlinear activation function on an electronic circuit such as a digital computer, and preparing the resulting optical state to be injected to the next stage of the ANN. 
     In the description of this example, the OIU is described by a number, N, of single-mode waveguide input ports coupled to the same number of single-mode output ports through a linear and lossless device. In certain embodiments, the device may also be extended to operate on differing numbers of inputs and outputs. OIUs in accordance with some embodiments of the invention implement directional propagation such that all power flows exclusively from the input ports to the output ports. In its most general form, devices implement the linear operation 
       ŴX in =Z out ,   (1)
 
     where X in  and Z out  are the modal amplitudes at the input and output ports, respectively, and Ŵ, or the transfer matrix, is the off-diagonal block of the system&#39;s full scattering matrix. 
     
       
         
           
             
               
                 
                   
                     ( 
                     
                       
                         
                           
                             X 
                             out 
                           
                         
                       
                       
                         
                           
                             Z 
                             out 
                           
                         
                       
                     
                     ) 
                   
                   = 
                   
                     
                       ( 
                       
                         
                           
                             0 
                           
                           
                             
                               
                                 W 
                                 ^ 
                               
                               T 
                             
                           
                         
                         
                           
                             
                               W 
                               ^ 
                             
                           
                           
                             0 
                           
                         
                       
                       ) 
                     
                      
                     
                       
                         ( 
                         
                           
                             
                               
                                 X 
                                 in 
                               
                             
                           
                           
                             
                               
                                 Z 
                                 in 
                               
                             
                           
                         
                         ) 
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
     The diagonal blocks are zero because forward-only propagation is assumed, while the off-diagonal blocks are the transpose of each other because a reciprocal system is assumed. Z in  and X out  correspond to the input and output modal amplitudes, respectively, if the device were run in reverse, i.e., sending a signal in from the output ports. 
     Operation and Training with Backpropagation 
     A key requirement for the utility of any ANN platform is the ability to train the network using algorithms such as error backpropagation. Such training typically demands significant computational time and resources and it is generally desirable for error backpropagation to be implemented on the same platform. 
     The operation and gradient computation in an ANN in accordance with an embodiment of the invention is illustrated in  FIG. 2 . In this example, propagation through a square cell (e.g.,  215 ) corresponds to matrix multiplication, while propagation through a rounded region (e.g.,  220 ) corresponds to activation. The ⊙ (e.g.,  225 ) indicates clement-wise vector multiplication. 
     The top row  205  corresponds to the forward propagation steps in the operation and training of an ANN. For the forward propagation step, processes in accordance with numerous embodiments of the invention begin with an initial input to the system, X 0 , and perform a linear operation on this input using an OIU represented by the matrix Ŵ 1 . In several embodiments, processes can apply an element-wise nonlinear activation, f l (·), on the outputs, giving the input to the next layer. This process can be repeated for each layer l until the output layer, L. Written compactly, for l=1 . . . L 
         X   l   =f   l ( Ŵ   l   X   l−1 )≡ f   l ( Z   l ).   (3)
 
     Once forward propagation is completed, a cost function is computed to train the network. Cost function   is an explicit function of the outputs from the last layer,  = (X L ). To train the network, cost functions can be minimized with respect to the linear operators, Ŵ 1 , which may be adjusted by tuning the integrated phase shifters within the OIUs in accordance with some embodiments of the invention. In a variety of embodiments, training methods can operate without resorting to an external model of the system, while allowing for the tuning of each parameter to be done in parallel, therefore scaling significantly better with respect to the number of parameters when it compared to a brute force gradient computation method. 
     Once a cost (or loss) function is computed, backward propagation is performed to adjust the model based on the computed loss. Bottom row  210  of  FIG. 2  illustrates the backward propagation steps. In a number of embodiments, backpropagation processes can derive an expression for the gradient of the cost function with respect to the permittivities of the phase shifters in the OIUs. In the following, ε l  the permittivity of a single phase shifter in layer l, as the same derivation holds for each of the phase shifters present in that layer. Note that Ŵ l  has an explicit dependence on ε l , but all field components in the subsequent layers also depend implicitly on ε l . 
     As a demonstration, a mean squared cost function is calculated 
         =1/2( X   L   −T ) † ( X   L   −T ),   (4)
 
     where T is a complex-valued target vector corresponding to the desired output of a system given input X 0 . 
     Starting from the last layer in the circuit, the derivatives of the cost function with respect to the permittivity of the phase shifters in the last layer ε L  are given by 
     
       
         
           
             
               
                 
                   
                     t 
                      
                     
                       
                         d 
                          
                         
                             
                         
                          
                         ℒ 
                       
                       
                         d 
                          
                         
                             
                         
                          
                         
                           ɛ 
                           L 
                         
                       
                     
                   
                   = 
                   
                      
                      
                     
                       { 
                       
                         
                           
                             ( 
                             
                               
                                 X 
                                 L 
                               
                               - 
                               T 
                             
                             ) 
                           
                           † 
                         
                          
                         
                           
                             dX 
                             L 
                           
                           
                             d 
                              
                             
                                 
                             
                              
                             
                               ɛ 
                               L 
                             
                           
                         
                       
                       } 
                     
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
             
               
                 
                   = 
                   
                      
                      
                     
                       { 
                       
                         
                           
                             ( 
                             
                               
                                 Γ 
                                 L 
                               
                               ⊙ 
                               
                                 
                                   f 
                                   L 
                                   ′ 
                                 
                                  
                                 
                                   ( 
                                   
                                     Z 
                                     L 
                                   
                                   ) 
                                 
                               
                             
                             ) 
                           
                           T 
                         
                          
                         
                           
                             d 
                              
                             
                               
                                 W 
                                 ^ 
                               
                               L 
                             
                           
                           
                             d 
                              
                             
                                 
                             
                              
                             
                               ɛ 
                               L 
                             
                           
                         
                          
                         
                           X 
                           
                             L 
                             - 
                             1 
                           
                         
                       
                       } 
                     
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       = 
                       _ 
                     
                      
                     
                        
                        
                       
                         { 
                         
                           
                             δ 
                             L 
                             T 
                           
                            
                           
                             
                               d 
                                
                               
                                 
                                   W 
                                   ^ 
                                 
                                 L 
                               
                             
                             
                               d 
                                
                               
                                   
                               
                                
                               
                                 ɛ 
                                 L 
                               
                             
                           
                            
                           
                             X 
                             
                               L 
                               - 
                               1 
                             
                           
                         
                         } 
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
     where ⊙ is element-wise vector multiplication, defined such that, for vectors a and b, the i-th element of the vector a⊙b is given by a i b i .  {·} gives the real part, f l ′(·) is the derivative of the lth layer activation function with respect to its (complex) argument. The vector δ L ≡Γ L ⊙f L ′ is defined in terms of the error vector Γ L ≡(X L −T)*. 
     For any layer l&lt;L, the chain rule can be used to perform a recursive calculation of the gradients. 
     
       
         
           
             
               
                 
                   
                     Γ 
                     l 
                   
                   = 
                   
                     
                       
                         W 
                         ^ 
                       
                       
                         l 
                         + 
                         1 
                       
                       T 
                     
                      
                     
                       δ 
                       
                         l 
                         + 
                         1 
                       
                     
                   
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
             
               
                 
                   
                     δ 
                     l 
                   
                   = 
                   
                     
                       Γ 
                       l 
                     
                     ⊙ 
                     
                       
                         f 
                         l 
                         ′ 
                       
                        
                       
                         ( 
                         
                           Z 
                           l 
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       d 
                        
                       
                           
                       
                        
                       ℒ 
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         l 
                       
                     
                   
                   = 
                   
                      
                      
                     
                       
                         { 
                         
                           
                             δ 
                             l 
                             T 
                           
                            
                           
                             
                               d 
                                
                               
                                 
                                   W 
                                   ^ 
                                 
                                 l 
                               
                             
                             
                               d 
                                
                               
                                   
                               
                                
                               
                                 ɛ 
                                 l 
                               
                             
                           
                            
                           
                             X 
                             
                               l 
                               - 
                               1 
                             
                           
                         
                         } 
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   10 
                   ) 
                 
               
             
           
         
       
     
     This process is illustrated in the backward propagation of the second row  210  of  FIG. 2 , which computes the δ l  vectors sequentially from the output layer to the input layer. The computation of δ l  requires performing the operation Γ l =Ŵ l+1   T δ l+1 , which corresponds physically to sending δ l+1  into the output end of the OIU in layer l+1. In this way, processes in accordance with many embodiments of the invention ‘backpropagate’ the vectors δ l  and Γ 1  physically through the entire circuit. 
     In some embodiments, training a photonic ANN relies on the ability to create arbitrary complex inputs. Processes in accordance with several embodiments of the invention require an integrated intensity detection scheme to occur in parallel and with virtually no loss. In numerous embodiments, this can be implemented by integrated, transparent photo-detectors. 
     The problem of overfitting is one that can be addressed by ‘regularization’ in any practical realization of a neural network. Photonic ANNs in accordance with various embodiments of the invention provide a convenient alternative approach to regularization based on ‘dropout’. In various embodiments, in a dropout procedure, certain nodes can be probabilistically and temporarily ‘deleted’ from the network during train time, which has the effect of forcing the network to find alternative paths to solve the problem at hand. This has a strong regularization effect and has become popular in conventional ANNs. Dropout in accordance with some embodiments of the invention can be implemented in the photonic ANN by ‘shutting off’ channels in the activation functions during training. Specifically, at each time step and for each layer l and element i, one may set f l (Z l )=0 with some fixed probability. 
     Specific processes for training photonic neural networks in accordance with embodiments of the invention are described above; however, one skilled in the art will recognize that any number of processes can be utilized as appropriate to the requirements of specific applications in accordance with embodiments of the invention. 
     For example, the discussion above assumes that the functions f l (·) are holomorphic. For each element of input Z l , labeled z, this means that the derivative of f l (z) with respect to its complex argument is well defined. In other words, the derivative 
     
       
         
           
             
               
                 
                   
                     
                       df 
                       l 
                     
                     dz 
                   
                   = 
                   
                     
                       lim 
                       
                         Δ2 
                         → 
                         0 
                       
                     
                      
                     
                       
                         
                           
                             f 
                             l 
                           
                            
                           
                             ( 
                             
                               z 
                               + 
                               
                                 Δ 
                                  
                                 
                                     
                                 
                                  
                                 z 
                               
                             
                             ) 
                           
                         
                         - 
                         
                           
                             f 
                             l 
                           
                            
                           
                             ( 
                             
                               z 
                               - 
                               
                                 Δ 
                                  
                                 
                                     
                                 
                                  
                                 z 
                               
                             
                             ) 
                           
                         
                       
                       
                         2 
                          
                         Δ 
                          
                         
                             
                         
                          
                         z 
                       
                     
                   
                 
               
               
                 
                   ( 
                   11 
                   ) 
                 
               
             
           
         
       
     
     does not depend on the direction that Δz approaches 0 in the complex plane. 
     In numerous embodiments, the backpropagation derivation can be extended to non-holomorphic activation functions. In the backpropagation algorithm, the change in the mean-squared loss function with respect to the permittivity of a phase shifter in the last layer OIU as written in Eq. (5) is 
     
       
         
           
             
               
                 
                   
                     
                       d 
                        
                       
                           
                       
                        
                       ℒ 
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         L 
                       
                     
                   
                   = 
                   
                      
                      
                     
                       { 
                       
                         
                           Γ 
                           L 
                           T 
                         
                          
                         
                           
                             dX 
                             L 
                           
                           
                             d 
                              
                             
                                 
                             
                              
                             
                               ɛ 
                               L 
                             
                           
                         
                       
                       } 
                     
                   
                 
               
               
                 
                   ( 
                   12 
                   ) 
                 
               
             
           
         
       
     
     Where the error vector was defined as Γ L ≡(X L −T)* for simplicity and X L =f L (Ŵ L X L−1 ) is the output of the final layer. 
     To evaluate this expression for non-holomorphic activation functions, f L (Z) and its argument can be split into their real and imaginary parts 
         f   L ( Z )= u (α,β)+ iv (α,β),   (13)
 
     where i is the imaginary unit and α and β are the real and imaginary parts of Z L , respectively. 
     Evaluating 
     
       
         
           
             
               
                 dX 
                 L 
               
               
                 d 
                  
                 
                     
                 
                  
                 
                   ɛ 
                   L 
                 
               
             
             = 
             
               
                 
                   df 
                   L 
                 
                  
                 
                   ( 
                   Z 
                   ) 
                 
               
               
                 d 
                  
                 
                     
                 
                  
                 
                   ɛ 
                   L 
                 
               
             
           
         
       
     
     gives the following via the chain rule 
     
       
         
           
             
               
                 
                   
                     
                       df 
                       
                         d 
                          
                         
                             
                         
                          
                         ɛ 
                       
                     
                     = 
                     
                       
                         
                           du 
                           
                             d 
                              
                             
                                 
                             
                              
                             α 
                           
                         
                         ⊙ 
                         
                           
                             d 
                              
                             
                                 
                             
                              
                             α 
                           
                           
                             d 
                              
                             
                                 
                             
                              
                             ɛ 
                           
                         
                       
                       + 
                       
                         
                           du 
                           
                             d 
                              
                             
                                 
                             
                              
                             β 
                           
                         
                         ⊙ 
                         
                           
                             d 
                              
                             
                                 
                             
                              
                             β 
                           
                           
                             d 
                              
                             
                                 
                             
                              
                             ɛ 
                           
                         
                       
                       + 
                       
                         i 
                          
                         
                           
                             dv 
                             
                               d 
                                
                               
                                   
                               
                                
                               α 
                             
                           
                           ⊙ 
                           
                             
                               d 
                                
                               
                                   
                               
                                
                               α 
                             
                             
                               d 
                                
                               
                                   
                               
                                
                               ɛ 
                             
                           
                         
                       
                       + 
                       
                         i 
                          
                         
                           
                             dv 
                             
                               d 
                                
                               
                                   
                               
                                
                               β 
                             
                           
                           ⊙ 
                           
                             
                               d 
                                
                               
                                   
                               
                                
                               β 
                             
                             
                               d 
                                
                               
                                   
                               
                                
                               ɛ 
                             
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   14 
                   ) 
                 
               
             
           
         
       
     
     where be layer index has been dropped for simplicity. Here, terms of the form 
     
       
         
           
             dx 
             dy 
           
         
       
     
     correspond to element-wise differentiation of the vector x with respect to the vector y. For example, the i-th element of vector 
     
       
         
           
             dx 
             dy 
           
         
       
     
     is given by 
     
       
         
           
             
               
                 dx 
                 i 
               
               
                 dy 
                 i 
               
             
             . 
           
         
       
     
     Now, inserting into Eq. (12): 
     
       
         
           
             
               
                 
                   
                     
                       d 
                        
                       
                           
                       
                        
                       ℒ 
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         L 
                       
                     
                   
                   = 
                   
                      
                      
                     
                       { 
                       
                         
                           
                             Γ 
                             L 
                             T 
                           
                           ⊙ 
                           
                             
                               ( 
                               
                                 
                                   du 
                                   
                                     d 
                                      
                                     
                                         
                                     
                                      
                                     α 
                                   
                                 
                                 + 
                                 
                                   i 
                                    
                                   
                                     dv 
                                     
                                       d 
                                        
                                       
                                           
                                       
                                        
                                       α 
                                     
                                   
                                 
                               
                               ) 
                             
                             T 
                           
                         
                          
                         
                           
                             d 
                              
                             
                                 
                             
                              
                             α 
                           
                           
                             d 
                              
                             
                                 
                             
                              
                             
                               ɛ 
                               L 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   15 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       
                         + 
                         
                           
                             Γ 
                             L 
                             T 
                           
                           ⊙ 
                           
                             
                               ( 
                               
                                 
                                   du 
                                   
                                     d 
                                      
                                     
                                         
                                     
                                      
                                     β 
                                   
                                 
                                 + 
                                 
                                   i 
                                    
                                   
                                     dv 
                                     
                                       d 
                                        
                                       
                                           
                                       
                                        
                                       β 
                                     
                                   
                                 
                               
                               ) 
                             
                             T 
                           
                         
                       
                        
                       
                         
                           d 
                            
                           
                               
                           
                            
                           β 
                         
                         
                           d 
                            
                           
                               
                           
                            
                           
                             ɛ 
                             L 
                           
                         
                       
                     
                     } 
                   
                   . 
                 
               
               
                 
                   ( 
                   16 
                   ) 
                 
               
             
           
         
       
     
     The real and imaginal pairs of Γ L  are defined as Γ R  and Γ I , respectively. Inserting the definitions of α and β in terms of Ŵ L  and X L−1  and doing some algebra: 
     
       
         
           
             
               
                 
                   
                     
                       d 
                        
                       
                           
                       
                        
                       ℒ 
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         L 
                       
                     
                   
                   = 
                   
                      
                      
                     
                       { 
                       
                         
                           
                             Γ 
                             L 
                             T 
                           
                           ⊙ 
                           
                             
                               ( 
                               
                                 du 
                                 
                                   d 
                                    
                                   
                                       
                                   
                                    
                                   α 
                                 
                               
                               ) 
                             
                             T 
                           
                         
                          
                         
                           
                             d 
                              
                             
                               
                                 W 
                                 ^ 
                               
                               L 
                             
                           
                           
                             d 
                              
                             
                                 
                             
                              
                             
                               ɛ 
                               L 
                             
                           
                         
                          
                         
                           X 
                           
                             L 
                             - 
                             1 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   17 
                   ) 
                 
               
             
             
               
                 
                   
                     - 
                     
                       
                         ( 
                         
                           
                             Γ 
                             1 
                           
                           ⊙ 
                           
                             dv 
                             
                               d 
                                
                               
                                   
                               
                                
                               α 
                             
                           
                         
                         ) 
                       
                       T 
                     
                   
                    
                   
                     
                       d 
                        
                       
                         
                           W 
                           ^ 
                         
                         L 
                       
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         L 
                       
                     
                   
                    
                   
                     X 
                     
                       L 
                       - 
                       1 
                     
                   
                 
               
               
                 
                   ( 
                   18 
                   ) 
                 
               
             
             
               
                 
                   
                     - 
                     
                       
                         i 
                          
                         
                           ( 
                           
                             
                               Γ 
                               R 
                             
                             ⊙ 
                             
                               du 
                               
                                 d 
                                  
                                 
                                     
                                 
                                  
                                 β 
                               
                             
                           
                           ) 
                         
                       
                       T 
                     
                   
                    
                   
                     
                       d 
                        
                       
                         
                           W 
                           ^ 
                         
                         L 
                       
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         L 
                       
                     
                   
                    
                   
                     X 
                     
                       L 
                       - 
                       1 
                     
                   
                 
               
               
                 
                   ( 
                   19 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       
                         + 
                         
                           
                             i 
                              
                             
                               ( 
                               
                                 
                                   Γ 
                                   1 
                                 
                                 ⊙ 
                                 
                                   dv 
                                   
                                     d 
                                      
                                     
                                         
                                     
                                      
                                     β 
                                   
                                 
                               
                               ) 
                             
                           
                           T 
                         
                       
                        
                       
                         
                           d 
                            
                           
                             
                               W 
                               ^ 
                             
                             L 
                           
                         
                         
                           d 
                            
                           
                               
                           
                            
                           
                             ɛ 
                             L 
                           
                         
                       
                        
                       
                         X 
                         
                           L 
                           - 
                           1 
                         
                       
                     
                     } 
                   
                   . 
                 
               
               
                 
                   ( 
                   20 
                   ) 
                 
               
             
           
         
       
     
     Finally, the expression simplifies to 
     
       
         
           
             
               
                 
                   
                     
                       d 
                        
                       
                           
                       
                        
                       ℒ 
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         L 
                       
                     
                   
                   = 
                   
                      
                      
                     
                       { 
                       
                         [ 
                         
                           
                             Γ 
                             R 
                           
                           ⊙ 
                           
                             ( 
                             
                               
                                 du 
                                 
                                   d 
                                    
                                   
                                       
                                   
                                    
                                   α 
                                 
                               
                               - 
                               
                                 i 
                                  
                                 
                                   du 
                                   
                                     d 
                                      
                                     
                                         
                                     
                                      
                                     β 
                                   
                                 
                               
                             
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   21 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       
                         
                           
                             + 
                             
                               
                                 Γ 
                                 1 
                               
                               ⊙ 
                               
                                 ( 
                                 
                                   
                                     - 
                                     
                                       dv 
                                       
                                         d 
                                          
                                         
                                             
                                         
                                          
                                         α 
                                       
                                     
                                   
                                   + 
                                   
                                     i 
                                      
                                     
                                       dv 
                                       
                                         d 
                                          
                                         
                                             
                                         
                                          
                                         β 
                                       
                                     
                                   
                                 
                                 ) 
                               
                             
                           
                           ] 
                         
                         T 
                       
                        
                       
                         
                           d 
                            
                           
                             
                               W 
                               ^ 
                             
                             L 
                           
                         
                         
                           d 
                            
                           
                               
                           
                            
                           
                             ɛ 
                             L 
                           
                         
                       
                        
                       
                         X 
                         
                           L 
                           - 
                           1 
                         
                       
                     
                     } 
                   
                   . 
                 
               
               
                 
                   ( 
                   22 
                   ) 
                 
               
             
           
         
       
     
     As a check, if the conditions for f L (Z) to be holomorphic are set, namely 
     
       
         
           
             
               
                 
                   
                     
                       du 
                       
                         d 
                          
                         
                             
                         
                          
                         α 
                       
                     
                     = 
                     
                       dv 
                       
                         d 
                          
                         
                             
                         
                          
                         β 
                       
                     
                   
                   , 
                   
                       
                   
                    
                   
                     
                       and 
                        
                       
                           
                       
                        
                       
                         du 
                         
                           d 
                            
                           
                               
                           
                            
                           β 
                         
                       
                     
                     = 
                     
                       - 
                       
                         dv 
                         
                           d 
                            
                           
                               
                           
                            
                           α 
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   23 
                   ) 
                 
               
             
           
         
       
     
     Eq. (20) simplifies to 
     
       
         
           
             
               
                 
                   
                     
                       d 
                        
                       
                           
                       
                        
                       ℒ 
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         L 
                       
                     
                   
                   = 
                   
                      
                      
                     
                       { 
                       
                         [ 
                         
                           
                             
                               Γ 
                               R 
                             
                             ⊙ 
                             
                               ( 
                               
                                 
                                   du 
                                   
                                     d 
                                      
                                     
                                         
                                     
                                      
                                     α 
                                   
                                 
                                 + 
                                 
                                   i 
                                    
                                   
                                     dv 
                                     
                                       d 
                                        
                                       
                                           
                                       
                                        
                                       α 
                                     
                                   
                                 
                               
                               ) 
                             
                           
                           + 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   24 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       
                         
                           
                             Γ 
                             I 
                           
                           ⊙ 
                           
                             ( 
                             
                               
                                 - 
                                 
                                   dv 
                                   
                                     d 
                                      
                                     
                                         
                                     
                                      
                                     α 
                                   
                                 
                               
                               + 
                               
                                 i 
                                  
                                 
                                   du 
                                   
                                     d 
                                      
                                     
                                         
                                     
                                      
                                     α 
                                   
                                 
                               
                             
                             ) 
                           
                         
                         ] 
                       
                       T 
                     
                      
                     
                       
                         d 
                          
                         
                           
                             W 
                             ^ 
                           
                           L 
                         
                       
                       
                         d 
                          
                         
                             
                         
                          
                         
                           ɛ 
                           L 
                         
                       
                     
                      
                     
                       X 
                       
                         L 
                         - 
                         1 
                       
                     
                   
                   } 
                 
               
               
                 
                   ( 
                   25 
                   ) 
                 
               
             
             
               
                 
                   = 
                   
                      
                      
                     
                       { 
                       
                         
                           
                             [ 
                             
                               
                                 Γ 
                                 L 
                               
                               ⊙ 
                               
                                 ( 
                                 
                                   
                                     du 
                                     
                                       d 
                                        
                                       
                                           
                                       
                                        
                                       α 
                                     
                                   
                                   + 
                                   
                                     i 
                                      
                                     
                                       dv 
                                       
                                         d 
                                          
                                         
                                             
                                         
                                          
                                         α 
                                       
                                     
                                   
                                 
                                 ) 
                               
                             
                             ] 
                           
                           T 
                         
                          
                         
                           
                             d 
                              
                             
                               
                                 W 
                                 ^ 
                               
                               L 
                             
                           
                           
                             d 
                              
                             
                                 
                             
                              
                             
                               ɛ 
                               L 
                             
                           
                         
                          
                         
                           X 
                           
                             L 
                             - 
                             1 
                           
                         
                       
                       } 
                     
                   
                 
               
               
                 
                   ( 
                   26 
                   ) 
                 
               
             
             
               
                 
                   = 
                   
                      
                      
                     
                       { 
                       
                         
                           
                             [ 
                             
                               
                                 Γ 
                                 L 
                               
                               ⊙ 
                               
                                 
                                   f 
                                   l 
                                   ′ 
                                 
                                  
                                 
                                   ( 
                                   
                                     Z 
                                     L 
                                   
                                   ) 
                                 
                               
                             
                             ] 
                           
                           T 
                         
                          
                         
                           
                             d 
                              
                             
                               
                                 W 
                                 ^ 
                               
                               L 
                             
                           
                           
                             d 
                              
                             
                                 
                             
                              
                             
                               ɛ 
                               L 
                             
                           
                         
                          
                         
                           X 
                           
                             L 
                             - 
                             1 
                           
                         
                       
                       } 
                     
                   
                 
               
               
                 
                   ( 
                   27 
                   ) 
                 
               
             
             
               
                 
                   = 
                   
                      
                      
                     
                       { 
                       
                         
                           δ 
                           L 
                           T 
                         
                          
                         
                           
                             d 
                              
                             
                               
                                 W 
                                 ^ 
                               
                               L 
                             
                           
                           
                             d 
                              
                             
                                 
                             
                              
                             
                               ɛ 
                               L 
                             
                           
                         
                          
                         
                           X 
                           
                             L 
                             - 
                             1 
                           
                         
                       
                       } 
                     
                   
                 
               
               
                 
                   ( 
                   28 
                   ) 
                 
               
             
           
         
       
     
     as before. 
     This derivation may be similarly extended to any layer l in the network. For holomorphic activation functions, whereas the δ vectors were defined as 
       δ l =Γ l   ⊙f   l ′( Z   l ),   (29)
 
     for non-holomorphic activation functions, the respective definition is 
     
       
         
           
             
               
                 
                   
                     
                       δ 
                       l 
                     
                     = 
                     
                       
                         
                           Γ 
                           R 
                         
                         ⊙ 
                         
                           ( 
                           
                             
                               du 
                               
                                 d 
                                  
                                 
                                     
                                 
                                  
                                 α 
                               
                             
                             - 
                             
                               i 
                                
                               
                                 du 
                                 
                                   d 
                                    
                                   
                                       
                                   
                                    
                                   β 
                                 
                               
                             
                           
                           ) 
                         
                       
                       + 
                       
                         
                           Γ 
                           I 
                         
                         ⊙ 
                         
                           ( 
                           
                             
                               - 
                               
                                 dv 
                                 
                                   d 
                                    
                                   
                                       
                                   
                                    
                                   α 
                                 
                               
                             
                             + 
                             
                               i 
                                
                               
                                 dv 
                                 
                                   d 
                                    
                                   
                                       
                                   
                                    
                                   β 
                                 
                               
                             
                           
                           ) 
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   30 
                   ) 
                 
               
             
           
         
       
     
     where Γ R  and Γ I  are the respective real and imaginary parts of Γ l , u and v are the real and imaginary parts of f l (·), and α and β are the real and imaginary parts of Z l , respectively. 
     This can be written more simply as 
     
       
         
           
             
               
                 
                   
                     δ 
                     l 
                   
                   = 
                   
                     
                        
                        
                       
                         { 
                         
                           
                             Γ 
                             l 
                           
                           ⊙ 
                           
                             df 
                             
                               d 
                                
                               
                                   
                               
                                
                               α 
                             
                           
                         
                         } 
                       
                     
                     - 
                     
                       i 
                        
                       
                           
                       
                        
                        
                        
                       
                         
                           { 
                           
                             
                               Γ 
                               l 
                             
                             ⊙ 
                             
                               df 
                               
                                 d 
                                  
                                 
                                     
                                 
                                  
                                 β 
                               
                             
                           
                           } 
                         
                         . 
                       
                     
                   
                 
               
               
                 
                   ( 
                   31 
                   ) 
                 
               
             
           
         
       
     
     In polar coordinates where Z=r exp (iϕ) and f=f(r, ϕ), this equation becomes 
     
       
         
           
             
               
                 
                   
                     
                       δ 
                       l 
                     
                     = 
                     
                       exp 
                        
                       
                           
                       
                        
                       
                         ( 
                         
                           
                             - 
                             i 
                           
                            
                           
                               
                           
                            
                           φ 
                         
                         ) 
                       
                        
                       
                         ( 
                         
                           
                              
                              
                             
                               { 
                               
                                 
                                   Γ 
                                   l 
                                 
                                 ⊙ 
                                 
                                   df 
                                   
                                     d 
                                      
                                     
                                         
                                     
                                      
                                     r 
                                   
                                 
                               
                               } 
                             
                           
                           - 
                           
                             i 
                              
                             
                                 
                             
                              
                              
                              
                             
                               { 
                               
                                 
                                   
                                     Γ 
                                     l 
                                   
                                   ⊙ 
                                   
                                     1 
                                     r 
                                   
                                 
                                  
                                 
                                   df 
                                   
                                     d 
                                      
                                     
                                         
                                     
                                      
                                     φ 
                                   
                                 
                               
                               } 
                             
                           
                         
                         ) 
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   32 
                   ) 
                 
               
             
           
         
       
     
     where all operations are element-wise. 
     Gradient Computation 
     Computing gradient terms of the form 
     
       
         
           
             
                
                
               
                 { 
                 
                   
                     δ 
                     l 
                     T 
                   
                    
                   
                     
                       d 
                        
                       
                         
                           W 
                           ^ 
                         
                         l 
                       
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         l 
                       
                     
                   
                    
                   
                     X 
                     
                       l 
                       - 
                       1 
                     
                   
                 
                 } 
               
             
             , 
           
         
       
     
     which contain derivatives with respect to permittivity of the phase shifters in the OIUs, can be a crucial step in training an ANN. In certain embodiments, gradients can be expressed as the solution to an electromagnetic adjoint problem. 
     OIUs used to implement the matrix Ŵ l , relating the complex mode amplitudes of input and output ports, can be described using first-principles electrodynamics. This can allow for the gradient to be computed with respect to each ε l , as these are the physically adjustable parameters in the system in accordance with some embodiments of the invention. Assuming a source at frequency ω, at steady state, Maxwell&#39;s equations take the form 
       [{circumflex over (∇)}×{circumflex over (∇)}×− k    0   2 {circumflex over (ε)} r ] e =−i ωμ   0   ,j,    (33)
 
     which can be written more succinctly as 
         Â (ε r ) e=b.    (34)
 
     Here, {circumflex over (ε)} r  describes the spatial distribution of the relative permittivity (ε r ), k 0 =ω 2 /c 2  is the free-space wavenumber, e is the electric field distribution, j is the electric current density, and Â=Â T  due to Lorentz reciprocity. Eq. (34) is the starting point of the finite-difference frequency-domain (FDFD) simulation technique, where it is discretized on a spatial grid, and the electric field e is solved given a particular permittivity distribution, ε r , and source, b. 
     To relate this formulation to the transfer matrix Ŵ, source terms b i , i ∈ 1 . . . 2N can be defined, which correspond to a source placed in one of the input or output ports. In this example, it is assumed that there are a total of N input and N output waveguides. The spatial distribution of the source term, b i , matches the mode of the i-th single-mode waveguide. Thus, the electric field amplitude in port i is given by b i   T e, and a relationship can be established between e and X in , as 
       X in,i =b i   T e   (35)
 
     for i=1 . . . N over the input port indices, where X in,i  is the i-th component of X in . Or more compactly, 
       X in ≡{circumflex over (P)} in e,   (36)
 
     Similarly, 
         Z   out,i   =b   i+N   T   e    (37)
 
     for i+N=(N+1) . . . 2N over the output port indices, or, 
       Z out ≡{circumflex over (P)} out e,   (38)
 
     and, with this notation, Eq. (1) becomes 
       Ŵ{circumflex over (P)} in e={circumflex over (P)} out e   (39)
 
     Based on the above, the cost function gradient in Eq. (10) can be evaluated. In particular, Eqs. (34) and (39), 
     
       
         
           
             
               
                 
                   
                     
                       d 
                        
                       
                           
                       
                        
                       ℒ 
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         l 
                       
                     
                   
                   = 
                   
                     
                       - 
                        
                     
                      
                     
                       
                         { 
                         
                           
                             δ 
                             l 
                             T 
                           
                            
                           
                             
                               P 
                               ^ 
                             
                             out 
                           
                            
                           
                             
                               A 
                               ^ 
                             
                             
                               - 
                               1 
                             
                           
                            
                           
                             
                               d 
                                
                               
                                 A 
                                 ^ 
                               
                             
                             
                               d 
                                
                               
                                   
                               
                                
                               
                                 ɛ 
                                 l 
                               
                             
                           
                            
                           
                             
                               A 
                               ^ 
                             
                             
                               - 
                               1 
                             
                           
                            
                           
                             b 
                             
                               ϰ 
                               , 
                               
                                 l 
                                 - 
                                 1 
                               
                             
                           
                         
                         } 
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   40 
                   ) 
                 
               
             
           
         
       
     
     Here b x,l−1 is the modal source profile that creates the input field amplitudes X l−1  at the input ports. 
     The key insight of the adjoint variable method is that the expression can be interpreted as an operation in the field solutions of two electromagnetic simulations, which can be referred to as the ‘original’ (og) and the ‘adjoint’ (aj) 
         Âe   og   =b   x,l−1    (41)
 
       Âe aj ={circumflex over (P)} out    T δ,   (42)
 
     using the symmetric property of Â. 
     Eq. (40) can now be expressed in a compact form as 
     
       
         
           
             
               
                 
                   
                     
                       d 
                        
                       
                           
                       
                        
                       ℒ 
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         l 
                       
                     
                   
                   = 
                   
                     
                       - 
                        
                     
                      
                     
                       
                         { 
                         
                           
                             e 
                             aj 
                             T 
                           
                            
                           
                             
                               d 
                                
                               
                                 A 
                                 ^ 
                               
                             
                             
                               d 
                                
                               
                                   
                               
                                
                               
                                 ɛ 
                                 l 
                               
                             
                           
                            
                           
                             e 
                             og 
                           
                         
                         } 
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   43 
                   ) 
                 
               
             
           
         
       
     
     If it is assumed that this phase shifter spans a set of points, r ϕ  in the system, then, from Eq. (33). 
     
       
         
           
             
               
                 
                   
                     
                       
                         d 
                          
                         
                           A 
                           ^ 
                         
                       
                       
                         d 
                          
                         
                             
                         
                          
                         
                           ɛ 
                           l 
                         
                       
                     
                     = 
                     
                       
                         - 
                         
                           k 
                           0 
                           2 
                         
                       
                        
                       
                         
                           ∑ 
                           
                             
                               r 
                               ′ 
                             
                              
                             ϵ 
                              
                             
                                 
                             
                              
                             
                               r 
                               φ 
                             
                           
                           
                               
                           
                         
                          
                         
                             
                         
                          
                         
                           
                             δ 
                             ^ 
                           
                           
                             r 
                             , 
                             
                               r 
                               ′ 
                             
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   44 
                   ) 
                 
               
             
           
         
       
     
     where {circumflex over (δ)} r,r′  is the Kronecker delta. 
     Inserting this into Eq. (43), the gradient is given by the overlap of the two fields over the phase-shifter positions 
     
       
         
           
             
               
                 
                   
                     
                       d 
                        
                       
                           
                       
                        
                       ℒ 
                     
                     
                       d 
                        
                       
                           
                       
                        
                       
                         ɛ 
                         l 
                       
                     
                   
                   = 
                   
                     
                       k 
                       0 
                       2 
                     
                      
                      
                      
                     
                       
                         { 
                         
                           
                             ∑ 
                             
                               rϵ 
                                
                               
                                   
                               
                                
                               
                                 r 
                                 0 
                               
                             
                           
                            
                           
                             
                               
                                 e 
                                 aj 
                               
                                
                               
                                 ( 
                                 r 
                                 ) 
                               
                             
                              
                             
                               
                                 e 
                                 og 
                               
                                
                               
                                 ( 
                                 r 
                                 ) 
                               
                             
                           
                         
                         } 
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   45 
                   ) 
                 
               
             
           
         
       
     
     A schematic illustration of methods in accordance with many embodiments of the invention for experimental measurement of gradient information is illustrated in three stages  305 - 315  in  FIG. 3 . The box region  320  represents the OIU. The ovals (e.g.,  350 ) represent tunable phase shifters. Computation of the gradient is illustrated with respect to phase shifters  360  and  365 . 
     In the first stage  305 , the original set of amplitudes X l  is sent through the OIU  32 . The constant intensity terms is measured at each phase shifter. The second stage  310  shows that the adjoint mode amplitudes, given by δ l , are sent through the output side of the OIU. X* TR  is recorded from the opposite side, as well as |e aj | 2  in each phase-shifter. In the third stage, X l +X TR  is sent through the OIU, interfering e og  and e* aj  inside the device and recovering the gradient information for all phase shifters simultaneously. 
     Methods in accordance with numerous embodiments of the invention compute the gradient from the previous section through in situ intensity measurements. Specifically, an intensity pattern is generated with the form  {e og e aj }, matching that of Eq. (45). Interfering e og  and e* aj  directly in the system results in the intensity pattern: 
         I=|e   og | 2   +|e   aj | 2 +2   {e   og   e   aj },   (46)
 
     the last term of which matches Eq. (45). Thus, in many embodiments, the gradient can be computed purely through intensity measurements if the field e* aj  can be generated in the OIU. 
     The adjoint field, e aj , as defined in Eq. (42), is sourced by {circumflex over (P)} out   T δ l , meaning that it physically corresponds to a mode sent into the system from the output ports. As complex conjugation in the frequency domain corresponds to time-reversal of the fields, e* aj  is expected to be sent in from the input ports. Formally, to generate e* aj , a set of input source amplitudes, X TR , is found such that the output port source amplitudes, Z TR =ŴX TR , are equal to the complex conjugate of the adjoint amplitudes, or δ* l . Using the unitarity property of transfer matrix Ŵ l  for a lossless system, along with the fact that {circumflex over (P)} out {circumflex over (P)} out   T =Î for output modes, the input mode amplitudes for the time-reversed adjoint can be computed as 
         X*   TR   =Ŵ   l   T δ l .   (47)
 
     As discussed earlier, Ŵ l   T  is the transfer matrix from output ports to input ports. Thus, X TR  can be experimentally determined by sending δ l  into the device output ports, measuring the output at the input ports, and taking the complex conjugate of the result. 
     A process  400  for experimentally measuring a gradient of an OIU layer in an ANN with respect to the permittivities of the OIU layer&#39;s integrated phase shifters is conceptually illustrated in  FIG. 4 . Process  400  sends ( 405 ) original field amplitudes X l−1  through the OIU layer and measures the intensities at each phase shifter. Process  400  sends ( 410 ) δ l  into the output ports of the OIU layer and measures the intensities at each phase shifter. Process  400  computes ( 415 ) the time-reversed adjoint input field amplitudes. In numerous embodiments, time-reversed adjoint input field amplitudes are calculated as in Eq. (47). Process  400  interferes ( 420 ) the original and time-reversed adjoint fields in the device and measures the resulting intensities at each phase shifter. Process  400  computes ( 425 ) a gradient from the measured intensities. In a number of embodiments, processes compute gradients by subtracting constant intensity terms measured from the original field amplitudes and    l  (e.g., at steps  405  and  410  of process  400 ) and multiply by k 0   2  to recover the gradient, as in Eq. (45). 
     In many embodiments, the isolated forward and adjoint steps are performed separately, storing the intensities at each phase shifter for each step, and then subtracting this information from the final interference intensity. In a variety of embodiments, rather than storing these constant intensities, processes can introduce a low-frequency modulation on top of one of the two interfering fields, such that the product term of Eq. (46) can be directly measured from the low-frequency signal. 
     Specific processes for experimentally measuring a gradient of an OIU layer in an ANN in accordance with embodiments of the invention are described above: however, one skilled in the art will recognize that any number of processes can be utilized as appropriate to the requirements of specific applications in accordance with embodiments of the invention. 
     A numerical demonstration of a time-reversal procedure is illustrated in five panels  505 - 530  of  FIG. 5 . In this example, the procedure is performed with a series of FDFD simulations of an OIU implementing a 3×3 unitary matrix. These simulations are intended to represent the gradient computation corresponding to one OIU in a single layer, l, of a neural network with input X l−1  and delta vector δ l . In these simulations, absorbing boundary conditions am used on the outer edges of the system to eliminate back-reflections. 
     The first panel  505  shows a relative permittivity distribution for three MZIs arranged to perform a 3×3 linear operation. Boxes (e.g.,  550 ) represent where variable phase shifters could be placed in this system. As an example, the gradient information for a layer with X l−1 =[0 0 1] T  with unit amplitude and δ l =[0 1 0] T  is computed, corresponding to the bottom left and middle right port, respectively. 
     The second panel  510  illustrates the real part of a simulated electric field E z  corresponding to injection from the bottom left port. Specifically, stage  510  show s the real part of e og , corresponding to the original, forward field. 
     In the third panel  515 , the real part of the adjoint E z  is shown, corresponding to injection from the middle right port. The third panel  515  shows the real part of the adjoint field, e aj , corresponding to the cost function  = {δ l   T Ŵ l X l−1 }. 
     The fourth panel  520  shows a time-reversed adjoint field in accordance with some embodiments of the invention that can be fed in through all three ports on the left. Panel  520  shows the real part of the time reversed copy of e aj  as computed by the method described in the previous section, in which X* RT  is sent in through the input ports. There is excellent agreement, up to a constant, between the complex conjugate of the field pattern of panel  515  and the held pattern of panel  520 . 
     The fifth panel  525  shows gradient information 
     
       
         
           
             
               
                 d 
                  
                 
                     
                 
                  
                 ℒ 
               
               
                 d 
                  
                 
                     
                 
                  
                 
                   ɛ 
                   l 
                 
               
             
              
             
               ( 
               
                 ϰ 
                 , 
                 y 
               
               ) 
             
           
         
       
     
     as obtained directly by the adjoint method, normalized by its maximum absolute value. Panel  525  shows the gradient of the objective function with respect to the permittivity of each point of space in the system, as computed with the adjoint method, described in Eq. (45). 
     In the sixth panel  530 , the gradient information as obtained by methods in accordance with a variety of embodiments of the invention is shown, normalized by its maximum absolute value. Namely, the field pattern from panel  510  is interfered with the time-reversed adjoint field of panel  520  and the constant intensity terms are subtracted from the resulting intensity pattern. In certain embodiments, the results are then multiplied by an appropriate set of constants. Panels  525  and  530  match with high precision. 
     In a realistic system, the gradient must be constant for any stretch of waveguide between waveguide couplers because the interfering fields are at the same frequency and are traveling in the same direction. Thus, there should be no distance dependence in the corresponding intensity distribution. This is largely observed in this simulation, although small fluctuations are visible because of the proximity of the waveguides and the sharp bends, which were needed to make the structure compact enough for simulation within a reasonable time. In practice, the importance of this constant intensity is that it can be detected after each phase shifter, instead of inside of it. Intensity measurements in accordance with some embodiments of the invention can occur in the waveguide regions directly after the phase shifters, which eliminates the need for phase shifter and photo-detector components at the same location. 
     Numerically generated systems in accordance with many embodiments of the invention experience a power transmission of only 59% due to radiative losses and backscattering caused by very sharp bends and stair-casing of the structure in the simulation. Nevertheless, the time-reversal interference procedure still reconstructs the adjoint sensitivity with very good fidelity. Furthermore, a reasonable amount of this loss is non-uniform due to the asymmetry of the structure. 
     Choice and Implementation of Activation Functions 
     A major drawback of saturable absorption is that it is fundamentally lossy. Depending on the threshold power and waveguide implententation, an attenuation per layer of at least 1 dB can be expected. In a large scale photonic ANN with many layers, the compounding attenuation from each layer can bring the signal levels below the optical noise floor. Moreover, this scheme may require lowered activation power thresholds for successively deeper layers, which can be challenging to achieve for a fixed hardware platform and saturable absorption mechanism. 
     It is therefore of substantial interest to develop activation functions that are not subjected to the above limitations. Additionally, for simple implementation of the backpropagation algorithm described in this work, activation functions in accordance with various embodiments of the invention can have derivatives that allow the operation δ l =Γ l ⊙f′(Z l ) to be performed simply in the photonic circuit. 
     A possible alternative to saturable absorption is the rectified linear unit (ReLU) activation, which is a common activation function in conventional real-valued ANNs with several complex-valued variants. For example, the complex ReLU (c-ReLU) variant returns the output only if it is above a power threshold. This function is convenient because it is holomorphic (away from the discontinuity) and its derivative is simply 0 below the power threshold and 1 above the power threshold. Therefore, forward and backward propagation steps can be performed on the same hardware. For forward propagation, one would first measure the power of the waveguide mode with an electro-optic circuit and close the channel if it is below a threshold. For backward propagation, simply leave this channel either closed or open, depending on the forward pass. 
     The mod-ReLU variant similarly first checks whether the input is above a power threshold. If not, 0 is returned. Otherwise, it returns the input but with the power threshold subtracted from its amplitude. 
     Specific activation functions in accordance with embodiments of the invention are described above; however, one skilled in the art will recognize that any number of activation functions can be utilized as appropriate to the requirements of specific applications in accordance with embodiments of the invention. 
     Avoiding Internal Coherent Detection and Reconstruction 
     As presented thus far, the time-reversal interference method requires coherent detection and state preparation inside of the device. This introduces potential technical challenges and the need for additional components within the device. To mitigate this issue, methods in accordance with some embodiments of the invention (also referred to as a ‘linearization methods’) can recover gradient information without needing coherent detection within the device. In a variety of embodiments, methods can allow one to work entirely on the outsides of the ANN by selectively turning off or ‘linearizing’ the activation functions between layers in a controlled fashion. However, such methods can require an activation that may implement backpropagation through some protocol that, itself, does not require internal coherent measurement and state preparation. 
     A time-reversal interference method for a single linear OIU embedded in the network is described above. As described, time-reversal interference methods can require coherent detection to obtain X* TR , external computation of X TR , and subsequent coherent preparation of X TR  in the device. Thus, implementing this in a multi-layered network could require additional elements between each layer, complicating the experimental setup. An example of such a time-reversal interference technique for a layer embedded in a network is illustrated in  FIG. 6 . In this example, coherent detection, external computation, and state preparation is required between layers  610  and  615 , which can be undesirable. 
     To overcome this issue, methods in accordance with various embodiments of the invention can obtain the sensitivity of the internal layers without needing coherent detection or preparation within the network. The strategy relies on the ability to ‘linearize’ the activations, such that they become represented as simple transmission for both forward and backward passes. For a linearized activation given by f l (Z l )=Z l , Z l =X l  and δ l =Γ l , which greatly simplifies both forward and backward propagation. 
     A process for tuning a linear layer Ŵ l  is conceptually illustrated in  FIG. 7 . Process  700  linearizes ( 705 ) all activations after Ŵ l  and performs a forward pass with original input X 0 . Process  700  records ( 710 ) intensities within Ŵ l  and measures the output. Process  700  linearizes ( 715 ) all activations in network and sends the complex conjugate of the measured output into the output end of network and measures output b. Sending b* into the input end of the network recreates the desired X l−1  at the input of Ŵ l . Process  700  linearizes ( 720 ) all activations in network before Ŵ l  and performs backpropagation, sending Γ L  into the output end of network. Process  700  measures ( 725 ) intensities in Ŵ l  for subtraction and the output e. Sending e* into the input end of the network recreates the desired X TR  at the input end of Ŵ l  and Γ* l  at the output end of Ŵ l . Process  700  inputs ( 730 ) b*+c* into completely linearized network, which reproduces the desired interference term X 2 +X TR  at the input end of Ŵ l . Process  700  measures ( 735 ) the intensities in Ŵ l  and computes sensitivity. Methods for computing the sensitivity based on the measured intensities are described in greater detail above. 
     By linearizing the activation functions, large sections of the network can be made unitary. This allows the encoding of information about the internal fields in the externally measured states a, b. c. These states can be used to recreate the desired interference terms needed to obtain gradients as intensity measurements. 
     In numerous embodiments, methods in accordance with several embodiments of the invention can be used in a general network. In order to compute gradients for layer l of L, each OIU can be described by a matrix Ŵ l , and each activation can be described by a {circumflex over (F)} l  operator in the forward propagation and {circumflex over (B)} l  in the backward propagation. Note that even though the activations are nonlinear, depending on the result of a forward pass through the network, they can be set and then may resemble linear operations. 
     For the time-reversal interference method to work, an input to the system can be produced such that the mode X l +X TR  is created right before Ŵ l . Equivalently, the state Z l +δ* l  can be produced directly after Ŵ l . Using the notation introduced, these states can be explicitly described as 
         Z   l   =Ŵ   l ( {circumflex over (F)}   l−1   Ŵ   l−1    . . . {circumflex over (F)}   l   Ŵ   l ) X   0    (48)
 
       δ l =( Ŵ   l−1   T   {circumflex over (B)}   l−1    . . . Ŵ   L   T   {circumflex over (B)}   L )Γ L    (49)
 
     
       
         
           
             
               
                 
                   
                       
                   
                    
                   
                     
                       Z 
                       l 
                     
                     = 
                     
                       
                         
                           
                             W 
                             ^ 
                           
                           l 
                         
                          
                         
                           ( 
                           
                             
                               
                                 F 
                                 ^ 
                               
                               
                                 l 
                                 - 
                                 1 
                               
                             
                              
                             
                               
                                 W 
                                 ^ 
                               
                               
                                 l 
                                 - 
                                 1 
                               
                             
                              
                             
                                 
                             
                              
                             … 
                              
                             
                                 
                             
                              
                             
                               
                                 F 
                                 ^ 
                               
                               1 
                             
                              
                             
                               
                                 W 
                                 ^ 
                               
                               1 
                             
                           
                           ) 
                         
                       
                        
                       
                         X 
                         0 
                       
                     
                   
                 
               
               
                 
                   ( 
                   48 
                   ) 
                 
               
             
             
               
                 
                   
                       
                   
                    
                   
                     
                       δ 
                       l 
                     
                     = 
                     
                       
                         ( 
                         
                           
                             
                               W 
                               ^ 
                             
                             
                               l 
                               - 
                               1 
                             
                             T 
                           
                            
                           
                             
                               B 
                               ^ 
                             
                             
                               l 
                               - 
                               1 
                             
                           
                            
                           
                               
                           
                            
                           … 
                            
                           
                               
                           
                            
                           
                             
                               W 
                               ^ 
                             
                             L 
                             T 
                           
                            
                           
                             
                               B 
                               ^ 
                             
                             L 
                           
                         
                         ) 
                       
                        
                       
                         Γ 
                         L 
                       
                     
                   
                 
               
               
                 
                   ( 
                   49 
                   ) 
                 
               
             
             
               
                 
                   ? 
                 
               
               
                 
                   ( 
                   50 
                   ) 
                 
               
             
             
               
                 
                   
                     ? 
                   
                    
                   
                     indicates text missing or illegible when filed 
                   
                 
               
               
                 
                     
                 
               
             
           
         
       
     
     A stage diagram of a procedure for using the time-reversal interference method to measure sensitivities in the OIUs without internal coherent detection or preparation is illustrated in six stages  805 - 830  of  FIG. 8 . In the first stage  805 , the gradient measurement with respect to the phase shifters is shown in the l−3 layer, which requires creating an input state to recreate X 2 +X TR  at the input of Ŵ 3 . Empty ovals  850  correspond to ‘linearized’ activations, which can be represented as an identity matrix. The first stage  805  shows all of the channels following layer l are linearized. The output of the system when sending in the original input, X 0 , is computed, which is labeled a. 
         a =( Ŵ   L    . . . Ŵ   l )( {circumflex over (F)}   l−1   Ŵ   l−1    . . . {circumflex over (F)}   1   Ŵ   1 ) X   0 .   (51)
 
     The second stage  810  shows that all of the activation channels are linearized. Then, a* is sent into the output end of the network and the output is measured, which is labeled b. 
         b =( Ŵ   1   T    . . . Ŵ   L   T ) a*    (52)
 
     The complex conjugate of b is given by 
     
       
         
           
             
               
                 
                   
                     b 
                     * 
                   
                   = 
                   
                     
                       ( 
                       
                         
                           
                             W 
                             ^ 
                           
                           1 
                           † 
                         
                          
                         ⋯ 
                          
                         
                           
                             
                               
                                   
                               
                                
                               W 
                             
                             ^ 
                           
                           L 
                           † 
                         
                       
                       ) 
                     
                      
                     a 
                   
                 
               
               
                 
                   ( 
                   53 
                   ) 
                 
               
             
             
               
                 
                   = 
                   
                     
                       ( 
                       
                         
                           
                             W 
                             ^ 
                           
                           1 
                           † 
                         
                          
                         ⋯ 
                          
                         
                           
                             
                               
                                   
                               
                                
                               W 
                             
                             ^ 
                           
                           L 
                           † 
                         
                       
                       ) 
                     
                      
                     
                       ( 
                       
                         
                           
                             W 
                             ^ 
                           
                           L 
                           † 
                         
                          
                         ⋯ 
                          
                         
                           
                             
                               
                                   
                               
                                
                               W 
                             
                             ^ 
                           
                           l 
                           † 
                         
                       
                       ) 
                     
                      
                     
                       ( 
                       
                         
                           
                             F 
                             ^ 
                           
                           
                             l 
                             - 
                             1 
                           
                         
                          
                         
                           
                             W 
                             ^ 
                           
                           
                             l 
                             - 
                             1 
                           
                         
                          
                         ⋯ 
                          
                         
                             
                         
                          
                         
                           
                             F 
                             ^ 
                           
                           1 
                         
                          
                         
                           
                             W 
                             ^ 
                           
                           1 
                         
                       
                       ) 
                     
                      
                     
                       X 
                       0 
                     
                   
                 
               
               
                 
                   ( 
                   54 
                   ) 
                 
               
             
             
               
                 
                   = 
                   
                     
                       ( 
                       
                         
                           
                             W 
                             ^ 
                           
                           1 
                           † 
                         
                          
                         ⋯ 
                          
                         
                           
                             
                               
                                   
                               
                                
                               W 
                             
                             ^ 
                           
                           
                             l 
                             - 
                             1 
                           
                           † 
                         
                       
                       ) 
                     
                      
                     
                       ( 
                       
                         
                           
                             F 
                             ^ 
                           
                           
                             l 
                             - 
                             1 
                           
                         
                          
                         
                           
                             W 
                             ^ 
                           
                           
                             l 
                             - 
                             1 
                           
                         
                          
                         ⋯ 
                          
                         
                             
                         
                          
                         
                           
                             F 
                             ^ 
                           
                           1 
                         
                          
                         
                           
                             W 
                             ^ 
                           
                           1 
                         
                       
                       ) 
                     
                      
                     
                       
                         X 
                         0 
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   55 
                   ) 
                 
               
             
           
         
       
     
     The third stage  815  shows that sending b* into the input end of the network recreates the desired X l−1  at the input of Ŵ l . In the fourth stage  820 , only the activation channels before layer l are linearized and Γ L  is input into the output end of the system, measuring c. 
         c =( Ŵ   l   T    . . . Ŵ   l   T )( Ŵ   l+1   T   {circumflex over (B)}   l+1    . . . Ŵ   L   T   {circumflex over (B)}   L )Γ L    (56)
 
         c *=( Ŵ   1   †    . . . Ŵ   l   † )( Ŵ   l+1   †   {circumflex over (B)}*   l+1    . . . Ŵ   L   †   {circumflex over (B)}*   L )Γ* L    (57)
 
     The fifth stage  825  shows that sending c* into the input end of the network recreates the desired X TR  at the input end of Ŵ l  and δ* l  at the output end of Ŵ l . The sixth stage  830  shows that inputting b*+c* will output Z l +δ* l  directly after layer l, which will be sufficient for training via the time-reversal intensity measurement technique. To make the derivation more dear, these two terms are split up and the total output is defined as o≡o 1 +o 2  where 
       o≡(Ŵ l−1  . . . Ŵ 1 )(b*+c*)  (58)
 
       o 1 ≡(Ŵ l−1  . . . Ŵ l )b*   (59)
 
       o 2 ≡(Ŵ l−1  . . . Ŵ l )c*   (60)
 
     Inserting the form of b* from Eq. (55) into the expression for o 1 , 
     
       
         
           
             
               
                 
                   
                     o 
                     1 
                   
                   = 
                   
                     
                       ( 
                       
                         
                           
                             W 
                             ^ 
                           
                           l 
                         
                          
                         ⋯ 
                          
                         
                             
                         
                          
                         
                           
                             W 
                             1 
                           
                           ^ 
                         
                       
                       ) 
                     
                      
                     
                       ( 
                       
                         
                           
                             W 
                             ^ 
                           
                           1 
                           † 
                         
                          
                         ⋯ 
                          
                         
                             
                         
                          
                         
                           
                             W 
                             ^ 
                           
                           
                             l 
                             - 
                             1 
                           
                           † 
                         
                       
                       ) 
                     
                      
                     
                       ( 
                       
                         
                           
                             F 
                             ^ 
                           
                           
                             l 
                             - 
                             1 
                           
                         
                          
                         
                           
                             W 
                             ^ 
                           
                           
                             l 
                             - 
                             1 
                           
                         
                          
                         ⋯ 
                          
                         
                             
                         
                          
                         
                           
                             F 
                             ^ 
                           
                           1 
                         
                          
                         
                           
                             W 
                             ^ 
                           
                           1 
                         
                       
                       ) 
                     
                      
                     
                       X 
                       0 
                     
                   
                 
               
               
                 
                   ( 
                   61 
                   ) 
                 
               
             
             
               
                 
                   = 
                   
                     
                       
                         
                           W 
                           ^ 
                         
                         l 
                       
                        
                       
                         ( 
                         
                           
                             
                               F 
                               ^ 
                             
                             
                               l 
                               - 
                               1 
                             
                           
                            
                           
                             
                               W 
                               ^ 
                             
                             
                               l 
                               - 
                               1 
                             
                           
                            
                           ⋯ 
                            
                           
                               
                           
                            
                           
                             
                               F 
                               ^ 
                             
                             1 
                           
                            
                           
                             
                               W 
                               ^ 
                             
                             1 
                           
                         
                         ) 
                       
                     
                      
                     
                       X 
                       0 
                     
                   
                 
               
               
                 
                   ( 
                   62 
                   ) 
                 
               
             
             
               
                 
                   = 
                   
                     
                       
                         W 
                         ^ 
                       
                       l 
                     
                      
                     
                       X 
                       l 
                     
                   
                 
               
               
                 
                   ( 
                   63 
                   ) 
                 
               
             
             
               
                 
                   = 
                   
                     Z 
                     l 
                   
                 
               
               
                 
                   ( 
                   64 
                   ) 
                 
               
             
           
         
       
     
     as desired. Similarly, inserting the form of c* from Eq. (57) into the expression for o 2 . 
     
       
         
           
             
               
                 
                   
                     o 
                     2 
                   
                   = 
                   
                     
                       ( 
                       
                         
                           
                             W 
                             ^ 
                           
                           l 
                         
                          
                         ⋯ 
                          
                         
                             
                         
                          
                         
                           
                             W 
                             ^ 
                           
                           1 
                         
                       
                       ) 
                     
                      
                     
                       ( 
                       
                         
                           
                             W 
                             ^ 
                           
                           1 
                           † 
                         
                          
                         ⋯ 
                          
                         
                             
                         
                          
                         
                           
                             W 
                             ^ 
                           
                           l 
                           † 
                         
                       
                       ) 
                     
                      
                     
                       ( 
                       
                         
                           
                             W 
                             ^ 
                           
                           
                             l 
                             + 
                             1 
                           
                           † 
                         
                          
                         
                           
                             B 
                             ^ 
                           
                           
                             l 
                             + 
                             1 
                           
                           * 
                         
                          
                         ⋯ 
                          
                         
                           
                             
                               
                                   
                               
                                
                               W 
                             
                             ^ 
                           
                           L 
                           † 
                         
                          
                         
                           
                             B 
                             ^ 
                           
                           L 
                           * 
                         
                       
                       ) 
                     
                      
                     
                       Γ 
                       L 
                       * 
                     
                   
                 
               
               
                 
                   ( 
                   65 
                   ) 
                 
               
             
             
               
                 
                   = 
                   
                     
                       ( 
                       
                         
                           
                             W 
                             ^ 
                           
                           
                             l 
                             + 
                             1 
                           
                           † 
                         
                          
                         
                           
                             B 
                             ^ 
                           
                           
                             l 
                             + 
                             1 
                           
                           * 
                         
                          
                         ⋯ 
                          
                         
                             
                         
                          
                         
                           
                             W 
                             ^ 
                           
                           L 
                           † 
                         
                          
                         
                           
                             B 
                             ^ 
                           
                           L 
                           * 
                         
                       
                       ) 
                     
                      
                     
                       Γ 
                       L 
                       * 
                     
                   
                 
               
               
                 
                   ( 
                   66 
                   ) 
                 
               
             
             
               
                 
                   = 
                   
                     δ 
                     l 
                     * 
                   
                 
               
               
                 
                   ( 
                   67 
                   ) 
                 
               
             
           
         
       
     
     also as desired. 
     Thus, inputing b*+c* will give o=Z l +δ* l  at the output of layer l. Equivalently, this same input reproduces X l−1 +X TR  at the input end of later l, which is what we need to do time-reversal sensitivity measurements. The derivation presented here holds for an arbitrary choice of activation function, even those not representable by a linear operator. 
     Other Applications 
     In addition to the feed-forward ANNs discussed in this work, methods in accordance with some embodiments of the invention can be used to train recurrent neural networks (RNNs), which are commonly used to analyze sequential data, such natural language or time series inputs. Re-current networks, as diagrammed in  FIG. 9 , have a single linear layer and activation function. A schematic of a recurrent neural network (RNN) is illustrated in  FIG. 9  with a single OIU and activation function. A time series of inputs to the system are provided from the left. The output of this layer may be read out with a portion routed back into the input end of the network. In many embodiments, this splitting and routing may be done simply in the photonic platform using an integrated beam splitter. RNNs may also be trained using the same backpropagation method as described above. 
     Methods in accordance with several embodiments of the invention can be used to train Convolutional Neural Networks (CNNs). CNNs are another popular class of neural network that is commonly used to analyze image data, where there are many inputs with a large amount of correlation between adjacent elements. In CNNS, the matrix multiplications are replaced by convolution operations between the input values and a trainable kernel. This convolution may be performed on a mesh of MZIs similar to those presented here, but with far fewer MZI units necessary. In various embodiments, backpropagation for CNNs may be performed using methods as described above. 
     The training scheme presented here can also be used to tune a single OIU without non-linearity, which may find a variety of applications. Specifically, the linear unit can be trained to map a set of input vectors {X i } to a corresponding set of output vectors {Y i }. One example for an objective function for this problem is 
     
       
         
           
             
               
                 
                   
                     ℒ 
                      
                     
                       ( 
                       ϕ 
                       ) 
                     
                   
                   = 
                   
                     
                       1 
                       N 
                     
                      
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         N 
                       
                        
                       
                           
                       
                        
                       
                          
                          
                         
                           { 
                           
                             
                               Y 
                               i 
                               † 
                             
                              
                             
                               W 
                               ^ 
                             
                              
                             
                               X 
                               i 
                             
                           
                           } 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   68 
                   ) 
                 
               
             
           
         
       
     
     where φ is a vector containing all the degrees of freedom. Other choices that also maximize the overlap between X i  and Y i  for every i are, of course, also possible. 
     One application of this training is a mode sorter for sorting out a number of arbitrary, orthogonal modes. In particular, for an N×N linear system, N orthogonal inputs X i  can be chosen. Define Y i =B i  as the unit vector for the i-th output port. While this specific problem may also be solved sequentially by existing methods, gradient-based implementations in accordance with some embodiments of the invention may have benefits in speed and scaling with number of parameters in the system, especially when the degrees of freedom can be updated in parallel. 
     Training protocols in accordance with numerous embodiments of the invention may also be used for optimizing a single (potentially random) input into a desired output, i.e.,  = {δ † ŴX}. This type of problem arises for example when considering optical control systems for tuning the power delivery system for dielectric laser accelerators. In this application, a series of waveguides carry power to a central accelerator structure. However, it is likely that these signals will initially consist of randomly distributed powers and phases due to the coupling, splitting, and transport stages earlier in the process. Thus, the OIUs can be used as a phase and amplitude sorting element, where now X is an initially random amplitude and phase in each waveguide, and δ is a vector of target amplitudes and phases for optimal delivery to the dielectric laser accelerator. The adjoint field is directly given by the radiation from an electron beam, so the target vector may be generated physically by sending in a test electron beam. In several embodiments, a similar system can be used for electron beam focusing and bunching applications. 
     In numerous embodiments, OIUs can also be used to implement reconfigurable optical quantum computations, with various applications in quantum information processing. In such systems, linear training with classical coherent light can be used to configure the quantum gates, e.g., by setting up a specific matrix Ŵ described by complete, orthonormal sets {X i } and {Y i }. After the tuning, systems in accordance with numerous embodiments of the invention can be run in the quantum photonic regime. 
     Methods in accordance with numerous embodiments of the invention work by physically propagating an adjoint field and interfering its time-reversed copy with the original field. In a number of embodiments, the gradient information can then be directly obtained out as an in-situ intensity measurement. While processes are described in the context of ANNs, one skilled in the art will recognize that the processes are broadly applicable to any reconfigurable photonic system. Such a setup can be used to tune phased arrays, optics delivery systems for dielectric as accelerators, or other systems that rely on large meshes of integrated optical phase shifters. Methods in accordance with some embodiments of the invention can implement the nonlinear elements of the ANN in the optical domain. 
     Although the present invention has been described in certain specific aspects, many additional modifications and variations would be apparent to those skilled in the art. It is therefore to be understood that the present invention may be practiced otherwise than specifically described. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive.