Deep Learning Solution of the Eigenvalue Problem for Differential Operators

1 INTRODUCTION . Eigenfunctions and eigenvalues of the Laplacian ( among other operators ) are important in various applications ranging , inter alia , from image processing to computer vision , shape analysis and quantum mechanics . It is also of major importance in various engineering applications where resonance is crucial for design and safety [ Benouhiba & Belyacine ( 2013 ) ] . Laplacian eigenfunctions allow us to perform spectral analysis of data measured at more general domains or even on graphs and networks [ Shi & Malik ( 2000 ) ] . Additionally , the M -smallest eigenvalues of the Laplace-Beltrami operator are fundamental features for comparing geometric objects such as 3D shapes , images or point clouds via the functional maps method in statistical shape analysis [ Ovsjanikov et al . ( 2012 ) ] . Moreover , in quantum mechanics , the smallest eigenvalues and eigenfunction of the Hamiltonian are of great physical significance [ Han et al . ( 2019 ) ] . In this paper we present a novel numerical method for the computation of these eigenfunctions ( efs ) and eigenvalues ( evs ) , where the efs are parameterized by NNs with continuous activation functions , and the evs are directly calculated via the Rayleigh quotient . The resulting efs are therefore smooth functions defined in a parametric way . This is in contrast to the finite element [ Pradhan & Chakraverty ( 2019 ) ] and finite difference [ Saad ( 2005 ) ; Knyazev ( 2000 ) ] methods in which the efs are defined on either a grid or as piecewise linear/polynomial functions with limited smoothness . In these matrix-based approaches one has to discretize first the problem and to represent it as an eigenvalue problem for a matrix . This in itself is prone to numerical errors . Following [ Bar & Sochen ( 2019 ) ] , we suggest an unsupervised approach to learn the eigenpairs of a differential operator on a specified domain with boundary conditions , where the network simultaneously approximates the eigenfunctions at every entry x . The method is based on a uniformly distributed point set which is trained to satisfy two fidelity terms of the eigenvalue problem formulated as theL2 andL∞-like norms , boundary conditions , orthogonality constraint and regularization . There are several advantages of the proposed setting : ( i ) the framework is general in the sense that it can be used for non linear differential operators with high order derivatives as well . ( ii ) Since we sample the domain with a point cloud , we are not limited to standard domains . The problem can be therefore solved in an arbitrary regular domain . ( iii ) The framework is generic such that additional constraints and regularizers can be naturally integrated in the cost function . ( iv ) Unlike previous methods , the suggested framework solves simultaneously multiple eigenpairs . This means that we handle a family of PDEs ( one for each eigenvalue and finding the eigenvalues themselves ) in one network that solves these multiple PDEs together . The method is applied in 1D and 2D for both known and multiple unknown eigenvalues of the Laplacian operator . Quantitative analysis demonstrates the robustness of the method compared with the classical matrix-based methods . 2 RELATED WORK . Many recent approaches have shown promise in using the power of NNs to approximate solutions of differential equations . Classical methods are often prone to weakness due to the discretization of the domain Ω . In [ Bar & Sochen ( 2019 ) ] , the authors propose a solver for both forward and inverse problems , using NNs to model the solution , and penalizing using both the automatic differentiation , and boundary conditions . In [ Raissi et al . ( 2017 ) ] , a similar approach was taken to solve both continuous and discrete time models . In [ Chen et al . ( 2018 ) ] , differential equation solvers are used as part of the network architecture , and are shown to enhance the smoothness and convergence of the solutions . In order to properly solve differential equations , a representation that captures highorder derivatives is desired . Recently , [ Sitzmann et al . ( 2020 ) ] proposed a network architecture that illustrates these requirements using periodic activation functions with the proper initialization . Additionally , [ Rippel et al . ( 2015 ) ] proposed leveraging Discrete Fourier Transform ( DFT ) to represent the network in spectral space . The well-known power method and its variants [ Eastman & Estep ( 2007 ) ] has been the main method for addressing the eigenvalue problem . The method works on specific Linear operators , L : L2 ( Rd ) → L2 ( Rd ) . It is done after the continuous equation is reduced numerically to an eigenpair equation for matrices . This process introduces numerical errors even before the solution of the eigen problem . The usage of the power method for spectral operators on Hilbert Spaces was recently shown inn [ Erickson et al . ( 1995 ) ] . In [ Hait-Fraenkel & Gilboa ( 2019 ) ] a modified method for nonlinear differential operators was proposed . Furthermore , most power method variants for operators , converge to a single eigenpair . Finding the M smallest eigenpairs can be both computationally and algorithmically challenging . Advanced methods addressing the eigenvalue problem via deep networks were recently introduced . These methods are based on variational Monte Carlo ( VMC ) and diffusion Monte Carlo ( DMC ) methods . VMC relies on leveraging physical knowledge to propose an ansatz of the eigenfunction and incorporates the essential physics [ Han et al . ( 2019 ) ; Hermann et al . ( 2019 ) ; Pfau et al . ( 2019 ) ; Choo et al . ( 2019 ) ] . Recently , [ Han et al . ( 2020 ) ] formulated the eigenvalue problem by the stochastic backward equation using the DMC method , where the loss function optimizes the eigenvalue , eigenfunction and the scaled gradient of the eigenfunction . The loss function consists of L2 norm of two fidelity terms with additional normalization . The algorithm yields the first eigenpair with an optional second eigenpair given some mild prior estimate of the eigenvalue . In the suggested work , we formulate the eigenvalue problem in a direct setting with flexible number of eigenpairs . Additionally , we use L∞ norms for fidelity and boundary condition terms to accomplish a strong ( pointwise ) solution . 3 PRELIMINARIES . Let H be a Hilbert space where the inner product for u , v ∈ H is 〈u , v〉 . Let A ∈ O ( H ) be an operator . Let A∗ be the adjoined operator defined by 〈A∗u , v〉 = 〈u , Av〉 ∀u , v ∈ H. Then A is said to be self-adjoint if A = A∗ . We start with a short Lemma on self-adjoint operators [ Conway ( 1985 ) ] . Lemma 3.1 LetH be a Hilbert space . Let A ∈ O ( H ) be a self-adjoint operator . Then all eigenvalues of A are real . In this work we focus on self-adjoint operators . An eigenpair of an operator L is defined as : ( u , λ ) s.t . λ ∈ R , where u is the eigenfunction of L and λ is the corresponding eigenvalue . Let L be a self-adjoint operator L : L2 ( Rd ) → L2 ( Rk ) . Our objective is to search for eigenpairs { ui , λi } such that Lui + λiui = 0 ∀i . ( 1 ) The proposed algorithm approximates the eigenfunction set ui ( x ) by a NN ui ( x ; θui ) parameterized by θui which denotes the network weights and biases . The network consists of few fully connected layers with smooth activation function ϕ and linear sum in the last layer . For example , four layers architecture is given by u ( x ) = W5ϕ ( W4ϕ ( W3ϕ ( W2ϕ ( W1x+ b1 ) + b2 ) + b3 ) + b4 ) + b5 , ( 2 ) and ϕ ( · ) = tanh ( · ) or SIREN . The input to the network is x ∈ Rd , one input node for each dimension . The network is trained to satisfy the PDE with boundary conditions by minimizing a cost function . 4 SINGLE KNOWN EIGENVALUE . We first address the problem of finding a single eigenfunction u ( x ) given its corresponding eigenvalue λ . We approximate u ( x ) by a NN and optimize the following cost function F1 ( u ( x , θu ) ) = α‖T u‖22 + µ‖T u‖∞ + δ‖u− u0‖1 , ∂Ω + β ∣∣∣‖u‖22 − c∣∣∣+ ρ‖θu‖22 , ( 3 ) where T u : = Lu+ λu , with the Laplacian operator L = ∆ . The first two terms in equation 3 are the L2 and L∞ fidelity terms where the latter promotes a pointwise solution such that the equation is satisfied for isolated points as well [ Bar & Sochen ( 2019 ) ] . The third term imposes boundary conditions and the forth normalizes the squared length of u to c = 1 due to scale invariance of the eigenfunction ( since ( αu , λ ) is a valid eigenpair for α 6= 0 ) . The final term is the standard weight decay term which stabilizes the network weights . The L2 norm ‖.‖22 is defined as the Monte-Carlo Integral approximation norm : ‖u‖22 = Vol ( Ω ) N xN∑ x=x1 |u ( x ) |2 ∼ ∫ Ω |u ( x ) |2dx . ( 4 ) In the first example we apply the Laplacian operator in 1D where λ = 4 and u ( 0 ) = u ( π/2 ) = 0 . Then , T u = u′′ ( x ) + 4u ( x ) . The normalized analytical solution is therefore u ( x ) = 2√ π sin ( 2x ) . Figure 1 demonstrates the outcome of the algorithm at iterations 1 , 100 and 2500 . As can be seen , the approximated solution approaches the sin function with Relative Mean Square Error ( RMSE ) = 6.52e−4 and peak signal-to-noise ratio ( PSNR ) = 28.84 . 5 SINGLE EIGENPAIR WITH THE SMALLEST EIGENVALUE . Next , we address the case where the eigenvalue is not known in advance . We therefore limit ourselves to the smallest nontrivial eigenpair . This approach is analogue to power method approach , where only one dominant eigenpair is found . Recall the Rayleigh quotient defined as [ Miller ( 2016 ) ; Feld et al . ( 2019 ) ] R ( u ) : = −〈Lu , u〉 〈u , u〉 . ( 5 ) It can be shown that the eigenfunction u is a critical point of R ( u ) , where R ( u ) is its corresponding ( nontrivial ) eigenvalue λ . Furthermore , if ũ is a function which is close to u thenR ( ũ ) approximates λ . In the following cost function we replace λ by R ( u ) . Then equation 3 is modified to F2 ( u ( x , θu ) ) = α‖T u‖22 +µ‖T u‖∞+ δ‖u−u0‖1 , ∂Ω +β ∣∣∣‖u‖22− c∣∣∣+ ρ‖θu‖22 + γ‖R ( u ) ‖22 , ( 6 ) where T u = Lu+R ( u ) u . The last term minimizes R ( u ) and therefore attracts the solution to the lowest nontrivial eigenvalue . The ground truth eigenpair is given by ( √ 2 π sin ( x ) , 1 ) for Ω = [ 0 , π ] . Figure 2 shows the outcome of the proposed method at three iterations . The approximated eigenfunction is obtained up to a sign . The eigenvalue which is the value of the Rayleigh quotient converges to the true value λ = 1 with decreasing standard deviation along the mini-batches . Quantitative results are shown in row 1 of tables 1 and 2 where MAE stands for Mean Absolute Error and MRE for Mean Relative Error .

Summary: The authors frame the decomposition of the Laplacian equation as an unsupervised regression problem that is using a 5-level (and fully connected?) neural network as regression function.  A cost function to be used in the optimization is proposed that is expanded to eigendecomposition problems of increasing complexity. A comparison with a classical method indicates that the proposed approach is comparable to or better than the former for the given task.