Deep Learning Solution of the Eigenvalue Problem for Differential Operators
Authors: Ido Ben-Shaul
Leah Bar
Nir Sochen
Venue:  (2021)
Keywords: Eigenvalue problem
Unsupervised learning
Laplacian operator

Abstract: Solving the eigenvalue problem for differential operators is a common problem in many scientific fields. Classical numerical methods rely on intricate domain discretization, and yield non-analytic or non-smooth approximations. We introduce a novel Neural Network (NN)-based solver for the eigenvalue problem of differential self-adjoint operators where the eigenpairs are learned in an unsupervised end-to-end fashion. We propose three different training procedures, for solving increasingly challenging tasks towards the general eigenvalue problem. 

The proposed solver is able to find the M smallest eigenpairs for a general differential operator. We demonstrate the method on the Laplacian operator which is of particular interest in image processing, computer vision, shape analysis among many other applications. 
Unlike other numerical methods such as finite differences, the partial derivatives of the network approximation of the eigenfunction can be analytically calculated to any order. Therefore, the proposed framework enables the solution of higher order operators and on free shape domain or even on a manifold. Non-linear operators can be investigated by this approach as well.

Under review as a conference paper at ICLR 2021
DEEP LEARNING SOLUTION OF THE EIGENVALUE PROBLEM FOR DIFFERENTIAL OPERATORS
Anonymous authors Paper under double-blind review
ABSTRACT
Solving the eigenvalue problem for differential operators is common problem in many scientific fields. Classical numerical methods rely on intricate domain dis- cretization, and yield non-analytic or non-smooth approximations. We introduce a novel Neural Network (NN)-based solver for the eigenvalue problem of differ- ential self-adjoint operators where the eigenpairs are learned an unsupervised end-to-end fashion. We propose three different training procedures, for solving increasingly challenging tasks towards the general cigenvalue problem. The pro- posed solver is able to find the M smallest eigenpairs for general differential operator. We demonstrate the method on the Laplacian operator which is of par- ticular interest in image processing, computer vision, shape analysis among many other applications. Unlike other numerical methods such as finite differences, the partial derivatives of the network approximation of the eigenfunction can be ana- lytically calculated to any order. Therefore, the proposed framework enables the solution of higher order operators and on free shape domain or even on a manifold. Non-linear operators can be investigated by this approach as well.
INTRODUCTION
Eigenfunctions and eigenvalues of the Laplacian (among other operators) are important in various applications ranging, inter alia, from image processing computer vision, shape analysis and quan- tum mechanics. It is also of major importance in various engineering applications where resonance crucial for design and safety [Benouhiba & Belyacine (2013)1. Laplacian cigenfunctions allow us to perform spectral analysis of data measured at more general domains or even on graphs and networks (Shi Malik (2000)] Additionally, the M-smallest eigenvalues of the Laplace-Bell Beltrami operator are fundamental features for comparing geometric objects such as 3D shapes, images or point clouds via the functional maps method in statistical shape analysis [Ovsjanikov et al. (2012)]- Moreover, in quantum mechanics, the smallest eigenvalues and eigenfunction of the Hamiltonian are of great physical significance [Han et al. (201 this paper we present a novel numerical method for the computation of these eigenfunctions (efs) and eigenvalues (evs), where the efs are parameterized by NNs with continuous activation functions, and the evs are directly calculated via the Rayleigh quotient. The resulting efs are therefore smooth functions defined in parametric way. This is in contrast to the finite element [Pradhan & Chakraverty (2019)) and finite difference [Saad] (2055; Knyazev [2000)] methods in which the ets are defined on either grid as piecewise lin- car/polynomial functions with limited smoothness. In these matrix-based approaches one has to discretize first the problem and to represent it as eigenvalue problem for matrix. This in itself is prone to numerical errors.
Following [Bar Sochen [2019]] we suggest an unsupervised approach to learn the eigenpairs of a differential operator on a specified domain with boundary conditions, where the network simul- taneously approximates the eigenfunctions at every entry x. The method based on a uniformly distributed point set which is trained to satisfy two fidelity terms of the eigenvalue problem formu- lated as the L2 and -lke norms, boundary conditions, orthogonality constraint and regularization. There are several advantages of the proposed setting: (i) the framework is general in the sense that can be used for non linear differential operators with high order derivatives as well. (ii) Since sample the domain with point cloud, we are not limited to standard domains. The problem can be therefore solved in an arbitrary regular domain. (ii) The framework is generic such that additional constraints and regularizers can be naturally integrated in the cost function. (iv) Unlike

---

Under review as a conference paper at ICLR 2021
previous methods, the suggested framework solves simultaneously multiple eigenpairs. This means that we handle a family of PDEs (one for each eigenvalue and finding the eigenvalues themselves) in network that solves these multiple PDEs together. The method is applied in ID and 2D for both known and multiple unknown eigenvalues of the Laplacian operator. Quantitative analysis demonstrates the robustness of the method compared with the classical matrix-hased methods.
RELATED WORK
Many recent approaches have shown promise in using the power of NNs to approximate solutions of differential equations. Classical methods are often prone to weakness due to the discretization of the domain Q2. In [Bar& & Sochen [019)]] the authors propose solver for both forward and inverse problems, using NNs to model the solution, and penalizing using both the automatic differentia- tion, and boundary conditions. In [Raissi et al. (2017]]. similar approach was taken to solve both continuous and discrete time models. In [Chen et al. (2018)), differential equation solvers are used as part of the network architecture, and are shown enhance the smoothness and convergence of the solutions. In order to properly solve differential equations, a representation that captures high- order derivatives is desired. Recently, [Sitzmann et al. (2020)] proposed a network architecture that illustrates these requirements using periodic activation functions with the proper initialization. Ad- ditionally, [Rippel et al. (2013)] proposed leveraging Discrete Fourier Transform (DFT) to represent the network in spectral space.
The well-known power method and its variants (Eastman & Estep (2007)) has been the main method for addressing the eigenvalue problem. The method works on specific Lineur operators, f: (:2((R)) + T2((1R) It is done after the continuous equation is reduced numerically to an eigen- pair equation for matrices. This process introduces numerical errors even before the solution of the eigen problem. The usage of the power method for spectral operators on Hilbert Spaces was recently shown inn [Erickson et al. [1995]]. In [Hait-Fraenkel & Gilboa [ modified method for non- linear differential operators was proposed. Furthermore, most power method variants for operators, converge to a single eigenpair. Finding the M smallest eigenpairs can be both computationally and algorithmically challenging.
Advanced methods addressing the eigenvalue problem via deep networks were recently introduced. These methods are based on variational Monte Carlo (VMC) and diffusion Monte Carlo (DMC) methods. VMC relies on leveraging physical knowledge propose an ansatz of the eigenfune-
tion and incorporates the essential physics (Han et al. (2019); Hermann et al. (2019); ; Pfau et al.)
2019:; ; Choo et al. ]. Recently, (Han al. (2020p] formulated the eigenvalue problem by the stochastic backward equation using the DMC method, where the loss function optimizes the eigenvalue, eigenfunction and the scaled gradient of the eigenfunction. The loss function consists of Le norm of two fidelity terms with additional normalization. The algorithm yields the first eigenpair with an optional second eigenpair given some mild prior estimate of the eigenvalue, In the suggested work, we formulate the cigenvalue problem in direct setting with flexible number of cigenpairs. Additionally, we use Loe norms for fidelity and boundary condition terms to accomplish a strong (pointwise) solution.
3 PRELIMINARIES
Let H. be Hilbert space where the inner product for u, E H is (u. Let A E O(H) be an operator. Let A' be the adjoined operator defined by (A'u, u, v) (u, Au) Yu, E H. Then A is said to be self-adjoint if A = A". We start with short Lemma on self-adjoint operators (Conway 1985)].
Lemma 3.1 Let H be Hilbert space. Let A E O(H) be self-adjoint operator. Then all eigenval- ues of A are real.
In this work we focus on self-adjoint operators. An eigenpair of an operator L is defined as: (u, A) s.t. X E R, where u is the eigenfunction of L and À is the corresponding eigenvalue. Let L be a self-adjoint operator L: LR()) + (R*).. Our objective is search for eigenpairs {u,, > such that
Lus Apt Vi.
2

---

Under review as a conference paper at ICLR 2021
FA3 64p4
03
Figure 1: The solution u" 4u 0 with u(0)) u(Tr2) iterations (from left to right) 1, 100 and 2500. At the last iteration RMSE = 6.52c-4 and PSNR = 28.84.
The proposed algorithm approximates the eigenfunction set (z) by a NN (ı 0u.) parameterized by 9.., which denotes the network weights and biases. The network consists of few fully connected layers with smooth activation function yp and linear sum in the last layer. For example, four layers architecture given by
u (x) Wop Way [Wspp W22 Wi2 + b1) + by + bs) + +ba) + bs+
and (p.) tanh(-) or SIREN. The input to the network is æ Rd, one input node for each dimension. The network is trained to satisfy the PDE with boundary conditions by minimizing cost function.
SINGLE KNOWN EIGENVALUE
We first address the problem of finding a single eigenfunction u(x) given its corresponding eigen- value We approximate u(z) by NN and optimize the following cost function
Fi (u(2, 0w)) a Tull Tll + ōllu voll,an + 1lul| cf ellou11.
(3)
where
Tu: Lu + Au,
with the Laplacian operator L A. The first two terms in equation are the Le and Loo fidelity terms where the latter promotes pointwise solution such that the equation is satisfied for isolated points well [Bar & Sochen [] The third term imposes boundary conditions and the forth normalizes the squared length u to = due to scale invariance of the eigenfunction (since (ar,, A) is valid eigenpair for a j 0). The final term is the standard weight decay term which sta- bilizes the network weights. The La norm ||.1[3 is defined as the Monte-Carlo Integral approximation norm:
Vol(22 N
@N Σ
|
12(x))2
(4)
In the first example we apply the Laplacian operator ID where 4 and u(() u(m/2) = 0.
Then,
=u= + Au(x).
The normalized analytical solution is therefore
2
u(x) =
sin sin(2r).
Figure demonstrates the outcome of the algorithm at iterations 1, 100 and 2500. As can be seen, the approximated solution approaches the sin function with Relative Mean Square Error (RMSE) 6.52e-4 4 and peak signal-to-noiss ratio (PSNR) 28.84.

---

Under review as a conference paper at ICLR 2021
0.8
0.75
0.0
0.6
0.6
er7or
0.50
0.4
0.4
0.2
0.00
00.25
4,0
0,0
-0.8
-08
Figure 2: The solution to u" + Au 0 with u00) u(TT) 0 at iterations 1, 500 and 1000. The
associated eigenvalues are À = 70 + 53, 5.5 + 0.5, + 0.03. Note that the solution is up to a sign.
SINGLE EIGENPAIR WITH THE SMALLEST EIGENVALUE
Next, address the case where the eigenvalue is not known in advance. We therefore limit our- selves to the smallest nontrivial eigenpair. This approach is analogue to power method approach, where only one dominant eigenpair found. Recall the Rayleigh quotient defined as [Miller 2016); Feld et al. (2019)
R(u) (Cu,
(5)
(u, u)
can be shown that the eigenfunction u is critical point R(u), where R(u) corresponding (nontrivial) eigenvalue A. Furthermore, if ū is a function which is close to then R(ū) approximates
X. the following cost function we replace À by R(u). Then equation 3 modified to
F2(u(x, 0.)) = a T u112 #ll + ōl|u 1.ll + + I1ul2 -
-c of el10.113 + xl) (6)
where
Tu Cu + R(u)u.
The last term minimizes R(u) and therefore attracts the solution to the lowest nontrivial eigenvalue.
The ground truth eigenpair is given by (VZsin(z>) 1) for f = [0, π). Figure shows the outcome of the proposed method at three iterations. The approximated eigenfunction is obtained up to a sign. The eigenvalue which is the value of the Rayleigh quotient converges to the true value À 1 with decreasing standard deviation along the mini- batchss Quantitative results are shown row of tables [ and [ where MAE stands for Mean Absolute Error and MRE for Mean Relative Error.
MULTIPLE EIGENPAIRS WITH THE SMALLEST EIGEN VALUES
A generalization of the former case is to find M eigenpairs with the corresponding bottom-M eigen- values. Following [Hait-Fraenkel & Gilboa (2019)). and using the orthogonality property of the eigenfunctions optimize the following cost function:
M
Fu(u, (z. Bw)) Σ
(al/Tuill + p1l + 8||u, vollaan + B1 c + + [1(R(
(wi Uy)
4

---

Under review as a conference paper at ICLR 2021
0.8
0.4
0.4
0.2
0.2
-0.4
-0.4
-0.6
-0.6
500
1000
1500
Epoch
Figure 3: Left panel: eigenfunctions for M = 3. Right panel: eigenfunctions for M = 4 are shown on the left and the convergence of eigenvalues are shown on the right. Clearly the convergence of the functions are un((x) v sin(na), A, = n²2 up a sign.
Table 1: Dimension= 1, Eigenfunction Error Analysis
M
Mean PSNR (per ef.)
Mean RMSE (per ef.)
1
28.84
1.30e-3
2
56.58
2.21e-6
3
52.21
8.54e-6
4
48.78
2.05e-5
where the last term is explicitly given by
M ΣΣ Σ
(u(x))u
with point size N.. For each eigenfunction u, impose the T.2 and Toxc terms, boundary ditions and normalization as before. The weight of the Rayleigh quotient is multiplied by % = 1/i since we a monotonic penalty of the size of Aę. The last term enforces the orthogonality tinct eigenfunctions. Figure deemonstrates the outcome of the algorithm in ID. In this case we have one network with M output values, one for every eigenfunctuion. The ground truth eigenfunctions of the Laplacian with u(0)= = u( = 0 are given by
dis-
2 sin sin) An n² 1,2,3
Un (x)
The left panel of Figure in the right panel shows the convergence of the four eigenvalues which as expected have the values 1, 4, and 16. Quantitative results summarized in tables I and
3 shows the results for M 3 and in the right for M = 4. The right figure
Next, we tested the method two-dimensions where we trained A different networks simultane- ously, each with single output, one for each eigenfunction. We found this architecture adequate for the multiple eigenpairs problem in the 2D case. The ground truth solution is then
Wram (xr, y) = sin(ner) sin(my), Anm = n² + m² n,, m 1,2,3...
for [0, 2 and u(x, 0) = u(zz, T) = u(0. v) = u( 3) = 0. Figure shows the results at different iterations for M 1. The expected eigenvalue is the lowest one (n 1, m 1).
5

---

Under review as a conference paper at ICLR 2021
Table 2: Dimension= 1, Eigenvalue Error Analysis
M
Ground Truth
Predicted
Mean MAE (per ev.)
Mean MRE (per ev.)
1
1.02
0.02
0.02
4
1.11, 4.09
0.10
0.07
3
1, 4,
1.08, 8,93, 4.13
0,09
0.04
1,4, 9, 16
1.12, 4.09, 9.03, 15.95
0.07
0.04
21
Figure 4: The solution to the 2D Laplace eigevalue problem, with M = 1 at iterations (from left to right) 1, 100 and 1000.
As the algorithm converges, the expected eigenfunction is clearly seen (right image). We further tested our performance for M = 4, see Figure 5 The figures from left to right stand for the four eigenfunctions. The rows from top to bottom are iterations 500 and 1000 respectively. As can be easily shown, these results are with accordance with the theoretical eigenfunctions (n, m) (1, 1), )) (1, 2), (2, 1), (2, 2). Quantitative results are summed up tables and where SSIM stands for structural similarity index measure.
As the proposed method does not depend on the discretization of the domain, it can be easily adapted to free-form domains. It demonstrated with M = 1, using the same square domain, with circular piece removed. The boundary conditions along the circular curve are not provided. We tested three different sizes of circles inferred at the whole rectangle. The ground truth is on the right column and is as for the full shape. that the error concentrates the missing
be seen in Figure 6 The left column is the outcome of the algorithm
The error is depicted in the central column. It is easy regions, and is extrapolated in smooth fashion.
7 COMPARISON TO HATRIX-BASER METHODS
In this section we compare algorithm with matrix-based methods where the continuous operator discretized by finite differences scheme to form the 2D Laplacian matrix with spacing h. The ap- proximation of the continuous operator, therefore, may yield significant numerical errors [Knyazev] [. In addition, the incorporation of the boundary condition is not straightforward since the boundary conditions may affect the construction of the matrix approximation of the operator. We calculate the eigenpairs of the matrix by standard Matlab solver with = 0.03. Quantitative results are shown in the last lines of tables and 4 As can be seen, the proposed algorithm outperforms in the cigenvalue analysis (Table 4 while in the eigenfunction analysis the results are comparable. Next, we compare our results to the power method. This method is a well known iterative algo- rithm where given diagonalizable matrix A the algorithm finds the maximal eigenvalue and its corresponding eigenfunction, while the inverse power method finds the lowest one [Bronson et al. 2.14).. We adopted the inverse power method to problem following [Bozorgnia (2016)). Fig- ure shows the outcome the inverse power method implemented in Matlab with h 0.03 and initial eigenvalue λ 1 in two dimensions. On the left is the estimated eigenfunction which cor- responds to the lowest eigenvalue (m = 1, n = 1). the middle the error |u (z, y) u(x y) and in the right is the convergence plot of the eigenvalue which converged to 2.02. In our

---

Under review as a conference paper at ICLR 2021
Figure The solution to the 2D Laplace eigenvalue problem M 4. The Iterations 1, 100, 1000 are shown from top to bottom, for each of the eigenfunctions.
Figure 6: Free-form domain results for excluding points in the right half circles. Left: The outcome of the proposed algorithm inferred in the full rectangle. The solution in the missing domain seems to be smooth extrapolation. Middle: error from the full ground truth |u full(u The colorbar is referred only this column. Right: Ground Truth. is clear that the error is condensed into the missing domain, where the information is missing.

---

Under review as a conference paper at ICLR 2021
Figure 7: Inverse power method. Left: Approximated eigenfunction, Middle: error lut u/. Right:
convergence of the eigenvalue. À = 2.02, RMSE=8.706 76e PSNR-20.57, SSIM=0.99
Table 3: Dimension= 2,
Eigenfunction Error Analysis
Mean PSNR(per ef.)
Mean SSIM(per ef.)
Mean RMSE(per
36.67
0.93
2.14e-4
2
31.66
0.89
1.33e-3
3
24.84
0.84
3,92e-3
18.97
0.52
1.52e-1
(Laplacian matrix)
19.39
0.75
2.45e-1
periments, the algorithm seemed to be sensitive to discretization and initialization. Table shows the performance of the proposed algorithm compared with the inverse power method. Better results are obtained in almost all quantitative measures. Furthermore, the inverse power method finds only single eigenpair, while the proposed method outputs the M smallest eigenpairs. Although it is possible to calculate more eigenpairs based on the previous one via orthogonality constraints, an undesired accumulated error may emerge.
IMPLEMENTATION DETAILS
Our network was constructed dense fully connected network with hidden layers architecture, each with a varying number neurons, from 26-100. Our code was implemented PyTorch. Each of our models was trained for 5000 iterations NVIDIA QUADRO RTX-5000 Graphics card. In the 2D case all the networks were trained simultancously. As for the activation function, since we are modeling smooth functions, and experimentally. For the ID experiments we used the tanh activation function, and in the 2D experiments we used the SIREN (Sitzmann al. (2020)] activation function. The SIREN activation has been shown to excel in modeling complex signals, and their higher-order derivatives. We used an Adam optimizer [Kingma & Ba (2014)) with default parameters. starting LR (Learning Rate) of 4c-3 was used such that it was reduced every 100 epochs by the factor of 0.7 until the min LR reached 5e-5.
found that ReL.U and its variants were less suitable both theoretically
We found that the weight initialization Katanforoossh & Kunin (019) was important for conver- gence. In our 1D experiments Gaussian initialization with 0.0 mean and 1.0 std was used. In the 2D experiments, we used the standard initialization proposed in [Sitzmann et al. (222)). Our hyperparameters were set as follows: a = le-1, = 1 = 5e-1,
1.5,c 1,p =
le-8, x = 1, 2. Our point set consisted of 45, 000 inner-points, and 1200 boundary points. In the one-dimensional case, the boundary conditions had two actual points. the two-dimensional case, points were taken along the boundary of the square. Instead of using exact T u/| bol we used a relaxed approximation: t Ekewp Le], with 40 for broader effect the Los fidelity term.
8

---

Under review as a conference paper at ICLR 2021
Table 4: Dimension= 2, Eigenvalue Error Analysis
M
GT
Predicted
MAE
MRE
Rayleigh Quotient std
2
2.01
0.0073
0.0037
0.0170
2
1.99
0.05
0.01
0.0432
4.91
2
1.94
0.07
0.019
0.0603
5.08
5
4.93
4
2
1.98
0.0490
0.0093
0.0234
4.93
5
4.96
7.91
Laplacian Matrix
1.96
0.1
0.02
5
4.90
4.90
8
7.84
Table 5: Comparison to the Inverse Power Method where Agt = 2
PSNR
SSIM
RMSE
Inverse power method
2.02
20.57
0.99
8.76e-3
Proposed
2.01
36.67
0.93
2.14e 4
9 CONCLUSION
Interesting works have been recently published the field of solving ODEs and PDEs with NNs [Raissi et al. (2017): Flamant et al.) (2020); Bar & Sochen (2019); Siriinano & Spiliopoulos [2018]] We propose novel method exploit these new innovations, along with specific constraints that exist in the mathematical nature of such problems, and shed some light the well-known eigen- value problem. Finding eigenpairs both linear and non-linear differential operators a problem that appears in many different areas in science and in many problems that arise in nature [Davydov (1976); [pez-Blanne Blanco et al.] (2013)]. Due to the possible complex nature such operators. finding robust method for solving this problem is challenging. Additionally, most such problems do not necessarily have a rectangular domain, and will therefore make the task more difficult across meth- ods. Our method fits any free-form domain, and is able alleviate these constraints. Furthermore, the proposed method can be used in principle on higher order, non-linear operators and on higher dimensional manifolds. When using the Rayleigh quotient learn both the eigenfunction and its corresponding eigenvalue, the end-to-end learning process seems to assist the convergence, each acting as constraint towards the other. When learning multiple eigenpairs at once, the orthogonal- ity term eliminates degeneration. Empirically, it seems that these constraints also assist both the efs and the evs to converge towards more accurate solutions. This suggests that the orthogonality term serves regularizer each eigenpair during the training process.
The training process produces smooth solutions, due to the fact that the networks are composed of linear layers and smooth activation functions (namely tanh and SIREN). Since these solutions are deterministic and parameterized by the learned weights, they are infinitely differentiable. The fact that the solution is given in explicit form, enables exact analytical differentiation. Rigorous analysis of the approximation error and its relation to the network architecture and design are under current study. Future research includes higher dimensions and non- linear operators.

---

Under review as a conference paper at ICLR 2021
REFERENCES
I.. Bar and N. Sochen. Unsupervised deep learning algorithm for PDE--ased forward and inverse problems, arXiv 1904.05417v1, 2019.
N. Benouhiba and Z. Belyacine. On the solutions of the (p.q)-laplacian problem at resonance. Nonlinear Analysis, 77, 2013.
F. Bozorgnia. Convergence of inverse power method for first eigenvalue of p-laplace operator. Nu- merical Functional Analysis and Optimization, 37, 2016.
R. Bronson, G. B. Costa, and T. Saccoman. Chapter eigenvalues, eigenvectors, and differential equations. In Linear Algebra (Third Edition), pp. 237 288. Academic Press, third edition edition, 2014. ISBN 978-0-12-391420-- doi: https://ooi.org/11.. B978-0-
URL http: / /wwww swwencedirect..c.mmmmmmoo.o wwtpreett..c//cceenneeecee
42000000440000044
R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. Neural ordinary dif- ferential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 6571-6583, Curran Associates, Inc., 2018. URL http:://aapet www.sssiiss..c..co..o cctpaper//ppr////////p// fifferenttee welll-euutiinssso.e. paf
K. Choo, A. Mezzacapo, and G. Carleo. Fermionic neural-network states for ab-initio electronic structure. arXiv 1909. 12852, 2019.
B. Conway. Course Functional Analysis. Springer- Verlag New York, 1985.
A. Davydov. Quantum mechanics. In Quantum Mechanics. Translated, edited, and with additions by D. ler Haar (2nd ed.). pp. 237 - 2888 Oxford: Pergamon Press, second edition, 1976.
S. Eastman and D. Estep. A power method for nonlinear operators. Applicable Analysis, 86(10): 1303-1314, 2007, doi: 10. 000100000033681101666600000
URL httts:://eei.orr//10////
000368 66666666666
M. A. Erickson, R. S. SMITH, and A. J. LAUB. Power methods for calculating eigenvalues and cigenvectors of spectral operators on hilbert spaces. International Journal of Control, 62(5);
1117-1128, 1995. doi: 00108000000011199099255555
URL/attps://aci.orr
108C/
00207179508921586
T. Feld, F. Aujol, G. Gilboa, and N. Papadakis. Rayleigh quotient minimization for absolutely one-homogeneous functionals. Inverse Problems, 2019.
C. Flamant, P. Protopapas, and D. Sondak. Solving Differential Equations Using Neural Network Solution Bundles. arXiv e-prints, art. arXiv:2006. 14372, June 2020.
E. Hait-Fraenkel and G. Gilboa. Numeric solutions of eigenvalue problems for generic nonlinear operators. arXiv 1909.12775 2019.
Han, L. Zhang, and Weinan. Solving many-electron Schrödinger equation using deep neural networks. Journal of Computational Physics, 2019.
J. Han, J. Lu, and M. Zhoh. Solving high-dimensional eigenvalue problems using deep neural networks: diffusion Monte Carlo like approach. arXiv 2002.026001., 2020.
Hermann, Z. Schätzle, and Frank Noé. Deep neural network solution of the electronic Schrödinger equation. arXiv 1909.08423, 2019.
Katanforoosh and Kunin. Initializing neural networks, 2019.
D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, art. arXiv: 111.:1412,690. December 2014.
A. Knyazev. Toward the optimal preconditioned eigensolver: Locally optimal block preconditioned conjugate gradient method. SIAM Journal Scientific Computing, 23(2), 2000.
10

---

Under review as a conference paper at ICLR 2021
J.R. Lpez-Blanco, Blanco, R. Reyes, J. I. Aliaga, R. M. Badia, P. Chacon, and E. S. Quintana Orti. Ex- ploring large macromolecular functional motions on clusters of multicore processors. Journal of Computational Physics, 246, 2013.
G. Miller.
Spectral Graph Theory and The Laplacian Paradigm, Lecture 27, 2016.
http:/ www cs. cmu
IreEacchcchaa//aaayaaaaaaaloleie pdf
M. Ovsjanikov, M. Ben-Chen, J. Solomon, A. Butscher, and L. Guibas. Funetional maps: a flexible representation of maps between shapes. ACM Transactions Graphics, 2012.
D. Pfau, J. S. Spencer, A. G. de G Matthews, and W. M. C. Foulkes. Solution of the many-electron Schrödinger equation with deep neural networks. arXiv 1909.02487, 2019.
K. K. Pradhan and S. Chakraverty. Chapter four finite element method. In Computational Struc- tural Mechanics, pp. 25 28, Academic Press, 2019. ISBN 978-0-12-8154922 doi: https://doi. /100::00066689978000---8555-2220
URL http: www www.scccenceiieec.oo com/
M. Raissi, Perdikaris, and Em Karniadakis. Physics Informed Deep Learning (Part
I): Data-driven Solutions of Nonlinear Partial Differential Equations. arXiv e-prints, art. arXiv:1711.10511, November 2017.
O. Rippel, J. Snoek, and R. Adams. Spectral Representations for Convolutional Neural Networks. arXiv e-prints, art. arXiv: N0v::55060555 506.03667 June 2015.
M. H. Saad. 5 formulation and solution strategies. MARTIN H. SADD (ed.), Elasticity, pp. 83 102. Academic Press, Burlington, 2005. ISBN 18--80--2-60811111 1-6. doi: https://oi.orr/110 10.
101609977-001600811146500666 3. URL htttt:::: http:://wwww.... sc encedi rect. wttettcccm/cccceccceee
article//111/997701122888 26058 1600066
J. Shi J. Malik. Normalized cuts and image segmentation. IEEE IRENTAAAATOONN ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 22, 2000.
J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339-1364, December 2018. doi: 10.101 jcp.2018.08.029.
V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein. Implicit Neural Representations with Periodic Activation Functions. arXiv 2006. 09661, 2020.


Reviews:
[{"review_id": "wjFOE3SPAm2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors frame the decomposition of the Laplacian equation as an unsupervised regression problem that is using a 5-level (and fully connected?) neural network as regression function.  A cost function to be used in the optimization is proposed that is expanded to eigendecomposition problems of increasing complexity. A comparison with a classical method indicates that the proposed approach is comparable to or better than the former for the given task.\n\nI like the overall approach and the idea of framing the solution of the eigenvalue decomposition as an ANN regression problem, and I would see that this is of interest to ICLR. From an application perspective, the authors refer to computer vision tasks, I would be interested in a somewhat deeper discussion of constraints of the approach, for example:\n\nAs both methods are somewhat on par in terms of various performance indicators I was wondering whether the proposed approach is faster?\n\nIn real world problem critical information is contained in the boundary conditions. Could the authors discuss pros and cons of their approach with respect to them? And, again, compare to existing methods?\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_L2lwbfwnIR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript proposes a deep learning solver for the eigenvalue problem of differential self-adjoint operators. Specifically, the aim is to calculate M lowest eigenvalues and their corresponding eigenfunctions. This work is a natural follow up of the work by Bar and Sochen (2019) for solving PDE-based problems.  The unsupervised loss resembles the loss suggested in Bar and Sochen (2019), with addition of mainly two terms: (a) The Rayleigh Quotient term and (b) the orthogonality constraint.  In practice, the proposed framework is tested on the 1D and 2D Laplace operators on regular domains. \n\nPros. \n1.\tThe authors address an important and practical problem \n2.\tThe manuscript is self-contained, written very well and easy to follow \n3.\tThe proposed framework has the potential to handle eigenvalue problem of self-adjoint differential operators over irregular domains. \n4.\tMesh-free approach\n\n\nCons. \n\n1.\tIncremental novelty. I feel that that the proposed framework has a quite limited novelty as it naturally extends the work by Bar and Sochen (2019)\n2.\tGeneralization.  Practically, the proposed framework is an optimization framework and not a training framework. In other words, each new instance requires re-training of the neural net, in case of modification of the coefficients of the differential operator  and /or the boundary conditions.  Moreover, in the 2D case, M networks are optimized simultaneously.  \n3.\tI think that  it is worthwhile to refer to existing works, solving the eigenvalue problem by neural net.\n4.\tLimited experimental work.  The proposed framework has the potential to handle irregular domains. However, as far as I can see the experiments are done over regular domains for a special instance of differential operator, namely the Laplace operator.\n5.\tComparison.   The comparison is made versus the inverse power method. I think that a comparison to a very relevant solver, LOBPCG, is missing.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WOZOnsEJ8XK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a framework to estimate the eigenfunctions and eigenvalues of (non-linear) differential operators. The main idea is to parameterize the eigenfunction with a neural network and minimize a loss function which enforces the definition of eigenfunctions along with auxiliary terms enforcing boundary conditions, smoothness and getting rid of de-generate solutions. \n\nMy main concern is the lack of novelty/understanding what the contribution is. Going through the existing literature (section 2.1 in related work), it seems that the standard way to solve a differential equation using deep networks is to parameterize the solution using a deep network and optimize a loss which captures the physical relation (that is the PDE itself) along with auxiliary terms for smoothness, boundary condition etc. In that sense, it is not clear to me what the contribution of the paper is. The main loss in the paper is definition of eigenfunction since that is the problem they're interested in and in the literature for solving PDEs, the main loss is the definition of the PDE itself. The authors should put their contribution in context with existing literature and highlight the changes they've made. \n\nIn addition, it's unclear to me why we are particularly interested in the lowest M-eigenpairs. Is there a reason to concentrate on the lowest eigenvalues instead of the highest? I don't think this is motivated in the paper. Further, section 7 says \" Furthermore, the inverse power method finds only a single eigenpair, while the proposed method outputs the M smallest eigenpairs.\". Can we not find the smallest pair and run the method again with a constraint that the eigenfunction should be orthogonal to the one we already found to find the next smallest eigenfunction? (similar to what we do in power method?)\n\nFinally, the experiments only concentrate on a Laplacian operator. In order to claim that is general and works across multiple operators, I believe that the results should be demonstrated  on multiple differential operators (of varying complexity levels). Unfortunately, I do not know enough about differential operators to suggest what these test cases should be. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DrHiJzYgXSm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper proposes to use deep learning neural network to model eigenfunction while solving a generic eigenvalue problems. Since such problems is very generic this application of such framework are huge and the experiments show some promising results. \n\n##########################################################################\n\nReasons for score: \n \nOverall, I vote for rejecting (see cons for more details). The idea is very good, but many important information are missing. For example it is impossible to reproduce the results since we don't have any clue on the used networks.\n\n##########################################################################\n\nPros:  \n\n1. This paper proposes a new framework for generic eigenvalue problem which seems promising.\n\n2. The framework is able to deal with several eigenfunction and almost assure the orthogonality of the function.\n\n3. This paper provides comprehensive experiments, that shows that such methods cam catch complex eigenfunction. \n\n##########################################################################\n\nCons: \n\n1. The notation is confusing on some parts. First please use <,> for scalar product instead of (,) since you have functions with two arguments. Second I don't w_u is a good notation for the weights , for example in equation 6 we don't know which weights it is. Perhaps using a function which gives the associated weights could help.\n\n2.  When dealing with multiple eigenfunction, there is en additional term is the equation 6 for the orthogonality constraints. However it is not clear on how it is approximate, I assume that it is something similar to equation (3). Please clarify.\n\n3. Again on the orthogonality constraint, the complexity of such constraint is huge. It is impossible to deal with many eigenfunction. Such limitation can drastically reduce the interest of the framework, I think there should be some discussion on this point.\n\n4. On the implementation details, we don't have any clue on which kind of neural network are used. Are they just dense network?\n\n5. Again on the implementation details, the loss function of equation 6 is highly non-convex and I assume that there is several way to try solving it. I don't think it is possible to deal with all the networks (the u) at once, so I figure there is a solving scheme (most likely alternate minimization) behind. Please clarify as it may answer to one of my previous remarks. \n\n##########################################################################\n\nQuestions during rebuttal period: \n \nPlease address and clarify the cons above \n\n#########################################################################\n\nSome typos: \n\nall pages: be careful with the citation, when citing a paper please add parenthesis\n\npage 2: There is an extra parenthesis in equation 2", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of solving for the eigenpairs of a self-adjoint differential operator. This problem, of course, is classical; the main innovations here are 
a) the use a parametric form of the (pointwise) solution using a (shallow) neural network so as to avoid discretization, and 
b) obtaining multiple eigenpairs simultaneously as outputs. 
The techniques themselves (i.e., the architecture, the loss function, and the training procedure) are fairly standard.

There was some variance in the review scores. The reviewers appreciated the importance of the problem and the direction adopted by the authors. However, concerns were raised regarding: the limitations of the experimental evaluation; a lack of sufficient distinction from prior work; *very* limited comparisons with prior approaches; and the limited demonstration of generalizability. I agree with all these criticisms, and therefore vote to reject. 

Decision: Reject