Accelerating Optimization using Neural Reparametrization
Authors: Nima Dehmamy
Csaba Both
Jianzhi Long
Rose Yu
Venue: ICLR 2022 Submitted (2022)
Keywords: optimization
graph neural networks
neural reparameterization
neural tangent kernel

Abstract: We tackle the problem of accelerating certain optimization problems related to steady states in ODE and energy minimization problems common in physics. 
We reparametrize the optimization variables as the output of a neural network. 
We then find the conditions under which this neural reparameterization could speed up convergence rates during gradient descent.
We find that to get the maximum speed up the neural network needs to be a special graph convolutional network (GCN) with its aggregation function constructed from the gradients of the loss function.
We show the utility of our method on two different optimization problems on graphs and point-clouds. 

Under review as a conference paper at ICLR 2022
ACCELERATING OPTIMIZATION REPARAMETRIZATION
USING NEURAL
Anonymous authors Paper under double-blind review
ABSTRACT
We tackle the problem of accelerating non-linear non-convex optimization prob- lems. We discover that reparametrizing the optimization variables as the output of neural network can lead to significant speedup. We examine the dynamics of gradient flow of such neural reparametrization. We find that to obtain the maxi- mum speed up, the neural network architecture needs to be specially designed graph convolutional network (GCN). The aggregation function of the GCN is con- structed from the gradients of the loss function and reduces the Hessian in early stages of the optimization. We show the utility of our method on two optimization problems: network synchronization and persistent homology optimization, and find impressive speedup, with our method being 4 ~ 80x faster.
INTRODUCTION
Gradient-based optimization lies at the heart of machine learning. For non-linear, non-convex large- scale optimization problems, gradient-based optimization can suffer from slow convergence and significant computational bottleneck. In linear systems, preconditioning Axelsson (1996); Saad & Van Der Vorst (2000) can accelerating convergence by multiplying a symmetric positive-definite preconditioner matrix to the original problem. In effect, preconditioning re-scales the loss landscape to have more circle-like level sets, hence changing the condition number of the system. Adaptive gradient optimizers such as AdaGrad Duchi et al. (201 1) and Adam Kingma & Ba (2014) uses the same idea but updates the preconditioner during iterative optimization with adaptive step size, which leads to great improvement in convergence speed.
Our goal is improve the convergence of non-convex optimization further. First, we examine the dynamics of the gradient flow and identify the condition number that dictates the speed of conver- gence. We discover that changing the condition number naturally requires the reparametrization to be a type of graph neural networks (GNN). Based on the analysis, we introduce novel neu- ral reparametrization scheme, which generalizes preconditioning to non-linear systems. Note that our method is complementary to existing gradient-based optimization algorithms and can be used in conjunction with any adaptive gradient optimizer (e.g. Adam). We test our hypothesis on two highly nonlinear optimization problems and obtain impressive speedups.
A few earlier work Sosnovik & Oseledets (2019); Hoyer et al. (2019) have demonstrated that convo- lutional neural networks can serve as priors to improve the parameterization of a class of structural optimization problems. But the theoretical foundation behind these techniques are not well under- stood. Concurrently, the implicit acceleration of peee-parametrizationn in linear neural networks have been analyzed in Arora et al. (2018); Tarmoun et al. (2021). Specifically, Arora et al. (2018) shows that reparametrizing the linear weights with deep linear networks impose a pre-conditioning scheme on gradient descent. Tarmoun et al. (2021) further exploits the conservation laws in gradient flow and derive convergence rate for two-layer linear networks.
In contrast, our method applies to highly nonlinear problems, guided by theoretical insights. Our contributions can be summarized as follows
1. We analyze the dynamics of the gradient flow and derive how a neural reparametrization (NR) can lead to a speedup.
2. We prove that maximum speedup is achieved if the NR is a special GCN which uses gradi- ent of the loss in its aggregation function.

---

Under review as a conference paper at ICLR 2022
3. For early steps, show that the NR should use the Hessian of the loss, leading to an efficient hybrid implementation.
4. We showease NP on two optimization tasks: network synchronization and persistent ho- mology and observe consistent speedup.
Our method also demonstrates a novel usage and unique perspective for GNN, specifically GCN (Kipf & Welling, 2016). The majority of literature treat GNNs representation learning tool for graph-structure data (see surveys and the references Bronstein et al. (2017); Zhang et al. (2018); Wu et al. (2019); Goyal Ferrara (2018)). However, our analysis shows that GNN can be used to modify optimization dynamics in problems without explicit graph data.
2
METHODOLOGY
We consider the problem of minimizing a smooth, lower-bounded loss function L(w) abounded variable I
E
R on
or
When the optimization variables are matrices or tensors, w represents flattened vector containing all variables. w also represent all trainable parameters in a deep neural network. For supervised learning with a dataset Z = y)) the loss function [(w) becomes L(w; Z). The dataset and the model class (e.g. architecture of neural network) together define loss landscape function of the optimization variables w.
Gradient-based optimization updates the variables w by taking repeated steps in the direction of the steepest descent -=+ w² -E "ww" the dynamics of w(( R" during the optimization process, treating / continuous time. Using steepest descent for the optimization, dw/di follows the gradient flow:
ac for a learning rate e. With very small time steps öt, we examine
OC dw, -Ei ow;"
where denote the learning rate as a diagonal matrix of learning rates, because modern optimizers such as RMSProp (Tieleman et al., 2012) or Adam (Kingma & Ba. 2014) choose adaptively for each wP Specifically, in Adam and RMSProp, which we use in experiments, we have
Eig (c,t)
dij
(2)
L. +
where dij is the Kronecker delta, and and & are small numbers. The expected value E is calculated using stochastic samples for V.L over multiple iteration steps, with more weights given more recent steps. Equation is also an ordinary differential equation (ODE). The local minima of L(w) correspond to the stationary solutions of the ODE, i.c. dw/dt = 0.
We are interested in the expected convergence rate B[dC/dt] of the loss func- tion. This expectation can be either approximated a fashion similar to adaptive gradient optimizers (i.e. exponential time average) be over random initializations under a distribution (0). We will assume the expectation value for and E/dC/ dt) are evaluated with the same method, meaning assume [E[e] E. The convergence rate then given by
Convergence rate
d((w) dt
OC dw:
OC Oc ow; Ow,
E
E
Tr [cM]
Mij E
(3)
ow: dt
where used our assumption get B[eM] [MM with My V.LV, being the "square gradients" matrix and M = =EMM]. With this definition, equation 2 becomes e = m// diag M + + Note that M time-dependent and can change during optimization.
Estimating M If we know the distribution (tw(t)) for some we can approximate M. One way to do this is similar to Adam as weighted average over the past few steps. Also, if we know the distribution of the initialization P(w(O)), evaluate M for early stages, which is what
2

---

Under review as a conference paper at ICLR 2022
Reparametrizaionn
Neural Network
Liv
L
Networks
Neural Reparam.
data
deta
W
W
Figure 1: Original (a) Neural Reparametriiation (b) of the optimization problem.
will in the current paper. When using Xavier initialization (Glorot & Bengio, 2010), have (0) (O, In this case can show (see appendix A.1)
02L 8L
(ty(t + 100000mm Elww Ow,dw, ow,uw, + +0
[?[
+ +0(n
(4)
1-00
1,
where Hiy 0 ((w) /Ow, is the Hessian of the loss at t 0.
The eigenvectors of M with zero eigenvalue are modes that do not evolve during optimization. As gradients are zero in these directions, GD can never find and explore those directions. Therefore, we may exclude them from the beginning and claim that M is full rank.
The rate dun:/ can different along different directions of the parameter space, with fast and slow modes. Next, we quantify this difference in rates and suggest ways for acceleration.
2.1 FAST AND SLOW DYNAMICS OF GRADIENT FLOW
M is positive semi-definite because for any vector R" we have v" Mu E
= 2 0.
Thus, M is Hermitian with spectral expansion M = Σ; m.ttsaf. Using equation 1, we can show that the dynamics of w along different 4/; are orthogonal to each other
dw
dw di
)
E
e² = M eseym;dij.
(5)
dt
e is constant and not adaptive, as evolves faster along modes 1/s with larger eigenvalues m,, since if my >> my we have
( dw
27
dw 2
m = EE
=mmm
(6)
di
dt
This is precisely what optimizers such as Adam try to address. Ideally, having mm, would make up for the rate difference, resulting in uniform GD where no mode vb; is evolving more slowly than others. However, calculating m; is ((²²) and can be quite expensive during optimization, hence the approximate version equation 2 is used. show here that in many cases it is worthwhile to do full correction to GD using M, at least in early stages of the optimization.
The eigenvalues of M determine the rate of evolution of the overlaps along each of its eigenvectors. In particular, if some eigenvalue Matow < mean;/mu is much smaller than the mean, the evolution of w along d'stow will be much slower than other directions. Therefore, we will refer to all d'stow as the slow modes of the optimization problem. Conversely, the v/r fast where m mfast fast mean; [m.] will be referred to as the fast modes.
When running GD, the maximum change in w is bounded to ensure numerical stability. Because of the orthogonality equation 5, we can enforce a numerical bound via
dwo
dw dt
max E
< <T²
Eman
E
dt
v Milmaar
M-naz
where Myma max; m, is the largest eigenvalue of M. Therefore, the learning rate is bounded by the largest eigenvalues (fastest modes). Thus, order to speed up convergence, we need to focus on the slow modes, which can be achieve with reparametrization of w.

---

Under review as a conference paper at ICLR 2022
2.2 NEURAL PARAMETRIZATION OF OPTIMIZATION PROBLEMS
Using neural networks to reparametrize optimization problems have been suggested in (Hoyer et al., 2019; Sosnovik & Oseledets, 2019). The idea is that the optimization variable w be defined as the output of a deep neural network (0) = (()) with trainable parameters 0. Then, rather than optimizing over the original variables directly, we can instead optimize over the neural network parameters 0. We will take this idea further and analyze the type of neural network architectures that can accelerate the optimization.
Modified convergence rate ing updating rule for 0:
The neural reparametrization of the variables w(0) leads the follow-
dla
OC Ea 00.
OC ow; Ow, 00,
= E
(8)
dt
The convergence rate is then given by dC ow; d0a
dC dl
OC Oc Σ Ea 00, 00.
ow; dw
Σ
Σ W
Ow, 00, dl.
Ona own
i,a
Σ 1
Mi - [MKT]
Kij =
Σ
Ea 00, 00.
Iw dw;
(9)
Here K' is the neural tangent kernel (NTK) (Jacot et al., 2018) of w(f), and Mij is the squared gradients. Note, we have absorbed &a into K for convenience of notation.
The expected convergence rate E[dC/d( equation 3 depended on M = E M] the reparametrized one from equation 9 depends on B[MK]. In order to quantify any improvements in EldC/dd dt] from equation 9, we need to be able to express E K] in terms of E[M], which is not possible for general K.
Expected convergence rates
we assume M and to independent from each other, i.e. B[MK] becomes E [[M] Since M is function of w, K must not be function of w. One way is to have a b) any other similar neural network. We will discuss explicit choices for w below sec. 2.3. In summary, an important constraint that will put w() is that it yields K independent of w
choose (0) s.t.:
M[MK] = [[M]E[]]
(10)
Now we can compute the expected rate of change of (()) along the previous fast and slow mode eigenvectors th of M, similar to equation 6. Using equation and equation 8 we have
dw
Vi Σ
dw do,. 08, dt
6i Σ Ea 00_
Ow Σ Oun dL T K
-
Oc
(E R)
(11)
00, Ow,
Ow
where 0L C is the gradient vector. Multiplying equation 11 by its transpose, denoting K E K and using equation 10 the expected value becomes
dwo
E
WKKKK' U. E (o Ke,)
(12)
di
which shows how the neural reparametrizatioon may speed up the convergence.
First, note that NR changes the rate of evolution of modes ti; in equation 7. We can quantify this change when K, M = 0. In this case, K and M can be diagonalized simultaneously with shared eigenvectors Vi. Plugging the spectral expansion K Σ, kaby into equation 12 and using e < n/ Memax (equation 7) we have
dur
mak? Mmnax
if:
K, M =0
E
= k? my < n?
(13)
dt
Thus, one way to speed up the convergence, is to have [K, M = and have increase the slowest rates (eosttoddd//dl while keeping the fastest rate mostly unchanged. As result, have:
4

---

Under review as a conference paper at ICLR 2022
Theorem 1. With neural reparametrizaaion yielding M ideal maximum rate max; || hl////////l dw/dttl < 7,. need to have
O. to make all modes th reach the
1/2
M M-max
K
(14)
Proof. This is because plugging kg ~ mmaa/ (m/ + E)(E (5 < 1) into equation 13 makes the r.h.s. equal r/.
Next, we discuss how we can achieve this in practice. Specifically, we derive what architecture for the neural network w(t) f(0) will result in K satisfying the condition in equation 14.
2.3 PRACTICAL IMPLEMENTATION
To find a neural network model that satisfies equation 14, we need to consider the fact that (ww(t) changes during optimization. To minimize the overhead from constructing w()) (and hence K) as function of M, the architecture has to be simple as possible and involve steps which do not increase the time complexity of each iteration significantly. We will now discuss when such neural reparametrization is feasible and how to implement equation 14 in practice.
Hybrid optimization use equation at points other than the initial steps, we need to dynami- cally estimate and update M. Though estimating (t)) efficiently with ideas from Adam is possible, in the current work we focus on the speeding up at early stages using equation 4. Thus, our main proposal is to a two stage hybrid optimization: 1) use the Hessian to reparametrize w(() in early stages for some number of iterations and then 2) switch to the original optimization over w.
Simple implementation Because M may depend on w()), as in equation 10, need a K that independent of 0 to ensure [[ = [[]] The simplest architecture a linear model w. Σ. Oa. which yields K = KK Ps (M [M/mman (t/mmaa))1/// - We need K to a full rank matrix, meaning that x needs to be at least n x n. Since we need the least computationally expensive K, we matrix. This way, = VK (M/ (M/mmax)-1// For the initial stages, where M H2/ ///nnnn we have (H/hhaaz) -1/2 with H. being the Hessian.
choose 0 E RR. and let 6. be symmetric
Per-step time complexity Since w E R", the complexity of computing L(w) is (mp) where depends on the sparsity structure of L. M is an n x n matrix and computing M - is (( which is very expensive when >> 1. The eigenvalue mynar can be estimated iteratively in O(gn²). Hence, computing K by exactly following equation 14 is not feasible, unless M is fixed and can be calculated offline. Even then, evaluating M 1/2 only makes sense if the expected GD steps for convergence more than O(n), otherwise we don't gain any speedup. We may instead try to approximate M-1/2 and mmaz iteratively with complexity (gn2) where 9 & n is the number of iterations. Hence note that using equation 14 to speedup the optimization may generally be useful if the the per step complexity of GD on has a time complexity of least O(gn²).
Implementation constraints In summary, we have the following constraints on the structure of the neural network w+))
Computing w=()= E RD must at most O(qn²) with q & n
2. To have K independent M, Ow/06 must be independent of w and hence, only linear maps w w0)) = x& are allowed, with k independent of w.
3. 0 R with m n. We choose m n,, which yields k VK (M /Mmaz) -1/4
4. Instead of /MM//=aaaa (0/mmaa))-1/// which is O(r²), should use O(gn²) approximations.
5

---

Under review as a conference paper at ICLR 2022
Relation to GCN Since both A R" and E R", our linear = nō architecture is essentially a Graph Convolutional Network (GCN) (Kipf & Welling, 2016) with the aggregation rule k or a weighted adjacency matrix. In fact, all of our derivations remain unchanged if w [exd and 0 E R= (i.e. if there are d and m features per node in and 0). Thus, claim is that we can speed up the optimization process by reparametrizing w using a GCN with linear activation, the loss gradients. However, evaluating this & during GD can be quite expensive (O(n")) for large (GD steps (nn)). Ideally want approximation for 6 which is (gqn²) with q ~ O(1)
(M (M/mman //aaz)
) 1-141,
w k8, where 0 are trainable and is the aggregation function derived from
Computationally efficient implementation We will focus speeding up the early stages, in which we use the Hessian H in M H2/n. Define H = -1 ())H))H/llaa where Imaa = v Mmaa and ş < 1. To get an O(qn²) approximation for x = (M ) H we can take the first q terms in the binomial expansion (see Appendix A.2). A rough estimate would be
1
3 (1- 4
H 111 -1
(15)
2
Since H is positive semi-definite and its largest eigenvalue is 1 - { < 1, the sum in equation 15 can be truncated after ~ O(1) terms. The first terms of equation 25 can be implemented as a q layer GCN with aggregation function fMM = My and residual connections. We choose to be as small or 2, larger q may actually slow down the optimization. Note that computing (Mo is (gqn²) because we do not need to first compute J(M) (which is (qn²)) and instead use 1 f(M) (O(n²)) times with in = 0.
Lastly, to evaluate My need to estimate the leading eigenvalue maz-- Given any vector R and using spectral expansion, we have
1T 1TH1
E H2
H0v= Σ m 12 (Ufe) v)
hmaz
(16)
Σij His
where, since H is positive semi-definite, for q > the leading eigenvector Umaz quickly dominates the sum and we have Hdv s hi V) Here we chose = 2 to get crude approx-
imation for hmaa- The generalized Perron-Frobenius theorem (Berman & Plemmons, 1994), state the components of the leading eigenvector tima should be mostly positive. Therefore, we chose v = 1/Vn, where 1 = (1....,,)) which should be close to the actual Vmaz-
If we think of M as the weighted adjacency matrix of graph with vertices, the vector component D, = [7(1]; = Σj Hig is the weighted degree of node i. Hence equation 16 states H M/mmmax R D D)/1 H. This is similar the graph diffusion operator D-1H and D-1/2HD- 1/2 which are used as the aggregation functions in GCN (here Dij = D2ō is the degree matrix).
When using modern optimizers, the adaptove learning rates is a term within K. This will interfere with the optimal choice for and hence our GCN aggregation function. Therefore, practice, we do not fix the coefficients for 7 and H22 in our GCN. Instead, we use single or two layer GCN with aggregation function (++ = D D-1/2
3 EXPERIMENTS
We showcase the acceleration effect of neural reparametrization two non-convex optimization problems: network of Kuramoto oscillators which are widely-used for modeling synchronization phenomena; and persistent homology, mathematical tool that computes the topology features of data that persist across scales. We use Adam Kingma & Ba (2014) optimizer and compare different reparameterization models. We implemented all models with Pytorch.
3.1 NETWORK OF KURAMOTO OSCILLATORS
Kuramoto model Kuramoto (1975: 1984) are widely used for synchronization problems, which have profound impact on engineering, physics and social systems (Pikovsky et al., 2003). As shown in Fig. 3(rihht), Kuramoto model describes the behavior of large set of coupled oscillators. Each

---

Under review as a conference paper at ICLR 2022
50
Teut
2505 stuae
CCN
1.0
-50
3000
BOOK
UnIT
3000
00e0 X1
Dnw
Figure 2: Optimizing Kuramoto model 25 x 25 square lattice (a) Loss over time in seconds for different methods. (b) Evolution of the phase variables over iterations. (c) Level of synchroniza- tion, measured by global order parameter time. Neural reparameteriizttion with GCN achieves the highest speedup.
oscillator is defined by an angle 0, wit + di where w; is the frequency and @; is the phase. We consider the case where = 0. The coupling strength is represented by graph adjacency matrix with elements Ai E R. Defining Aij = 0i oj. the dynamics of the phases (t()) the Kuramoto model follows
dØ:
E Σ ==1
Ags sin Aij,
[ = Σ
Agi cos Aij.
(17)
dt
fJ=1
Hopf- Kuramoto (HK) model Lauter et al. (2015) is a more general version of the Kuramoto model. includes second neighbor interactions. HK has rich phase space with qualitatively different solutions. The phase space includes regions where simulations becomes slow and difficult. We use the HK model to showcase our method in more complex and nonlinear scenarios. The HK model's dynamics follows
dob dt
= e Σ Aji (cos Dij 81 A:y]
+ $2
Σ (A.jAJA [sin (A,r + A₃k) sin (Axr - A5k)l + AiA As sin (Afr + Ass)) (18)
k.j
with the following loss function
L C= -
AAx Ag (sin Auy +1 cos A.y +
$2 Σ AyA fcos (Axr + Dzz)) + cos (A₁r - Azk)]
28 k.k.j
1
(19)
Our goal is minimize the phase drift dé:/dt synchronize the oscillators. Existing numerical methods directly optimize the loss L()) with gradient-based algorithms, which we refer to as linear. We apply our method reparametrize the phase variables and speed up convergence towards synchronization.
Implementation. For early stages, we use a GCN with the aggregation function derived from the Hessian which for the Kuramoto model simply becomes (0)(0) = 8² L/O0,00 the graph Laplacian of A. We found that NR in the early
Ai
E. Asgō L,j. where D stages of the optimization gives more speed up. We implemented the hybrid optimization described earlier, where we reparemtrize problem in the first 100 iterations and then switch the original linear optimization for the rest of the optimization.
We experimented with three Kuramoto oscillator systems with different coupling structures: square lattice, circle graph, and tree graph. For each system, the phases randomly initialized between and 2T from uniform distribution. We let the different models run until the loss converges (10 patience steps for early stopping, 10-15 loss fluctuation limit).
Results of Kuramoto Model. Figure 2 shows the results of Kuramoto model on a square lattice. Additional results on circle graph, and tree graph can be found in Appendix B. Figure 2 (a) shows that NR with one- one-layer GCN (GCN-1) and GCN with residual connection (RGCN-1) achieves sig- nificant speedup. In particular, found 3.6 + 5 speed improvement for the lattice, 6.1 + for the

---

Under review as a conference paper at ICLR 2022
Hopt Surumeto mote!
0.64
0.11
10
0044
100 1ror -805
-20000
A.
0.05
25000
-20000
RCCUAA
RCCHA
speedu
8./0
Figure 3: Hopf-Kuramoto model on square lattice (50 x 50). a) Speedup in the final loss value difference function. Points color correspond to the regions of the phase diagram (b), also, the number above each phase pattern are the global order parameters. We fixed c = 1 for the simplicity. c) Coupled oscillator system.
circle graph and 2.7 .3 for tree graphs. We also experimented with two layer (GCN/RGCN-2) and three layer (GCN/RGNN-3) GCNs. As expected, the overhead of deeper GCN models slows down optimization and offsets the NR speedup gains. Figure 2 (b) visualizes the evolution of the oscillator phase & on a square lattice over iterations. We can see that even though different reparametrization models reach the same loss value, the final solutions are quite different. The linear version (without NR) arrives at the final solution smoothly, while GCN models form dense clusters at the initial steps and reach an organized state before 100 steps. To quantify the level of synchronization, we measure a quantity known as the "global order parameter" (Sarkar & Gupte (2021)): P = N ΣS e Fig- ure 2 (c) shows the convergence of the global order parameter over time. We can see that one-layer GCN and RGCN gives the highest amount of acceleration, driving the system to synchronization.
Results of Hopf-Kuramoto Model. We report the comparison of NR models on synchronizing Hopf-Kuramoto dynamics. According to the Lauter et al. (2015) paper, we identify two different main patterns on the phase diagram Fig. 9(b): ordered (small s2/c, smooth patterns) and disordered (large 82/c, c. noisy) phases. In all experiments we use the same lattice size 50 x 50, with the the same stopping criteria (10 patience steps and 10-10 loss error limit) and switch between the Linear and GCN model after 100 iteration steps. Fig. 3 (a) shows the loss at convergence versus the speedup. We compare different GCN models and observe that GCN with A² as the propagation rule achieves the highest speedup. Also, the disordered phases. Furthermore, we observed that the Linear and GCN models converge into a different minimum state in few cases. However, the patterns remain the same. (b) shows how the level of ordering changes region by region. If the global order parameter is closer to O, we have more of disordered phase while the parameter is closer to 1, meaning it a more organized pattern.
see that we have different speedups in each region, especially in
We know that running dynamics in a large lattice are computationally expensive, especially close to the phase transition points where difficult to separate the slow and fast modes.
3.2 PERSISTENT HOMOLOGY
Persistent homology Edelsbrunner et al. (2008) is an algebraic tool for measuring topological fea- tures of shapes and functions. It provides precise definition of qualitative features, is computable with linear algebra and robust to perturbation of input data (Otter et al., 2017), see more details in Appendix B.3. An example of persistent homology is point cloud optimization Gabrielsson et al. (2020); Carriere al. (2021). Given a random point cloud w that lacks any observable characteris- tics, we aim to produce persistent homological features by optimizing the position of data points.
((w) = - +
ww r
(20)
PED
where p E D [bb,, di), denotes the homological features in the the persistence diagram D. consisting of all the pairs of birth b, and death d, filtration values of the set of k-dimensional homological features Ig. TA is the projection onto the diagonal A and Σ, d( S) constrains the points within the a square centered at the origin with length 2r (denote r as range of the point cloud). Carriere et al. (2021) optimizes the point cloud positions directly with gradient-based based optimization.
8

---

Under review as a conference paper at ICLR 2022
Training Spocd-Up
Spood-Up
Loss
10
10
-0.5
100 pts
100 pts
Dura
200
8-
200
-1.0
Buselie
Speeed Up
-1.5
-2.0
-2.5
1.C Bange
4.0
Bange
Time (x)
Figure 4: Speedup: a.b) Training and total time speedup; e) GCN speeds up convergence.
GCN Model
NAE Ooud
Optiieed Putr Coud
2:00 pts
300 pts
Lo00 Value
52
-5
tw0.5
1=1.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
me40
Figure 5: a) Loss vs speed-up of GCN model. The converged loss value is determined by point cloud range only, while the speed-up is affected by both point cloud range and size. b) Initial random point cloud and GCN-optimized point cloud.
We will refer theirs as the baseline linear model. In contrast, we integrate GCN and reparameterize the point cloud positions, and will refer ours as the GCN model.
Implementation. To measure the speed-up of GCN model, we used the same Gudhi library for computing persistence diagram as Gabrielsson et al. (2020); Carriere et al. (2021). The runtime of learning persistent homology is dominated by computing persistence diagram in every iteration, which has the time complcxity linear model are very similar, and we demonstrate that the GCN model can reduce convergence time by factor of ~ 4. We ran the experiments for point cloud 100,200,300 points, with ranges of 0.5.1.0,..0.4.0. The hyperparameters of the GCN model are kept constant, including network dimensions. The result for each setting is averaged from consecutive runs.
( Thus, the runtime per iteration for GCN model and
Results. Fig. and 5(a) show that the speedup of the GCN model is related to point cloud density. Training converges faster as the point cloud becomes more sparse, but the speedup gain saturates as point cloud density decreases. On the other hand, time required for initial point cloud fitting increases significantly with the range of point cloud. Consequently, the overall speedup peaks when the range of point cloud is around 4 times larger than what is used be in Gabrielsson et (2020); Carriere et al. (2021), which spans over an area 16 times larger. Further increase in point cloud range causes the speedup to drop as the extra time of initial point cloud fitting outweighs the reduced training time. The loss curve plot in Fig. shows the convergence of training loss of the GCN model and the baseline model in one of the settings when GCN is performing well. Fig. 5 shows the initial random point cloud and the output from the GCN model. In the Appendix B.3. we included the results of GCN model hyperparameter search and a runtime comparison of the GCN model under all experiment settings.
Conclusion We propose neural reparametrization scheme to accelerate a large class of optimiza- tion problems. Our method is grounded on analysis that the dynamics of gradient flow are related to the condition number of the system. By reparametrizing the optimization problem with a graph convolutional network, we can modify the condition number and obtain the maximum speed up. The aggregation function of the GCN is constructed from the gradients of the loss function and re- duces to the Hessian in early stages of the optimization. We demonstrate our method on optimizing synchronization problems and persistent homology of pointclouds. Depending on the experiment, obtain best case speedup that ranges from 4 to 80.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning, pp. 244-255 PMLR, 2018
Owe Axelsson. Iterative solution methods. Cambridge university press, 1996.
Abraham Berman and Robert Plemmons. Nonnegative matrices in the mathematical sciences.
SIAM, 1994.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geomet- ric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-44 2017.
Mathieu Carriere, Frédéric Chazal, Mare Glisse, Yuichi Ike, Hariprasad Kannan, and Yuhei Umeda. Optimizing persistent homology based functions, In International Conference on Machine Learn- ing, pp. 1294-1303. PMLR, 2021.
Nima Dehmamy, Albert-Lásza Barabási, and Rose Yu. Understanding the representation power of graph neural networks in learning graph topology. Advances in Neural Information Processing Systems, 2019.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning reseurch, 12(7), 2011.
Herbert Edelsbrunner, John Harer, et al. Persistent homology-a survey. Contemporary mathematics, 453:257-282, 2008.
Rickard Brüel Gabrielsson, Bradley Nelson, Anjan Dwaraknath, and Primoz Skraba. A topology layer for machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 1553-1563, PMLR, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Palash Goyal and Emilio Ferrara. Graph embedding techniques, applications, and performance: A survey. Knowledge-Baud Based Systems, 151:78-94, 2018.
Stephan Hoyer, Jascha Sohl-Dickstein. and Sam Greydanus. Neural reparameteriizaton improves structural optimization. arXiv preprint arXiv: 140.:049999 2019.
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and gen- cralization in neural networks. In NeurIPS, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint arXiv:1412.6980 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- 2016.
works. arXiv preprint
John Michael Kosterlitz and David James Thouless. Ordering, metastability and phase transitions in two-dimensional systems. Journal of Physics C: Solid State Physics, 6(7):1181, 1973.
Yoshiki Kuramoto. Self-entrainment population of coupled non-linear oscillators. Interna- tional symposium mathematical problems in theoretical physics, pp. 420-422 422. Springer, 1975.
Yoshiki Kuramoto. Chemical turbulence. Chemical Oscillations, Waves, and Turbulence, pp-
111-140. Springer, 1984.
Roland Lauter, Christian Brendel, Steven JM Habraken, and Florian Marquardt. Pattern phase di- agram for two-dimensional arrays of coupled limit-cycle oscillators. Physical Review E, 92(1): 012902, 2015.
10

---

Under review as a conference paper at ICLR 2022
Nina Otter, Mason A Porter, Ulrike Tillmann, Peter Grindod, and Heather A Harrington. A roadmap for the computation of persistent homology. EPJ Data Science, 6(17), 2017.
Arkady Pikovsky, Jurgen Kurths, Michael Rosenblum, and Jürgen Kurths. Synchronization: uni-
versal concept nonlinear sciences. Number 12. Cambridge university press, 2003.
Yousef Saad and Henk A Van Der Vorst. Iterative solution of linear systems in the 20th century. Journal of Computational and Applied Mathematics, 123(1-2):1-333 2000.
Mrinal Sarkar and Neclima Gupte. Phase synchronization in the two-dimensional kuramoto model:
Vortices and duality. Physical Review E, 103(3):032204,22 2021.
Ivan Sosnovik and Ivan Oseledets. Neural networks for topology optimization. Russian Journal of Numerical Analysis and Mathematical Modelling, 34(4):215-23,, 2019.
Salma Tarmoun, Guilherme Franca, Benjamin Haeffele, and Rene Vidal. Understanding the dynamics of gradient flow in overparameterized linear models. In International Conference on Machine Learning, pp. 10153-10161. PMLR, 2021.
Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning. 4(2):26 31, 2012.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey graph neural networks, arXiv preprint 1rXvv:991..0056, 2019.
Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning on graphs: survey. arXiv preprint arXiv:/812.042022 2018.


Reviews:
[{"review_id": "xdpgZPpFM4T", "reviewer": "Reviewer_e51J", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduced a new network reparametrizing method for accelerating optimization for nonlinear problems. Overall, the reviewer finds the paper is a bit hard to follow, and the presentation of the paper can be significantly improved. The experiments are interesting but the comparison is not quite comprehensive. \nFirst, the reviewer is not fully convinced by the benefits of reparametrizaiotn. The reparameterization using a neural network can improve convergence speed, but on the other hand, the memory cost could be higher.  \nSecond, it is a bit unclear to the reviewer why in Section 2.2, the authors considered the NTK. The weights for NTK require an ultrawide network and the weights barely change. It is a bit abrupt without much explanation of the motivations behind it.\nThird, the speed up in Figure 2 does not seem impressive. The authors only compared with a very baseline optimizer. More comprehensive comparisons are needed to draw the conclusion. \n", "summary_of_the_paper": "This work proposed a neural reparametrization scheme to accelerate a large class of nonconvex nonlinear optimization problems. The proposed method is grounded on analysis that the dynamics of gradient ﬂow are related to the condition number of the system. More specifically, by reparametrizing the optimization problem with a graph convolutional network (GNN), the proposed method can modify the condition number and obtain convergence speed up, the acceleration is demonstrated on optimizing synchronization problems and persistent homology of point-clouds.", "summary_of_the_review": "Overall, the reviewer finds the paper is a bit hard to follow, and the presentation of the paper can be significantly improved. The experiments are interesting but the comparison is not quite comprehensive. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "u8w1ug2H_y", "reviewer": "Reviewer_WWWw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Main comments**\n\n-Overall, I think the paper is quite novel and the experiments fairly convincing.\n\n-I really enjoy how much the authors walk through the individual steps of the gradient math which derives their neural reparamaterization in 2.1 and 2.2. It is easy to follow and clear.\n\n-However, one drawback of this approach is that it appears that it seems to only help the early stages of optimization, as this is how it is used in the experiments. I think the authors should take more care to make this point more clear. In particular, what prevents one from using this hessian approximation for $\\bar{M}$ as in Section 2.3 in early stages of training when using Adam? It would be nice to see ana ablation of the different components of their method, to understand exactly what component of the approach is contributing to the improved performance.\n\n-How does this approach compare to gradient-based optimization in terms of memory consumption? How would this scale to large-scale datasets with larger parameter spaces, e.g. deep network training? \n\n**Minor Points**\n\n-The authors seem to pose the title and introduction to refer to any non-convex optimization problem, but in some parts of the paper they seem only focused on neural network optimization (e.g. Fig 1). It would be good to smooth out these inconsistencies. \n\n-The abstract on OpenReview and the abstract in the article do not match. \n\n-In the experiments, why is the term \"linear\" used to refer to the gradient-based baselines? I am not sure this is the best term to use and was confusing to me upon my first read. ", "summary_of_the_paper": "The authors derive a neural reparameterization of non-convex optimization problems in order to accelerate their convergence. They do this by deriving how the slowest components of the optimization variables can have their convergence rate improved by preconditioning with a NTK-based matrix. They make connections between this approach and Group Convolutional Networks. Experimentally, they show this approach improves upon baseline gradient-based optimization on a two datasets. ", "summary_of_the_review": "Overall, I lean slightly towards acceptance. This is due to the clarity and novelty of the paper, as well as encouraging experimental results. However, I think some more experimental verification is needed for ablating the different components of the proposed approach and for demonstrating its applicability to a broader range of problems. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QnLh8vaBQ0O", "reviewer": "Reviewer_ue9h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In terms of its strength, this paper contains interesting thoughts about the intriguing idea that a temporally varying linear reparametrization of the unknown can accelerate the gradient flow based optimization. The general topic, the combination of theoretical analysis and numerical experiments, and the bridge between the two by using efficient numerical approximations of what the theory demands, are strength of this paper.  And although the numerical experiments are certainly not exhaustive, there is some proof of concept of the benefit in the particular applications considered here. \n\nUnfortunately, the paper also has some clear drawbacks. In particular, I found the paper difficult to follow and the main idea from an optimization perspective appears to be unnecessarily hidden in a framework on \"neural\" reparametrizations. Unless I misunderstood the main idea significantly, the \"neural reparametrization\" illustrated by a neural network in Fig. 1(b), later turns out to be a linear parametrization only, i.e., considering the gradient flow for theta in $w(t) = \\kappa(t) \\theta(t)$ instead of in the original variable w.  Before considering this to be a graph neural network, I would have been interested in how this idea relates to other classical optimization methods: Has the idea of temporally changing but linear reparametrization not been considered in the optimization literature before? As kappa turns out to be the square root of the inverse of the Hessian, is there a relation to Newton or quasi-Newton methods? For me, the paper would have been easier to follow from this more classical optimization perspective. In particular, the considered gradient flow resulting from the linear reparametrization seems to be $\\partial_t \\theta(t) = \\kappa(t)^T \\nabla L (\\kappa(t) \\theta(t))$, and should be stated explicitly. If now the change in $\\kappa$ is negligible slow in comparison to the change in $\\theta$, and if $\\kappa$ represents the (scaled) square root of the inverse Hessian, isn't that the flow arising from Newton's method? I would much rather prefer a clear motivation and presentation of the paper from such a classical perspective before delving into graph neural networks. \n\n\nSome minor aspects\n- In equation (2) there is an $\\epsilon_{i,j}$, but I think the way eq. (1) is written it is unclear what 'off-diagonal' elements in $\\epsilon$ mean. (Of course the delta ensures there are no off-diagonals, but then I would avoid the notation). \n- \"Equation 1 is also an ordinary differential equation\"; I would call it a partial differential equation.\n- I am sometimes not sure which quantities are random variables and which ones are not. In eq. (4), for instance, random variables seem to have been dragged out of the expectation, which I do not understand. \n- An example of why the paper was a little difficult for me to follow are sentences like \"When running GD, the maximum change in w is bounded to ensure numerical stability.\" This sounds like a modification of GD (like gradient clipping), but it is actually meant as a condition to limit the step size you are using. Thus, isn't the reasoning flipped, i.e., in order to ensure numerical stability, we have to bound ...?\n- In the entire analysis, it could be made clearer that M is time dependent. The first sentence of section 2.3 is the first time where it is really prominent. The discretization for time dependent matrices  might of course make the behavior of the actual algorithm differ from the (continuous) gradient flow. \n- Before eq. (10) it is exemplified that $w = \\sigma(A\\theta + b)$ would be a valid choice. $A$, $b$, and $\\sigma$ are, however, not defined and if $\\sigma$ refers to a (nonlinear) activation function, I do not see how this is true.\n- page 2, \"abounded\"\n- page 6, \"adaptove learning rates\"\n- If the numerical experiments are carried out with Adam, shouldn't the theory also consider effects like (adaptively scaled) momentum?\n- In Fig. 2, why does GCN-1 seemingly start with a much lower loss function value than the other methods? Does it have a sharp drop at the beginning?\n- \"GCN with $A^2$ as the propagation rule achieves the highest speedup\". What is $A^2$? Please define. \n- \"where difficult to separate the slow and fast modes\" >> \"where it is difficult ...\"\n- The numerical results are, to my mind, not a strong indication of the proposed approach being a universal way to accelerate gradient-based methods. In particular, I am wondering how specific the acceleration results are to the applications? Also, what amount of hyperparameter tuning is required for the proposed approach to work well?\n", "summary_of_the_paper": "This work studies the question to what extend a reparametrization of an optimization problem, i.e. representing the original parameters w to optimize for as a function of some other parameters theta, can accelerate the convergences of the gradient flow / gradient descent for nonconvex optimization problems. It studies the dynamics of the flow via eigenvectors of a matrix M formed as the expectation over the outer product of the gradient of the loss with itself to reveal 'slow' and 'fast' modes of the evolution. It subsequently derives sufficient conditions for the reparametrization (which is chosen to be linear but time varying) to balance the decay on all modes. After discussing an efficient approximation of the theoretically derived scheme, numerical results demonstrate the effectiveness of the proposed reparametrization in two exemplary applications.  ", "summary_of_the_review": "Although I like the general idea and do believe that reparametrization can balance out different convergence speeds of different modes to some extend, I found the presentation to be a little confusing. The appoach seems to reduce to a linear reparametrization, which seems to relate it to other (more classical) approaches. Along with the list of minor aspects that make the paper a little difficult to follow, I need some clarification on this aspect.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0FTrk1mPD0V", "reviewer": "Reviewer_BGYL", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The idea of reparameterization is nice.\n\nWeaknesses:\n\n* The experimental evaluation consists of two problems that are not of interest to the ICLR community. I have certainly never seen either of them used in a ML paper. I have no idea how they relate to actual optimization problems I care about (i.e., training deep neural networks).\n* The experimental work doesn't look thorough -- where are the learning rate sweeps, comparisons to other optimizers, etc etc?\n* The paper spends a substantial of space (pg 2-4) deriving well known results (under assumptions that amount to strong convexity lambda_max to lambda_min ratio controls covergence). I strongly suggest that the authors use the results and language of optimization, rather than going from first principles for no good reason.\n* The final reparameterization is not very interesting -- although much ado is made about \"using a neural network parameterization\", it's just a linear map at the end of the day. \n* Since the reparameterization is linear, this makes the overall idea very similar to a preconditioner. This should be touched on, and compared to e.g., KFAC, shampoo, the many other linear preconditioners that people use. As with the optimization comment above, I think this work needs to be grounded more in the literature.\n* GCNs are tangentially relevant, but don't seem to be used in any really meaningful way.\n\nTechnical comment: right after eqn 15, it says that H is positive semidefinite. Where does thos come from? Isn't the base problem meant to be non-convex, in which case by definition H should have some negative eigenvalues at some point?", "summary_of_the_paper": "This paper proposes a reparameterization of non-linear non-convex optimization problems. This reparameterization amounts to a linear map (i.e., \"optimization params = linear operation of a different set of parameters). These linear maps are interpreted as a graph convolution network. The experimental results are validated on \"Kuramoto models\" and \"persistent homology models\".", "summary_of_the_review": "This paper is clearly unready for publication. The main idea -- using a structured linear reparameterization -- is under-developed, and the experimental results are on problems that the ICLR audience don't really care about.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes speeding up certain optimization problems common in physics by reparameterizing their parameters as the output of a graph neural network. The reviewers appreciate the idea, but are not convinced enough to recommend the paper for acceptance. They point out the following weaknesses:
* The method amounts to linear preconditioning, and hence it's reasonable to expect a fairly complete comparison to the many linear preconditioning approaches that have been proposed previousl. The reviewers are not satisfied with the currently provided comparison. 
* The main idea is not presented clearly enough. In particular, it's not obvious the proposed method is best described as neural reparameterization, since it seems to amount to linear preconditioning.
* The experiments are not persuasive enough: The presented problems may not be relevant to all of the target audience of ICLR, and the experimental evaluation does not seem sufficiently exhaustive.

The suggested areas of improvement provided by the reviewers seem reasonable to me: I therefore recommend not accepting the paper in its current form. To make the paper more accessible and appealing, the authors may consider rewriting the paper to more closely match the perspective taken by the reviewers, and to provide a more thorough comparison to the previous approaches and the existing literature.
Decision: Reject