Symmetry Teleportation for Accelerated Optimization
Authors: Bo Zhao
Nima Dehmamy
Robin Walters
Rose Yu
Venue: NeurIPS 2022 Accept (2022)
Keywords: optimization
parameter space symmetry
teleportation

Abstract: Existing gradient-based optimization methods update parameters locally, in a direction that minimizes the loss function. We study a different approach, symmetry teleportation, that allows parameters to travel a large distance on the loss level set, in order to improve the convergence speed in subsequent steps. Teleportation exploits symmetries in the loss landscape of optimization problems. We derive loss-invariant group actions for test functions in optimization and multi-layer neural networks, and prove a necessary condition for teleportation to improve convergence rate. We also show that our algorithm is closely related to second order methods. Experimentally, we show that teleportation improves the convergence speed of gradient descent and AdaGrad for several optimization problems including test functions, multi-layer regressions, and MNIST classification.

Symmetry Teleportation for Accelerated Optimization
Bo Zhao
Nima Dehmamy
University California, San Diego bozhaoQucad.cem edu
IBM Research nima. dwwmamymmmm..m bm com
Robin Walters Northeastern University
Rose Yu
University of California, San Diego roseyu@ucsd. edu
wallters@@orttheasteen... edu
Abstract
Existing gradient-based optimization methods update parameters locally, in direction that minimizes the loss function. We study different approach, symmetry teleportation, that allows parameters travel large distance the loss level set, order to improve the convergence speed in subsequent steps. Teleportation exploits symmetries in the loss landscape of optimization problems. We derive loss-invariant group actions for test functions in optimization and multi-layer neural networks, and prove necessary condition for teleportation to improve convergence rate. We also show that our algorithm is closely related to second order methods. Experimentally, we show that teleportation improves the convergence speed of gradient descent and AdaGrad for several optimization problems including test functions, multi-layer regressions, and MNIST classification. Our code is available at /https://githii com/ -osee-STT-Laabll Saaese-SLLLLaaSyymeeetetaeae htw.ymmmttty-ellprrtttti.....
Introduction
Consider the optimization problem of finding argmin (( where L loss function and w are the parameters. While finding global minima of L(w) is hard for non-convex problems, we can use gradient descent (GD) to find local minima. In GD we apply the following update rule at every step:
W++1 w. V,C,
(1)
where is the learning rate. Gradient descent is first-order method that only gradient information. is easy to compute but suffers from slow convergence. Second-order methods such as Newton's 's method use the second derivative to account for the landscape geometry. These methods enjoy faster convergence, but calculating the second derivative (Hessian) can be computationally expensive over high dimensional spaces (Hazan 2019
m
Figure 1: Left right: gradient descent, second-order methods, proposed method.
In this paper, we propose a new optimization method based on parameter space symmetries, which are group actions the parameter space that leave the loss unchanged. Our algorithm takes advantage of higher-order landscape geometry but uses only gradient information in most steps, thus avoiding the computational cost of second-order methods.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Specifically, we look beyond local optimization and ask: what if allow the parameters to make big jump once in a while? As shown in Figure ! during optimization, we teleport the parameters to another point with steeper gradients using loss-invariant symmetry transformation. After teleportation, the loss stays the same, but the gradient and hence the rate of loss decay changes. The increased magnitude of the gradient can reduce the number of steps required to converge, leading to acceleration of the gradient descent.
The locality of gradient descent is reflected in its formulation in terms of a proximal term; our method circumvents this locality by teleporting to new locations with the same loss. step of GD is equivalent to the following proximal mapping (Combettes and Pesquet 2011). Let (t, denote the inner product, we have
u++1 = argmin...
(V((V))) 2n)
2 |1:0 we w:112
The term 2//w - wella is the proximal term that keeps W close to w. Adaptive gradient methods define the proximal term using the Mahalanobis distance ||w well2 1/2 to account for landscape geometry. For example, in AdaGrad, G is the sum of the outer product of gradients (Duchi et al.) 2011). Our proposed teleportation technique relaxes the requirement from the proximal term. We teleport to points on the same level set of ((w)), but allow w to be far from wy in Euclidean distance.
In summary, our main contributions are:
We propose symmetry teleportation, an accelerated gradient-based optimization algorithm that exploits symmetries the loss landscape.
We derive loss- ioss-invariant group actions of test functions and multi-layer neural networks.
We provide necessary conditions and examples of when teleportation improves convergence rate of the current and subsequent steps.
We show that our algorithm is closely related to second-order methods.
Experimentally, we show that teleportation improves the convergence speed of gradient descent and AdaGrad in various optimization problems.
2
Related Work
Continuous parameter space symmetries have been identified in neural networks with homogeneous (Badrinarayanan al. 2015 Du al. 2018) and radial activation functions (Ganev et al. [02T]. The
effect of symmetry transformations on gradients has been examined for translation, scale, and rescale symmetries in Kunin et al. (202T) We contribute this line of work by deriving loss-invariant group actions in multi-layer neural networks with invertible activation functions.
Several works exploit symmetry to facilitate optimization. For example, Path- Neyshabur ct al. 2013) improves optimization in ReLU networks by path regularization. G-SGD Meng ef al. 2019) performs weight updates in ccale-invrriant space, which also exploits the symmetry in ReLU networks. Van Laarhoven (2017) analyzes the effect of L.2 regularization in batch normalization which gives scale-invariant functions. Bamlerr and Mandt (2018) transforms parameters in the symmetry orbits to address slow movement along directions of weakly broken symmetry and find the optimal g E G that minimizes C(g -)). In comparison, we search within G-orbits for points which maximize [d.[g w)) = VLg w)
Kingma and Ba 2015), and their approximations Martens and Grosse 2018; Gupta et al.) 2018) improve the direction of parameter updates instead of directly transforming parameters. If the group acts transitively on the level set, and we teleport to the point with maximum gradient, then our update direction is the same as that in Newton's method. We prove that our algorithm is connected second-ordes optimization methods and show that teleportation can be used to improve these methods empirically.
Natural gradient Amari 1998). adaptive gradient methods (Duchi et al.
The concept of neural teleportation was first explored using quiver representation theory AArmntta and Jodoin 2021 Armenta et al.) 2020). These works provide way to explore level curves of the loss of neural networks and show that random teleportation speeds up gradient descent experimentally and theoretically. Our algorithm improves neural teleportation by searching for teleportation destinations that lead to the largest improvement in the magnitude of gradient.

---

Several works study how parameter initialization affects convergence rate (Saxe et a.. 2014 Tarmoun et al. 202T Min al., 202T). If we apply a group transformation only initialization, our method is similar to that of Tarmoun et al. ). We do not guarantee that the transformed parameters lead to faster convergence rate throughout the entire training. However, we accelerate convergence least for short time after initialization. Additionally, our method is not restricted to initialization and can be applied at any time during training.
Most contemporary neural networks are overparametrized. While this has been shown to improve generalization (Belkin al. 2019), an important question is how overparametrization affects opti- mization. A series of works starting from Arora et al. (2018) show that overparametrization resulting from the depth of neural network accelerates convergence. Another view is that the symmetry created by overparametrization poses constraints on trajectories in the form of conserved quantities (Guuch and Urbanke 202T). Additionally, the symmetry generates additional trajectories. When the trajectories created by overparametrization are equivalent, model compression by removing symmetry reduces training time (Ganev et al. 202T). However, when trajectories are not equiva- lent, gradient flows on some paths are faster than others. We search for the better paths created by overparametrization.
3 Symmetry Teleportation
We propose symmetry teleportation, an accelerated gradient- based optimization algorithm that exploits symmetries in the loss landscape. Symmetry teleportation searches for the best gradient descent trajectory by teleporting parameters to a point with larger gradients using a group action. The resulting algorithm requires only gradient computations but is able to account for the global landscape geometry, leading to faster loss decay.
Let the group G' be set of symmetries which leave the loss function L invariant: Lig (wx, X)) C(w, X). where g E G. We perform gradient descent for bmar steps. We define an index set K C. {0. 1, kmax 1} as teleportation schedule. At epochs that are in the schedule, we transform parameters using group element y E G to the location where the gradient largest, then continue with gradient descent. Algorithm| I describes the details of this procedure. Note that the loss does not change after teleportation (Line 2-5) since L is G-invariant.
Algorithm 1: Symmetry Teleportation
Input: Loss function [(ur), learning rate y, number of epochs bmaz, initialized parameters wo. symmetry group G, teleportation schedule K.
Output: Wimas.
for i +- to tmax - do
2
if E K then
9 argmax,eci |I [VL))
Wt + g wi
end if
Wi+1 < wi m (VL),,
end for return Wt Wimax
Algorithm Caan be generalized apply teleportation stochastic gradient descent (Appendix A. We discuss some design choices below and provide detailed analysis in the next two sections.
When the action of G is continuous, teleportation can be implemented by parameterizing and performing gradient ascent on For example, the SO(2) group can be parameterized by the rotation angle 0. Small transformations g E GLa(RR) (general linear group) can be parameterized as = + eT where ε < and are arbitrary d x matrices. For discrete groups, search algorithms random sampling can be used to find group element that improves the magnitude of the gradient.
Although g (w, X) does not change X the cases we consider in this paper, Algorithm | can be extended to allow transformations on both parameters and data. The group actions on data during teleportations can be precomposed and applied the input data inference time. The algorithm extension that allows transformation the input can be found in Appendix A.2
3

---

The teleportation schedule K is a hyperparameter that determines when to perform teleportation. We define K as a set to allow flexible teleportation schedules, such as with non-fixed frequencies or teleporting only at the earlier epochs.
4 Symmetry Groups of Certain Optimization Problems
We give few practical examples to demonstrate how teleportation can be used accelerate optimiza- tion. Specifically, we first consider two test functions which are often used to evaluate optimization We then derive the symmetries of multi-layer neural networks.
algorithms BBck 1996p.
4.1 Test functions
Rosenbrock function. The Rosenbrock function originally introduced by Rosenbrock 1960) has characteristic global minimum that is inside a long, narrow, parabolicly-shapee flat valley. Finding the valley is easy but reaching the minimum difficult. On 2-dimensional space, the Rosenbrock function has the following form:
(x) = 100(a² x²)² (21 1)².
Booth function. The Booth function Jamil and Yang 2013) is also defined on I2 and has one global minimum at (1, 1) where the function evaluates to 0:
(o(21, (z1 + 2z² - 7) + (2x1 + 22 5)2.
(4)
The following proposition identifies the symmetry of these two test functions.
Proposition 4.1. The Rosenbrock and Booth functions have rotational symmetry. In other words, there exist action maps ar, ab SO(2) x R2² + R2? such that for all E SO(2),
L. (x1, v2) = L, (ar (g, [21, x2))) and Lu(x1, = Lo(as(g, x2))).
The exact forms of the action maps are deferred to Appendix B.1.2 and B.1.3
During the teleportation
(C(o-(ai..)) (2))))) ||.
step, our goal is to maximize the gradient within a level set the loss: max
dt
4.2 Multi-layer Neural Networks
Next, we consider feed-forward neural networks. Denote the output of the mth layer by hm e pdd. Xn, where dm is the hidden dimension and n is the number of samples. Denote the input by ho X E Rdxxn In terms of the previous layer output hm = Wm where W Rd xdm- (we absorb biases into Wm, m by adding an extra row of ones in hm the output hm
hm (h
(Wm hm-))
(5)
We assume the activation functions a + I are bijections. For instance, Leaky-ReLU bijective. For other activation, such Sigmoid Tanh, analytically extend them to bijective functions (e.g. tanh(z) + = x- for > 1). For linear activations, we have linear symmetries:
Proposition 4.2. A linear network is invariant under all groups Gm GLad, acting g (Wm, Wm-1) m..-t ggmm1)),
g WE = WE. Vk & {m, m 1}.
(6)
Similar, in the nonlinear case, we want to find Wm that keep the outputs hy for k * m 1-11 invariant. With nonlinearity, him Wm (W m- 1/mmmmmm.. 2). This network has GLd symmetry given by
g. (W,, m, hm-1) = (Wmgg) ghmm-1), g hm = Wmng ghgg- = hm-
Thus, have the following proposition regarding symmetries of nonlinear networks:
Proposition 4.3. Assume that hm is invertible. A multi-layer network with bijective activation o has GLam- symmetry. For 9m. Gm GLdm-,, (R) the following group action keeps h, with p 2 m invariant
Wm o (gmo
k = m k = m k & {m, m 1}
9m W =
(8)
We

---

Proofs can be found in Appendix B.2. Note that this group action depends on the input to the network as well the current weights of all the lower layers. Yet, since the action keeps the output of upper and lower layers invariant, multiple Gmm for different applied at the same time still keep the network output invariant. The proposition assumes that hm is square and full-rank. When n dm d and 2m-2 has rank identity axiom of a group action.
(with left inverse of hm-2) keeps the loss invariant but does not satisfy the
5
Theoretical Analysis
5.1 What symmetries help accelerate optimization
We now discuss the conditions that need be satisfied for teleportation accelerate GD, For brevity, we denote all trainable parameters (e.g. [W1.... Wp) } for the p-layer neural network) by a single flattened vector w E R". Consider symmetry G of the loss function [(w), meaning for g G, [(w)) = [gg w). We quantify how teleportation by G changes the rate of loss decay, given by
dl(w)
iw" dt ) - [V -
@C dw
where 17 is the matrix of learning rates (which must be positive semi-definite) and || u² my is the Mahalanobis norm. constant learning rate means r] = 1.
The following proposition provides the condition symmetry needs to satisfy to accelerate optimiza- tion. (Proofs can be found in Appendix C.I)
Proposition 5.1. Let w' = g w be a point we teleport to. Let J Ow'' /ow be the Jacobian. Symmetry teleportation using 9 accelerates the rate of decay in L if it satisfies
||-1|² VL(w) | ||VL()))
(10)
the action of the symmetry group G C GLn is linear we have w/ = w = gu and J g. follows that if G is subgroup of the orthogonal group, dL/dt will be invariant:
Corollary 5.2. Let Ory denote the orthogonal group of invariances of the inverse of the learning rate, n meaning for np g" '1' n Then
dL(g w) dt
f(w)
Vg E On
(11)
dt
In the simple case where the learning rate is constant, O, (n) becomes the classic orthogonal group (e.g. rotations) with g² = I. In general, when preserves the norm of the gradient, symmetry teleportation has no effect on the convergence rate.
From Theorem 3.2 in Hazan 2019), assuming that is f-smooth and is bounded by |C| s M, the gradient norm in gradient descent converges 23 Σ || (VL) < 2M. Teleportation increases
(VL) w therefore requiring less time reach convergence.
5.2 Improvement subsequent steps
Since teleportation moves the parameters point with larger gradient, and subsequent GD steps are local, we would expect that teleportation improves the magnitude the gradient for a few future steps as well. The following results formalize this intuition (proofs in Appendix C.2
Assumption 5.3 (Lipschitz Continuity). The 12 norm of the gradient is Lipschitz, continuous with constant E IR200 which is
0c
ac
S T. w1 W2ll2,
(12)
Ow
Owo
where W1, W2 two points in the parameter space and L is the Lipschitz constant.
5

---

Proposition 5.4, Consider the gradient descent with G-invariant loss L(w) and learning rate ?i RR. Let w, be the parameter at time t and w/ = g w, the parameter teleported by g G. Let WL+T and W be the parameters after T more steps of gradient descent from w, and w/ if and
respectively. Under Assumption
5.3
|aC/Ow//1y 11
(1 + nL)
2
(13)
@CC/0well
(1 my)
then
ac
ac
2
(14)
Ow
:w
The proposition provides sufficient condition for teleportation to improve 7 future steps. The condition is met when L is small, 1) is small, or the initial improvement,
AC
ac
is large.
Ore
Sw,
5.3 Convergence analysis for convex quadratic functions
Teleportation improves the magnitude of gradient for the current step. We have also shown that the magnitude of gradient stays large for a few subsequent steps (Proposition 5.4). In this section, we analyze a class of functions where teleporting once guarantees optimality all future times, We consider a trajectory optimal if for every point on the trajectory. the magnitude of gradient is at local maximum the loss level set that contains the point.
Consider a positive definite quadratic form Cx(w) = w" A'w, where w E R" is the parameter and A E IR nx is a positive definite matrix. The gradient of L. is VLA 2Aw, and the Hessian of L H = 2A. Since is defined to be positive definite, LA is convex. The function LA has global minimum at single point w= O.
Let p be representation of (n)) acting on RR. For g E On), we define the following group action:
- A w.
(15)
Then it can be shown that (w) admits a O(n) symmetry:
Calg w) = wT (( AA p(g)) wT Aw CA(w).
(16)
Let S. = (w: (a(w)= c} be level set of CA. We show that after teleportation, every point on the gradient flow trajectory is optimal in its level set (with full proof in Appendix C.3
Proposition 5.5. If at point w. || wI? 2 is maximum in Sea(w). then for any point w' on the gradient flow trajectory starting from w. || w/ 1|2 is maximum S (Cw(w))
Finally, we observe that for CA. teleportation moves the parameters closer to the global minimum in Euclidean distance. other words, maximizing the magnitude of gradient minimizes the distance to w² in loss level set.
Proposition 5.6. Consider a point w in the parameter space. Let arg max max,E gEG L Then w arg mi l|ww w='l3
5.4 Relation to second-order optimization methods
Since our algorithm involves optimizing the gradients, symmetry teleportation is closely related to second-order optimization methods. the target point to teleport to, gradient descent becomes equivalent to Newton's 's method (proof in Appendix C.4)
Proposition 5.7. Let Sc, = {w: L(w) Lo} } be level set of C. af a particular w E Sc., we have [V[))ll 2 [VL(w')ll for all w' in small neighborhood of w in Sc,, then the gradient L(w)) has the same direction the direction from Newton's 's method, H H())
Proposition 5.7 provides an alternative way to interpret teleportation. Instead of computing the Newton's 's direction, we search within the loss level set for point where the gradient has the same

---

b
d
50
10
1.1-
dluot
12"
10"4
1p-3
200
600
600
300
1000
200
420
600
500
1900
Epoch
Epoch
g
1.0
101
101
20-
Loss
10-
10-1
10
10
02
Ideaet
6 teleper
Epach
Epoch
Figure 2: Optimization of the Rosenbrock function (top row) and Booth function (bottom row) using (a) gradient descent and (b) the proposed algorithm. Contours represent the level sets of the loss function. Loss L and convergence rate dc dt are shown in (c) and (d). Teleportation helps move the parameters towards the target.
direction as Newton's 's direction. However, symmetry teleportation does not require computing the full Hessian matrix. The second derivative required for optimizing over continuous groups is obtained by taking derivatives with respect to parameters to taking the derivative with respect to parameters twice. This makes the computation significantly more feasible than Newton's method on neural networks with large number of parameters.
then with respect the group element, as opposed
In practice, however, the group action used for teleportation is usually not transitive. Additionally, we do not teleport using the optimal group element since it can be unbounded. We also do not apply teleportation in every gradient descent step. Therefore, Proposition serves as intuition instead of an exact formulation of how teleportation works. We provide empirical evidence in Section that these approximations do not erase the benefits of teleportation completely, and leave theoretical investigations of connection to second-order methods under approximations as future work.
Experiments
6.1 Acceleration through symmetry teleportation
We examine the effect of symmetry teleportation on optimization. We illustrate teleportation the parameter space two test functions and show speedup in regression and classification problems using multilayer neural networks. For test functions, we compared with GD for illustration purposes. For multi- layer neural networks, we also include AdaGrad as more competitive baseline.
Rosenbrock function. We apply symmetry teleportation to optimize the 2-variable Rosenbrock function 3 The parameters æ1, 22 initialized (-1, -1). Each algorithm is run 1000 steps with learning rate 10-3: We teleport the parameters every 100 steps. The group elements are found by gradient ascent 8, the parameter for the SO( group, for 10 steps with learning rate 10-!
The trajectory of parameters and the loss level sets are plotted in Figure Lb. The blue star denotes the final position of parameters, the green star denotes the target, and orange dots are the positions from which symmetry transforms start. While gradient descent is not able to reach the target in 1000 steps, teleportation allows large steps and reaches the target much earlier. Figure 2 shows that teleportation improves the norm of gradients in the following step, and m shows its effect on the loss value. Teleportations clearly reduce the number of steps needed for convergence.
Booth function. We also test symmetry teleportation on the Booth function defined in Eqn. We initialize the parameters ,2, to (5, -5). Each algorithm is run 10 steps with learning rate 0.08. We perform symmetry teleportation on the parameters once, before epoch The group elements found by gradient ascent 0 for 10 steps with learning rate 0.001. 0 is initialized uniformly
7

---

b
107
107
10
307
Aceo ootdoogat
0 10%
Adacres
Adadresi sport
dl/stt
305
Multraatt opart
el/dtt
10
201
101-
10
Aietirpen Madin Adyotolrtopppt
1000
100
Epoch
Epoch
Figure 3: Multi-layer network optimization using gradient descent with and without teleportation. Teleportation reduces both the number of epochs and the total computational time required reach convergence. At the same loss values, the teleported version has larger gradient.
at random over (0, x) Similar to the Rosenbrock function, teleportation moves the parameters to a trajectory with a larger convergence rate (Figure bottom row).
Multilayer neural network regression. We further evaluated our method on a three-layer neural network with a regression loss minw,, Wa || W3 (W2o ())ll. The dimension of weight matrices W3 E _ W2 R2x6 and W1 E _ X E gxx4 is the data, Y e g8x4 is the target, and a is the LeakyReLU activation with slope coefficient 0.1. Additional hyperparameter details can be found in Appendix D.2.
Teleportation of the (L) group can be performed by finding an element its Lie algebra, such that transforming w by the group element g exp(x) improves the gradient dc We use the first order approximation of the exponential map. Between each pair of weight matrices, we replace Im by + T and 9m by T T in H. where T E Rd xdm initialized to O. Then we perform gradient ascent steps on T with objective defined in Line 3 of Algorithm D and update the pair of weights.
dt
Figure 3h and BD show the training curves plotted against epochs and time. Shaded area denotes one standard deviation from 5 runs. Since GD and AdaGrad use different learning rates, they are not directly comparable. However, the addition of teleportation clearly improves both algorithms. Figure 3 and Dd shows the squared norm of gradient from single run. Teleportation increases the magnitude of gradient, and the trajectory with teleportation has a larger dc/ dt value the same loss values, which demonstrates that teleportation finds a better trajectory.
We apply symmetry teleportation on the MNIST classification task (Deng) 2012). We split the training set into 48,000 for training and 12,000 for validation. The input data has dimension 28 28 and is flattened into vector. The output of the neural network has dimension 10 corresponding to the 10 digit classes. We used a three-layer neural network with hidden dimension [512, LeakyReLU activations, and cross-entropy loss. Learning rate 2 x 10-3 3, and learning rate for teleportation is 10 10-3 Each optimization algorithm is run 80 epochs with batch size of 20. Immediately after the first epoch, we apply teleportation using data from one mini-batch. and repeat for 4 different mini-batches. For each mini-batch, 10 gradient ascent steps are used to optimize Im-
MNIST classification.
Figure L shows the effect of teleportation on training and validation loss, and Figure F.d shows the norm of the gradient in training. While teleportation significantly accelerates the decrease of the training loss in SGD, its effect on the validation loss is limited and detrimental for AdaGrad. Therefore, teleportation on MNIST makes training faster at the beginning but leads to earlier overfit and slightly worse validation accuracy. possible reason that regions with large gradients have sharp minima that do not generalize well.
6.2 Teleportation schedule
The effect of teleportation varies depending its time and frequency. Figure shows the result of teleportation on MNIST with different hyperparameters. In all experiments, we use SGD with batch size 20, learning rate 2 10 and 10 gradient ascent steps for each teleportation.
In Figure 5p, we randomly select 4 different mini-batches, and apply teleportation on each of them individually, but at different epochs. Teleportation before training has the worst performance. After epoch O, the effect of teleportation is stronger when is applied earlier. In Figure Sb. we again apply teleportations on 4 randomly selected batche, but repeat this with 5 different teleportation
8

---

a
b
d
100
10a
SGD Adaiiidd
MI
10
idd to sport
Lgg3
AduGrash sa--------eeee spatt
30+
10
10
10
301
10
Epoch
time
Epoch
Loss
Figure 4: MNIST classification using gradient descent with and without teleportation. Solid lines training loss and dashed lines are validation loss.
b
100
5GD
[0]
121
[4]
100
1.2.3.4.
0.3.3.7.91
100
SULD botes
batch batch
[16)
10-1
Loss
10-1_
Loss
10-1
10-2
10-2
10
10
40
80
120
40
80
120
40
80
120
Epoch
Epoch
Epoch
Figure 5: Teleportation (a) once at different epoch, (b) 5 times with different teleportation schedules, and (c) using different number of mini-batches. The lists in the legend of (a) and (c) denote the epoch numbers in teleportation schedule where teleportation happens.
schedules (hyperparameter K in Algorithm 2 as shown in the legend. All schedules have the same number of teleportations. Smaller intervals between teleportations accelerate convergence more significantly. In Figure 5b, we apply teleportation immediately after the first epoch, but use different numbers of aani-batches (hyperparameter B in Algorithm 2 and teleport using each of them individually. Using more mini-batches to teleport leads to faster decrease in training loss but is also more prone to overfitting.
6.3 Runtime analysis
The additional amount of computation introduced by teleportation depends on the implementation of Line 2-5 of Algorithm We discuss our implementation for teleporting multi-layer neural networks as an example. Teleportating each pair of adjacent weight matrices requires computing the inverse of the output from the previous layer. Denote the batch size n, the largest dimension of all layers as dmaz. and the number of layers as l. Assume that dmaz > n. Computing the inverse of the output of each layer has complexity Od n.). and computing the pseudoinverse for all layers has complexity OdP Note that all matrices we invert have dimensions at most dmax X n.
For one gradient ascent step on g. the forward and backward pass both have complexity O Pd n!). This is the same as the forward and backward pass of gradient descent on w because the architecture is the same except with approximately twice many layers. We perform t gradient ascent steps on g. Therefore, the computation cost for one teleportation is O (dl..
We show empirically that the runtime for teleportation scales polynomially with matrix dimensions and linearly with the number of layers. We record the runtime of gradient descent on a Leaky-Rell - neural network for 300 epochs, with a 10-step teleportation every 10 epochs. Each pair of adjacent weight matrices is transformed by a group action during teleportation. Figure ()) shows the runtime for 3-layer network using square weights and data matrices with different dimensions. Figure b() shows the runtime for 128-by-128 weight and data matrices with different number of layers.
Although the teleportation step has the same complexity dominated by teleportation due to larger constants in the complexity analysis. The trade-off between teleportation time and convergence rate depends on specific problems. In our experiments. the convergence rate improved by a small number of teleportation steps which does not add significant computational overhead.
gradient descent step, the runtime is

---

a
GD
Hteleeoor
@Dsteleport
Time (s)
10-
200 400
600
800
1000
12 14
Weight dimension
Number of layers
Figure 6: Gradient descent training time on a Leaky-ReLU neural network for 300 epochs, with 10-step teleportation every 10 epochs. (a) 3-layer network using square weights and data matrices, with different dimensions. (b) Weights and data are all 128 by 128 matrices, but the network has different number of layers.
7 Discussion and Conclusions
We proposed new optimization algorithm that exploits the symmetry in the loss landscape to improve convergence. It is interesting to note that optimizing dC/dt locally sometimes leads to an improved global path. One example the ellipse function (Figure D. where teleporting once ensures that dC/ dt is optimal at every L value along the trajectory. Another example is the matrix factorization problem L(U,V) | - UVI Tarmoun et al. (221)) shows that the convergence rate increases with the imbalance U V Consider the transformation U, V - Ug.9 'V. To optimize dU, V)/dt locally, we would need a large g E GL but a large also leads to large UTU V which is positively correlated with the overall convergence rate of the entire trajectory. Hence teleportation guaranteed to produce a better trajectory.
A potential future direction is to derive the exact expression for how teleportation affects the loss value at a later time in gradient flow, which may lead closed-forn solution of the optimal teleportation destination. Additionally, inspired by the landscape view of parameter space symmetry Simşek et al. (202T), teleportation using discrete (permutation) symmetries may allow us to reach better minimum. Finally, additional theory can be developed to explain the relationship between teleportation and second-order methods under the approximations we introduced, especially to quantify the improvement the overall convergence rate, and to derive its effect generalization bounds. Integrating teleportation with other advanced optimizers such as Adam and RMSprop would be another interesting future step.
Acknowledgments and Disclosure of Funding
This work was supported in part by U.S. Department Of Energy, Office of Science, U. S. Army Research Office under Grant W9 W911NF-20- 1-0334, Google Faculty Award, Amazon Research Award, and NSF Grants #2134274, #2107256 and 28.34178. R. Walters is supported by the Roux Institute and the Harold Alfond Foundation. We are grateful Iordan Ganev for many helpful discussions.
References
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251-276.
1998.
Marco Armenta and Pierre-Mare Jodoin. The representation theory of neural networks. Mathematics, 9(24), 2021. ISSN 2227-7390.
Marco Armenta, Thierry Judge, Nathan Painchaud, Youssef Skandarani, Carl Lemaire, Gabriel Gibeau Sanchez, Philippe Spino, and Pierre-Mare Jodoin. Neural teleportation. arXiv preprint arXiv:2012.01118, 2020.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning, pages 244-253, PMLR, 2018.
10

---

Thomas Back. Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programmingg, genetic algorithms. Oxford university press. 1996.
Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Semmetry-invariann optimization in deep networks. arXiv preprint arXiv:1511.01754 2015.
Robert Bamler and Stephan Mandt. Improving optimization for models with continuous symmetry breaking. In International Conference on Machine Learning, pages 423-432. PMLR, 2018.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- leaaninng practice and the classical varianee trade-off. Proceedings of the National Academy of Sciences, 116632):15849-15844 2019.
Patrick L Combettes and Jean-Christte Christophe Pesquet. Proximal splitting methods in signal processing. In Fixed-pore algorithums for inverse problems in science and engineering, pages 185-212 Springer, 2011.
Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-143 2012.
Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Neural Information Processing Systems, 2018.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Iordan Ganev, Twan van Laarhoven, and Robin Walters. Universal approximation and model compression for radial neural networks. arXiv preprint arXiv:2107.022550 2021.
Grzegorz Gluch and Rüdiger Urbanke. Noether: The more things change, the more stay the same. arXiv preprint arXiv:2104 04.05508, 2021.
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimiza- tion. In 35th International Conference on Machine Learning. 2018.
Elad Hazan. Lecture notes: Optimization for machine learning. arXiv preprint arXiv: urXiv:1909.03557) 2019.
Momin Jamil and Xin-She Yang. A literature survey benchmark functions for global optimisation problems. International Journal of Mathematical Modelling and Numerical Optimisation, 4(2): 150-194, 2013.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations, 2015.
Daniel Kunin, Javier Sagastuy- Brena, Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In International Conference Learning Representations, 2021.
Serge Lang. Algebra. Graduate Texts in Mathematics. Springer, 2002. ISBN 978-0387953854.
James Martens and Roger B Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on International Conference on Machine Learning, PMLR, 2018.
Qi Meng, Shuxin Zheng, Huishuai Zhang. Wei Chen, Zhi-Ming Ma, and Tie-Yan Liu. G-SGD: Optimizing relu neural networks in its positively scale-invariant space. International Conference on Learning Representations, 2019.
Hancheng Min, Salma Tarmoun, René Vidal, and Enrique Mallada. On the explicit role of initializa- tion on the convergence and implicit bias of overparametrized linear networks. In International Conference on Machine Learning. PMLR, 2021.
Behnam Neyshabur, Russ Salakhutdinov, and Nati Srebro. Path-SGD: Path-normalized optimization in deep neural networks. In Advances in Neural Information Processing Systems, 2015.

---

HoHo Rosenbrock. An automatic method for finding the greatest or least value of a function. The Computer Journal, 3(3):175-184, 1960.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning deep linear neural networks. International Conference on Learning Representations, 2014.
Berfin Şimşek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Sym- metries and invariances. In International Conference on Machine Learning, pages 9722-9732. PMLR, 2021.
Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal. Understanding the dynamics gradient flow in overparameterized linear models. In International Conference on Machine Learning, pages 10153-10161. PMLR, 2021.
Twan Van Laarhoven. 1.2 regularization versus batch and weight normalization. Advances in Neural Information Processing Systems, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "uDdwTYuLwwq", "reviewer": "Reviewer_ysBc", "summary": "This paper proposes moving along the level set during an optimization problem. Doing so may lead to faster gradient iterations, and is proven and shown empirically. A few examples of symmetries are discussed (modifications to the parameters that leave the output of the network invariant). For a nonlinear MLP, only an input-dependent symmetry is discussed. Runtime analysis is also performed.", "questions": " * Does the theory behind convergence take into account the extra steps needed to solve for the symmetry teleportation problem? I'm wondering if the symmetry teleportation problem itself has a different convergence rate, and the real convergence rate would be worse?\n\n * Practically, how would Figure a look if it is plotted against number of gradient computations or even wall clock time? The runtime analysis per teleportation is great to have, but since this cost of teleportation is amortized (done only every so GD iterations), it isn't completely clear to me how this actually affects practice.\n\n * Since only a first order approximation is used for exp(x), the output of the neural network isn't invariant anymore right? Moreover, if you take more teleportation steps, this should actually diverge or at least cause the parameters to be arbitrarily far away from the level set. What's the intuition behind this not occurring in practice; is it due to the fixed number of steps for optimizing the teleportation?\n\nMinor:\n\n * How did you settle on 10 steps for teleportation? How important is it to tune this number?\n\n * Given your expertise, would you use symmetry teleportation as a default when you run optimization problems with known symmetries? Why or why not.\n\n * Small suggestion for Figure 2 b and f: draw the symmetry teleportation trajectory. Right now some of the blue lines are gradient descent while others are teleportation + GD which makes it difficult to parse the details.\n\n * Type (line 62): dt -> dw", "limitations": ".", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n  - Paper is well-written.\n  - Problem setting (optimization of MLPs) is relevant.\n  - Proposed solution is simple to understand.\n\nCons:\n  - Runtime complexity is not fully reconciled / transparent.\n  - Some details of the symmetry teleportation is not completely clear to me.\n  - In practice, symmetries must be known for a particular optimization problem, and even when known, in many cases I imagine optimizing within this group can be very difficult. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tRMSpyjeJzS", "reviewer": "Reviewer_ag4f", "summary": "Existing gradient-based optimization methods update the parameters locally, in a direction that minimizes the loss function. The author proposes a different optimization method to improve the convergence speed by teleportation which transforms parameters while making the loss invariant. The author gives theoretical proof and experimentally shows that teleportation improves the convergence speed of GD and AdaGrad for several optimization problems.", "questions": "1. The meaning of ‘Acceleration’ should be stated clearly in this paper.\n2. Does the initialization have an impact on the convergence and the final result through teleportation? How and why?\n3. The proliferation of saddle points can slow down convergence. Can Symmetry Teleportation effectively escape saddle points?\n4. Is there any general guideline for designing symmetry groups?\n5. In some scenarios, GD with a learning rate schedule can also achieve a similar loss curve in Fig. 2(c). What advantages do you think GD with teleport has over GD with a learning rate schedule?\n6. As far as I can see, GD in deterministic settings is slightly different from GD in stochastic settings. How do you guarantee that SGD+teleport can accelerate convergence over SGD?\n7. There is a little abuse of notations. The learning rate, $\\eta$, and the learning rate matrix, $\\eta$. The steps, T, and the transformation matrix, T.\n\nTypos\n\nLine 73 leads -> lead\n\nLine 88 flow -> flows\n\nLine 130 are -> is\n\nLine 227 reduces -> reduce\n\nLine 272 has -> have\n\nLine 285 has -> have\n\nLine 315 minima -> minimum\n\netc.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The teleportation improves the convergence speed of subsequent steps by transforming parameters to another point with steeper gradients.\n2. The symmetry teleportation takes advantage of higher-order landscape geometry but uses only gradient information.\n\nWeaknesses\n1. It does not guarantee that the transformed parameters lead to a faster convergence rate throughout the entire training.\n2. It takes time to find g.\n3. When the number of weights and data is large, it may increase the complexity.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YQQkHBaa9Bz", "reviewer": "Reviewer_W8uy", "summary": "Using teleportation of parameters to a new point at the same loss value but with a larger gradient norm, the authors offer a new approach for accelerating the convergence of gradient-based optimization methods. This new scheme can accelerate the convergence of gradient-based optimization methods. An intuitive understanding of the connection between such a scheme and second-order methods is demonstrated by the authors. The major application discussed in the paper are deep neural networks and two rather simple classical functions. An empirical evaluation of the effectiveness of the authors' approach to solving these problems was provided, along with a comparison to gradient descent and AdaGrad.", "questions": "1) There are two main assumptions for unconstrained optimization problems: $L$-smoothness (Lipschitz-continuous gradient) and $M$-Lipschitz-continuous function. Both are defined globally but we are actually using them locally on the set around the starting point $x_0$ and the solution $x_{\\ast}$. The reason why we can do it is the monotonicity and compactness of gradient-based methods. With that local approach, we can make our function class much broader. For example, x4 is not $L$-smooth or $M$-Lipschitz, but we can constrain $L$ locally by some constant dependent on $x_0$, and the gradient method will converge for such a $L$. Note that the presented Rosenbrock function is not $L$-smooth or $M$-Lipschitz globally. The same problem arises for neural networks. Additionally, level sets can be unbounded. This leads us to the question of the existence of such teleportation because the maximization problem will also be unbounded. For example, one of the simplest examples of neural networks is a linear 2-layer network without activations for one data-point $(1,1)$ with MSE-loss.  $\\min (x \\cdot y - 1)^2$. All points $x = 1/y$ are local minima. Let us start from parameters $a = (2,1) $. The gradient norm at $b = (1000, 0.002)$ is bigger than the gradient norm of $a$. So, we'll teleport to $b$, but why is this location superior? Furthermore, because of smoothness, the learning rate at $b$ should be less than $1/(1000^2+0.002^2)<10^{-6}$, whereas the learning rate at $a$ is around $1/5$. So, my main concern is that this teleportation technique can actually make convergence worse and much slower. Because of that, the method should be used much more carefully and with clear limitations to its applicability. \n\n2) It is unclear to me, how complex the process of teleportation is in general. \n\n3) Some of the notations and objects were unclear to me. So, I would recommend adding a notation section to the appendix to clarify the introduced notation. For example, what does $g \\cdot w$ mean? Why are you using $dL/dt$ and not $dL/dw$? Some definitions of used groups or links to the literature where the reader can be educated about them will also be helpful. \n\n4) In experiments,\na) Figure 4: Why are SGD and AdaGrad starting from different points?\nb) How do you tune learning rates for gradient descent? Do \nc) Figure 5: It seems that the fact that teleportation has bad validation accuracy proves my claim from question 1.\nd) Figure 3 and 8: It seems that the gradients of the teleported version of AdaGrad are smaller than regular AdaGrad. Despite this, the teleported loss is smaller than regular. How does it correspond with the dependence \"bigger gradient -> faster convergence\"? \n\n5) The authors showed intuition about two-dimensional quadratic problems (ellipsoid). It will be interesting to understand what will happen with d-dimensional quadratic problems.\n\n#######################\nDuring the rebuttal, the authors addressed most of my concerns and questions. They improved the paper. Special thanks for the new Section D.1 about quadratic problems. I increased my overall score to 6.", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The symmetry teleportation technique is the main idea and contribution of the paper. It is based on the fact that after teleportation, the loss will be the same, but the landscape can be much better, and hence we will get faster convergence to the local minima. I liked the presented method's relations with second-order methods both theoretically and practically on Booth function. The idea is interesting and original. Some proofs are presented, and they seem to be correct. The authors show some experiments for small feed-forward networks with leak-relu activations on MNIST. The paper is well-structured and well-written. I have some concerns about the correctness and effectiveness of the teleportation technique presented in the paper. I will address them in the questions section.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RbJ68Cxazp-", "reviewer": "Reviewer_94ST", "summary": "This work proposes an accelerated gradient-based optimization algorithm, symmetry teleportation, which exploits symmetries in the loss landscape. It improves current neural teleportation methods by searching for teleportation destinies that lead to the largest improvement in the magnitude of gradient. This work also provides empirical evidence showing the improved convergence speed of gradient descent and AdaGrad in different optimization problems.", "questions": "How to exploit symmetries for different loss functions？", "limitations": "There are no potential negative societal impacts of the work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. This work proposes a novel optimization algorithm that applies symmetric teleportation to accelerate the convergence of gradient descent.\n2. The problem is well-motivated, and the work also provides an intuitive illustration of the ideas behind the proposed algorithm.\n\nWeakness:\n1. It would be better if the work could explain more about how to exploit symmetries for different loss functions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QrmutSIE2EB", "reviewer": "Reviewer_WGhM", "summary": "Authors consider the class of optimization problems having group of symmetry G, which is simple enough to optimize norm of the gradient over its orbits, and propose technique of \"teleportation\" from current point to the more suitable one on the same orbit before every gradient step. They present the examples of problems having symmetries, including simplest NN, and numerically show the competitiveness of proposed technique in these simple cases and more complicated test problems (MNIST). New technique is provided with sketchy theoretical justification of acceleration effect.", "questions": "Regarding second weakness — it would be good to find at least one case (class of functions + class of symmetries) when GD with teleportation is competitive with GD/Conjugate gradients method/Newton method with explicitly expressed effect of teleportation in final convergence rate. It seems to be possible.", "limitations": "Limitations seem to be obvious for the readers of the paper, and authors do not pretend to propose the panacea. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of using group theory in optimization for solving problems with known symmetries is not new (I've found review with DOI:10.1007/978-1-4614-1927-3_9 almost immediately, but it is hardly the earliest), but this topic was not well discovered for continuous convex case. The first strength of the paper is that it attracts attention to this blank space. The reason why it is blank is the unnaturalness of group of symmetry oracle — symmetries that cannot be used to simplify the model itself are often too complicated to be used in auxiliary problems as well. This is related with first weakness — paper does not consider question of automatic symmetry detection and does not say anything about practical NN architectures, but this is very important to show that this technique at least can be applied to any practical task. The second weakness is that sketch of theoretical analysis is too sketchy, there are no explicit convergence rate estimations in the paper, so the effect of teleportation cannot be appreciated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept