Enforcing Idempotency in Neural Networks
Authors: Nikolaj Banke Jensen
Jamie Vicary
Venue: ICML 2025 poster (2025)
Keywords: idempotent neural networks
non-convex optimisation
gradient-free optimisation
Primary area: optimization
TL;DR: An alternative optimisation method to gradient-based approaches for finding idempotent neural networks is proposed and evaluated.

Abstract: In this work, we propose a new architecture-agnostic method for training idempotent neural networks. An idempotent operator satisfies $f(x) = f(f(x))$, meaning it can be applied iteratively with no effect beyond the first application. Some neural networks used in data transformation tasks, such as image generation and augmentation, can represent non-linear idempotent projections. Using methods from perturbation theory we derive the recurrence relation ${\mathbf{K}' \leftarrow 3\mathbf{K}^2 - 2\mathbf{K}^3}$ for iteratively projecting a real-valued matrix $\mathbf{K}$ onto the manifold of idempotent matrices. Our analysis shows that for linear, single-layer MLP networks this projection 1) has idempotent fixed points, and 2) is attracting only around idempotent points. We give an extension to non-linear networks by considering our approach as a substitution of the gradient for the canonical loss function, achieving an architecture-agnostic training scheme. We provide experimental results for MLP- and CNN-based architectures with significant improvement in idempotent error over the canonical gradient-based approach. Finally, we demonstrate practical applications of the method as we train a generative network successfully using only a simple reconstruction loss paired with our method.

Enforcing Idempotency in Neural Networks
Nikolaj Banke Jensen'
Jamie Vicary?
Abstract
2022), video generation (Ma et al., 2020; Liu et al., 2021), sorting algorithms (Tambouratzis, 1999), compression al- gorithms (Namphol et al., 1996; Liu et al., 2021), image denoising (llesanmi & Ilesanmi, 2021; Liu et al., 2021; Mao et al., 2023), and image generation (Liu et al., 2021), among others.
In this work, we propose a new architecture- agnostic method for training idempotent neu- ral networks. An idempotent operator satisfies f(x) = (x)),,, meaning it can be applied itera- tively with no effect beyond the first application. Some neural networks used in data transformation tasks, such as image generation and augmenta- tion, can represent non-linear idempotent projec- tions. Using methods from perturbation theory we derive the recurrence relation K' + 3K 2K3 for iteratively projecting real-valued matrix K onto the manifold of idempotent matrices. Our analysis shows that for linear, single-layer MLP networks this projection 1) has idempotent fixed points, and 2) is attracting only around idempo- tent points. We give an extension to non-linear networks by considering our approach as a sub- stitution of the gradient for the canonical loss function, achieving an architecture-agnosti train- ing scheme. We provide experimental results for MLP- and CNN-based architectures with signif- icant improvement in idempotent error over the canonical gradient-based approach. Finally, we demonstrate practical applications of the method as we train generative networks on MNIST and CelebA successfully using only simple recon- struction loss paired with our method.
Some data transformation tasks admit only idempotent solu- tions (e-g., sorting), whilst other tasks admit no idempotent solutions (e.g., rotating image by 90"). This work is con- cerned with class of data transformation tasks which has both idempotent and non-idempotent solutions and where idempotency might be a desirable property. For example, in Section we study idempotency in generative networks where it is the formal requirement of one-step inference, but also denoising and image augmentation (e.g., application of effect- filterr-- are examples of tasks where idempotent solu- tions may be desirable (Liu et al., 2021; Mao et al., 2023), Since solutions are not inherently idempotent in this class, we explore actively enforcing idempotency as a component of the loss function used in training.
In this paper we are primarily concerned with networks So ER R", where 0 is collection of weight parame- ters. The condition that fe is idempotent is the following, for all x E R":
fo(x) = fo(o(x)))
If fo(x) = Wx (a single-layer, fully-connected network with no bias and the identity activation function) where W E RRxn is the weight matrix, then condition (1) reduces to the familiar notion from linear algebra where W = W and eigenvalues of W are either 0 or 1. Condition (1) also gives the correct notion for non-linear networks acting as idempotent projections, and can be optimized using a simple mean- mean-squared error loss, where x E R.
1. Introduction
Using neural networks as data augmentation tools is becom- ing more widespread in areas such as signal processing and generative artificial intelligence. In particular, networks of the form f X + X, mapping data within the same space X, are frequently used in image augmentation (Lu et al.,
Litem (x)
m Σ (folfo(x)) - -fo(x)²²
1
DDepartment of Computer Science, University of Oxford, Ox- ford, UK. *Department of Computer Science and Technology, University of Cambridge, Cambridge, UK. Correspondence to: Nikolaj Banke Jensen <niikllj siikollajjenseneee wwwsssnees.ox.cccoo cary <jamie.vican @cl. cl..m.ac..k>..
As we show in Section 3, minimizing this loss using canon- ical gradient descent can yield relatively poor improve- ment in the idempotent loss. Additionally, due to the sec- ondary application of fe the number of terms in the gradient V Liditem grows exponentially in the number of layers memoization is not used, making the approach computation- ally expensive for certain architectures. If memoization
Jamie Vi-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Enforcing Idempotency in Neural Networks
used, then this be reduced to linear growth, as discussed in Section 3.3.
K'
In this work, we propose an alternative method for train- ing neural networks to satisfy condition (1). Using ideas from Perturbation Theory (Kato, 1995) we derive a function g which solves K' = g(K) such that if K E R*** is an "almost" idempotent matrix, then K' E R"x* perfectly idempotent (i.e., (K')² K'). In this work, we focus on one such function:
-0.5
0.5
1.
g (K) = =K? 2K
(3)
Although we assume K is close to idempotent, we show that in practice g can be used to derive matrices which are within machine precision perfect idempotency even when the input matrix K relatively far from idempotent. At high level, this process is based on recurrence rela- tion K' K (K) K), taking small 7-sized steps in the direction of g()). While this recurrence relation derives idempotent matrices- -nd can therefore be used to train single-layer networks with identity activations to be idempotent-ww we also give a more general application of Eq. (3) as modification of the backpropagation algorithm, yielding an architecture agnostic and efficient algorithm for finding idempotent networks. As will see, this modifi- cation in general not only leads to significantly improved idempotent error reduction but also explores the loss land- scape differently from the canonical approach.
Figure I. Plot of K' =3K² - -2 in the case K is scalar.
to reason about complex dynamical systems often used in physics and quantum chemistry (Hirschfelder et al., 1964). We refer the reader to Kato 1995 for detailed treatment of the topic.
We first define the term near-idempotent used throughout: Definition 2.1 (Near-idempotent to order n). Let the ma- trix E R mx satisfy P = P Let E Rm x be arbi- trary (e.g., noise) where there exists some n E N such that D has coefficients with absolute value below < 1. We say that K = P + D is near-idempotent to order n.
Using Definition 2.1 we may define the following ansatz in terms of a mear-idempotent K:
In Section 2.1 we give a detailed description of the method used to derive Eq. (3) and alternative solutions. We also show that while there exists non-idempotent fixed points to Eq. (3), these points are repelling under the recurrence relation K' = K ((((K) K) for 0 < 7 < 1, giving credence to the use of such recurrence relation in practice. Finally, in Section 2.3 we derive a full training scheme for training arbitrary neural network architectures of the form So RD - R". In Section 3, we present experimental data for variety of fully-connected network architectures, show- ing that our method outperforms ordinary backpropagation under varied conditions. We also replicate the results of Shocher et al. 2023 by applying our method on a U-net style DCGAN model to successfully create generative networks for the MNIST and CelebA datasets. Lastly, Sections 4 and discuss how our method distinguishes itself from related approaches as well as future work.
(4) This poses K' the linear combination of higher or- ders of near-idempotent matrices. If we further constrain (K')² - K' 0. the result is a system of polynomial equa- tions in variables ag Importantly, for all equations of the system, any term in which D appears at least n + times can be considered "negligible" and ignored. This simplifi- cation vastly reduces the problem and allows approximate solutions. The coefficients a; can be thought of as parame- terizing a projection g such that K' {()) for arbitrary near-idempotent K. The requirement that K' be idempotent and that K is only near-idempotent implies that solution is a projection onto the manifold of idempotent matrices; we call an idempotent corrector as must "make K idempotent".
K' a. + + a,,K'.
2. Method
Note that Definition 2.1 places no restrictions on the distri- bution from which D is drawn, hence K and the underlying P have no presumed relation. Additionally, the equation 4 above also places no assumptions on the relationship be- tween P and K'
2.1. An idea from Perturbation Theory
Perturbation Theory comprises methods for finding an ap- proximate solution to a problem by starting from the exact solution of related, simpler problem and adding successive "perturbations" the system. is a diverse set of tools used
In the case when n = 1 we consider D2 x 0 and the ex- pression (K')² K' = can be expanded and reduced
2

---

Enforcing Idempotency in Neural Networks
by recursively applying the following assumptions, for all X, Y, Z matrices:
be constructed. The block-diagonal structure of im- poses up four equations per block of size (k x k) (see Appendix B):
D2 0, P²=P, XDYDZ 0.
(5)
A 31 - 221
(7)
When j S 2, there exists no solutions for Os. When j = there exactly one solution when 01 = 0, 02 3 and 03 -2, which gives precisely ! defined in Eq. (3).
6Λ 6A22
Only when k 2
(8)
0 62
Only when k 2 3.
(9)
For there exists families of solutions (see Appendix A), but we consider primarily the case when j = 3 as this requires fewer higher-order terms of K and is therefore generally less costly to evaluate for concrete values. Note also that in general, solving the above system polynomial equations NP-hard worse, but this not a concern for us in practice as the number of variables low (so all constraints have low degree also).
0 = 0 2
Only when k 2 4.
(10)
Clearly, this system of equations is inconsistent when A: > 2. hence algebraic multiplicity and geometric multiplicity of each eigenvalue have to be equal. This implies that J diagonalizable for any fixed point K. Furthermore, solutions which satisfy only Eq. (7) are:
E {0, 1..
(11)
2.2. Fixed Points and Stability Analysis
Therefore, any fixed point of 3K 2 2K must have eigenvalues in this Consequently, all idempotent ma- trices fixed points, but there exists also non-idempotent fixed points.
Undoubtedly, a required property of any idempotent correc- tor g is that every idempotent matrix is fixed point, but it may also be desirable to find if any non-idempotent ma- trices are fixed points. Concretely, we wish to characterize solutions to 3K2 2K*
Although the initial derivation of g(K) 3K - re- lies on K being hear-idempotent to the first order, we consider more generally the behaviour of g around the fixed points when applied repeatedly as recurrence re- lation. Let h(A) 3X 2X and observe its derivative 1 (((A) 6A 6X2 Then, for each fixed point of we have
In general, we place no restrictions on matrix K mx In particular, might not be directly diag- onalizable. It is well known, however, that for every square matrix K there exists invertible matrix P and Jor- dan normal form (H. Weintraub, 2009) J E Cmxm of K E R mx such that K = PJP-1 From this the dual problem.
W W((0) 0, N N((5.5) 1.5, N(11) 0. (12)
Since [h'(A)] < 1 for X E {0, 1} these points attracting whilst [}(()] for À 0.5, thus this point is repelling.
J = 3J
2J
(6)
34
30
-54
-03
80
3.5
1.5
Figure 10-time recursive application of h(A) 3x² 2 onto O, while orange areas denote points converging onto 1.
3 for each point on the complex plane. Black areas denote points converging
3

---

Enforcing Idempotency in Neural Networks
In other words, if the idempotent corrector g, applied as recurrence relation K, converges at some point K', then K' will be approximately idempotent unless K has eigenvalue of exactly 0.5.
term "Modified Backpropagation" refer to the canon- ical backpropagation algorithm with the rule (15) applied appropriately when computing gradients.
One way to understand why this approach is sensible is to consider that in the linear case we obtain exactly the directional information (3K²-2K3 K) of Eq. 13 from the previous section. In the case when fe is non-linear we wish for the network to act in an idempotent way around inputs taken from the training distribution with the expectation that enough such points yields idempotent behaviour for the rest of the distribution. We can approximately achieve this by enforcing the idempotency of the Jacobian Jg at x. In our scheme this would give the objective
Furthermore, Figure shows the result of applying the idem- potent corrector recursively 10 times for each point on the complex plane. The attracting regions around and are large, hence any matrix that "reasonably close" to idem- potent will be projected onto (within machine precision) idempotent matrix.
Whilst this analysis technically only applies in the lin- earized setting, we propose to also apply the method in non-linear settings using the following recurrence relation, for 0 S 77511 1: 1:
(x)) (x) Jx(x))x
(16)
K' (0(K) K).
(13)
which can be seen exactly as the linearized counterpart to A (x). Therefore, under the assumption that so behaves locally linearly we should expect the training scheme pre- sented in this section to also optimize for idempotency in the non-linear setting at least around the training samples.
This has the effect of taking small 7-sized steps the direc- tion of g at every time point.
2.3. Deriving a Training Scheme
In practice, the definition (15) can be implemented in com- mon machine learning frameworks, such as Jax and Py- Torch as user-defined automatic differentiation rule (see Appendix C).
Gradient-bar based optimization techniques use the gradient of an often non-convex loss function as the directional infor- mation used to update the hypothesis at each time step. This highlights a core difference between our approach and con- ventional gradient-basee based approaches, since the recurrence relation derived above (and shown in Figure 1) exactly de- scribes the "direction" to move in to reduce idempotent error. Our method need only evaluate g finding its derivative is unimportant.
3. Experimental Results
To evaluate the training scheme suggested in Section 2.3 compare relative performance between the two methods: "Ordinary Backpropagation" with the quantity ii-Cnt (y)) resolved at runtime by automatic differentiation, and "Mod- ified Backpropagation" with the modified backpropagation rule for 2 2(-Lidem (y)) To demonstrate the flexibility of the approach, we report results for four diverse MLP-style net- works, described in Table 1.
By
Consider neural network fe Rm + Rm together with its application to input E RM denoted (x). We might then consider the recurrence relation in Eq. (3) in the following form:
By
33=3fo -=ffo())22fo(o66
(14)
The dataset used for training in this section is drawn from normal distribution with mean 0 and standard deviation 1. To prevent concerns about overfitting, the distribution is sampled i.i.d. at each epoch during training. Furthermore, a batch size of 1000 used, although comparable results have been found for batch sizes between 32 and 10 000. The optimizer used is SGD.
This describes a desired change in the output of the net- work which we denote fx) =y' y. In other words, A (x) describes the change in y which moves y towards an idempotent projection much in the same way that the quantity 3(-Lidmm(()) describes the direction which reduces dy the idempotent loss function in Eq. (2). A central idea presented in this work is therefore the definition
((-Lidem (y)) = Ax)
3.1. Qualitative Differences
(15)
Oy
In this section we present suggestive evidence that Modi- fied Backpropagation searches the solution space differently from Ordinary Backpropagatiin.
as an alternative quantity to the traditional, analytical solu- tion to 0(- Lidemly)), To complete the scheme, we consider By how a change in the output y can be propagated to a change in the parameters f of fe- This, however, is a straightfor- ward application of the chain rule as it is calculated con- ventionally in backpropagation. In this paper we use the
For purposes of visualization, we employ the methods of Li et al. 2018 to compare the optimizer trajectories of Modi- fied Backpropagation and Ordinary Backpropagation. Con- cretely, we train a copy of the same network with either
4

---

Enforcing Idempotency in Neural Networks
Table Four neural networks for testing. Fach "I "Linear(n, m)" block is parameterized by its input dimension and its output dimension m, corresponding to the underlying W "x*x weight matrix. Every block has associated bias vector and LeakyReL.U(O2)) activation function. represents a trivial net- work, B2 represents a relatively deep network, B3 represents relatively wide network, and B4 represents a more realistic net- work.
Abs Cosine
0.
Separate
Airin DB
Alonn MD
00
500
1000
1500
2000 2500
Epoch
Figure 4. Absolute cosine similarity of gradients over time of representative training run with model "Along OB" optimizes the network with Ordinary Backpropagation and compares each timepoint with suggested gradient from Modified Backpropagation. "Along MB" optimizes the network with Modified Backpropagation and compares with suggested gradient from Ordinary Backpropa- gation. "Separate" compares gradients of each optimizer they independently optimize the network. Gradients suggested by Mod- ified Backpropagation remains significantly different from those suggested by Ordinary Backpropagation.
Identifier
Architecture
No. Parameters
B1
Linear(5, 5)
30
B2
Linear(128) 256)
263 296
Linear(256, 256)
Linear(256, 256)
Linear(256, 256)
Linear(256, 128)
B3
Linear(4096, 1024)
8933 728
Linear( 1024,, 1024, 4096)
Linear(784, 1024)
505 456
10
Undraxy Indegmaggtiio
Medified tnckpeopogrion
Linear( 1024, 2048)
(0024,
Nomn
Linear(2048, 784)
10"
10"
1500
2000
algorithm and record model parameters OMB and 0,8 at epoch t. A PCA analysis is then performed over the rela- tive change parameters from f and 00 (which are identical), from which we select the two most explanatory directions. Lastly, the loss landscape and trajectory paths
Epoch
Figure 5. Norm of gradients time of representative training run with model B2. The network is optimized independently by either algorithm at optimal learning rates (Figure 6). Modified Backpropagation gives consistently stronger gradient signal than Ordinary Backpropagation.
0.4
@ and 0 are projected onto the selected dimensions. An example is shown in Figure (and Appendix H).
Qualitative evaluation show that Modified and Ordinary Backpropagation often differ significantly in projected tra- jectories across the two most explanatory directions, but this is not always the case (e.g., B4 in Figure 16). Addi- tionally, optimization trajectories for Modified Backprop- agation can be explained by projection onto two direction with more than 90% variance explained, indicating that it exhibits the same behaviour Ordinary Backpropagation which has previously been suggested to largely operate in cow-dimensional subspaces (Li et al., 2018; Song et al., 2024). One should note, however, that the loss surface here represented under a dramatic dimensionality reduction which limits further conclusions.
2nd PC: 4,26 %
00 x 08
or K
1.34 19
a0 x 101
DC x 1852
o1
111
-0.1
-022
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.7
1st PC: 93.55
9
Modified Backgropagation
Ordinary Backpropagation
We now investigate how the gradients produced by Modified Backpropagation differ from those produced by Ordinary Backpropagation. We give here analysis over a single training run on network B2, but similar results hold for all networks in Table over repetitions of the experiment. As Figure 4 shows, gradients suggested by either algorithm re- main relatively dissimilar throughout training, which further
Figure 3. Representative projections of the optimizer trajectories over 2500 epochs of either algorithm on the B2 model at optimal learning rates (Figure 6). Total variance captured is 97.8% with cosine similarity of PC1 and PC2 less than 1.0 x 10-6, Optimizer trajectory of Modified Backpropagation deviates significantly from Ordinary Backpropagation.
5

---

Enforcing Idempotency in Neural Networks
indicates a difference in the expected optimization trajectory. Furthermore, evidenced by Figures 3 and 5, Modified Backpropagation travels faster (i.e., gives stronger gradient signal) than Ordinary Backpropagation, even when optimal learning rates are selected for both algorithms.
3.2. Quantitative Differences
We now give an evaluation of the relative efficacy of Mod- ified Backpropagation Ordinary Backpropagation. As shown in Figure 6, for networks B2-B4 Modified Backprop- agation achieves significantly lower absolute idempotent error on average at lower learning rate. For network B3 the difference more than one order of magnitude. As
Network BI
Network B2
Arg Lirrr
10
1000000
100000
00000000
000
030
010
0.92
176
0000000
014
055
60
2/2
13
136
Network B3
Network B4
10
Avg Error
10
1.06 10-2
479
1
DUSONN GUSONU
DID
02
03
015
1005
014
OAL
LTD
136
0
010
Leannng Rate
Leaming Rate
Ontinary Bxlgnopggtton
Modified Dxigropgriin
Figure 6. Average of 10 runs of each algorithm for a variety of learning rates. Networks are randomly initialized and trained for 500 epochs. Runs which did not return a network with lower idempotent error the average is over remaining runs. For networks B3 and B4, learning rates >> 0.22 and > 0.52 respectively had no runs with improvement in error. For Modified Backpropagation on B1, some runs resulted approximately which, due to floating-pont point imprecision, results in the error spikes.
initial value are discarded, and
Ordinary Backpropagation
Modified Backpropagation
9.54 10
B2
8.51
Avg
6,05
0.07
0.22
500
1000
2000
2500
1000
1500
2500
0.37
10
0.67
Error
0.82
0.97
1.42
10
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Lipoch
Epoch
Figure 7. On networks B2 and B3, the average idempotent algorithm. Each column of graphs represents one algorithm. Modified Backpropagation achieves lower idempotent error at lower learning rates than Ordinary Backpropagation. The biggest relative improvement between algorithms occurs in the first ~ 500 epochs.
10 runs for each learning reported for each
6

---

Enforcing Idempotency in Neural Networks
the tested networks represent varying architectures with commonly used activation function, these results suggest that Modified Backpropagation fares well in variety of training configurations.
datasets. Let D denote the distribution of dataset samples, while D' is distribution from which noise is sampled. For MNIST we use D' = (O 1) whilst for CelebA we use distribution of noise with similar frequency-statistiic as the dataset, following Shocher et al. 2023. Let O' be a copy of the trainable weights O at each time step, where O' is detached from the computational graph. In this training scheme, the loss function being optimized is
Although the dataset used here is i.i.d. samples drawn from a Gaussian (0, 1), we observe similar results when data comes from other distributions, such as the uniform distri- bution ( k) for & E N. Following Shocher et al. 2023, we also observe similar results when applying Fast Fourier Transform to MNIST data, finding the mean and variance of each frequency, and then apply an inverse FFT to get noise with similar frequency-statinn statistics as the underlying dataset.
[(0, 0') = Lrec ( + +A:iiomm (0, + Ar (righe(O,)))))))))) 00,0'.
(17) To see why employing two copies of the weights is useful, consider yx) ~ 'D and z ~ D' and the individual loss components:
Whilst the above results are promising, a natural concern is the quality of solutions produced. In particular, if a signifi- cant fraction of networks trained using Modified Backpropa- gation have weights close to the null matrix 0 or the identity matrix I then the algorithm might not be practically useful. We refer the reader to Appendix D which shows that the norm of trained weight matrices in general is comparable to those found by Ordinary Backpropagation.
Cree((x, y y): 0) = Ily' fo((x)l1
(18)
Lidem (z; 0, 0') = [[for(f(z))) fo(z))l
(19)
Cright 0, 0') = - -le
for (z)llı
(20)
For instance, the quantity (C: is only affected by 80
the inner application of ʃ above due to O' being detached from the computational graph. The relationship between loss components Lidem and Lright is adversarial in nature.
3.3. Relative Computational Cost
Theoretical analysis shows that Modified Backpropagation and Ordinary Backpropagation both require on the order of Okk) matrix multiplications for k-layer MLP under minimal memoization assumptions. In Appendix E we pro- vide the full analysis for this, as well as Figure 14 which shows empirically that the wall-clock running time of both algorithms is roughly the same for the networks B1-4.
The major difference in this work from Shocher et al. 2023 is that we use Modified Backpropagation for implementing both Lidem and Lright. As such, we evaluate both loss components as the Mean Squared Error (MSE) instead of the L1 loss, and we use the training scheme in Section 2.3 to perform the backwards pass (see also Appendix F). We use the same implementation for Cree as above.
Whilst we provide analysis for the MLP case here, we ex- pect similar findings for other architectures. Practically, the major difference between Ordinary Backpropagation and Modified Backpropagation is the way gradients of the loss with respect the output of the network is computed. In Modified Backpropagation, we require only forward passes of the network to calculate this quantity. whilst for Ordinary Backpropagation one must also find @fe(y) due to the sec- ondary application of f in the loss function (Eq. 2). Thus, in implementations using memoization one should generally expect training time of both algorithms to differ only by a constant factor, whilst without memoization we generally expect Modified Backpropagation to have a computational advantage.
We have successfully replicated several results of Shocher et al. 2023 under this training scheme. In particular, Figure 8 shows qualitative examples of noise drawn from D' being mapped to images resembling samples from the MNIST and CelebA datasets. While outputs remain largely similar between the first and second application of the network, in some cases we do also observe the same "self-correction" be- haviour after the second application as observed by Shocher et al. 2023, with some small defects in background, hairstyle and facial features being corrected. Figure 10 gives further evidence for this, as we demonstrate the ability to recover original dataset images after various degradations have been applied, such as noise, greyscale filters, and Gaussian blur (see Appendix F for details).
In Figure we visualize the effect of applying the trained network to noise linearly interpolated between two clear MNIST samples A, B E 28228 We again observe the secondary application of the network "cleaning up" images. For more uncurated examples of generated images, see Ap- pendix G.
3.4. Application to Generative Networks
As mentioned, one of the motivating factors for actively enforcing idempotency during training is to apply it as a sec- ondary optimization objective in conjunction with optimiz- ing for primary task. In this section we replicate the results of Shocher et al. 2023 as we train a U-net style DCGAN architecture (see Appendix F) on the MNIST and CelebA
We note that qualitative results in this training scheme for both Ordinary Backpropagation (as applied in Shocher et al.
7

---

Enforcing Idempotency in Neural Networks
O
/
9
3
/
S
9
f(x)
f((o))
D
/
9
3
/
S
O
9
f(a)
)l(())
Pigure Uncurated generations of the U-net style DCGAN model trained on MNIST and CelebA with Modified Backpropagation for optimizing idempotent and tightness losses. Rows denote samples drawn from D' whilst the second and third rows represent first and second application the network these samples, respectively.
90%
80%
70%
60%
50%
10%
30%
20%
30%
9
9
f(x)
[((())
3
6
9
9
S
Figure 9. Latent space manipulation for MNIST with the model used in Figure 8. Samples A and are selected randomly while remaining samples linear combinations of these. We give the first and second application of the model.
Noise
Grayscale
Sketch
n[A]
Tatai
s(A)
Asiki
+
J(la()
JUASKDI
Figure 10. Out-of-distributis mappings CelebA with the model used in Figure 8, To images in the dataset we apply transformations Shocher 2023 and plot the first and second application of the model. Characteristic our method the images appear "blurry" but the model recovers the original image relatively well.
8

---

Enforcing Idempotency in Neural Networks
2023) and Modified Backpropagation are heavily sensitive to the hyperparameters À chosen in Eq. (17). Whilst results for MNIST are comparable to those obtained by Shocher et al. 2023, our results on CelebA are less competitive with their work and the state-of-the-art. Nevertheless, with fine- tuned hyperparameters we believe our results could be im- proved significantly and potentially be competitive with state-of-the-ar models.
ferences called SPSA is suggested with competitive perfor- mance lower relative computational cost. Bandler et al. 1988; Do & Reynolds 2013; Scheinberg 2022 all further ex- plore this and similar approaches based on finite differences. Nevertheless, a core idea of Modified Backpropagation is the direct substitution of the gradient of Lidem with the quantity in Eq. (15). Although the exact relationship be- tween this quantity and the gradients produced by Ordinary Backpropagation is still unclear, it is certainly not trivial approximation as evidenced in Section 3.1.
Although the aim of our results is to demonstrate practical applications of Modified Backpropagation, we believe the results could be easily replicated with other datasets, such as Cifar10, or similar. Additionally, further work is required to fully ascertain the practical benefit of Modified Backprop- agation as a secondary optimization procedure in general. In particular, further experiments with other data modalities, primary optimization objectives, and datasets is needed to understand the wider applicability of the results of Sections 3.1 3.2.
5. Conclusion
In this work, we have given motivation for actively enfore- ing idempotency in arbitrary neural networks used in data transformation. The central idea presented is the idem- potent corrector g(K) 3K - -2K?, which has been de- rived by solving system of polynomial equations such that K' - ='= for a near-idempotent matrix K and a perfectly idempotent K'. We have also expanded the idea to a training scheme for arbitrary neural networks via a modification to the canonical backpropagation algorithm, termed Modified Backpropagation. Experimental results have shown that optimizer trajectories generally differ from those of Ordi- nary Backpropagation across wide variety of MLP net- work configurations. Furthermore, we showed that Modified Backpropagation outperforms Ordinary Backpropagation in finding neural networks with lower idempotent error by up to an order of magnitude. Lastly, we demonstrate that Modified Backpropagation can be used alongside Ordinary Backpropagation train generative models on MNIST and CelebA, following the example of Shocher et al. 2023.
4. Related Work
4.1. Algebraic Properties in Neural Networks
There has been significant work in actively enforcing alge- braic properties in weights of neural networks for a variety of reasons. For instance, in Mikolov et al. 2015; Le et al. 2015 it was found that enforcing part of the weights of an RNN to remain close to the identity throughout training can cause the network to capture more long-term information, yielding performance close to LSTMs for the same natural language modelling tasks. Arjovsky et al. 2016 have also gra-
found that RNNs can overcome the exploding/vanishie dient problem if weight matrices actively enforced to be approximately unitary throughout training. Similarly, Saxe et al. 2014; Jing et al. 2017: Kiani et al. 2022 all explore unitary RNNs in the same vein. Lastly, Ardizzone et al. 2019 demonstrate that under a specialized training scheme, inverse mappings of MLPs can be found that can recover correlations in the parameter space.
We believe the work presented here suggests that alternative methods to gradient- based optimization in neural networks are practically viable, and we hope that future work will explore other applications of the central ideas presented here.
A Acknowledgements
These methods generally attempt mitigate undesirable ef- fects inherent to particular architectures or training schemes, whereas the work we present intends to characterize be- haviour that the network should exhibit. As such, applica- tions of the work of Shocher et al. 2023 is closest to ours, hence we focus on this Section 3.4. Nevertheless, the prolific use of orthogonal/unitary weights in the literature invites future work into adapting Modified Backpropagation to enforce orthogonality ((K') (K') (K")("'"" =1)
The authors thank the anonymous reviewers for their helpful comments and suggestions.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
as opposed to idempotency (K') -KK) = 0).
4.2. Alternatives to Gradients
There is a large body of work focusing on approximating gradients. In Spall 1999, a stochastic variant of finite dif-
9

---

Enforcing Idempotency in Neural Networks
References
Le, Q. V., Jaitly, N., and Hinton, G. E. A Simple Way Initialize Recurrent Networks of Rectified Lin- ear Units, 2015. URL https: http://axvvvv.o/ org abs// 1504. 00941.
Ardizzone, L., Kruse, J., Wirkert, S., Rahner, D., Pelle- grini, E. W., Klessen, R. S., Maier-Hein, L., Rother, C., and Kthe, U. Analyzing Inverse Problems with Invertible Neural Networks. In International Confer- ence on Learning Representations, 2019. URL https: //arxiv. org/abs /80.8/18....0 04730.
Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. Visu- alizing the loss landscape of neural nets. In Proceedings of the 32nd International Conference Neural Informa- tion Processing Systems, NIPS'18, pp. 6391-6401 Red Hook, NY, USA, 2018. Curran Associates Inc.
Arjovsky, M., Shah, A., and Bengio. Y. Unitary Evolution Recurrent Neural Networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning- Volume 48, ICML 16, pp. 1120-1128. JMLR.org, 2016.
Liu, M.-Y., Huang. X., Yu, J., Wang, T.-C., and Mallya, A. Generative Adversarial Networks for Image and Vidco Synthesis: Algorithms and Applications. Proceedings of the IEEE, 109(5):839-863 2021. doi: 10. 10.110OJPRRC. 2021 1304999..
Bandler, W., Chen, S. H., Daijavad, S., and Madsen, K. Efficient optimization with integrated gradient approxi- mations. IEEE Iransactions on microwave theory and techniques, 36(2):444-455 455, 1988.
Lu, Y., Chen, D., Olaniyi, E., and Huang, Y. Generative Adversarial Networks (GANs) for image augmentation in agriculture: A systematic review. Computers and Electronics in Agriculture, 1072000 2022. ISSN 0168- 1699. doi: http:://oi.or///..0oof..ooapmppp URL htttp::/www.sccieet rect com/ (tterre//rticale/pp////111162229
Do, S. т. and Reynolds, A. C. Thcoretical connections between optimization algorithms based on an approxi- mate gradient. Computational Geosciences, 17:959-977 2013.
H. Weintraub, Jordan Canonical Form. Springer, 2009.
Ma, S., Zhang, X., Jia, C., Zhao, Z., Wang, S., and Wang, S. Image and Video Compression With Neural Networks: A Review. IEEE Transactions on Circuits and Systems for Video Technology. 30(6): 2020. doi: 10. 1 1109/TCSVT:20192211 2911011199
Hirschfelder, J. O., Brown, W. B., and Epstein, S. T. Recent Developments in Perturbation Theory. volume 1 of Ad- vances in Quantum Chemistry, pp. 255-374. Academic Press, 1964. doi: https///oi.orr//1..r w0tp////....1101/..0011 101.1000000000000070060000 60381-0.
Mao, Y., Wan, Z., Dai, Y., and Yu, X. Deep Idempotent Network for Efficient Single Image Blind Deblurring. IEEE Transactions on Circuits and Systems for Video Technology, 33(1):172-185, 2023. doi: 10.1109/TCSVT. 2022.3202361.
Ilesanmi, A. E. and Ilesanmi, T. O. Methods for im- age denoising using convolutional neural network: review. Complex & Intelligent Systems, 7(5):2179-
2198. 10 2021. ISSN 2198-6053 6053. doi: 10.1007/
440747-021-00022-44 URL https:tt//s/eet //doi../////////e roddoi.org//....
Mikolov, T., Joulin, A., Chopra, S., Mathieu, M., and Ran- zato, M. Learning Longer Memory in Recurrent Neural Networks. In Workshop Submission at ICLR 2015, 2015, URL https: arwiviiiiivv.//.ie .rgg
Jing, L., Shen, Y., Dubcek, T., Peurifoy, J., Skirlo, S., Le- Cun, Y., Tegmark, M., and Soljačić, M. Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs. In Proceedings of the 34th International Con- ference on Machine Learning- Volume 70, ICML'17, pp. 1733-1741. JMLR.org. 2017.
httgg//b///// abs////////// 111// 4
Namphol, A., Chin, S., and Arozullah, M. Image compres- sion with a hierarchical neural network. IEEE Transac- tions on Aerospace and Electronic Systems, 32(1):326- 338, 1996. doi: 10.112.77.48127..
Kato, T. Perturbation Theory for Linear Operators. Springer, 1995. ISBN 978-981-16-2527-5. doi: https:
Saxe, A. M., McClelland, J. L.,, and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2014. URL https: /ttss///axxiva org/ / abs / 111.. 6120.
Kiani, B., Balestriero. R., LeCun, Y., and Lloyd, S. pro- jUNN: efficient method for training deep networks with unitary matrices, In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 14448-14463 Curran Associates, Inc., 2022.
Scheinberg, K. Finite difference gradient approximation: To randomize or not? INFORMS Journal on Computing, 34(5):2384-2388, 2022.
10

---

Enforcing Idempotency in Neural Networks
Shocher, A., Dravid, A., Gandelsman, Y., Mosseri, I,, Rubinstein, M., and Efros, A. A. Idempotent Gener- ative Network. In The Twelfth International Confer- ence on Learning Representations, 2023. URL https: //arxiv.org abs /231 01462.
Song, M., Ahn, K., and Yun, C. Does SGD really hap- pen in tiny subspaces? In Proceedings of the Work- shop on High-dimenniooal Learning Dynamics (HiLD) at ICML 2024. PMLR, 2024. URL https:://aaaae org/ axg/abs/2a abaa 2405. 16002.
Spall. J. C. Stochastic optimization and the simultancous perturbation method. In Proceedings of the 31st Con- ference on Winter Simulation: Simulation-a Bridge to the Future Volume I, WSC '99, pp. 10-109, New York, NY, USA, 1999. Association for Computing Machinery. ISBN 0780357809. doi: 00.11444//4418.32411770 URL https:/ /ttp::::///ooooo/11////0 /ttp:///11111//11111111
Tambouratzis, T. novel artificial neural network for sort- ing. IEEE Transactions on Systems, Man, and Cyber- netics, Part (Cybernetics), 29(2):271-275, 1999. doi: 10.1 10.1109/3477.752799

---

Enforcing Idempotency in Neural Networks
A. Solutions to the Ansatz
We give detailed description of how to derive idempotent correctors for matrices that are near-idempotent to order with fixed dimension For given j, find mapping 9 making the input idempotent:
1. Assume K = D.
2. Assume that D x p P, and XDYDZ x 0 for all X, Y, Z.
3. Expand the expression (K')² K' 0.
j = 2 gives 34 terms
j 3 gives 154 terms
j = 6 gives 10,794 terms
4. Apply assumptions from 2 recursively.
j 2 reduces 34 to 16 terms
j 3 reduces 154 to 32 terms
j = 6 reduces 10,794 to 104 terms
5. Collect coefficients for D, P, DP, PD, and PDP (no other exist).
6. Create a set of equations from coefficients and solve as a set polynomial equations.
For 1 we give the first j 1,...,8 idempotent correctors which have been found using the above recipe and the "Reduce" method in Mathematica:
For j = 1, 2, there are no solutions.
For j = 3, there is one solution (found in ~ 130 ms):
g 3K² -2K
(21)
For j 4, there one solution (found 611 ms):
g (K) = K² + (4 - 23²) + +wv2
- -av2-3)) K2
(22)
For j = 5, there is one solution (found in ~ 3.8 s):
g (K) ag + K + (5 300 20g Kt + (20z + a0 - 4)
(23)
For j 6, there one solution (found 24 s):
g(K) a² a3 + K 4 + (6 4az 303 -Ka (3az + 2ag + as 5)
(24)
For j = 7, there one solution (found ~ 150 s):
g (K) = o2 + K + K + agK + (7 - 50. - 40g - 30s - 20g K
(25)
+ (40g 30g 20.4 as - K)
For j 8, there is one solution (found in 920 s):
g(K) a²k² + ag + ta + asK + tak
(8 602 50g 4a4 33r5 200 (ow))- K
(26)
(50g + 4aa + 3a4 + 2a5 + a6 7)
12

---

Enforcing Idempotency in Neural Networks
B. Jordan Normal Form Analysis
For a matrix E IR let (<A) denote algebraic multiplicity of eigenvalue and let 7(A) denote the geometric multiplicity of X. Given the equation
=33 -2KK
(27)
substitute K PJP-1
PJP PJJP-1 -
(28)
which can be simplified to Since P is invertible, this is equivalent
PJP 3PJ -PP'''
(29)
3J 2J
(30)
Thus for an arbitrary K, solving Eq. (27) equates to solving Eq. (30).
Since J is block-diagonal, the equation in (30) can be broken down into smaller equations for each block, J = 3J 2J
which we can write as system of equations:
2
2A
0
A3
312
31
0
0
0
X2
2
0
0
312
31
0
0 0
3
0
0
X²
0
2
0
A
312
0
(31)
:
:
0
X?
0
0
0
X3
Thus, for (Ae x k) Jordan block we have up to four equations:
A 31 213
(32)
3(2A) 2(322) 6Λ 6X2
Only when & 2 2.
(33)
3(1) -2(3A) =3- 6Λ
Only when k 2 3.
(34)
0 3(0) -((1) = -0-2 2
Only when & 2 4.
(35)
There never equations than this, since all other entries in Jordan block must be O.
Since Eq. (35) is a contradiction, we can have no solution which solves all Eqs. (32), (33), (34), and (35). Note also that there exists no solutions satisfying Eqs. (32), (33), and (34), nor do any solutions exist for both Eqs. (32) and (33). The following are solutions which satisfy only Eq. (32):
A = (0,0.0 0.5, 1}
(36)
The only situation where Eqs. (33), (34), and (35) do not arise is when (AA) 7(), which is precisely the case when K is diagonalizable. Therefore, any K which is a solution to (27) must have Jordan normal form which is a diagonal matrix:
A1
0 0
A2
(37)
0 A3
13

---

Enforcing Idempotency in Neural Networks
C. Automatic Differentiation Rule
To implement Modified Backpropagation in PyTorch we make use of custom autograd functions. description of the loss function for idempotency
The mathematical
Lidem (y) - -))
(38)
To practically implement the function, we note that PyTorch optimizers reduce - for some loss function L, but as per Section 2, our method reduces idempotent error when the loss function above is not negatively signed. This leads to the implementation the E. backward function in algorithm 1.
Algorithm Modified Backpropagation PyTorch rule.
class E (torch, autoorad. Function)
i cmethed def loss_fn fn (y, net):
loss torch. mean ( (net (y) y) ** 2)
return loss
@staticmethod
def (orward(ctx, y, net):
exx.save_for_backaat (y)
ctx.net net
return E. lolss_sfii (y, net)
@staticmethod
def backward outpputt
y. ctx.sared_tennors net ctx.net
y2 net (y)
y3 net (y2)
3+y2 2+y3
grads -e shape [0]
return grads * outpuu, None
class ELoss (torch. Module);
def _ (self, net, mode):
super (ELoss, self)__int init ()
self.net net
self mode mode
def (sel,, y) return E. E.apply (y, self.net)
14

---

Enforcing Idempotency in Neural Networks
We give the computational graph constructed by PyTorch for calculating gradients in Modified Backpropagation and Ordinary Backpropagation. The graphs constructed using the same single-layer neural network without biases and no activation function. We therefore have fo(x) Wx. PyTorch, however, represents the input x transposed, hence the following use (x) = Wx = xW" as the definition. We show the graphs describing the computation of gradients for W for a single optimization step.
In these graphs, the weight matrix W the blue box, and the green box at the bottom the graph is the gradient matrix 1 Yellow boxes of size 3 x 5 denote the imput x consisting of 3 samples with 5 features each.
UW
(a) Modified Backpropagation
(b) Ordinary Backpropagation
Figure 11. PyTorch computational graphs for gradient calculation.
The red line in Figure (a) denotes the intuitive split by the chain rule. Computation above the line corresponds to &Cadem (y) by
and computation below the line corresponds afa(x)
9w which altogether gives &Cistm (x))))
Chasing through the graph
&W
shows that indeed does use exactly the algorithm outlined above.
The graph Figure (bbb) shows a computation of aCidem (fo(x)) without using E. backward. Although the graph is less %W
obvious, chasing the graph shows that it computes the following quantity:
(2xW W xW") xW" + ((2xW"'W" - -xx") x 2(xW" xW xx"x
This is equivalent to the ordinary analytical solution @Citem for this particular network. &W
15

---

Enforcing Idempotency in Neural Networks
D. Idempotent Error and Norm for Test Networks
For each of the test networks outlined in Table I we report the distribution of absolute idempotent error and norm of resulting mapping in Figures 12 and 13. The learning rates used for training of each network is chosen be the value in Figure with the lowest idempotent error for each algorithm and network configuration.
Modified Backpropegation
Ordinary Backpropagation
50-
21-
40-
05-
20-
10
10"
Error
Ennor
(a) Network B1
Modified Backpropagation
Ordinary Backpropagation
15.0
17.5
12.5
10.0
12.5
Coune
35
5.0
5.-
3.5
0.0
Emor
Bino
(b) Network B2
Modified Backpropagation
Ordinary Backpropagation
$7.5
15.0
10-
CrnER
10.0
5.0
0.
$7-10
475-10
880 10
s.x 10
405-100 11
61-10"
13855 10
1386000 1888
1.55 to
1885 1186 t0
1.90 10
Timor
Crror
(c) Network
Modified Backpropagation
Ordinary Backpropagation
30
15
Ornzr
Conte
1.6
1.3
1.8-000
1.9
2.
21-110
222
10
1005
105
1,55 10
1.05
(d) Network B4
Figure For each network we report the distribution of absolute idempotent error after 2500 epochs training 250 randomly initialized networks. Note that distributions are narrow for cach algorithm and often separated by more than an order of magnitude.
16

---

Enforcing Idempotency in Neural Networks
Modified Backpropagation
Ordinary Backpropagation
25-
20-
10-
Norm
Norm
(a) Network
Modified Backpropagation
Ordinary Backpropagation
17.5
15.0
15-
Coout
1000
7.5
00
6-10 10
10
3x 10
Nor
(b) Network B2
Medified Backpropagation
Ordizary Backpropagation
20-
12-
2-
5>310
S1
5.2x
5.3x.10 10'
5.5
10
3.14-10 3.16 3.18 10 3.2x 3.22
3.24 10 3.26 3.25-10
(c) Network B3
Modified Bacápropagation
Ordinary Backpropsgation
15.0
12.5
(24m
100
18
2.5
00
2x100
Nom
Nora
(d) Network B4
Figure 13. For each network we report the distribution of norms after 2500 epochs training 250 randomly initialized networks. Note that distributions for either method not centred around zero, indicating that the trained network non-trivial idempotent mapping. To calculate the norm we pass each canonical basis vector of RR" as input by concatenating them into the identity matrix and report here the Frobenius norm of the output matrix.
17

---

Enforcing Idempotency in Neural Networks
E. Relative Computational Cost
Network BI
Network B2
Network B3
Network B4
17.5
14
92
9-
Ang Walt-Cee Time: (0
Ang. WiltCBook Tim:
00
41
8-
Wall-Ches Clock
10%
Wall Clock
88
159
40
ms
86
39
16.0
15.0
10
Modified
Ordinery
Modified UP
Ordinary
Medifid BP
Ondinary
Modified BP
Onlinory BP
Figure 14. Computational cost for networks B1-B4 in Table 1. Plot of average wall-clock running time consumed per training epoch across repetitions algorithm, with 98% confidence intervals. Across configurations, both algorithms take approximalely the same amount of time, with slight advantage to Modified Backpropagation.
The practical findings of Figure 14 supported by a theoretical argument which we give in detail here. Concretely, consider the growth in the number of matrix multiplications required as the number of layers increases in an ordinary MLP. Defining = fx)),, the loss function in Eq. 2 can be written as Lidem (y) = (fo(yy) - y) By the chain rule ay
aa ay By 8w Computing eW using backpropagation will generally use (kk) matrix multiplications for k-layer MLP. This quantity is computed in the same way for both Ordinary Backpropagation and Modified Backpropagation.
dC #W
For Ordinary Backpropagation, the quantity aCitem can also be unfolded via the chain rule and its evaluation requires By
@fe(y) By which each can be computed in O(k) time. Thus, assuming memoization used (as is the case by design in most automatic differentiation frameworks, like PyTorch), Ordinary Backpropagation compute &C in (k) time.
computing y = fx(x), fo(),, as well as
9W
For Modified Backpropagation we compute سسسةm 3foly) - 2 y. Again, assuming memoization, each of by
these terms take (k) time to evaluate, so Modified Backpropagation also runs in (kk) time.
Therefore, both Ordinary Backpropagation and Modified Backpropagation should require the same number of matrix multiplications under realistic assumptions.
18

---

Enforcing Idempotency in Neural Networks
F. Training Scheme for Generative Models
follow example Shocher et al. 2023 and train U-Net style DCGAN architecture on the MNIST dataset. Let y denote the latent space and X' denote the sample space, Then, letting G y + X and D X - y be the Generator and Discriminator of the network respectively, apply the network G(D(x)) = y.
Table 2 explicitly shows the architecture we use, and Table 3 shows the hyperparameters chosen for training. The core difference between our configuration and that of Shocher et al. 2023 is the use of dropout layers at some layers.
Table 2. U-Net style DCGAN Network architecture.
Layer
Size
Stride
Padding
Features
Dropout?
BN?
Activation Func
Diseriminator
Conv2l)
4 x
64
LeakyReL_U(0.2)
Conv2D
4
128
LeakyReLU(0.2))
Conv2D
3 X 3
0
256
V
LeakyReLU(0.2))
Conv2D
3 x 3
0
512
V
LeakyReLU(0.2)
Conv2D
3 x 3
0
512
X
None
Conv TranpposDD
3 x 3
0
256
V
ReLU
Generator
ConsSraaspose211
3 3
0
128
ReLU
Conv ConnTranspose21
3 3
0
64
V
ReLU
Conv TranppossD
4
32
V
ReLU
Conv 2ranspose2D
4
Tanh
Table 3. Training parameters.
Parameter
Value
Reconstruction loss metric
Li = ly* ypred ||
Loss terms weighting (MNIST)
A, 20, A = 0.1, A = 0.1
Loss terms weighting (CelebA)
A, = 20, x; = 0.006, Ar = =0.02
Optimizer
Adam(Ir =10 10-4, 31 = =0.5, f2 0.999)
Dropout probability
0.05
Batch Size
512
Epochs
100
Weight initialization
Default Kaiming Uniform initialization: u(-Vk, vk), for
k /1/n for features.
Degradations applied for out-of-distribution mappings
All degradations are implemented in Shocher et 2023 and are restated here for clarity. Let be an arbitrary sample from the dataset.
Noise: Let nxx
n, where is sampled from (0, 0.15).
Greyscale: Let (x) be the function that averages each pixel value across channels and returns the result.
Gaussian Blur: Let sxx)
Muussianbluu(g(xxt1 210 1, where gaussian blur(z, 21) applies Gaussian Blur with
g(x+1)
kernel size 21
19

---

Enforcing Idempotency in Neural Networks
G. Generative Model Sample Images
3
3
3
3
8
9
3
0
J
3
3
9
<
<
3
3
3
3
6
6
5
/
2
5
2
3
C
(
2
5
I
5
/
S
5
/
/
3
3
9
S
5
O
0
/
7
3
2
/
3
e
9
5
3
0
/
P
9
3
5
/
3
S
5
3
3
7
/
U
3
3
J
/
/
S
U
5
2
3
3
3
9
3
Q
/
S
3
/
/
9
3
3
3
3
9
9
3
/
9
]
)
/
O
3
e
e
O
9
S
9
9
3
3
3
D
)
3
J
I
O
9
5
/
/
b
/
3
9
9
5
0
3
Figure 15. Uncurated generations of U-net style DCGAN model trained on MNIST with Modified Backpropagation for optimizing
idempotent and tightness losses. All samples are drawn from a random distribution with mean 0 and variance 1, and the result of applying the network is shown.
20

---

Enforcing Idempotency in Neural Networks
H. Optimizer Trajectory Plots
Network B1
Network B2
0.07
0.3
0.05
2nd PC: 15,85 %
2nd PC: 4,26
0.83
200
000
0010
000
x000
x
601
00 222811
nt x-9
0.0
1.36
or
0,02
311
-0.1
0.11
100
50'
0.00
-0.2
-0.01
-0.3
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.0
0.5
1.0
1st PC: 74.87
%
1st PC: 93.55 %
Network B3
Network B4
0.02
06 -80
900
2nd PC: 2.49 %
1.0
2nd PC: 1.48
1000%00 02
1.78 to
0.5
55E
0.0g
K77 10
5.10
0.
20
0.02
-0.5
-15
-1.0
PC: 93.52
1st PC: 96.18 %
Modified Backpropaaationn
Ordinary Backpeopagacion
Figure 16. Representative projections of the optimizer trajectories 2500 epochs of either algorithu each of the B1-B4 models optimal learning rates (Figure 6). Total variance captured is >> 90% with cosine similarity of PCI and PC2 less than 1.0 x 10 across all plots. Optimizer trajectory of Modified Backpropagation often deviates significantly from Ordinary Backpropagation, but may sometimes overlap (e.g., B4).
21


Reviews:
[{"review_id": "03sQ0JURKu", "reviewer": "Reviewer_kYga", "summary": "The paper presents a new approach to enforcing idempotency in neural networks through a modification of the backpropagation algorithm, termed Modified Backpropagation. The key idea is the derivation of an idempotent corrector function $g(K) = 3K^3 - 2K^2$, which iteratively projects a real-valued matrix onto the manifold of idempotent matrices. The authors extend this idea to general neural network training by modifying the canonical gradient-based optimization approach. Experimental results demonstrate that this method reduces idempotent error and outperforms ordinary backpropagation in multiple MLP and CNN architectures.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable feedback. We appreciate the concerns raised, and we aim to address these below.\n\n**Theoretical development.** We employ a novel theoretical framework which allows gradient-free training of an idempotent property. For us it is interesting that there exists a unique order-3 polynomial iterator $\\mathbf{K}' = 3 \\mathbf{K}^2 - 2 \\mathbf{K}^3$ for the idempotence property. While the stability analysis may be technically straightforward, we have not seen perturbative techniques used for similar purposes in the machine learning community, and we believe our methods could be extended to other algebraic structures beyond idempotence. We therefore suggest that our work could be of theoretical interest to others in the community.\n\n\n**Computational cost.** A theoretical analysis shows that ordinary backpropagation and modified backpropagation have the same computational complexity. These results are reflected in practical experiments (https://imgur.com/a/S8FSExJ), which show that both algorithms run in approximately the same amount of time. These results show, for networks B1-B4 in Table 1, average wall-clock running time consumed per training epoch across 250 repetitions of each algorithm, with 98\\% confidence intervals. Across all configurations, both algorithms take approximately the same amount of time, with a slight advantage to Modified Backpropagation with relative improvement ranging from $1-38\\%$.\n\nThe theoretical argument goes as follows. We here consider the growth in the number of matrix multiplications required as the number of layers increases. Defining $y=f(\\mathbf{x})$, the loss function from Eq 2 can be written as $L(\\mathbf{y}) = \\frac{1}{m} \\sum (f(\\mathbf{y}) - \\mathbf{y})^2$. By the chain rule $\\frac{\\partial{L}}{\\partial{\\mathbf{W}}} = \\frac{\\partial{L}}{\\partial{\\mathbf{y}}} \\frac{\\partial{\\mathbf{y}}}{\\partial{\\mathbf{W}}}$. Computing $\\frac{\\partial{\\mathbf{y}}}{\\partial{\\mathbf{W}}}$ using backpropagation will generally use $O(k)$ matrix multiplications for a $k$-layer MLP. This quantity is computed in the same way for both Modified Backpropagation and Ordinary Backpropagation.\n\nFor Ordinary Backpropagation, the quantity $\\frac{\\partial{L}}{\\partial{\\mathbf{y}}}$ can also be unfolded via the chain rule and its evaluation requires computing $\\mathbf{y} = f(\\mathbf{x})$, $f(\\mathbf{y})$, as well as $\\frac{\\partial{f(\\mathbf{y})}}{\\partial{\\mathbf{y}}}$, which each can be computed in $O(k)$. Thus, assuming memoization is used (as is the case by design in most autodiff frameworks, like PyTorch), Ordinary Backpropagation can compute $\\frac{\\partial{L}}{\\partial{\\mathbf{W}}}$ in $O(k)$ time. (In Section 1 around Eq. 2 in the paper we argue that the computational cost of evaluating this gradient in Ordinary Backpropagation grows exponentially in the number of layers. Memoisation reduces this to $O(k)$ time, and we propose to clarify this point in the paper.) For Modified Backpropagation we compute $\\frac{\\partial{L}}{\\partial{\\mathbf{y}}} = 3f(\\mathbf{y}) - 2f(f(\\mathbf{y})) - \\mathbf{y}$. Again, assuming memoization, each of these terms take $O(k)$ time to evaluate, so Modified Backpropagation also runs in $O(k)$ time.\n\nIn the final paper, if accepted, we will include content on the theoretical discussion on relative running time, as well as the graph showing our practical experiments which demonstrate this point.\n\n\n**Error spikes in Figure 6-B1.** The spikes are caused by some runs of Modified Backpropagation converging (within machine precision) to the zero-matrix during training. Due to floating-point imprecision this is rounded down and causes the effect. We chose not to exclude these runs from analysis as we deem it important to expose this behaviour. Note that this only occurs on the small architecture B1. The last sentence in the caption for Figure 6 addresses exactly this concern. In the final paper, if accepted, we will add a further sentence in the main body to make this point more clearly.\n\n\n**Application to other network architectures.** We appreciate the relevance of exploring more complex network architectures. This work proves the principle that a fundamentally new approach to learning idempotent networks is possible. In this early work it has been important to explore behaviour of Modified Backpropagation in controlled environments with few external factors in order to give convincing evidence of the method's efficacy. To this end, we have evaluated our method on a variety of MLP networks and synthetic datasets which illuminates behaviour such as sensitivity to network size, dataset distributions, and running time. Nevertheless, wider application of the method in practice will depend on generalization to the architectures mentioned by the reviewer, and we are eager to explore this in future work.", "claims_and_evidence": "The claims made in the submission supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method makes sense for the problem and its application.", "theoretical_claims": "I have checked all details of the theory part. There is no actual proof.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "This paper provides a new way to enforcing idempotency in neural networks through a modification of the backpropagation algorithm.", "essential_references_not_discussed": "I believe that most relevant works have been cited in the paper.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-written and it is easy to follow.\n\n- The introduction of an idempotent correction function and its integration into neural network training is a promising research topic as it has not been explored deeply in the literature. The relation with perturbation theory is interesting.\n\n- The paper includes experiments demonstrating improved idempotent properties in trained networks compared to ordinary backpropagation.\n\n- The method has been applied to generative models.\n\n**Weaknesses**\n\n- The theory part of the paper is weak. The relation of the proposed method to the perturbation theory as well as the stability analysis is quite easy and straightforward.\n\n- The paper primarily focuses on MLPs and CNNs, but it does not provide results on more complex architectures like transformers or diffusion models.\n\n- The computational cost of enforcing idempotency through Modified Backpropagation is not discussed in detail. Does the method introduce significant overhead compared to traditional approaches?\n\n- It is unclear how much each component (e.g., the choice of recurrence relation, hyperparameter γ) contributes to the observed improvements.", "other_comments_or_suggestions": "No.", "questions_for_authors": "- How does the proposed method generalize to architectures beyond MLPs and CNNs, such as transformers or graph neural networks?\n\n- In Figure 6-B1, why the error of the modified backpropagation it not stable. Is there any reason why?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PClSKw65Au", "reviewer": "Reviewer_hBam", "summary": "This paper introduces a novel approach for training idempotent neural networks.\nLeveraging techniques from perturbation theory on idempotent matrices, the authors propose a new method for projecting matrices onto the idempotent manifold.\nThey further extend this approach to nonlinear neural networks.\nFinally, the paper presents experiments, primarily on toy datasets and MNIST generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable feedback. We are happy to respond to the questions that have been raised.\n\n**Extended experiments with CelebA.** We agree with the reviewer that extending experiments to cover other large datasets is interesting. In the graph (https://imgur.com/a/w89mL6I) we demonstrate generation of CelebA images from noise, following the experimental protocol of IGN, adjusted to use our Modified Backpropagation method. We train the same U-Net DCGAN architecture in the way outlined in our paper. Hyperparameters $\\lambda$ were chosen as $\\lambda_r = 20$, $\\lambda_i = 0.006$, $\\lambda_t = 0.02$ by trial and error. As with MNIST (and IGN), we sample noise with similar frequency-statistics as images from the dataset.\n\nAlthough the results are qualitatively inferior to state-of-the-art generative models, we believe that this is partly due to suboptimal hyperparameter selection, which would likely be improved with further fine-tuning. Nevertheless, the provided graph clearly demonstrates similar behaviour to that observed in our paper for the MNIST dataset (Figures 8 and 9), and in the IGN paper for MNIST and CelebA (Figure 4.) In particular, the images $f(\\mathbf{z})$ and $f(f(\\mathbf{z}))$ are highly similar as one expects from an idempotent mapping. Furthermore, we also observe the desired self-correcting property, with some small defects in background, hairstyle, and facial features visible in the image $f(\\mathbf{z})$ then being corrected in the image $f(f(\\mathbf{z}))$.\n\nA key result of the IGN paper is the ability to correct degraded images and map them into the distribution. In the graph (https://imgur.com/a/j4d20Ni) we demonstrate how a model trained using our method can correct images from the CelebA dataset after applying added noise, a greyscale filter, and a sketch filter. As in IGN, or model does not perfectly recover the original image, but characterizing features are clearly recovered in many cases.\n\nThe results on CelebA were not initially included in the paper as we wish to focus primarily on the task of finding general idempotent mappings, as opposed to focusing on generative applications of the method. However, as experiments on CelebA was requested by several reviewers, we propose to include a short section on these results in the final paper, if our work is accepted.\n\n**Figures 3-7 in larger-scale settings.** The larger-scale generative setting we consider in this paper has a loss function which is composed of several components, some of which are adversarial in nature. It is well known that training generative models requires careful balancing of the hyperparameters in order to achieve good qualitative results, and it is rarely beneficial to let the idempotent loss factor too greatly in the combined loss. On that basis, even if our approach would be able to find models with lower idempotent loss, we believe that reproducing Figures 6-7 in this setting would be misleading, as they cannot be used meaningfully to judge the quality of the trained network.\n\nFigures 3-5 could indeed be reproduced meaningfully for our MNIST dataset, and we agree with the reviewer that it would be interesting to see whether the behaviour we observe on toy networks also applies to the larger-scale setting. We plan to do this thoroughly in future work, where we can give a fuller analysis of our method for more complex architectures.", "claims_and_evidence": "I find the claims to be convincing.\nA key challenge arises when extending linear idempotency to the nonlinear case.\nTo address this in a practical and reasonable manner, the authors approximate the gradient of the loss as a local linear matrix and incorporate its direction into the gradient update to reduce idempotency error. I believe this approach is fairly reasonable.", "methods_and_evaluation_criteria": "Yes. More discussions on \"Experimental Designs Or Analyses\" section.", "theoretical_claims": "Theory in idempotency seems to be correct.", "experimental_designs_or_analyses": "- The visualizations of optimizer trajectories (Figure 3), absolute cosine similarity (Figure 4), and gradient norms (Figure 5) demonstrate that idempotent guidance facilitates the training of IGN. These visualizaton (Figures 3–7) provide valuable insights into the properties emerging from this idempotency guidance, effectively illustrating the benefits of the modified backpropagation.\n\n- A major limitation of this paper is that the experiments are restricted to toy datasets and MNIST. Could the authors extend the experiments to CelebA, following the experiment protocol in IGN?", "relation_to_broader_scientific_literature": ".", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": ".", "other_comments_or_suggestions": ".", "questions_for_authors": "Could the authors provide a similar analysis for the MNIST experiments as presented in Figures 3–7? It would be valuable to see whether the observed properties hold in a larger-scale setting.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BsP1PqhPad", "reviewer": "Reviewer_pCL4", "summary": "Idempotent Generative Networks (IGN) require an operator $f$ to satisfy $f = f \\circ f$. This paper addresses such idempotency by \n analysis from perturbation theory, identifying the polynomial $3K^2 - 2K^3$ as one that projects matrices to the idempotent matricse manifold. The authors then adapt this approach to the non-linear case. Specifically, they override the gradient of $||f(f(x))-f(x)||$ with $3f(y) - 2f(f(y)) - y$, thus simplifying backprop. Experiments on MLPs and on MNIST with a U-net style DCGAN show reduced idempotent error.\n\nI am listing strengths, weaknesses, and questions throughout these review fields, marked as **(S), (W), (Q)**. \n\n\n## update after rebuttal \nAs mentioned in my comment below, after some of my concerns have been addressed I would like to maintain my score and to state that this paper is safely in the \"4\" zone.  \nThanks again to the authors.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable feedback. We are pleased that the reviewer finds this to be an important contribution with much potential, and with a strongly interdisciplinary character. We appreciate the concerns raised, and we aim to address these below.\n\n**(W2) Runtime and memory cost.** Please see our response to reviewer \"kYga\" for a discussion of the runtime cost of Modified Backpropagation vs. Ordinary Backpropagation.\n\n**(W4 + W5) Dataset variety and out-of-distribution experiments.** Please see our response to reviewer \"hBam\" where we give indicative results on CelebA and replicate the out-of-distribution experiment of the IGN paper.\n\n**(W3) Reliance on near-idempotency.** Our illustration Figure 2 shows a relatively large domain of convergence around the fixed points for the linear case. In the generative setting, other elements of the loss function already help the network to learn an approximately idempotent behaviour, moving the model weights to a near-idempotent regime where our method can be most effective. Questions of convergence in the general non-linear case are fascinating to consider and we propose to investigate this in further work.\n\n**(Q1).** The reviewer's perspective on how our method extends the idempotence condition from the linear to the non-linear case is very well-stated, and agrees with our current understanding. We are grateful to the reviewer for setting this out, and if accepted, we propose to revise the start of Section 2.3 to include this perspective.\n\nWhen extending Modified Backpropagation to the non-linear case, we wish for the network to act in an idempotent way around inputs taken from the training distribution (and hope that enough such points yields idempotent behaviour for the rest of the distribution). Indeed, by treating the behaviour of the network around inputs as locally linear, Eq. 15 becomes the ``obvious'' choice for optimizing idempotency in the context of the method we discuss in Section 2 of our paper. We therefore agree with the reviewer's suggestion that setting $\\frac{\\partial{L}}{\\partial{\\mathbf{y}}} = 3f(\\mathbf{y}) - 2f(f(\\mathbf{y})) - \\mathbf{y}$ in Modified Backpropagation effectively trains the first order approximation of the network to act as an idempotent mapping. In particular, what we are optimizing is the idempotence property of the Jacobian of $f$ around input points.", "claims_and_evidence": "**(W1)** The paper claims the polynomial-based gradient step outperforms naive backprop in achieving $f(f(x)) \\approx f(x)$, demonstrated with synthetic MLPs and a small U-net on MNIST. While the results are persuasive for those cases, there is no large-scale benchmark comparison.", "methods_and_evaluation_criteria": "**(S1)** The authors propose a new method—“modified backprop”—that replaces $\\partial L/\\partial y$ by $3f(y)-2f(f(y))-y$, avoiding the usual chain rule on $f\\circ f$. This is simple to implement and apparently stable.\n\n**(W2)** Evaluation focuses on how quickly $\\|\\,f(f(x)) - f(x)\\|\\,$ shrinks. There is no direct measure of runtime or memory cost versus naive double-backprop, which would have clarified efficiency gains.", "theoretical_claims": "**(S2)** The linear case for $K$ is well understood: $3K^2 - 2K^3$ projects $K$ onto the set of idempotent operators. The paper extends this to non-linear networks via the gradient substitution. This is conceptually clean, though no convergence proof is provided for deep architectures.\n\n**(W3)** The derivation relies on near-idempotency perturbation arguments but does not fully address whether one can recover from being far from idempotent.", "experimental_designs_or_analyses": "**(S3)** They present ablation across different MLP depths and widths, showing a consistent drop in idempotent error. On MNIST, they adapt IGN method with “tightness” and show visually plausible results.\n\n**(W4)** The dataset variety is minimal (mostly random MLP inputs, plus MNIST). Additional experiments on complex images or tasks would strengthen the claims.", "relation_to_broader_scientific_literature": "**(S5)** The paper references Shocher et al. (Idempotent Generative Networks) and classical perturbation theory for linear idempotent operators. This is interdisciplinary, connecting different fields and gets benefits. This topic is mostly self-contained so not too much context is needed, I think the authors describe the relation well", "essential_references_not_discussed": "None that are strictly missing, but further parallels to other “fixed-point” networks or to full second-order approaches could be informative.", "other_strengths_and_weaknesses": "**(S6)** This is a very elegant idea and an impressive solution. At first glance it seems impossible that gradients taken only through one copy of the network yied optimization equivalent to taking the through recursion of two copies of the network. This is a solid mathematical solution.\n\n**(S7)** What the authors propose provides stability and freedom for IGN. It allows architectures that couldn't work with the original IGN and makes training more stable. This is a step towards making IGN practical.\n\n**(W5)** I would want to see projection of out-of-distribution data as shown in the original IGN paper, I think it is a key feature of IGN.\n\n**(S8)** the convergence diagram at Fig.2 is eye opening and very insightful.", "other_comments_or_suggestions": "I find this paper an important contribution, taking a step with a currently not very practical generative model, making it more feasible. I think this has much potential, having in mind the first generative diffusion models paper (Sohl-Dickstein 2015) that took some time to adapt until it became a major thing. I am also impressed with the Interdisciplinarity and the idea of bringing in perturbation theory to solve this problem. I find it novel, creative and out of the main stream works. However, the experimental side of this work is not strong enough. I would expect a step up in terms of scale, data, quality w.r.t. the previous IGN paper. I see the experimental and evaluation side as problematic. Nevertheless I think this is a definite accept.", "questions_for_authors": "**(Q1)** I was trying to interpret what is being optimized by the modified backprop step, and I speculate it can be seen as the idempotency of the Jacobian of $f$ at location $x$. Specifically, it is the distance between the Jacobian and the polynomial of that Jacobian $||(3J_\\theta(x)^2 - 2J_\\theta(x)^3 - J_\\theta(x))x||^2$.\n1. In the linear setting, where $f(x) = Kx$ is constant, directly enforcing $K = 3K^2 - 2K^3$ yields idempotency.  \n2. For the non-linear case, one could want each local Jacobian$J_\\theta(x)$ to act similarly, but that Jacobian depends on $\\theta$ and $x$ and is not fixed. So what one can do is train the network weights so that the Jacobian is close to the polynomial, providing the objective  $L = E_x[||(3J_\\theta(x)^2 - 2J_\\theta(x)^3 - J_\\theta(x))x||^2] = E_x[||3f(y) - 2f(f(y)) - y||^2]$.  \n3. Taking the gradient of this loss with respect to $y$ in such a construction is proportional to $3f(y) - 2f(f(y)) - y$, which matches the polynomial update the authors propose in their modified back-propagation.  \n4. This equivalence suggests the network is being trained so that, at for all inputs (noise or images) in the training distribution, the first order approximation is close to its projection onto the manifold of idempotent matrices.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies a modified backpropagation method based on perturbation theory to enforce idempotency (f(f(x)) = f(x)) in generative neural networks. The majority of reviewers agree that the approach is novel and elegant, and experiments support that the idempotency error improves across various neural network architectures. However, some reviewers point out that the experimental scope is somewhat limited, e.g., lacking evaluation on more complex models (Transformers, diffusion models) or larger, more diverse datasets (e.g., CelebA). Focusing on the experimental part would definitely make the paper stronger.

In addition to reviewers' comments there are several inaccuracies which need to be fixed in the final version:

- After equation (4) it is stated that it leads to a linear program (which is actually a linear program with quadratic constraints (with terms of the form \alpha_i \alpha_j for any i,j -- including i=j). Does this affect derivations?

- Around (6) d_\lambda is defined as the multiplicity of an eigenvalue, but it is rather the size of the corresponding Jordan block. This leads to stating that "geometric multiplicity of every eigenvalue must also be exactly 1", which is clearly incorrect (excluding, e.g., the identity matrix, which is clearly a solution).
Decision: Accept (poster)