Title: Nearly Optimal Bounds for Cyclic Forgetting
Venue: NeurIPS 2023
Accepted: yes
Keywords: catastrophic forgetting

linear systems
Abstract: We provide theoretical bounds on the forgetting quantity in the continual learning setting for linear tasks, where each round of learning corresponds to projecting onto a linear subspace. For a cyclic task ordering on $T$ tasks repeated $m$ times each, we prove the best known upper bound of $O(T^2/m)$ on the forgetting. Notably, our bound holds uniformly over all choices of tasks and is independent of the ambient dimension. Our main technical contribution is a characterization of the union of all numerical ranges of products of $T$ (real or complex) projections as a sinusoidal spiral, which may be of independent interest.

Nearly Optimal Bounds for Cyclic Forgetting

Abstract

We provide theoretical bounds on the forgetting quantity in the continual learning setting for linear tasks, where each round of learning corresponds to projecting onto a linear subspace. For a cyclic task ordering on T tasks repeated m times each, we prove the best known upper bound of O ( T 2 /m ) on the forgetting. Notably, our bound holds uniformly over all choices of tasks and is independent of the ambient dimension. Our main technical contribution is a characterization of the union of all numerical ranges of products of T (real or complex) projections as a sinusoidal spiral, which may be of independent interest.

1 Introduction

While machine learning often focuses on learning from a static dataset, one is often interested in acquiring data on the fly and adapting to the present context. This is known throughout the literature as 'continual learning' or 'lifelong learning' [Wan+23; CL18; Par+19; TM95].

One challenge of continual learning is 'catastrophic forgetting' [Had+20; VT19; Kem+18]: A model may learn useful information in context A but then lose much of this knowledge when adapting to context B . If context A is not representative of future contexts, then this is a correct adjustment for distribution shift. However, if contexts similar to A arise repeatedly, this may be undesirable..

Building on prior work, we aim to better understand the catastrophic forgetting phenomenon when faced with a series of linear regression problems that are revisited in cyclic order.

In machine learning, many data sets display cyclic or periodic patterns. These patterns often arise from factors such as the 'day of the week effect' [QR22] observed in financial or search engine data [Yan+22]. Additionally, the methods of cyclic alternating projections by Von Neumann [Von49] and Halperin [Hal62] are well-studied methods for solving linear systems [Kar37]. Equivalently, one can think of this as studying residual bounds for cyclic block Kaczmarz-type algorithms. Our work can be thought of as studying the worst-case forgetting of these popular methods. [Evr+22] mentions this perspective, but leaves open the problem of obtaining tight convergence bounds. In particular, the dimension dependence on the ambient data in the bounds of [Evr+22] could be problematic since high-dimensional data is so ubiquitous in machine learning. In contrast, our new bound for the worst-case forgetting does not scale at all with the dimension of the ambient data.

1.1 Problem Setup

The learning algorithm we consider fits a linear map to a set of data points of the form ( ⃗ x, y ) with ⃗ x ∈ R d , y ∈ R .

Our data is split into 'tasks', which are lists of elements from our dataset (with repeats allowed). These tasks are fed to our learning algorithm one at a time. Our algorithm updates its prediction after being given each task.

We assemble the r t inputs of the t th task into a matrix X t ∈ R r t ,d , where each row is an input vector ⃗ x T , and their corresponding outputs into a column vector ⃗ y t ∈ R r t . We aim to to find a column vector ⃗ w ∈ R d such that X t ⃗ w = ⃗ y t for all t .

The learning algorithm we consider applies a block Kaczmarz update at each step. Explicitly, we initialize ⃗ w 0 ∈ R d anywhere, and for t ∈ Z ≥ 1 we set ⃗ w t to be the projection of ⃗ w t -1 onto the solution space P t for the t th regression problem. That is,

where + denotes the Moore-Penrose pseudoinverse.

Geometrically, the set of vectors ⃗ w ∈ R d such that the map R d → R : ⃗ x ↦→ ⃗ x T ⃗ w agrees with all the data in task t is an affine subspace P t . This algorithm obtains w t +1 by projecting w t onto P t +1 . The set of vectors in R d satisfying all tasks is P := ⋂ t P t . The algorithm never leaves the maximal affine hyperplane through ⃗ w 0 perpendicular to P , so if the w t converge to a solution then it converges to the solution closest to w 0 ..

It is well-known that running gradient descent to interpolation with quadratic loss on task t starting from w t produces w t +1 .

Typically, projecting onto the solution space for task t moves ⃗ w i -1 off the solution space for task t -1 . We wish to understand the amount of information 'forgotten' over the course of training. Following [Evr+22], we define the forgetting for a sequence of tasks and initialization S at iteration n as

That is, F S ( n ) is the average loss of w n on all previous tasks (including task n ), where each task is weighted equally regardless of how many data points it has. As mentioned in the introduction, large forgetting is not always undesirable, but 'forgetting' in humans is not always undesirable either, so the name 'forgetting' not imply an inappropriately negative connotation.

In this work, we focus on cyclic task orderings. That is, we have T tasks ( X 1 , ⃗ y 1 ) , . . . , ( X T , ⃗ y T ) we visit cyclically, i.e., X t = X t mod T +1 and ⃗ y t = ⃗ y t mod T +1 for all t . Then the sequence of iterates converges to a cycle of length T in the sense that, for any t ∈ Z ≥ 0 , w t , w t + T , w t +2 T , . . . converges, either linearly or after a finite number of iterations; We justify this in the supplementary material.

In particular, if there is at least one simultaneous solution to all tasks, then the limit must be a solution; We justify a generalization of this in the supplmentary material, so we get linear convergence to a solution which implies linear convergence of forgetting.

However, this does not guarantee that, for any collection of T tasks in m cycles, the forgetting can be bounded above by O ( c -m ) for some constant c , even after normalizing the tasks in the way we describe later: The choice of tasks that give the largest forgetting for one value of m may not be the tasks that give the largest forgetting for another value of m . Here we ask whether we can obtain a worst-case bound on F S ( mT ) that only depends on the number of tasks T and the number of cycles m .

We focus on bounding the forgetting after a whole number of cycles; The forgetting in the middle of a cycle is a weighted average of the losses at each task, and therefore can be treated in the same way as getting a bound without normalizing the tasks. For a quick bound, the weights are close to 1, so we may rescale the tasks to get all coefficients to be 1 or less without increasing our bound much. The forgetting for T tasks after m cycles is

as each of the T tasks is visited equally often.

This can be made arbitrarily large by scaling the data points, so we describe normalizations in the next section. Given these normalizations, [Evr+22] showed that such a worst-case bound is possible, but the bounds they provided are not tight. Here, we improve on the forgetting bounds from [Evr+22] and, as a byproduct, show the union of the numerical ranges of all products of T orthogonal (real or complex) projections forms a sinusoidal spiral.

1.1.1 Normalizations

Following [Evr+22], we restrict to the setting where P is nonempty (i.e., the union of all tasks is consistent with a single linear function) and let ⃗ w ∗ ∈ P . So every data point ( ⃗ x, y ) satisfies ⃗ x T ⃗ w ∗ = y , and the planes P i we project onto all pass through ⃗ w ∗ .

Unlike [Evr+22], we translate the setup so w ∗ = 0 . This modifies each data point ( ⃗ x, y ) by setting y = 0 but preserving ⃗ x . This translates the P i to pass through 0, so they are subspaces, and P i is the kernel of X i , and the loss of task t at ⃗ w is now ∥ X t ⃗ w ∥ 2 , turning forgetting into.

where the product is the categorical product. Thus, for a fixed collection of tasks X 1 , X 2 , . . . , X n , abusing notation to let P t denote the projection onto the plane P t , the largest forgetting over all initializations ⃗ w 0 is

Rescaling ⃗ w 0 rescales all iterates, and hence the forgetting, by the same amount. So we normalize ∥ ⃗ w 0 ∥ = 1 . Similarly, rescaling all tasks X t by the same scalar scales each term in the forgetting by that factor, so we normalize so that max t ∥ X t ∥ = 1 .

As ∥ ⃗ w t ∥ decreases in t , this implies the loss of ⃗ w n at task t is ∥ X t ⃗ w n ∥ 2 ≤ 1 , and hence the forgetting is at most 1.

In the cyclic setting, for fixed T and m , these normalizations make the domain of optimization (the space of T -tuples of tasks with max t ∥ X t ∥ ≤ 1 and initializations ⃗ w 0 ∈ S d -1 ) compact, so forgetting attains a maximum, which is what we bound in this paper.

In the supplementary, we describe a tighter normalization that gives the same forgetting bounds.

1.2 Observations

In this section, we describe some observations that help us get our bound.

As noted in the previous section, using our normalization, there is a trivial bound on the forgetting of 1.

The data matrices X t only affect the iterates ⃗ w i via their rowspan (i.e., the span of the data points in the task), so we may replace X i by any other data matrix with the same rowspan, or equivalently the same kernel, without affecting the iterates subject to our normalization constraint. As ∥ X i ∥ ≤ 1 , it is a contraction, so the norm of the image of any ⃗ w is at most its distance to the kernel, and this is realized when X i is a projection away from its kernel. Therefore, the worst-case bound is not affected by restricting X i to be orthogonal projections with kernel P i .

With this restriction, for any ⃗ w , ∥ X i ⃗ w ∥ is the distance from ⃗ w to P i . Therefore, the forgetting can be interpreted as the average distance to each P i over all previous tasks (weighted equally).

1.3 Prior Work

Bounds on forgetting. [Evr+22] considered forgetting bounds both for random task orderings and cyclic task orderings. However their bound for cyclic task orderings is not tight. After m cycles of T

tasks, they show an upper bound of min { T 2 √ mT , T 2 d 2 mT } on the forgetting, leaving a gap with their lower bound of Ω( T 2 mT )

A key step of their proof is bounding ∥ A m ⃗ u ∥ 2 -∥ ∥ A m +1 ⃗ u ∥ ∥ 2 where A is a product of T projections and u is a unit vector. Their dimension-independent bound gives a T 2 / √ mT dependence on the forgetting after m cycles. Their other bound improves the m dependence, but only uses that A is a contraction. This forces a linear dependence on the dimension d , as this is possible if the supremum is taken over all contractions. Indeed, for any m a multiple of d , one can have

by choosing A to be an operator that maps

and taking ⃗ u = ⃗ e 1 , where ⃗ e i denotes the i th standard unit vector.

[RZ23] bound ∥ ∥ A m ⃗ u -A m +1 ⃗ u ∥ ∥ 2 for A a product of projections. However they are primarily interested in the asymptotics with respect to m . They supply two proofs which rely on either the bound in [BS16] or in [Cro08], and hence do not avoid the exponential dependence in T .

[Lin+23] study forgetting from a different perspective. They assume Gaussian i.i.d. tasks ( called features in their paper) at each iteration and explicitly compute the expected forgetting and generalization error. We analyze the worst-case forgetting without any assumptions on the distribution of tasks, so our work is quite different from [Lin+23].

As mentioned in [Evr+22], the iterate w t can be equivalently obtained by running the (block) Kaczmarz method [NT14] with the cyclic row selection rule for solving a consistent system y t = X t w, ∀ t ∈ [ T ] . However, most id=MK]of the convergence analyses of Kaczmarz-type methods [Elf80; NT14; SV09] bound the distance from the least-norm solution, whereas in our setting we interested in the residual errors. Moreover such analyses depend on the condition number of the system, so are unlikely to adapt to our setting, since our bounds hold for all choices of tasks (i.e. blocks), with no condition number dependence.

Bounds on the numerical range. Lemma 5.1 in [Cro08] gives a bound on the numerical range of any linear transformation A expressible as a product of T projections. This, combined with other results in that same work, yields a bound on ∥ ∥ A m ⃗ u -A m +1 ⃗ u ∥ ∥ 2 . When combined with the results of [Evr+22], the result is a bound of the form O ( f ( T ) /m ) on the forgetting. However, the function f ( T ) turns out to be exponential when applying the results of [Cro08] directly. This is due to Lemma 5.1 of [Cro08] being loose. There they show that the numerical range of A is contained in a region of C with a boundary point at one where the tangent lines at 1 have exponential slope. In this paper, we give the smallest region that is guaranteed to contain the numerical range of A given only T .

[BS16] also gives a bound on the numerical range of a product of projections. Specifically they show that the numerical range is contained in a Stoltz domain with the angle at 1 depending on a quantity called the Friedrich's number of the (subspaces associated to the) projections. They then combine their bound with that of [Cro08] to obtain somewhat tighter control over the numerical range. However the Friedrich's number for T subspaces can be made arbitrarily close to 1 so for our purposes their bound will not yield better results than the bounds in [Cro08].

1.4 Results

We give a nearly optimal bound on the forgetting for cyclic task orderings, removing the dimension dependence of [Evr+22]and avoiding the exponential dependence on T that would arise from applying results in [Cro07].

Theorem 1.

where S ranges over all (task, initialization) pairs where the tasks are given by matrices X 1 , X 2 , . . . , X T with max t ∥ X t ∥ ≤ 1 and ⃗ w 0 ∈ R d with | ⃗ w 0 | = 1 .

To prove this, [Evr+22] show that, if A denotes the linear map given by one cycle, letting ∆ m ( ⃗ u ) := ∥ A m ⃗ u ∥ - ∥ A m +1 ⃗ u ∥ ,

We provide a geometric interpretation of their proof in the supplementary material. [CP17] showed that there exists a constant Q ≤ 1 + √ 2 such that

for all polynomials p . Our approach is to determine ⋃ A W ( A ) where A ranges over all products of k complex orthogonal projections. As this includes the class of products of k real orthogonal projections, combining these inequalities gives our bound.

We will show in Section 3 that the problem of bounding the numerical range of the product of k -1 complex orthogonal projections is equivalent to understanding the range of the product

where the ⃗ v i range over complex vectors in C d of norm 1 . We give a complete description of the range of P as the filled sinusoidal spiral

This shape has a simple geometric description. Let P k be the convex hull of the k th roots of unity. Then the sinusoidal spiral above is the image of P k under the map z ↦→ z k .

The main challenge in computing P 's range Γ k is in computing its boundary since a topological argument (given in the supplementary material) shows that Γ k is simply connected. To compute the boundary, we first observe that it suffices to consider input vectors in C 2 . To see this, P maps a sequence of vectors to the cyclic product of pairwise inner products, so if any of the input vectors is not coplanar with its neighbors, then projecting it onto the plane spanned by its neighbors and scaling it to have unit norm increases the magnitude of the corresponding factors. So it suffices to consider the case when all vectors are coplanar, which is equivalent to considering input vectors in C 2 . In this case, P is a smooth map from the (real) manifold ( S 1 ) n ⊂ C n to the (real) manifold C , so any input that gets sent to a boundary point of Γ k has singular Jacobian. That is, the directional derivative in all directions tangent to the domain must be parallel. It turns out that these algebraic conditions can be manipulated to characterize all critical points and critical values of P , and the computation is made simpler by using quaternions.

As a side effect of our proof, we also fully describe the tuples ⃗ v 0 , . . . , ⃗ v k -1 ) that map to boundary points of the image of P and show that they naturally correspond (up to multiplying each ⃗ v t by a complex unit ϕ t ) to certain quaternionic roots of unity.

Using this result we obtain the following bound on the increments between consecutive powers of A when A is a product of projections. We believe this result may be independently useful.

Lemma 2. For any linear map A : C n → C n expressible as a product of k complex projections,

where C is an absolute constant which can be taken to be 0 . 4 .

This lemma, with the inequality of [Evr+22], immediately implies Theorem 1.

In particular, for fixed T , we obtain an asymptotically optimal bound in terms of iteration count m , independent of dimension . In contrast, the dimension-independent bound given in [Evr+22] was sub-optimal by a factor of √ m .

Finally, . So our results can be interpreted as giving a bound for this setting as well.

variable name

element of

a, b, z

C

⃗ v,

⃗ w

C

2

p, q

H

.

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

1.0

0.5

0.0

0.5

10

1.5 Notation for proof

We use C and H to denote the complex numbers and the quaternions. While the previous sections considered real vector spaces, our proof works by obtaining a bound over all complex projections. So for sections 2 and 3, all vector spaces will be over C . In these sections, we use the following guidelines for variable names:

Our complex inner products are always conjugate linear in the first argument, so ⟨ ⃗ v, ⃗ w ⟩ = ⃗ v ∗ ⃗ w where v ∗ denotes the conjugate transpose of ⃗ v . The argument of a complex number is taken to be in ( -π, π ] . The numerical range of a matrix A ∈ C d × d is defined by

2 Bounding the range of P

In this section, we fully describe the range of P given by (2) and obtain a full description of the inputs that map to boundary points. We will use the following notation throughout the next argument.

The following identity is a straightforward consequence of the definition.

Our first theorem shows that the range of P is a filled sinusoidal spiral. This result is of mathematical interest on its own, and will also be applied to cyclic forgetting bounds in the next section.

We begin with a lemma characterizing the critical points of P treated as a smooth map R 2 k → R 2 . We defer the proof to the supplementary material.

Lemma 5. For any critical value c ∈ C of P , there exists a sequence of vectors ( a 0 , b 0 ) = (1 , 0) , ( a 1 , b 1 ) , . . . , ( a k -1 , b k -1 ) such that P (( a m , b m ) k -1 m =0 ) = c . Furthermore, there exist α, β ∈ C such that

⟨ ( a m , b m ) , ( a m +1 , b m +1 ) ⟩ = α for all m

(where indices are treated mod k ). Furthermore, any sequence (( a m , b m ) k -1 m =0 ) attaining P ((( a m , b m ) k -1 m =0 )) = c can be obtained by taking a sequence satisfying the above conditions and multiplying each vector by a complex unit.

Given this characterization of P 's critical points, our next result computes the boundary points of P 's image, from which we obtain the entire image of P as a consequence.

Theorem 6. Let P ( ⃗ v 0 , . . . , ⃗ v k -1 ) = ⟨ ⃗ v 0 , ⃗ v 1 ⟩ ⟨ ⃗ v 1 , ⃗ v 2 ⟩ . . . ⟨ ⃗ v k -2 , ⃗ v k -1 ⟩ ⟨ ⃗ v k -1 , ⃗ v 0 ⟩ and let Γ k be the range of P as each ⃗ v i ranges over unit vectors in C d . Then

which is a filled sinusoidal spiral.

Proof. We show this result for vectors in C 2 . In the supplementary material, we show that increasing the dimension does not affect the range, so considering vectors in C 2 is sufficient. The result is trivial for k ≤ 2 , so we will assume k ≥ 3 .

The domain of P is compact, so Γ k is also compact. P is also smooth (as a function R 4 → R 2 ), so its boundary consists of critical values. We will give a complete description of the critical values and critical points and show that the outermost critical values form a sinusoidal spiral, which is enough to show that Γ k is contained in the claimed region. We defer showing that Γ k contains the entire filled spiral to the supplementary material.

Fix a critical value c of P and let a i , b i be as in Lemma 5 above. As ( a m , b m ) and ( a m , b m ) ⊥ are orthogonal, they form a basis for C 2 , so letting α = a 0 , β = b 0 , we get a recurrence relation

Representing each vector ( a, b ) ∈ C 2 as a quaternion via ( a, b ) ↦→ a + bj , it follows from direct computation that ( a, b ) ⊥ ↦→ j ( a + bj ) and, for ϕ ∈ C , ϕ ( a, b ) ↦→ ϕ ( a + bj ) = ( a + bj ) ϕ . In particular, multiplication on the left by a complex number is unambiguous.

Our recurrence relation can be expressed as

In other words, the vectors must be the partial products

for m = 0 , . . . , k -1 (which is the empty product of m = 0 ), where α, β satisfy the equation

We now show that any sequence of vectors ( a 0 , b 0 ) , . . . , ( a k -1 , b k -1 ) in a solution to the above system of equations can be obtained by slightly modifying a solution in the case ω = 1 (possibly with a different value of k ).

Let ζ k be a (not necessarily primitive) complex k th root of unity. Consider the sequence

This also satisfies the two conditions in Lemma 5, so we can run the same logic as above. The corresponding change in the resulting system is that this multiplies α, β by ζ k and ω by ζ 2 k , so by multiplying each vector by these complex units we may freely modify ω by ζ 2 k for any k th root of unity ζ k , or equivalently any power of e 2 πi/k .

If k is odd, this is enough to show that any nonzero critical value can be obtained by ω = 1 , and furthermore that any critical point can be expressed as ( a 0 , b 0 ) , ( a 1 ζ k , b 1 ζ k ) , ( a 2 ζ 2 k , b 2 ζ 2 k ) , . . . , ( a k -1 ζ k -1 k , b k -1 ζ k -1 k ) for some k th root of unity ζ k .

If k is even, let ζ 2 k be a 2 k th root of unity. Consider the sequence

This satisfies the conditions in Lemma 5 for 2 k in place of k . The corresponding values of α, β in the resulting system are the same as in the original multiplied by ζ 2 k , and the resulting value of ω is the same as the original multiplied by ζ 2 2 k . As ζ 2 2 k can be chosen to be ω -1 , we can reduce to the case

where ω = 1 , or equivalently

Remark 7. These reductions can also be interpreted using quaternions, which we demonstrate in the supplementary material.

For any integer m , the m th roots of unity in H are precisely those numbers whose real parts are the same as those of the real parts of the m th roots of unity in C . So ( α + βj ) m = 1 if and only if α lies on one of the vertical diagonals of the m -gon whose vertices are the m th roots of unity (where the vertical diagonals are those with constant real part, including sides of the n -gon).

Therefore, the values of α in the phase-shifted sequences are vertical diagonals in the regular k -gon if k is odd and every other vertical diagonal in the regular 2 k -gon if k is even, where the vertices are the corresponding roots of unity. Inverting the phase shift gives that, regardless of whether k is even or odd, the values of α in the solutions to

form the diagonals (including the sides) of the k -gon whose vertices are the k th roots of unity.

As the inner product of consecutive vectors at such a sequence is α , the value of P at such a sequence is α k . Therefore the critical values of f are then k th powers of these diagonals. Furthermore, when k is odd the critical points are sequences of the form ( a m , b m ) = ϕ m ( α + βj ) m where α + βj is a quaternionic k th root of unity and ϕ m are units in C ; When k is even, the critical points are sequences of the form ( a m , b m ) = ϕ m ( α + βj ) m where α + βj are quaternionic 2 k th roots of unity and ϕ m are units in C ,

By the maximum principle for holomorphic maps, the outer boundary of Γ k is obtained by taking the k th powers of the sides (where each side maps to the same curve). Using polar coordinates, the resulting curve is the sinusoidal spiral cos( π k ) r -1 /k = cos( θ -π k ) .

The same proof shows that all other critical points other than those that P sends to 0 arise from the other diagonals of the polygons and their corresponding critical values are their k th powers, which are also sinusoidal spirals. A computation shows that a sequence of vectors that P sends to 0 is a critical point if and only if at least two of the inner products are 0.

3 Application to cyclic forgetting bounds

As an immediate application of Theorem 6, we can control the numerical range of a product of k orthogonal projections.

Corollary 8. Let A = P k P k -1 . . . P 1 be a product of k orthogonal complex projections C d → C d . Then W ( A ) ⊆ Γ k +1 .

Proof. We first observe a general fact. Let P : C d → C d be an orthogonal projection and let ⃗ z ∈ C d . Let ⃗ q = P⃗ z ∥ P⃗ z ∥ . Then

Now let ⃗ z be an arbitrary unit vector and let

Iterating the fact stated above, for all i ∈ [ k ] . Hence

In the supplementary material, we use our characterization of P -1 ( ∂ Γ k +1 ) to characterize the pairs ( A,⃗ z ) such that ⃗ z ∗ A⃗ z ∈ ∂ Γ k +1 .

We now bound ∥ A m ( I -A ) ∥ uniformly over all linear maps A : C n → C n that can be expressed as a product of k complex projections by applying the above bound on the numerical range into the result of [CP17] that there exists a constant Q ≤ 1 + √ 2 such that

for all polynomials p . We defer the calculation to the supplementary material.

Lemma 9.

A more precise version of this bound in which 4 eπ 2 can be improved in terms of k m will be given in the supplementary material.

This, combined with Crouzeix's inequality, gives Lemma 2, which we restate below more precisely, and hence Theorem 1.

Lemma 2. For any linear map A : C n → C n expressible as a product of k complex projections,

where Q is the constant of Crouzeix's inequality as presented above.

4 Real projections

In the application to forgetting, all relevant projections are real projections. Therefore one may ask whether our bound can be improved by restricting to products of k real projections.

As we show in the supplementary material, the union of the numerical ranges is the same even if we restrict to real orthogonal projections R 4 → R 4 , and it is possible to get the maximal asymptotic decay rate on ∥ ∥ A m -A m +1 ∥ ∥ , but any real or complex sequence of orthogonal projections attaining a point on ∂ Γ k in its numerical range can be orthogonally decomposed into invariant (under all projections) subspaces U ⊕ V where ∥ A m ⃗ u ∥ -∥ ∥ A m +1 ⃗ u ∥ ∥ decays quickly on U . In particular, any indecomposable such sequence must have quickly decaying forgetting.

5 Future Work

It is not clear that this yields a completely optimal bound for forgetting. Is it possible to improve O ( T 2 /m ) to O ( T/m ) matching the lower bound of [Evr+22]?

Finally one might wonder how rare the worst-case bound is. For example if one visits the T tasks cyclically, but in a random order is it possible to beat the O ( T 2 /m ) bound, perhaps with some weak dependence on the ambient dimension?

6 Acknowledgment

The authors would like to thank Itay Evron for interesting discussions about [Evr+22].

References

Peer reviews:
comments: SUMMARY:
Authors provide theoretical bounds on the forgetting quantity in the continual learning setting for linear tasks, where each round of learning corresponds to projecting onto a linear subspace.
For a cyclic task ordering on $T$ tasks and an arbitrary iteration $m$, they prove the upper bound of $O(T^3/m) $ on the forgetting. For the proof they use a bound on the numerical range of a product of T projections.

STRENGTHS:
-near optimal bounds are proven for the worst case for cyclic task ordering(also improving the bounds for less general settings if I understood the related work section correctly): their upper bound $O(T^2/m)$ compared to a $\Omega(T/m)$ lower bound and the previous known upper bounds $O(T^2/\sqrt{mT})$.

-also provides a bound on the numerical range of a product of $T$ projections

WEAKNESSES:
(Common) notations could be defined earlier or there could be a reference to the noation section when first notaions are used in the introduction.

There is still a gap of T if I understood everything correctly.

As I am unfamiliar with some pieces of related work and Math/other details were not carefully checked so it is hard to rate the novelty.

QUESTIONS:
What is the differnce between the $T^3/m$ bound in the abstract and the $T^2/m$ bound presented in the paper?

LIMITATIONS:
As they mention in the conclusion there are some gaps for real projections and  for forgetting, all relevant projections are real projections.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the setting of continual learning for linear tasks, and in particular the phenomena of catastrophic forgetting. \nThey prove the best known upper bound on forgetting for the setting of cyclic tasks.\n\nSTRENGTHS:\nThe paper demonstrates an upper bound on Forgetting that is independent of the dimension.\nThis is a strong improvement for settings with very large dimension spaces.\n\nWEAKNESSES:\nI think the main weakness is the significance of contribution, compared to previous work. \nThis result build on recent work by [Evron et al.22] and demonstrates improvement in a certain setting (cyclic).\nThe paper would be strengthened if either the results would be extended to other setting, and incorporated experimental evaluations - both of which similarly to [Evron et al.22].\n\nAdditional comments:\n\n1. Not clear why the paper use the term “near-optimal” : \n- The known lower bound from [Evron et al.22] at T^2/m. The bound given in this work has a worse dependence on T compared to [Evron et al.22]. The improvement comes for large dimensions, which is indeed important but not necessarily near-optimal ? \n\n2. Forgetting vs. Regret:    by Evron et al., in the cyclic setting both quantities of Forgetting and Regret will go to 0 as m increases. Since regret is a well studied quantity, it is interesting to compare these and examine how will the convergance rate behave ? a connection between them may allow to build on the vast literature on regret. \n  \n\nMinor comments: \n- Notation: I assume you mean $X_i \\in \\mathbb{C}^{r_i \\times d}$. Maybe say r_i refers to samples each with d dimensions. \n- Line 92 : “We give nearly a nearly optimal bound”\n- Citations do not indicate the names of all authors, incorrect format\n- Lines 52-53 - why write T^2/*mT instead of T/m ?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nnone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes bounds for cyclic forgetting when an overparametrized linear model is fit successively to a series of tasks. The exact same setting has been studied fairly recently before in the , and the main contribution here is to improve dimension dependence of the bounds.\n\nSTRENGTHS:\nThe theory of Theorem 5 is clean and beautiful, and certainly of mathematical interest in its lack of dependence on ambient dimension. The core idea is creative and the writing is good.\n\nLines 56-62 of the text are valuable for intuition as to why the suboptimal dimension dependence arises, and how to fix it. This is a very interesting insight which could potentially be exploited for more data-instance-dependent bounds in the future.\n\nWEAKNESSES:\nCatastrophic forgetting arises especially when there are certain types of dependencies/correlations between the tasks. This entire manuscript is about studying that situation theoretically, but it completely neglects to provide. The manuscript relies on the publication [Evr+22] to set precedent for the problems studied. However, that paper was pathbreaking in setting up the problem and establishing that cyclic task orderings don't suffer catastrophic forgetting, and included much more interpretation and exposition linking the theory result to the motivation for studying the problem. The current manuscript does not give any of this. Instead, the main result obviates all dependencies on the tasks other than their number, so it does not lend much further insight into the problem.\n\nThe proofs are far too long and technical for the main body of the paper - they should be moved to the appendices. The entire discussion leading up to Theorem 5 is full of mathematical beauty arising from symmetry - but none of this is in the service of the task of learning from different tasks, only using the generic fact that the tasks are projections.\n\nQUESTIONS:\n- Please make a pass for key typos. The abstract says T^3 instead of T^2; lines 52-53 shouldn't have T in the denominator anywhere; and many more. \n- Is there scope to extend these results to parametrize dependencies between tasks?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConsider an overparametrized system solving a periodic sequence of tasks in linear (least squares) regression in the following way: starting from a weight vector $w_t$ for task $t$, perform gradient descent  to solve task $t+1$, which, due to overparametrization, will eventually return an exact solution $w_{t+1}$ for task $t+1$. Here $w_0 = 0$ and each task is specified by a set of input vectors $X$ and output labels $y$.\n\nThe procedure amounts to projecting $w_t$ onto the solution (hyper-)space of task $t+1$ and implies a loss of information about task $t$ and all previous tasks. This raises the question just how much is forgotten after n tasks have been visited, which can be quantified as the error of $w_n$ averaged over all previous datasets. While this \"forgetting\" depends on the data, it is still possible to give a worst case bound for the worst possible periodic task sequence with period $T$.\n\nPrevious work gives a bound of $T^2/\\sqrt{nT}$ and a dimension dependent bound, as well as a lower bound of order $T^2/n$, the paper at hand gives order $T^3/n$, which has optimal dependence on n, though not on T.\n\nThis result is obtained by a reduction of the problem to a bound on the numerical range of a polynomial in $T$ projection operators, which is obtained by an intricate analysis, which (because of time constraints) I didn't completely verify, although I didn't find any flaws.\n\nSTRENGTHS:\nThe paper approaches an interesting problem with an equally interesting mathematical technique. The given bound is a relevant improvement and to me seems an important contribution to the subject of continual learning.\n\nThe problem is clearly stated, the minor limitations of the main result are nicely exposed already in the title by the word \"nearly\".\n\nWEAKNESSES:\nThe paper is technically very heavy in the proofs of Lemma 4 and Theorem 5. If possible the authors might sketch the basic ideas of the proof of Theorem 5 in a short paragraph. \n\nThe statement of Lemma 4 is somewhat opaque and its relevance to the problem is not immediately clear. It doesn't help that $\\Gamma_k$ shows up in the proof of Lemma 4 (l154), while its definition comes on the next page in the statement of Theorem 5. \n\nThe paper leans strongly on the reference [Evr+22], without which it is very difficult to understand. At least Theorem 11 in [Evr+22] could be reproduced.\n\nQUESTIONS:\nThe various bounds are sometimes expressed in terms of the number of iterations and sometimes in terms of the number of cycles. This may cause some confusion. Would it be possible to unify this?\n\nThe $m$ in l 43 and in (1) seems to be a typo.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}