Question: Bt and Γt appears rather abuptly and  can you explain them?
Evidence: Unlike the conventional smooth nonconvex finite-sum problem where each mini-batch is iid-sampled from the dataset P ∪ C, the replay memory based continual learning encounters a non-iid stream of data C and has access to a small sized memory M . Algorithm 1 presents the pseudocode of NCCL by Equation. By the limited access to P , the gradient update for f (x) in Equation is a biased estimate of the gradient ∇f (x t ). Specifically, at the timestep t,
where e t and e Mt denote the error term ∇f It (x t ) − ∇f (x t ) and the expectation value over I t given M t , respectively. We note that the given replay memory M t with small size at timestep t induces the inevitable overfitting bias. We first state an intermediate result for a single gradient update of NCCL.
For ease of exposition, we define the overfitting term B t and the catastrophic forgetting term Γ t as follows:
B t = (Lα 2 Ht − α Ht ) ∇f (x t ), e t + β Ht ∇g Jt (x t ), e t ,
where L is a constant for Lipschitz smoothness. Under Assumption 1, a single gradient update by Equation 7 satisfies the following bound by letting x ← x t+1 and y ← x t :
This reveals the basic qualitative difference between the conventional nonconvex SGD and NCCL in the convergence rate. Compared to the nonconvex SGD, there exist two terms B t and Γ t in Equation. We group the terms containing e t as B t and the other terms as Γ t . We note that the catastrophic forgetting term Γ t has ∇f It (x t ), ∇g Jt (x t ) , which is the key aspect of interference and transer, and B t includes the error term between the batch of M and the entire dataset P . Then, we can quantify the amount of overfitting by tracing B t .
To compute the expectation over the stochasticity of NCCL, we derive the expectation of ∇f Mt (x t ) over the sampling rule of M t . The episodic memory M t = M 0 for all t is uniformly sampled once from the random sequence of P , and ER-reservoir iteratively samples the replay memory M t by the selection probability
If M 0 is uniformly sampled from P , then both episodic memory and ER-reservoir satisfies
We provide the detailed proof in Appendix B. Note that taking expectation iteratively with respect to the history M [0:t] is needed to compute the expected value of gradients for M t . Since taking the expectation over the stochasticity of NCCL implies the total expectation
Algorithm 1 Nonconvex Continual Learning (NCCL)
Compute learning rates α Ht and β Ht with ∇f It (x t ) and ∇g Jt (x t )
Store J t into M t+1 by the rule of replay memory scheme end for
is also 0. We note that considering a random choice of M 0 allows us to analyze the convergence on f (x), even if M 0 cannot access the all datapoints in P to compute ∇f (x t ) 2 . Our first main result is the following lemma that provides the stepwise change of upper bound. Lemma 2. Suppose that Assumption 1 holds and 0<α Ht ≤ 2 L . Then for x t updated by Algorithm 1, we have the following bound
where the effect of B t vanishes by
The proof is presented in Appendix B. Surprisingly, taking the expectation over M 0 ⊂ P in Equation to handle the stochasticity of choosing M 0 allows us to analyze the convergence of f (x) in Section 3.3. We also note that the individual trial with a randomly given M 0 cannot cancel the effect by B t , although its total expectation over the whole possible trials E[B t ] is zero. More specifically, the worst case sup M0 ∇f (x t ) 2 definitely contains the non-zero value of B t in the upper bound term. We discuss the more details of the overfitting to memory in Appendix D.
This reveals the basic qualitative difference between the conventional nonconvex SGD and NCCL in the convergence rate. Compared to the nonconvex SGD, there exist two terms Bt and Γt in Equation 8. We group the terms containing et as Bt and the other terms as Γt. We note that the catastrophic forgetting term Γt has h∇fIt (x t ), ∇gJt (x t )i, which is the key aspect of interference and transer (Riemer et al., 2018), and Bt includes the error term between the batch of M and the entire dataset P. Then, we can quantify the amount of overfitting by tracing Bt.
Answer: We added some explnation below the Equation 8 for readibility.