LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning

Authors: Liangzu Peng, Juan Elenter, Joshua Agterberg, Alejandro Ribeiro, Rene Vidal
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Continual Learning; Pretrained Models; Overparameterization; Generalization; Random Feature Models

Abstract:
The goal of continual learning (CL) is to train a model that can solve multiple tasks presented sequentially. Recent CL approaches have achieved strong performance by leveraging large pre-trained models that generalize well to downstream tasks. However, such methods lack theoretical guarantees, making them prone to unexpected failures. Conversely, principled CL approaches often fail to achieve competitive performance. In this work, we aim to bridge this gap between theory and practice by designing a simple CL method that is theoretically sound and highly performant. Specifically, we lift pre-trained features into a higher dimensional space and formulate an over-parametrized minimum-norm least-squares problem. We find that the lifted features are highly ill-conditioned, potentially leading to large training errors (numerical instability) and increased generalization errors. We address these challenges by continually truncating the singular value decomposition of the lifted features. Our approach, termed LoRanPAC, is stable with respect to the choice of hyperparameters, can handle hundreds of tasks, and outperforms state-of-the-art CL methods on multiple datasets. Importantly, our method satisfies a recurrence relation throughout its continual learning process, which allows us to prove it maintains small training and test errors by appropriately truncating a fraction of SVD factors. This results in a stable continual learning method with strong empirical performance and theoretical guarantees. Code available: \url{https://github.com/liangzu/loranpac}.

Decision: accept (poster)
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729694953339

thread_type:
official_review

<thread_object>
creator: reviewer_p9vj

<content>
summary:
The paper tackles the problem of class-incremental learning (CIL).
To this end, the paper improves RanPAC (McDonnell et al., 2023), making it more numerically stable as a least-squares problem, by truncating small singular values from the SVD of the (pretrained) embeddings of the samples from *all* previous tasks. The truncation might also positively impact the generalization itself.
The authors link their method to a theoretical framework (ICL by Peng et al., ICML 2023) which derived sufficient conditions to preventing forgetting in linear models, and use this connection to derive some bounds on the training and generalization error of the proposed algorithm.

strengths:
1. The proposed method seems to empirically improve upon the existing RanPAC method (McDonnell et al., NeurIPS 2023).
1. The proposed method is easy to implement.

weaknesses:
# Major concerns

1. **Proposed analysis feels contrived.**   
To me, the analytical connection between ICL (Peng et al., ICML 2023) and the proposed method feels overly strained.
    1. To begin with, the ICL formulations presented in Line 111 and 116, are purely theoretical (formalizing a sufficient condition to prevent forgetting), and *cannot* be considered continual learning approaches, since they require storing the data from all previous tasks.
    1. In Line 105 (Eq. 1), the raw feature matrices $X_t$ are randomly mapped to the "effective" matrices $H_t=\text{relu}(PX_t)$ using a random matrix $P\in\mathbb{R}^{E\times d}$ where $E\gg d$.
To facilitate the comparison to the ICL framework, in Line 113, the authors assume that $E$ is so large that it makes the *entire* task sequence feasible (jointly!). However, for most datasets, this would practically make the time and space requirements of the proposed method infeasible.
    1. Theorem 1 also implicitly requires such extreme overparameterization or that the data can already be explained linearly almost perfectly. Otherwise, $\Vert\mathcal{E}_{1:t}\Vert$ would be large, making the bound uninformative ($\frac{M_t-k_t}{M_t}$ is not small).  
However, if the data is linearly explained by some model $W^{*}_t$, then ICL trivially obtains 0 loss (since all previous data is stored and optimized simultaneously) and all that is left to analyze is the error stemming from the truncation (which implicitly assumes that the smaller singular values are very small, which might not be the case for arbitrary data).   
In my opinion, this is **not** a significant theoretical contribution.

1. **Unclear if accuracy improvements are significant.**  
 It seems like the results in the main Table 1 are of a single run per model-dataset combination. No standard deviation is reported. Given that most of the improvements upon RanPAC are somewhat small, it is unclear if these are indeed statistically significant.

---

## Summary of weaknesses

 Considering all of the above, I believe the paper constitutes a nice incremental improvement over RanPAC (using truncation instead of regularization), but I am not convinced that the analysis makes the paper valuable enough for ICLR.

-----

# Minor concerns

I do not expect any response on the following concerns but I recommend addressing them in any future revised version.

1. Many times (e.g., Line 160), the authors specify constant values like $\lambda = 10^{4}$ and $\lambda = 10^{-5}$  and refer to them as being very small or very large. However, I find these references problematic since the regularization strength in least-squares problems is meaningless without taking into account the scale of the data and the fitting error. Similarly, I would even go further and say that the main benefit from the proposed method is not that it gets rid of the smallest singular values, but that it remedies the *condition number* of the matrices. The smallest singular values can only be considered "small" when comparing to the largest singular values. An alternative method could have limit the condition number instead of using a fixed percentage $\zeta$.

1. I think Figure 1c would be better if it showed the training accuracy, since this is the origin of the problem there.

1. The time complexity analysis should appear immediately after the algorithm, rather than as a side note on page 9. Space complexity should also be specified. Moreover, at some point, given enough tasks, $M_t = k_{t-1} + m_t$ can become larger than $E$, which then makes the time complexity $E^3$ (notice that the $M_t - $E$ smallest singular values are zeros), just like in RanPAC.

1. In Line 130, the authors say "the theoretical justification for the strong empirical performance of RanPAC remains limited". I kindly disagree and claim that it is trivially justified, since according to the formula in Line 124, it just minimizes *all* the losses simultaneously.

1. To be slightly more precise, step 3 in Algorithm 1 should use the floor operator $\lfloor(1-\zeta)M_t\rfloor$.

1. The paper would be more readable if the dimensions are specified when matrices are first presented, e.g., the authors should specify the dimensions of ${\widetilde{U,\Sigma,B}_{1:t}}$ in Eq. (3).

1. The sentence in Line 319 (about the values of $\delta$) is unclear.

1. The message of Figure 6 is unclear.

1. In Theorem 1, I believe $\frac{M_t-k_t}{M_t}$ is simply $(1-\zeta)$.

1. In Theorem 1, specify the subscript for the norm used for $\mathcal{E}_{1:t}$.

1. The presentation is often overcomplicated and inconsistent. For instance, in line 105 (Eq. 1), a projection matrix $P$ is introduced, while most of the analysis throughout the paper, especially in Section 5, completely ignores that matrix and deals with arbitrary deterministic feature matrices $H_t$. Moreover, I would consider presenting the ICL details only after the algorithm itself, as a retrospect.

questions:
Other than what I wrote above,

1. I could not immediately understand why both RanPAC and the proposed method outperform the $LC(H_{1:T})$ benchmark (trained with the original cross entropy loss like explained in Appendix J). Even if the data matrices are ill-conditioned, early stopping (e.g., when performing a fixed number of steps) should create a remedying effect and help generalization. Is it possible that a better hyperparameter tuning is needed here? Did the authors check the training *and* test curves throughout the training of this competitor?

1. In Figure 4a, what is going on with the scaling of the x-axis? What was the maximal truncation percentage that you examined? I would expect seeing a sharp decrease at the for high values, but it is not apparent. Moreover, why can we truncate so much without losing performance even though the eigenvalues do not decrease so quickly (as seen from Figure 1a)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730442717316

thread_type:
official_review

<thread_object>
creator: reviewer_ivyo

<content>
summary:
This paper introduces a new algorithm, ICL-TSVD, designed to bridge the gap between theory and practice in continual learning (CL) with pre-trained models. It integrates an empirically strong approach RanPAC within the Ideal Continual Learner (ICL) framework. Specifically, ICL-TSVD lifts pre-trained features into a higher dimensional space, then addresses the ill-conditioning of these lifted features through continually SVD truncation, and solves the truncated form of an over-parameterized minimum-norm least-squares problem. Theoretically, it is shown that the method achieves small training and generalization error under certain conditions. Empirically, ICL-TSVD demonstrates stability across various hyperparameter settings and outperforms other CL baselines across multiple datasets.

strengths:
1. The paper clearly demonstrates the contribution of the proposed approach and is easy to follow.

2. Extensive experiments across multiple datasets show that ICL-TSVD remains stable regarding hyperparameter selection (including regularization parameter $\lambda$ and truncation percentage $\zeta$) and consistently outperforms previous CL baselines, particularly RanPAC.

3. It provides theoretical guarantees for the proposed method ICL-TSVD by revealing a recurrence relation throughout the CL process, showing that it has small estimation and generalization errors when a suitable fraction of SVD factors are truncated.

weaknesses:
1. The continual Implementation of $\overline{\bf{W}}_t$ involves the computation of $\widetilde{\bf{B}}_t$, which has the form of $\widetilde{\bf{B}}_t = [\widetilde{\bf{U}}\_{1:t-1}\widetilde{\bf{\Sigma}}\_{1:t-1}, \widetilde{\bf{H}}_t]$ when $t\geq 2$ according to Eq. (3). Appendix C.1 briefly mentions that it is empirically found continually updating all SVD factors $\widetilde{\bf{U}}\_{1:t}, \widetilde{\bf{\Sigma}}\_{1:t}$ and $\widetilde{\bf{V}}\_{1:t}$ lead to large test errors. However, it remains unclear why authors use $\widetilde{\bf{B}}_t = [\widetilde{\bf{U}}\_{1:t-1}\widetilde{\bf{\Sigma}}\_{1:t-1}, \widetilde{\bf{H}}_t]$ instead of $\widetilde{\bf{B}}_t = [\widetilde{\bf{U}}\_{1:t-1}\widetilde{\bf{\Sigma}}\_{1:t-1}\widetilde{\bf{V}}\_{1:t-1}, \widetilde{\bf{H}}_t]$ from a theoretical perspective. What are the difference and connection between these two implementations? Why just maintaining $\widetilde{\bf{U}}\_{1:t-1}$ and $\widetilde{\bf{\Sigma}}\_{1:t-1}$ can serve as a substitution for all SVD factors, what is the motivation for that?

2. In Section 6.2,  ICL-TSVD chooses embedding dimension $E = 10^5$ while RanPAC uses the default choice $E=10^4$. It is not so convincing that this is a fair comparison although the authors claim that ICL-TSVD’s implementation is more scalable and more efficient than RanPAC’s. Figure 8 and Table 13 in Appendix K indeed show that ICL-TSVD does not outperform RanPAC on about half of the datasets.

3. 
  - For Theorem 1, take Figures 3 and 4 for example, the truncation percentage is set to 25%. Then based on eigenvalues vs. eigenvalue index shown in Figure 1, $a_t$ should be at least in the order of $10^5 \sim 10^6$, and $\gamma_t$ is around or slightly larger than 1. In this case, the bound for the estimation error can be very large even if the unknown ground-truth $\\|\mathbf{W}_t^*\\|_F^2$ and noise $\\|\mathcal{E}\_{1:t}\\|^2$ are reasonably small, which means that the theoretical guarantee does not match with empirical results. And similar issue exists in Theorem 2 as well. 
  - Furthermore, without independent i.i.d. Gaussian vectors assumption on $\bf{h}$ and the columns of $\bf{H}\_{1:t}$ (which is unrealistic for practice), we may not guarantee that $\\|\Lambda - \frac{1}{M_t}\bf{H}\_{1:t}\bf{H}\_{1:t}^{\top}\\|$ converges to 0 as $M_t\rightarrow\infty$. 

4. Figure 5 shows the comparison of training times for varying embedding dimensions $E$ between ICL-TSVD and RanPAC. How about comparison with other CL baselines in terms of training time? Typically, the computational cost of SVD-type algorithms is large, is ICL-TSVD still faster than other CL methods under the current experimental settings?

questions:
1. In Table 2, why the final accuracy of RanPAC on StanfordCars dataset is only 1.19%? Could you provide an explanation for it?

2. Figure 5 does not plot the training time for RanPAC when embedding dimension $E = 50\mathrm{K}, 100\mathrm{K}$, is this because the training process is too slow?

3. How about the running time of ICL-TSVD compared to other CL baselines, especially those that do not utilize SVD in their implementation?

4. For original definition of $\overline{\bf{W}}_t$ in Eq. (2) and practical implementation of $\widetilde{\bf{W}}_t$ in Eq. (4), do they close to each other? In other words, is there any theoretically guarantee showing that $\\|\overline{\bf{W}}_t - \widetilde{\bf{W}}_t\\|_F$ is small in general?

**Minor issues and typos**

- In Eq. (5) of Section 5, the denominator of $\gamma_t$ should be $\max\_{1\leq i\leq t-1}\mu\_{k_{i+1}}(\widetilde{\bf{B}}_i\widetilde{\bf{B}}_i^{\top})$ instead of $\max\_{1\leq i\leq t-1}\mu\_{k\_{i}+1}(\widetilde{\bf{B}}_i\widetilde{\bf{B}}_i^{\top})$.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692921667

thread_type:
official_review

<thread_object>
creator: reviewer_xhqx

<content>
summary:
This paper proposes a new method to address instability issues of RanPAC. Part of the contribution is a diagnosis of the cause of the instability which is attributed to the condition number of the features exploding as the number of tasks increases. The proposed method truncates the smaller singular values of the features in an online or continual way, so that it doesn't have to store all of the previous tasks samples. Effectiveness of the new method is validated empirically.

strengths:
The problem and setup is well motivated. The results are natural and the writing is clear. The experimental results are extensive and seem significant.

weaknesses:
It seems to me that the main contribution of this work is an algorithm for continual principal component regression, with an application to training with random features. Specially, the properties of the features that arise from pretraining, random projections and nonlinearity don't seem to matter much to the derivations theoretically. On the other hand, if I view the work mainly as an algorithm for continual principal component regression, then the guarantees given in theorems 1 and 2 are a black box, since the quantities $\gamma_t$ and $a_t$ which have an unpacked recursive form. It is not clear to me how they behave in general.

questions:
In the ICL framework of Peng et al. (2023), they are maintaining a minimum norm solution based on proof of Proposition 5 in that work where they are using the pseudoinverse. So I think the statements on top of page 3 might not be quite accurate. 


In lines 202-208 the cause of instability of RanPAC is attributed to the double descent phenomenon. I think it would be helpful to explain this more to the reader.


New notation that is used in lines 242-243 has not been introduced yet.

To judge the significance of the experimental results, it would be helpful if the reported numbers included also a measure of uncertainty.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703148740

thread_type:
official_review

<thread_object>
creator: reviewer_exjv

<content>
summary:
This paper proposes a method for continual learning. In this method, the singular values are continuously truncated. The authors claim that their method has theoretical guarantees as well as good practical performance.

strengths:
The authors provided comprehensive theoretical and empirical results.

weaknesses:
A simple baseline in continual learning is experience replay. But I did not see the authors providing the results for experience-replay-based method.

questions:
Did the authors compare with experience replay method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731512090625

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Our Algorithmic Contributions and Improvements in Comparison to RanPAC

comment:
We begin by claiming that our method improves over RanPAC in two major ways:
- **Our method is more scalable: Fig 5 shows our method is up to 100-1000 times faster for large embedding dimension $E$**.
- **Our method is more stable: It has 10 points higher accuracy on average than RanPAC in Table 2.**

These improvements are not seen in follow-up works such as Ahrens et al. (TMLR 2024) and Prabhu et al. (NeurIPS 2024), which we discussed in detail in the manuscript.

Below, we explain why RanPAC is unstable and inefficient and how our method improves it.

# RanPAC is unstable and inefficient
- RanPAC is unstable (cf. Lines 197-200), and it fails when its validation set (chosen to be a small fraction of a training set of the current task) is small and not representative of the test data. The validation set is indeed small in class-incremental learning with small increments. **This is a new understanding of RanPAC's failure pattern**, to our knowledge. 
- RanPAC is inefficient for large embedding dimension $E$ (Remark 8, Appendix C). For each task $t$, it solves a system of linear equations that amounts to inverting the $E\times E$ (regularized) covariance matrix $H\_{1:t} H\_{1:t}^\top + \lambda I$. This would take $O(E^3)$ time in general. Moreover,  for every task $t$ it solves such equations more than $10$ times (for different $\lambda$ taken from a predefined set) for the purpose of cross-validation.

# Why we need large embedding dimension $E$ and why prior methods couldn't handle large $E$
Roughly speaking, a larger $E$ yields higher accuracy (cf. Fig 8, Appendix K.4). This motivates developing scalable methods for handling cases with large $E$.

On the other hand, as **Reviewer P9vj** rightly pointed out, a larger $E$ would bring significant computational challenges. Indeed, RanPAC and its variants (Ahrens et al. TMLR 2024, Prabhu et al. NeurIPS 2024) all have $O(E^3)$ complexity and can not handle large $E$. More specifically:
- The running times of RanPAC with $E=10K$ (default) and $E=25K$ (the choice of Prabhu et al.) are in Fig 5. In this case our method is up to $100$ times faster.
- For $E=50K$, RanPAC is too slow. E.g., it takes more than **20 hours** on ImageNet-A in the setting of Fig 5 (similarly on other datasets). In contrast, our method takes about **1 minute** on ImageNet-A under the same setting. (This answers a question of **Reviewer iVyo**)
- For $E=100K$, RanPAC runs out of memory.

# Our contribution: a scalable method that works for extremely large embedding dimension $E$

For task $t$, our method performs QR decomposition on an $E\times  (k\_{t-1}+m\_t)$ matrix and SVD on an $(k\_{t-1}+m\_t)\times  (k\_{t-1}+m\_t)$ matrix, where $k\_{t-1}$ is the number of SVD factors preserved after task $t-1$ and $m_t$ is the number of samples in task $t$. This takes $O( E(k\_{t-1}+m\_t)^2)$ and $O( (k\_{t-1}+m\_t)^3 )$ time, respectively. 

Importantly, we can control the time complexity by adjusting the values of $k\_{t-1}$ and $m\_t$. E.g., we can set $k\_{t-1}$ small, or if $m\_t$ is too large, we can split task $t$ into several subtasks. In particular, $k_{t-1}$ can be extremely small, as Figure 4a shows the accuracy is not affected too much even if we truncate 90\% SVD factors. 

- **Reviewer iVyo** is concerned that SVDs can be costly. This is true in general. But here, essentially, our method performs *low-rank* SVD, and its complexity $O( (k\_{t-1}+m\_t)^3 )$ could be affordable when $k\_{t-1}+m\_t$ is not large, and it is advantageous compared to $O(E^3)$ algorithms when $E$ is far larger.

- **Reviewer P9vj** is concerned that $k\_{t-1} +m\_t$ and $k_t$ would eventually be larger than $E$. But note that we need at least $k_t < \min\\{ E, M_t\\}$ for the truncation to have effects; here $M_t$ is the total number of samples seen after task $t$. This is why, in our full implementation, Algorithm 4 of Appendix C (Line 1098), we have an extra hyperparameter $r\_{max}\in[0,E)$, which denotes the *maximum allowable rank*, and we set $$k_t\gets \min (r\_{max}, (1- \zeta) M_t),$$ We apologize for this not being clear in Algorithm 1.

- **Reviewer P9vj** is concerned that a large $E$ would make *the proposed method infeasible for most datasets*. This would indeed be an issue for other methods, but not for ours. We showed our method is feasible and fast on multiple datasets with large $E$, as it sets $k\_{t-1}+m\_t$ small and has a linear time complexity in $E$.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731515895957

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Our Theoretical Contributions to CL with Pre-trained Models: Three-stage Interrogation

comment:
We consider a 3-stage interrogation of our theoretical contributions. By doing so, we hope the reviewers could appreciate our perspectives and theoretical results a little bit more.

# 1. Status of the field (continual learning)
Here we rehearse what we wrote in Lines 054-059:
- Many methods have recently been developed for CL with pre-trained models (cf. Table 1). **However, almost all these methods in Table 1 are purely empirical, and there is little theory derived in their paper.** These include RanPAC and its follow-up works (Ahrens et al. TMLR 2024) and (Prabhu et al. NeurIPS 2024).

- On the other hand, **theoretical studies on CL have been focused on algorithms or models that have little to no implications in practical CL scenarios such as class-incremental learning**. These include Evron et al. (COLT 2022), Lin et al. (ICML 2023),  and Peng et al. (ICML 2023).

# 2. Significance of our theory

Given the current status of the field, and while many CL papers provide theorems that do not match the algorithms that are in practical use, **we believe our theoretical results are the first of their kind in the field, in the sense that they apply precisely to the proposed CL algorithm, which has strong performance at the same time**.  This reduces the gap between theory and algorithms. We believe this is an important conceptual contribution and will inspire follow-up works to consider similar frameworks to develop empirically strong CL methods with guarantees.

Furthermore, we claim that our theory is novel, useful, and of sufficient technical depth.

- **Our theory is novel** in light of related works even in the statistics community (in Remarks 4 and 6). To our knowledge, **our theory is the first such result that analyzes the errors of incremental truncated SVD**, even though the basic techniques of incremental SVD have been known for decades since Bunch & Nielsen (1978) (cf. Appendix C).
- **Our theory is useful** as the upper bound is small if we truncate suitably. This can be understood from Section 5 and all the figures.
- **One theory is of sufficient technical depth**, as we analyze the effects of continual truncation, while prior published works in the theory community truncate only once (the offline case); cf. Remarks 4 and 6. 

While **Reviewer P9vj** was *not convinced that the analysis makes the paper valuable enough for ICLR*,  and while we appreciate the very high standard **Reviewer P9vj** holds for theoretical works, we should note that our work is not purely theoretical, and we consider our algorithmic contributions to be even more important, as we will justify in the next post (*Our Algorithmic Contributions and Improvements in Comparison to RanPAC*).

# 3. Limitations of our theory
As the reviewers acutely pointed out, our theorems have limitations, e.g., the bounds might be loose; the linear model assumptions might be restrictive. We will address them in our responses to individual reviewers. Here, in order to better appreciate these limitations, we clarify why our theorems are as such and the rationale behind them:
- We have a method that truncates SVDs continually, so **we want to analyze the effects of continual truncation**. However, such analysis in its full generality is a formidable task, as the full network architecture is complicated: the input images have a complicated distribution; they are fed to a pre-trained model to yield pre-trained features $X\_{1:t}$, which is lifted to random ReLU features $H\_{1:t}$; our method continually truncates $H\_{1:t}$.
- Since we don't know the distribution of $X\_{1:t}$, any randomness assumptions on $X\_{1:t}$ would sound unrealistic to practitioners. Hence, we want to derive theorems with $X\_{1:t}$ being deterministic, even though a rich body of theory work needs Gaussian-type assumptions on $X\_{1:t}$ or $H\_{1:t}$ (cf. Remarks 4 and 6). Crucially, we find we can do the analysis under a linear model assumption conditioning on $H\_{1:t}$. While the linear model assumption might be considered limited, note that:
    - **doing so achieves our goal, as the resulting theorems do reveal the effects of continual truncation**. Specifically, the role of continual truncation is made explicit via two quantities, the accumulative error $a\_t$ and eigengap $\gamma\_t$. Moreover, we can understand the behaviors of $a\_t$ and $\gamma\_t$ via the spectrum of (truncated) random ReLU features, which we visualized and analyzed. 
    - **the roles of pre-trained models, random embedding, and nonlinearity are also captured by our theory (implicitly)**, as they affect the spectrum of (truncated) random ReLU features and thus the values of $a\_t$ and $\gamma\_t$.
    - **it puts no restrictions on our method**. Rather, it *predicts* the errors our method could produce. This is important, as the prior work, RanPAC, is unpredictable and unstable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731550861272

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Reviews and Our Rebuttal

comment:
We thank all reviewers for their time in reading the manuscript and writing comments. 

First of all we are happy to read that the reviewers commented on the following strengths of our paper:
- Experiments are extensive, comprehensive, or significant (**Reviewer EXjv**, **Reviewer XHQX**, **Reviewer iVyo**)
- The writing is clear and easy to follow (**Reviewer XHQX**, **Reviewer iVyo**). All reviewers scored *3 (good)* for *Presentation*.

We received multiple comments on weaknesses from the reviewers, and we also received multiple questions. To address them, we write our rebuttal that consists of:
- a statement at the end of this post, identifying our non-conventional, “bridging fields" contributions;
- one post that clarifies our theoretical contributions (*Our Theoretical Contributions to CL with Pre-trained Models: Three-stage Interrogation*);
- one post that clarifies our algorithmic contributions (*Our Algorithmic Contributions and Improvements in Comparison to RanPAC*);
- posts to individual reviewers addressing their specific concerns;
- a revised version of our paper based on the comments. We made minimum modifications to the original submission (from the main paper to Appendix K). All revisions and new contents needed to address reviewers' concerns are in Appendix L. We believe this would make our revisions more accessible, and can be easily compared to the original version. Minor comments, such as grammar or typos, will be fully addressed later (e.g., after the decisions are out).

We are glad that most comments are constructive and can be addressed to their full extent with our best efforts. Resolving some other comments does require interacting with the reviewers and relying on their active participation in the discussion phase. Given the current divergent scores, and given our belief in the quality of our work, we would be grateful if the reviewers could join the discussion, and if **ACs** could take a look at the discussions, our rebuttal, and our manuscript (especially, the intro, Fig 5, Table 2, Table 16).


# Non-conventional, “bridging fields" contributions

This year, AISTATS helps the reviewers identify *non-conventional contributions* by the following questions:
> *Does the submission contain non-conventional research contributions? Examples: novel ideas with not widely accepted assumptions, new problems and/or tasks, “bridging fields” contributions.*

We bring it to the attention of our ICLR reviewers, and state our non-conventional, “bridging fields" contributions that we indicated in the title (as they are often overlooked):
- Our paper bridges the gap between the theoretical studies and empirical methods in continual learning;
- Our paper connects continual learning to recent theoretical studies on random feature models and overparameterized linear models (cf. Remarks 1, 4, 6, and Appendix H.2). Given this connection, we believe existing theory in random feature models and overparameterized linear models will play an important role in enhancing theoretical foundations of continual learning. Conversely, as **Reviewer XHQX** recognizes, our theory can also be viewed as advances in principal component regression, an important topic in statistics.
</content>
</thread_object>

</thread>
<thread>
creation date:
1767190514204

thread_type:
public_comment

<thread_object>
creator: ~xiaoyu_sun6

<content>
title:
About the title difference between the paper and BibTex

comment:
Dear authors:
Thank you for your inspiring work! I was copying the bibtex provided and found the title in bibtex "TSVD: Bridging Theory and Practice in Continual Learning with Pre-trained Models" is different from the paper title: "LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning". I wonder is this bibtex valid or I need to change the title field to LoRanPAC title? Thank you very much for your reply.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper combines RanPAC into the Ideal Continual Learner (ICL) framework, providing theoretical results and experiments. The new method (and its empirical performance), along with theoretical results, are key contributions of the paper. 

Reviewers were generally more positive for this paper after rebuttal than before. Key strengths are that the method is well-motivated, the paper is well-written, and the experiments are extensive. The experiments are particularly impressive given the number of pages devoted to theory in the paper; perhaps the authors could consider giving more space to the results. 

There are key questions regarding the theoretical contribution of this work, however. Reviewer XHQX is missing comparison to online PCA algorithms: the authors should clearly compare their theoretical analysis to existing work in this space (such as the PCA-OGD algorithm that has been discussed already). Reviewer P9vj does not find the theoretical results very useful, and notes that the authors have promised many changes to the paper. My own estimation of the theoretical contribution is somewhere between the authors' and the reviewers'. Overall, given the lack of such results in continual learning, I think the field would benefit from this paper.