Quantized Gradient Projection for Memory-Efficient Continual Learning
Authors: Dongjun Kim
Seohyeon Cha
Huancheng Chen
Chianing Wang
Haris Vikalo
Venue: ICLR 2026 Poster (2026)
Keywords: Continual Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose QGPM, a memory-efficient and privacy-preserving continual learning framework that compresses task subspaces via quantization.

Abstract: Real-world deployment of machine learning models requires the ability to continually learn from non-stationary data while preserving prior knowledge and user privacy. Therefore, storing knowledge acquired from past data in a resource- and privacy-friendly manner is a crucial consideration in determining their viability. We introduce Quantized Gradient Projection Memory (QGPM), a systematic framework for continual learning that compresses and preserves the previous gradient subspace. QGPM integrates three key components: (i) distribution-aware, basis-wise quantization to minimize storage overhead, (ii) a Quantization Error-Aware (QEA) gradient projection that selectively relaxes orthogonality to mitigate gradient drift caused by accumulated quantization noise, and (iii) an on-the-fly sparse sketching strategy that improves runtime memory and computational efficiency. Experiments across multiple benchmarks demonstrate that QGPM achieves state-of-the-art performance under fixed memory budgets, highlighting its effectiveness in scalable, privacy-preserving continual learning.

Published as a conference paper at ICLR 2026
QUANTIZED GRADIENT PROJECTION FOR MEMORY- EFFICIENT CONTINUAL LEARNING
Dongjun Kim', Seohyeon Cha', Huancheng Chen', Chianing Wang", Haris Vikalo' University Texas at Austin Toyota InfoTech Lab USA [dongjungim20, seohyeon. cha, huanchengch, hvikalo @utexas edu, johnny, www.wtoyota.ccomommmmm com
ABSTRACT
Real-world deployment of machine learning models requires the ability to contin- ually learn from non-stationary data while preserving prior knowledge and user privacy. Therefore, storing knowledge acquired from past data in a resource- and privacy-friendly manner is crucial consideration in determining their viability. We introduce Quantized Gradient Projection Memory (QGPM), systematic frame- work for continual learning that compresses and preserves the previous gradient subspace. QGPM integrates three key components: (i) distribution-aware, basis- wise quantization to minimize storage overhead, (ii) Quantization Ewror-Aware (QEA) gradient projection that selectively relaxes orthogonality to mitigate gradi- ent drift caused by accumulated quantization noise, and (iii) an on-the-fly sparse sketching strategy that improves runtime memory and computational efficiency. Experiments across multiple benchmarks demonstrate that QGPM achieves state- of- the-hear art performance under fixed memory budgets, highlighting its effectiveness in scalable, privacy continual learning. Our code is available here.
INTRODUCTION
Deep neural networks (DNNs) have achieved remarkable success across domains, particularly in computer vision (He al., 2015: Simonyan & Zisserman, 2014; He et al., 2017). However, their standard training paradigm assumes access to the full dataset upfront-an an unrealistic assumption in real-world scenarios where data arrives sequentially and evolves over time. In contrast, intelligent agents must learn continuously: acquiring new skills while retaining prior knowledge. This motivates the field of continual learning (CL) (Lange et al., 2019), which aims to enable models to adapt to new tasks without overwriting existing representations. Naively applying stochastic gradient descent (SGD) in this setting leads to catastrophic forgetting (McCloskey Cohen, 1989), where learning new tasks disrupts previously acquired knowledge. To mitigate this, several CL strategies have been proposed, including: (1) Regularization basedd methods (Kirkpatrick et al., 2017), which penalize updates important weights; (2) Expansion-based methods (Rusu et al., 2016), which allocate new parameters per task; (3) Rehearsal-based methods (Rolnick et al., 2019), which replay stored or synthetic data (Gao & Liu, 2023); and (4) Projection-base methods (Saha et al., 2021; Farajtabar et al., 2020), which constrain updates to subspaces orthogonal to prior gradients. Each strategy encodes prior knowledge in specific form: importance weights, dedicated modules, rehearsal buffers or generative models, and gradient subspaces, respectively. As Rebuffi et al. (2017) emphasize, viable incremental learner must keep memory and compute demands bounded or slowly growing. Thus, memory efficiency is a key factor in determining CL. method's practical deployability.
forgget
Gradient Projection Memory (Saha et al., 2021) maintains a dedicated memory structure the GPM where it stores a set of core bases that span the gradient subspaces associated with previous tasks. When learning a new task, incoming gradients are projected onto the orthogonal complement of the subspace spanned by the stored bases, thus minimizing interference with previously acquired knowledge. These projection- based methods demonstrate saate-ofthhe-e art stability against catas- trophic forgetting (Liang & Li, 2024). Furthermore, the method's 's inherent stemming from not storing raw data or intermediate representations, makes it well-suited for continual learning in privacy-sensitiv fields such the medical domain (Verma et al., 2023).
preeeccireeniinnggg nature,

---

Published as a conference paper at ICLR 2026
In this paper, we demonstrate that the memory efficiency of continual learning be significantly improved by quantizing the bases stored in GPM, while preserving its core benefits. However, this introduces two key challenges: (1) the distribution of individual bases is often heavy-tailed, leading to large quantization errors; and (2) subspace distortion can cause projected gradients to deviate from their intended direction, resulting in what call gradient drift. To overcome these issues, propose two complementary techniques: Centered Inlier Normal Float (CINF) quantization, which reduces the influence of outliers during quantization, and Quantization Error-Aware (QEA) gradient projection, which adaptively relaxes orthogonality constraints based on estimated deviation from the desired update direction.
The main contributions of this paper are:
1. We propose Quantized Gradient Projection Memory (QGPM), novel framework for memory- -efoccieeti continual learning that leverages basis-wise quantization. Its core compo- nent, Centered Inlier Normal Float (CINF) quantization, enhances robustness to outliers and improves codebook utilization.
2. To mitigate performance degradation caused by accumulated quantization errors, we intro- duce QEA gradient projection, a technique that balances orthogonality with alignment to the desired gradient direction.
3. We propose On-the-Fly Sparse Sketching to accelerate the SVD computation and reduce intermediate training-time memory overhead during QGPM construction.
4. We conduct extensive experiments demonstrating that QGPM achieves strong performance under tight memory budgets, with detailed analysis of its memory-accuracy tradeoffs.
RELATED WORK
Continual Learning. To address catastrophic forgetting, numerous continual learning (CL.) methods have been proposed. Among these, regularization based methods add constraints to prevent significant changes to parameters deemed important for earlier tasks. For example, Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) adds a quadratic penalty term weighted by the Fisher Information Matrix, while Synaptic Intelligence (SI) (Zenke et al., 2017) computes parameter importance during training and penalizes deviations proportionally. Despite their simplicity, such methods often suffer from low stability and require parameter importance masks that scale with model size, limiting memory efficiency. Expansion-based methods allocate task-specific parameters or sub-networks to isolate learning across tasks. Progressive Neural Networks (PNN) (Rusu al., 2016) grow the model by adding new columns of neurons for each task while freezing carlier ones, whereas Hard Attention to the Task (HAT) (Serrà et al., 2018) learns task specific binary masks to protect important neurons. While effective, these approaches incur linearly growing memory overhead, violating the goal of bounded resource usage in CL. Rehearsal-bast methods store and replay data or intermediate features from previous tasks to reinforce memory. Experience Replay (ER) (Rolnick et al., 2019) maintains a buffer of past samples and interleaves them with current training data. DER++ (Buzzega al., 2020) and FDR (Benjamin al., 2019) combine rehearsal with distillation- based regularization, penalizing shifts in logits or model outputs. These methods can perform well given a moderate buffer size but raise privacy concerns due to the storage of raw data. Finally, gradient projection-baser methods constrain updates to lie in subspaces orthogonal to previous gradients. OGD (Farajtabar et al., 2020) and OWM (Zeng al., 2019) pioneered this idea, while GPM (Saha et al., 2021) improves scalability by projecting in the space of input representations. Recent works (Yang et al., 2023; Liang Li, 2023; Lin al., 2022; Chen et al., 2024) build on this foundation enhance performance and efficiency.
Memory-Effiiien Continual Learning. Memory-efficient approaches to incremental learning remain relatively underexplored. Parameter isolation via pruning has been used to allocate disjoint parameter subsets per task (Zhao et al., 2022). Iscen et al. (2020) reduces the rehearsal buffer by storing compact representations (e.g.. lightweight features) instead of raw data, combined with knowledge distillation. However, storing intermediate features remains vulnerable to inversion attacks (Jacobsen et al., 2018). While Zhou et al. (2023) expands only the last few task-specific layers, the overall memory footprint still grows linearly with the number of tasks. To eliminate memory buffers altogether, data-free knowledge distillation has been proposed (Smith al., 2021; Chung
2

---

Published as a conference paper at ICLR 2026
et al., 2022; Choi al., 2021). Although it effectively addresses privacy concerns, full copy of the network must be retained for distillation, leading to additional memory overhead. Ermis et al. top of pretrained transformers, in conjunction with knowledge distillation;
(2022) utilizes adapters however, this is only applicable in scenarios where a pretrained foundation model is available. Many existing works leverage knowledge distillation to reduce memory usage from rehearsal model expansion. However, this typically requires either a dedicated auxiliary network or full copy of the backbone model, introducing significant memory and computational overhead aspect often overlooked when reporting overall memory budgets. To the best of our knowledge, no prior work has explored the use of quantization compress gradient subspaces, despite its potential offer strong privacy guarantees and improved memory efficiency.
3 QUANTIZED GRADIENT PROJECTION MEMORY (QGPM)
In this section, we present the core ideas behind our approach memory-efficient continual learning via basis-wise quantization of Gradient Projection Memory (GPM). The main challenge lies in the fact that the subspace spanned by the stored bases serves the reference for orthogonal gradient updates, making it highly sensitive to coantizationninducce
Varla M A4-1 (M!-z)' Valle
distortions. As shown in Theorem 3.2. the deviation between the ideal orthogonal update- - computed based on the true subspace and the update derived from a quantized subspace grows quadratically with the quantization er- ror. We find that standard linear quantization schemes are inadequate in this setting. In response, we propose set of techniques to mitigate quantization effects while preserving the core functionality of GPM. Algo- rithm 1 summarizes the full QGPM procedure, which integrates CINF quantization, QEA gradient projec- tion, and On-the-Fly sparse sketching into unified CL framework.
Vihe
Mi (M,->" Vidhe
f Viyle
Subspaaee spanned
by
M (M -yplir
M Al-1 (M!-1)
Vuple
Figure QEA Gradient Projection.
3.1 PRELIMINARIES
Gradient Projection for Continual Learning. The objective of continual learning is to solve minw T E L- (w: D(r)), where L, and (r))= = (x(r)) denote empirical loss and dataset of the 7-th task. Let w C w, denote I-th layer parameters trained on task and let R, be the input representation to layer at that point. The output activation of layer after training on task T is given by w! R We would like this activation to remain unchanged even after learning task + + 1, i.e. w/ R = the orthogonality condition Aw' R 0. To enforce this constraint, we store set of core basis vectors that span the subspace of R (see lines 13-16 of Algorithm 3 in Appendix B), and restrict future gradient updates to lie in the orthogonal complement of this subspace (see line 5 of the same algorithm). We denote this memory (i.e., GPM) for layer 1 after task r by M/.
w/ 7+1
R = (w,, + Aw')- R., This requires that the weight update Aw' satisfies
k-bit NormalFloat (NFK) Quantization. The authors of QLoRA (Dettmers al., 2023) introduced NFK, an information-theorttica.. optimal quantization data type for normally distributed data. NFk is variant of quantile guantization, which ensures that each quantization bin contains an equal number of data points mapped from the original distribution. For k-bit quantization, NFk stores 2 codes in
pre dre-end codebook:
[c1. C C2+ [-1, 1 Given vector u (a1, a2...., ag) R B the quantization proceeds as follows: (1) Compute M = max; |a,]. (2) Normalize each entry by M scale the values into the range [-1, 1]. (3) For each normalized value, find the index j of the closest codebook value Cj. i.e., 03 is mapped/diserretzzd according to the index j = argmin, Icg 4 M7
3.2 CENTERED INLIER NORMAL FLOAT (CINF)
key limitation of naive NFk quantization is its vulnerability to outliers. normalizes the input vector by its maximum absolute value, which can become large in the presence of outliers. This
3

---

Published as a conference paper at ICLR 2026
compresses the bulk of values toward zero, distorting the distribution into highly uub-Gaussian. As result, the quantization codebook is used inefficiently; most values are mapped to bins near zero, while only few occupy the extreme bins near -1 and 1, leading significant information loss (Yoshida, 2023). To address this issue, we propose Centered Inlier Normal Float (CINF) quantization, which improves robustness outliers by restricting normalization central quantile range. Given input vector = (A1, 02 ag,...,, QB), CINF performs the following steps:
1. Compute the mean of the input, μ B Zi=1 dg.
2. Center the vector components by subtracting the mean, as,centered a; J..
3. Compute the å and 1 quantiles of the centered values, and Q1-5-
4. Values outside the quantile range [gs, q1-6] are stored in full precision. The remaining inlier values are quantized using NFk, where normalization is performed using the scale factor
8
max (a 1/1 (5)) rather than the global maximum M max; (agl.
Further details on CINF codebook construction are provided Appendix E.I. We now analyze how this method reduces quantization error. Assume entries of vector follow standard normal distribution, i.c., 0; ~ (0, 1). Let M = max; |a: and define the quantile threshold % = inf 2 0: B Σ -= 1((; - r) 2 5), with scaling factor s = max(|gsl. (l). Partition the data into outliers O {i |al 2 s} and inliers I OC_ Let [c1, C2. ..... c2s denote the fixed NFk
codebook. The dequantized value under naïve NFK is
0
NFk
M (1)
(0()) = arg min C
a M
iEOUZ.
(I)
je jk
In contrast, the dequantized value under CINF is given by
Oj:
iEO,
a
CINF
j"(1) arg min
Cj
(2)
(C) + +M,, i I,
je (1....22))
The quantization errors of NFk and CINF can be respectively modeled as:
NFK
Σ (a, as aNFk?,
CINF
Σ (a: a CIINF
OUUI
I
(E(OUZ)
where the CINF error CINF) is computed over inliers only, as outliers are stored losslessly. To understand how normalization impacts quantization error, we compare the NFK scale factor M = max, |a,] to the CINF quantile threshold 98- The next result from order statistics offers useful approximation for this comparison.
Theorem 3.1 (Expected Values of Normal Order Statistics (Harter, 1961)) Consider
(21.a2..... aB). a set of independent random variables drawn from a standard nor- mal distribution, i.e. ~ (O, Denote the expected value of the i-th order statistic by Eii B). Then, for sufficiently large B, this expectation can be approximated as:
Fli : B Rs t
---0-- 21)
i
B-20 B-2a+
where -1 is the inverse cumulative distribution function (CDF) of (O, 1), and a = 0.375.
By definition, E[M E B ~ ( (--211)) As B grows, M becomes large since B-a + As previously discussed, this leads to overly aggressive normalization of the data via a;/M, compressing most values near zero and resulting in inefficient utilization of quantization bins. In contrast, by excluding small fraction of outliers (e.g., the top 1%), the expected scale for baseer-based normalization becomes E[s] = E00.99 B ~ x
B-2a+I
000(0.000) 1170.98Ba B B-2a+1 which remains
bounded above by t -0'(0.99) = 2.32. This results in and improved codebook utilization compared to the NFk case. Consequently, CINF produces a lower quantization error than NFk:
evenly spread normalized distribution
CINF C
NFK
(5)
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 QGPM Training Algorithm
Input:
fw the NN model, Diruin the training dataset,
12:
Bng D. Duusn
ry the learning rate, and cti, threshold value. Initialize, MMg.. a. St. Zg. Oj + D. for all
13:
(R. )i t Corward(B... fw)
14:
1,2,.... T/, and f- w.-
15:
for VI E 21,,2.... _.3 do
1: for r =1,2.... . T do
16:
R R M (M', T -
2: for v/ E 21. L') do
17:
Ú Σ V + SVD(RR)
3:
M -ea z z!-1)
18:
r criterion RR, Ri, Fen)
4: P - Diag(O;-1) (M,-
19:
U2 st, o² + Quant (;
5:
end for
6:
repeat
20:
Mar <- Mo, --11 U2
7:
Bn Dirain
21:
S + Concat S 1,
8:
g Vw J--
22:
Z Concat
9:
g g P P
23:
O + Concat -oncat(OO--1,11 10
<- W 17 g
24:
end for
11:
until convergence
25:
end for
Now, we define the basis-wise CINF quantization and dequantization functions for a single input vector I2d as 8, H, A) = QCINF(u) and ú 2CINF (u, s.). where u, the quantized vector, s is the inlier scale, is the mean, and À is orthogonality weight (introduced later). In our implementation, the Quant function in Algorithm applies the column- wise operator QCINF to the top-r left singular vectors Ü 1 r] obtained from SVD. Formally,
U2 i), st 2.1. of = QCINE (UH i)), Vi (1....,r).
(6)
Here, U U2-- = [u, .......... m [r)] d. stores the quantized basis vectors, while the auxiliary arrays s" = [s1...... s, 2 = [pr..... Mr]. and o' = [AI,.... Apl record the corresponding scales, offsets, and orthogonality weights used in dequantization and QEA projection.
[m (1)
3.3 QUANTIZATION AEROO-AWARRE (QEA) GRADIENT PROJECTION
QEA Gradient Projection. In standard GPM-based CL, when training on the 7-th task using dataset (r)). the raw gradient g = V. is computed via stochastic gradient descent (SGD). This gradient is then projected to be orthogonal to the subspace spanned by the basis vectors stored in M (see line 5, Algorithm 3 in Appendix B). In QGPM, the basis vectors quantized to reduce memory footprint, forming Ma... To perform gradient projection (line 9, Algorithm 1), the quantized basis vectors must be dequantized into full-precision (or optionally, a compact higher-precision format such BF16) for use in tensor operations. This is handled by the Dequant function in line 3 of Algorithm 1, which reconstructs each basis vector by applying column-wise 2 operator
M M .. a] (Mr d]. S.- z4-1()) Vi E (L....,IM,.r-)
(7)
where 1M2y-1 denotes the number of basis vectors in the QGPM. However, quantized bases accumulated over multiple tasks, the quantization error compounds. We observe that this accumulated distortion destabilize the learning process: when the subspace spanned by the quantized GPM is significantly misaligned from its original counterpart, the resulting gradient projection deviates from the true orthogonal direction. This leads update directions that are longer interference- free with respect to past tasks - we refer to these as gradient drift.
Theorem 3.2 (Quantization Error Accumulation) Let M, = [ui...... Uml E Raam be the full- precision GPM, and let E [eI...., Em] E &Rax be random Gaussian error matrix, where each column Ej ~ (0, LL)) Define the quantized GPM M. M, + E. Let detge IR" be a gradient vector, and let 8. and ge denote its orthogonal components with respect to the subspaces spanned by M, and M., respectively. The expected deviation between the projected gradients satisfies:
E. [ll.. BB [2] [g. -ge = m..0. 20 ||g||2
(8)
5

---

Published as a conference paper at ICLR 2026
implying that the projection error introduced by quantization grows linearly with the number of basis vectors m and quadratically with the quantization noise level a. The proof is provided in Appendix G.I.
QGPM mitigates quantization errors by relaxing the strict orthogonality constraint of standard GPM, allowing a controlled parallel gradient component. This relaxation is adaptively scaled based on the quantization fidelity of each basis vector, which may vary across vectors due differences in their distributions and compressibility. To capture this fidelity, the Quant function evaluates the quantization error individually for each of the r new basis vectors Ü i] to be incorporated into the QGPM. Specifically, immediately after quantization via Eq 6, each quantized basis vector o [' i] is dequantized to obtain an approximate reconstruction that incorporates quantization error:
Ü C (Ü* i))), Vi (1, (1,.....r)
To assess the distortion introduced by quantization, we compute the cosine similarity between each original full-precision basis vector [t, i)) and its dequantized counterpart (ÜLL. i). Finally, the quantization error for the i-th basis is defined as:
e simeos L. (.( =1 U. U. T all 10 a|1l2
(10)
The value e, quantifies how much the dequantized vector U [+ a] deviated from the original basis Üf :]. To incorporate this into the gradient projection mechanism, we introduce a hyperparameter that scales the error, and define an orthogonality weight A, = 1 - a ei. The Quant function distinct orthogonality weight for every
computes A for each of the new basis vectors, resulting column in the QGPM. These weights are then used to specify the projection strength during gradient updates, controlling the extent which the orthogonality is enforced. Formally,
V L V.L, M AL-1 (M, T. W V. P P V.Lr (11)
where A!_, Diag(A),... .......... A E RIMM 1 11, as illustrated in Figure 1. When the quantization error e; is large, the corresponding X; is reduced, allowing a greater component of the gradient to pass through in the parallel direction thereby increasing plasticity along the associated basis vector. This adaptive mechanism mitigates the impact of projection distortion due to quantization error, effectively preserving both model stability and past knowledge.
3.4 QGPM CONSTRUCTION WITH ON-THE-FLY SPARSE SKETCHING
After completing the 7-th task, QGPM is updated to M by incorporating new basis vectors. To do so, subset of training samples B., C Dirain is passed through the network fw, producing input representations at each layer, R2 [R2, RR,..., RR).. We extract the orthogonal component
of the new representation relative to the current GPM subspace M i.e., R = R M (M/-1)T R/,, where M (M/-1)T R² is the component that already exists in M To extract orthogonal basis vectors from R,, we apply Singular Value Decomposition (SVD) and retain the top-r singular vectors, Ú Si V
7,P 7.r 7. = R!,, Rs Rí, where the value of r is chosen by the
function criterion:
M
(M. R
T
|R RR
+
cth-
(12)
This ensures that the retained directions collectively explain least eea. of the total variance in R/ The selected r left singular vectors are then quantized, yielding r quantized singular vectors U/ along with auxiliary arrays, (s'_ z o')), ) introduced in Section 3.2. Finally, the newly quantized basis vectors and their associated metadata are appended to the QGPM: Ma St. 2 and O
On-the-Fly Sparse Sketching. In the standard GPM update. the representation matrix R\, is constructed by collecting local activation patches from the entire feature representation. Each patch is flattened into a column vector r,, and these vectors are concatenated to form R [ri...., P.]] A

---

Published as a conference paper at ICLR 2026
key challenge is that N can become extremely large, especially in convolutional or transformer blocks popular components of modern neural networks. As shown in Table 10 in Appendix C, this results in a high- nigh-dimmnsional matrix that significantly slows down SVD and incurs substantial memory overhead during training. To address this, we propose an On-the-Fly Sparse Sketching strategy, which constructs a low-dimensional approximation of R using sparse sketch in a streaming manner.
Theorem 3.3 ((1 + e)-22 Subspace Embedding via Sparse Sketching) Let
RTXN be sparse embedding matrix constructed using hash functions h [N] [r] and a: [ The i-th column of S has a single non-zero entry at row h(t), with value (i). Then, for any rank-k
matrix A E I. INNx, ifr = O( ( then with probability at least 1 6. S is (1 + e)-l2 subspace embedding for the row space of A in the En-norm. Equivalently, for all E R",
(1 =) || S ||SAx|12 S (1++) +++
(13)
Proof is in Appendix G.2. Leveraging the (1+c)-l2 subspace embedding property of sparse sketching (Theorem 3.3), we apply column-wiss sketching for each new representation vector r; by setting r= n. This reduces the size of the representation matrix from RR to R=*"" while approximately preserving the geometric structure of the original subspace. As result, the peak intermediate memory required to construct the QGPM is reduced by factor of N/n at each layer. In addition, the computational cost of SVD, the primary bottleneck in standard GPM, is reduced by the same factor. Specifically, the time complexity of computing the SVD of an x N matrix is O(N x n²) when N > n. Notably, with this sketching scheme, the representation matrix for ViT-S can be compressed from 1.62GB to just 66MB. Detailed implementation and analysis are provided in Appendix C.
4 EXPERIMENTS
Setup. We evaluate QGPM on three standard CL benchmarks: 10-split CIFAR-100, 100, 5-Datasets (Ebrahimi et al., 2020), and 10/20-split minilmageNet (Vinyals et al., 2016). These benchmarks capture complementary challenges of continual learning. including large label spaces, cross-domain heterogeneity, and scalability. We follow standard architectures for each benchmark: AlexNet (Serra et al., 2018) for CIFAR- ResNet-18 (He et al., 2016) for 5-Datasets, and pretrained VIT-S (Tou- vron et al., 2022) for minilmageNet. All models are trained in a multi-head setting, with a dedicated classification head per task. Further architectural and dataset details are in Appendix A.2.
Baselines. We compare QGPM against rehearsal- based continual learning methods that support flexible memory buffer sizes, unlike regularization-
expansion-based approaches. Specifically, we include Average GEM (AGEM) (Chaudhry et al., 2019), Function Distance Regularization (FDR) (Benjamin et al., 2019), Experience Replay (ER) (Rolnick et al., 2019), and Dark Experience Replay (DER++) (Buzzega et al.. 2020), which represent widely used and competitive baselines. We also evaluate two GPM variants: the original full-precision version (GPM-FP) and memory- constrained variant with reduced <th (GPM-FC-MC). For fairness, all methods are matched to QGPM's memory budget by adjusting the buffer size to yield an equivalent overall overhead.
Metrics. We evaluate performance using two standard continual learning metrics (Lopez-Paz & Ranzato, 2017): average accuracy (ACC) and backward transfer (BWT). ACC measures the overall test accuracy across all tasks the end of training, while the BWT quantifies the degree of forgetting (with negative values indicating performance loss on earlier tasks). They are defined as:
T E aco BWT T
T-1 Σ (acet - acci).
ACC = T
(14)
where acc? is the final accuracy on task t after learning all T tasks, and ace is the accuracy on task t immediately after it is learned.
4.1 PERFORMANCE ANALYSIS
Main Results. Table reports results for 8-bit QGPM (8QGPM) and 4-bit QGPM (4QGPM) compared to all baselines. Across benchmarks, 8QGPM matches full- precision GPM (GPM-FP), with ACC and BWT reduced by less than 0.5%, while requiring x less memory on average. Under
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison continual learning benchmarks. 8-bit QGPM (8QGPM) matches full- precision GPM (GPM-FP) with <0.5% drop ACC/BWT while reducing memory by 3.83x on average. Under the same memory budget, rehearsal-bard baselines (AGEM, FDR, ER, DER++) suffer substantial forgetting (higher BWT), 4-bit QGPM (4QGPM) remains robust (<1% drop vs. GPM-FP) with 6 memory reduction, demonstrating scalability under stricter constraints.
10-split CIFARI00
5-Datasets
10-split miniImageNNt
20-split miniImageNeet
Methods
ACC (t)
BWT (1)
ACC (1)
BWT )
ACC(1)
BTT
ACC (f)
BWT (t)
Memory
3.13 MB
27.67 MB
9.85 MB
11.07 MB
GPM-FP
71.11
-0.98
89.52
-1.83
73.84
-3.30
80.28
-2.73
Memory
0.81 MB
7.24 MB
2.59 MB
2.88 MB
GPM-FP-MC
64.58
-12.88
77.82
-66.14
71.48
-8.
77.06
-6.69
AGEM-MC
50.72
-25.44
80.45
-14.81
57.43
-23.74
60.42
-24.02
DER++-MC
60.63
-14.82
84.08
-8.10
68.77
-11.51
76.
-9.35
ER-MC
56.41
-16.77
80.97
-11.79
67.95
-14.37
75.56
100088
FDR-MC
63.07
-13.74
83,54
-9.88
67.80
-12.93
70.73
113,94
8QGPM (Ours)
70.70
-0.81
89.41
-2.08
73.78
-3.54
80.23
-2.77
Memory
0.48 MB
4.27 MB
1.52 MB
1.76 MB
GPM-FP-MC
62.33
-20.23
70.86
-25.15
70.90
-9.48
76.36
-7.64
4QGPM (Ours)
69.74
-2.62
88.51
-3.79
73.77
-4.50
80.25
-3.38
the same memory budget, rehearsal-basee baselines suffer substantial forgetting due to limited buffer size, leading much higher BWT. With aggressive compression, 4QGPM shows only modest drop (< 1% ACC/BWT relative to GPM-FP) while achieving a 6.44 x memory reduction. Since 8QGPM already outperforms all baselines equal memory, we focus on the comparisons involving 8-bit results; 4-bit results highlight the scalability of QGPM under stricter memory constraints.
Memory Profile. Figure 2(a) reports the memory footprint of QGPM compared full-precision GPM (GPM-FP), with model parameter sizes included for reference. Specifically, the maximum memory footprint of the fully occupied GPM can be compressed as follows: (1) ViT-S: 66.3MB - 17.8MB (8-bit) / 10.2MB (4-bit); (2) ResNet- 18: 34.27MB - 8.93MB (8-bit) 5.29MB (4-bit); and (3) AlexNet: 22.27MB -> 5.74MB (8-bit) 3M4 (4-bit). These reductions approach the theoretical 4x (8-bit) and 8x (4-bit) limits, with minor deviation due to overhead from storing auxiliary metadata (S), z Ol) and full-precision outliers. Additional experimental setup and hyperparameter details be found in Appendix A.
Runtime Analysis. Figure 2(b) reports normalized training time under matched memory budgets for AlexNet, ResNet- 18, and ViT-S. Sparse sketching is applied GPM-FP and 8QGPM but not GPM-FP. SGD is included as a lower bound, since does not incorporate any continual learning mechanisms. On AlexNet and ResNet-18, 18, GPM-family models achieve the lowest training time, as their projection operation is cheaper than the additional forward/backward passes and distillation losses required in rehearsal-bacee based baselines. The quantiaaion/dequanizatii is negligible, as shown by the similar runtimes of 8QGPM_S and GPM-FP s. For ViT-S, however, GPM-FP exhibits a significant slowdown because the SVD on large representations of transformer blocks dominate training cost. Applying sparse sketching mitigates this bottleneck, accelerating both SVD and overall training.
overhead in QGPM
Table 2: Effect of quantization bitwidth on QGPM.
Table 4: Effect of QEA scaling factor Aang. ACC, and BWT. Moderate a improves performance by relaxing or- thogonality based on quantization emror
bit
Cave
Cmee
ACC(%)
BWT
4-bit
0.544
1.173
25.03
-31.27
5-bit
0.126
0.254
38.97
-22.94
b-bit
0.033
8.bit
0.004
0.011
65.01
0.61
Xang
ACC(%)
BWT
FP
0.59
SGD HP
59.77
-18.88
65.13
-1.07
Table 3: Effect of outlier proportion on QGPM.
1.0
44.20
-20.69
Cavg
Emaz
ACC(%)
BWT
40
10
0.962
0.849
58.73
40.54
-25.44
-8.6
0.0
0.573
1.205
41.23
-24.00
60
0.775
65,58
-3,87
0.434
0.650
49.71
-16.29
0.699
67.26
-2.91
0.401
0.553
50.26
-15.84
100
0.622
66.59
4.24
1.5
0.373
0.485
55.02
-11.76
150
0.433
65.91
-5.79
0.353
55.36
-11.07
65.17
-7.28
8

---

Published as a conference paper at ICLR 2026
PRIS3S
10
Triinig Time (Hormalizenl
CPAER
MODOL
EIP ROCAA
0.4
sccome sesers secone aycort sead
accemve sccone
all st re comee radke
accone spcces sigored Padia
VIT-5
Alexleet (b)
ResNet18
Figure 2: (a) Memory profie of QGPM vs. full- full-precision GPM. QGPM approaches the 4x (8-hit) and 8x (4-bit) compression limits with only minor metadata overhead, yielding substantial savings across all architectures. (b) Normalized training time under matched memory budgets. QGPM adds negligible overhead compared GPM-FP, and sparse sketching mitigates the ViT-S bottleneck by accelerating SVD operations.
4.2 EFFECT OF BITWIDTH AND OUTLIERS ON QUANTIZATION FIDELITY
We examine how quantization bitwidth and the outlier proportion influence error characteristics and, ultimately, QGPM performance by analyzing experiments 10-split CIFAR-100. To this end, introduce two metrics to quantify quantization error: average quantization error across all stored bases (Eavg) and average per-layer maximum error (Emex), defined respectively as:
L
M
L max IsiSiMa.
Eang
Et M OT
Σ 1=1
Emaz
4
(15)
1=1
Effect of quantization bitwidth. Table reports Eaug. Emaxs ACC, and BWT across different bitwidths. To isolate bitwidth effects, disable QEA projection (a 0) and set the outlier proportion to zero (p = 0). As expected, larger bitwidths substantially reduce both error metrics, with accuracy converging full-precision levels. A sharp performance drop occurs below bits, caused by increased projection distortion and resulting gradient drift.
Effect of outlier proportion. In Table fix the quantization to 4-bit with 0 while varying Even with low Cang. a single poorly quantized basis vector (large emaa at 0) can destabilize training by inducing gradient drift. Introducing a small outlier proportion (e.g.. p 0.5%) markedly reduces eaoz and improves performance relative to the zero-outlier case.
4.3 EFFECT OF QEA PROJECTION ON QUANTIZATION ROBUSTNESS
To counteract guantization error accumulation, QGPM integrates Quantization Aware (QEA) projection, which adaptively relaxes orthogonality based on quantization fidelity. We study its effect by varying the error scaling factor a, which controls the strength of the orthogonality constraint. Table 4 reports performance where QEA has the most pronounced impact. All other hyperparameters match those used in Tables and 3. For each basis vector, the orthogonality weight is A; emror from Eq. 10. We also report the average of orthogonality weight across all basis vectors,
function of a, with the outlier proportion fixed at p 1%, setting
aeg where is the quantization
L
IM
Aoeg EL Q.
Σ 1=1
X
(16)
Given a fixed average quantization error (e.g., Cavg 0.376 set by the bitwidth and p), increasing a lowers Aoug. thereby relaxing the orthogonality constraint and permitting greater flexibility in gradient updates. This flexibility compensates for subspace distortion: consistently improves, showing that the parallel gradient component offsets quantization error. However, beyond a threshold, performance declines due to over-relaxation, which increases inter-task interference. Thus, a must be calibrated to match the severity of quantization. In practice, larger a values are required under higher quantization error, e.g., Table 9 in Appendix A.3 shows that 4QGPM
rises from zero, accuracy

---

Published as a conference paper at ICLR 2026
EOGPN
DOG hargings
GPM /veoting
46.0
$1.
15.0
72
c- 055
c-
0-
17.5
36.0
63
DWT
-25
SWT
-10
-20
195
3.22
11.09
24.87
92.33
3.91
Memory [MB]
Memory [MB]
(4)
(b)
[G]
Figure 3: Memory accuracy tradeoffs of QGPM. (a.b) QGPM consistently maintains full-precision level performance, outperforming the ER baseline in the lower-memory regime ViT-Tiny and ViT-B/16. (c) accuracy curve with varying value on VÝT-S. QGPM performance remains stable e decreases; 4QGPM moderate achieves
efficiency, while maintaining full-precision level performance.
generally uses larger than 8QGPM. Overall, QEA proves most beneficial under aggressive quantization settings, where error-aware relaxation can significantly stabilize learning.
4.4 MEMORY-ACCURACE TRADEOFFS IN QGPM
The memory characteristics QGPM depend on two factors: (i) the embedding dimension (i.e., model size) and (ii) the threshold e (i.e., subspace approximation fidelity). We utilize the 10-split ImageNet-R (Hendrycks et al., 2021) benchmark effectively evaluate models of varying sizes, including ViT-Tiny, VīT-S, and ViT-B/16.
Effect of model size. We evaluate QGPM on ViT-Tiny (192-dim) and ViT-B/16 (768-dim) using 10-split ImageNet-R, with ER (Rolnick et al., 2019) representative rehearsal baseline (Fig. 3(a) and (b)). GPM memory scales sharply with model size, as dimensionality matches the embedding dimension of the transformer. For example, fully populated GPMs reach 34MB for ViT-Tiny and 515MB for ViT-B/16. On smaller models, even GPM-FP outperforms ER under equal memory. On larger models, however, ER surpasses GPM-FP at 92.59MB. By contrast, QGPM compresses memory to as little as 14.4MB while still outperforming ER, with negligible accuracy loss. This validates QGPM's effectiveness in scaling large models under tight memory budgets.
Effect of threshold c. Figure 3(c) shows performance as c decreases from 0.85 in steps of 0.03, evaluated ViT-S. Smaller yields coarser subspace approximation (fewer bases), reducing QGPM size but potentially increasing forgetting. Performance remains largely stable across wide range of e, suggesting that memory be reduced further with modest trade-offs. Notably, 4QGPM with moderate is often superior to 8QGPM or GPM-FP with an aggressively small e.
5 CONCLUSION
We introduced QGPM, scalable and memory-efficient framework for continual learning that integrates: (i) distribution-aware quantization of gradient subspaces, (ii) Quantization Ewaor-Aware (QEA) projection that adaptively relaxes orthogonality constraints, and (iii) On-the-Fly Sparse Sketching reduce memory and computation during training. Theoretical analysis and extensive experiments demonstrate that QGPM achieves strong performance under tight memory budgets, consistently outerforming prior methods both accuracy efficiency. These results position QGPM a practical and scalable solution for memory-efficient, privacy-preserving continual learning.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Ari S. Benjamin, David Rolnick, and Konrad P. Körding. Mcasuring and regularizing networks in function space. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htt.httts////peeeeieeee Lis-uumidd--
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33: 15920-15999, 2020.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: //openreview.cooommmmor/o net htr.enrreeieeenee/ouummttaaa fidimmmiiddiiiiissF_eFFF
Huancheng Chen, Jingtao Li, Weiming Zhuang, Chen Chen, and Lingjuan Lyu. Replay-free continual low-rank adaptation with dynamic memory. 2024. URL https: //api//ppppp semanticschole org/Corpus 273798055.
Yoojin Choi, Mostafa El- El-Khamy, and Jungwon Lee. Dual-teacher class inssemeacre learning with data- free generative replay. 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 13388,-57,-20 2021.
Jaeyoung Chung, Kanggeon Lee, Sungyong Baik, and Kyoung Mu Lee. Meil-nerf: Memory-efficient incremental learning of neural radiance fields. CoRR, abss/2212.08328, 2022. doi: 10.48550/ ARXIV.2212.08322.. URL https:/ pestps://doks //wii 48550//....5/////
221...2221... 08328.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. Advances in neural information processing systems, 36:10088-10115, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representattinss, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: http:::///ppeneeiieeee net forum? id= Yi YicbFaNTTy.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning. In Computer Vision-ECCo 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 386-402. Springer, 2020.
Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cedric Archambeau. Memory efficient continual learning with transformers. Advances in Neural Information Processing Systems, 35: 15:19929-10642 2022.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In Silvia Chiappa and Roberto Calandra (eds.). The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pp. 3762-3773. PMLR, 2020. URL http: http:://poocceiineeeoo www.nnn.......................ww mmrr. press v108/ fara aaaraattabaraaaa html.
Rui Gao and Weiwei Liu. Ddgr: Continual learning with deep diffusion- based generative replay. In International Conference on Machine Learning, pp. 10744-10763. PMLR, 2023.
Leon Harter. Expected values of normal order statistics. Biometrika, 48(1/2):151-165 1961.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
11

---

Published as a conference paper at ICLR 2026
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. 2017.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international 2021.
conference on computer vision, pp.
Ahmet Iscen, Jeffrey O. Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficiet in- cremental learning through feature adaptation. In European Conference on Computer Vision, 2020.
Jörn- Jörn-Herrik Jacobsen, Arnold W. M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In 6th International Conference on Learning Representations, ICLR 2018. Vanconver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018, URL https://openreval
net. Forum?id=HJs. kkMbOZ.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinsker et al. Overcoming catastrophic forgetting in neural networks. Proceedings the national academy of sciences, 114 (13):3521-3526, 2017.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory G, Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:3366-3385 2019.
Yan-Shuo Liang and Wu-Jun Li. Adaptive plasticity improvement for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7816- 7825, 2023.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
23638 23647, 2024.
Sen Lin, Li Yang. Deliang Fan, and Junshan Zhang. Beyond not-forgetting: Continual learning with backward knowledge transfer. Advances in Neural Information Processing Systems, 35: 16165-16177, 2022.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Kanti Mardia. Measures of multivariate skewness and kurtosis with applications. Biometrika, 57: 519-530, 1970.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology Learning and Motivation, 24:109-165, 1989.
Sylvestre-AIvy Alvise Rebuffi Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, fas/1600.046611 2016. URL http: / arxiv. org/ abs /1666. 046.1166666..... 04671.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:/ /ttp////pp///eiee/e////e /ttttps////peeeeeeeee 3A0JORCNC2.
net forum?
12

---

Published as a conference paper at ICLR 2026
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pp- 4548 4557. PMLR, 2018.
Joan Serrà, Dídac Surís, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, 2018.
Samuel S. Shapiro and M. B. Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52:591-61 -611, 1965.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1 409. 1556, 2014.
James Smith, Yen-Chang Hsu, John C. Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming: A new approach for data-free class-incremental learning. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9354-9364, 2021.
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou. Three things everyone should know about vision transformers. In European Conference Computer Vision, pp. 497-515. Springer, 2022.
Tanvi Verma, Liyuan Jin, Jun Zhou, Jia Huang, Mingrui Tan, Benjamin Chen Ming Choong, Ting Fang Tan, Fei Gao, Ximxing Xu, Daniel S Ting, al. medical image classification: a comparative analysis. Frontiers in Medicine, 10:1227515, 2023.
continual learning methods for
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Enneng Yang. Shen, Zhenyi Wang. Shiwei Liu, Guibing Guo, and Xingwei Wang. Data aug- mented flatness-aware gradient projection for continual learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5630-5639, 2023.
Davis Yoshida. NF4 isn't i information theoretically optimal (and that's good). CORR, abs/2306.06965. 2023. doi: 10.48550/ARXIV.20666600 2366.. URL /tttps::/doi..rrr http://0..055///aaaa//. 2306. 06965.
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing neural networks. Nature Machine Intelligence, 1(8):364-33 372, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of machine learning research, 70:3987-3995, 2017.
Yuqing Zhao, Divya Saxena, and Jiannong Cao. Memory-efficient domain incremental learning for internet of things. Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, 2022.
Da-We Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory -fficceeef class-incres incremental learning. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kiguli, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: /awppeveeeeeeeeeeeee/ee/eeeeeeee ew net forum? d= fosss07fAAAe feA QHgM.
13

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS
A.1 DATASETS
Tables and 6 summarize the dataset configurations used the main experiments. To match the memory constraint of QGPM, the number of samples in the replay buffer is adjusted accordingly. Here, buffer size indicates the number of replay images stored in the buffer. For CIFAR-100, the memory overhead of QGPM is 0.81 which is equívalent to 264 training samples. Therefore, the buffer size is set to 280. For the 5-Datasets benchmark, we assume all five datasets are equally represented in the buffer. This results an effective memory usage of 1.7KB per sample. To match QGPM's ' memory constraint of 7.24MB, total of 4,259 samples can be stored in the buffer. For minilmageNet, to match the memory overhead of 2.59 and 2.88MB, 128 and 143 samples are used. Table 5: Dataset statistics of CTFAR10M and minilmageNet.
Split CIFAR-100 Split-minilmageNet
num. of tasks
10
20
input size (C'x H)W)
3 32 x 32
3 84 x 84
# Classes/task
10
10
# Test samples/task
1,000
500
Memory per sample
3,072KB
21.168KB
Table 6: 5--Datasets statisties. MNIST images are replicated across all RGB channels that each image has channels.
CIFAR-10
MNIST
SVHN
Fashion-MNIST
notMNIST
Classes
10
10
10
# Training samples
47,500
57,000
69,595
57,000
16,011
Validation samples
2,500
3,000
3,662
3,000
842
# Test samples
10,000
10,000
26,032
10,000
1,873
Memory per sample
3.073KB
0.785KB
3.073KB
0.785KB
0.785KB
A.2 MODEL ARCHITECTURES
The model architectures for the 5-layer AlexNet variant and the 20-layer ResNet-18 variant follow those used in Saha al. (2021). All networks use ReLU activations and softmax layer combined with cross-entropy loss at the final classification layer. AlexNet and ResNet-18 are trained from scratch.
For ViT, we adopt a ViT-S, a variant of ViT proposed in Dosovitskiy et al. (2021), with a 384- dimensional patch embedding, 6 transformer blocks forming total of 25 learnable layers and 6 attention heads. The MLP expansion ratio within each attention block is set to 4, and dropout is applied with a probability of 0.1. ViT-S uses pre-trained weights trained on ImageNet-2Ik. It takes an input image size of 224 x 224 with patch size of 16. Thus, each input image is resized to 224 x 224.
Table 7 shows the fully occupied GPM configuration at each backbone and the corresponding full precision memory overhead.
A.3 HYPERPARAMETER SETTINGS
In this section, we organize hyperparameters used in the experiment in Section In the experiments with AlexNet and ResNet- 18, learning rate of 0.1 is used for the first task and 0.01 is used for all subsequent tasks. For the experiment with AlexNet, 100 training epochs are used. For the ResNet18, 50 training epochs For the ViT-B/16, a 0.005 learning rate used with 20 epochs. In all experiments, SGD used as an optimizer with 64 mini-batch size.
used. For the ViT-S and ViT-Tiny, 0.01 learning rate is used with 10 epochs.
In the Table neetood-specifie hyperparameters are introduced. 6t is threshold value of k-rank approximation. is the QEA scaling factor for the QEA gradient projection. All experiments were
14

---

Published as a conference paper at ICLR 2026
Table 7: Maximum size of GPM each layer and the number of GPM parameters for each network.
Network
Size of maximum M
Memory in Bytes
48 48, 576 x 576, 512 x 512, 1024 x 1024,
AlexNet
22.28MB
2048 2048
27 x 27, 180 x 180, 180 x 180, 180 X 180,
180 x 180, 180 180, 360 360,
20 x 20, 360 x 360, 360 x 360,
ResNet18
360 x 360, 720 720, 40 x 40,
34.27MB
720 x 720, 720 x 720, 720 x 720,
1440 1440, 80 x 80, 1440 x 1440,
1440 x 1440
768, 384x384, 384x384, 384x384, 1536x 1536,
384x384, 384x384, 384x384, 1536x 1536,
384x384, 384x384, 384x384,
1536x 1536,
VIT-S
66.37MB
384x384, 384x384, 384x384,
1536x 1536,
384x384, 384x384, 384x384, 1536x 1536,
384x384, 384x384, 384x384, 1536x 1536
768 x 768, 768 x 768, 768 x 768, 768 x 768,
3072 3072, 768 768, 768 x 768, 768 768,
768 x 768, 3072 x 3072, 768 x 768, 768 x 768,
768 x 768, 768 768, 3072 3072, 768 768,
768 x 768, 768 768, 768 x 768, 3072 x 3072,
768 x 768, 768 x 768, 768 x 768, 768 x 768,
ViT-B/16
515.25MB
3072 3072, 768 768, 768 x 768, 768 768,
768 x 768, 3072 x 3072, 768 x 768, 768 x 768,
768 x 768, 768 768. 3072 x 3072, 768 768,
768 x 768, 768 x 768, 768 x 768, 3072 x 3072,
768 x 768, 768 768, 768 768, 768 x 768,
3072 x 3072
Table 8: Model parameters and GPM size comparison.
Network
Category
Memory in Bytes
Model Param. size
26. 1MB 22.28MB
AlexNet
GPM max size
Model Param. size GPM_max
4.4MB
ResNet-18
max size
34.27MB
Model Param. size GPM_max size
42.18MB
ViT-S
66.3MB
Model Param. size @PM_maax size
21.15MB
ViT-Tiny
34MB
Model Param. size GPM max size
327.59MB
VIT-B/16
515MB
conducted on single AMD Vega 20 GPU.
B GRADIENT PROJECTION MEMORY (GPM)
15

---

Published as a conference paper at ICLR 2026
Table Hyperparameterss used for all baselines.
Methods
Hyperparameters
GPM-FP
Eu:: 0.9 (10 cifar)00), 0.93 (5datasets), 0.93 (miniimagenet)
GPM-MC
cch: 0.7 (10 cif 0. 0.79MB), 0.75 (5 datasets, 8.36MB), 0.52 (10 miniimagenet, 2.66MB), 0.51 (20 minimagenet,, 2.88MB)
AGEM-MC
Buffer size: 280 (10 cifar 100), 4259 (5datasets), 128 (10 miniimagenet), 143(20 miniimagenet)
DER++MC
Buffer size: 280 (10 cifar 100), 4259 (5datasets), 128 (10 minimagenet), 143 (20 minimmgennet
weight for MSE loss on replay logits: 0.1, weight for CE loss on replay samples: 0.5
ER-MC
Buffer size: 280 (10 cifar 4259 (5datasets), 128 (10 minimmgennet.. 143 (20 miniimagenet)
FDR-MC
Buffer size: 280 (10 cifar100), 4259 (5datasets), 128 (10 miniimagenet),
143 (20 minimmgenet) weight for FDR replay loss: 0.6
8QGPM
a: 10 (cifar100), 10 (5datasets), 5 (10 miniimagenet), 5 (20 minimagenet) outlier: 1% (cifar 100), 1% (5datasets), 1% (10 miniimaenntt, 1% (20 miniimagenet)
4QGPM
a: 20 (cifar100), 20 (5datasets), 15 (10 miniimagenet), 15 (20 miniima- genet)
outlier: 3% ccifar100), 3% (Sdatasets), 2% (10 miniimagenet), 2% (20 miniimagenet)
Algorithm 2 On-the-Fly Sparse Sketching
1: Let R. be the representation matrix and be the i-th representation vector having n dimension.
Let's assume that there are N numbers of representation vector and they arrive one by one.
2: R. < O axn
3: for Vi E (1,2....., N do
4: seed f-
5: idx < handom_scleddn (1, ........... n,, seed)
6: sign t Random Random_scleaat( (olecc((-11,+10) +1,, seed)
7: R|E ...dd) +- RR: idæ) + sign.
8:
end for
We now provide detailed explanation of the Gradient Projection Memory (GPM) scheme Saha et al. (2021). In continual learning, aim minimize the average loss over sequence of tasks; however, without access to earlier data, models tend to forget previous tasks as they learn new ones. GPM combats this catastrophic forgetting by projecting gradients from the new tasks onto the subspace orthogonal to past task representations, without any data replay.
Let w C w. denote I-th layer parameters trained task and R. be the input representation layer at that point. The output activation of layer / after training on task 7 is given by w - We would like this activation remain unchanged even after learning task T + i.e., w - w R = (w- + Aw") R.. This requires that the weight update Aw satisfies the orthogonality condition Aw R 0. To enforce this constraint, the learner stores a set of core basis vectors that span the subspace R', and restricts future gradient updates to lie in the orthogonal complement of this subspace. We denote this memory (i.c.. GPM), for layer after task 7 by M..
Algorithm outlines continual learning procedure using GPM. For each task r, the learner: (1) performs projected gradient descent using the existing basis memory (M}-,} to remove interference (Lines 1-7); (2) gathers the layer-/ activations R., and orthogonally project components
16

---

Published as a conference paper at ICLR 2026
64
32
14
Activation patch
32
Figure Feature map of AlexNet and activation patch
Algorithm GPM Algorithm
Input: fw the NN model, Drrain the training dataset, the learning rate, and ézi. the threshold value. Initialize, My for all / = ........ T,', and w <- Wo-
1: for 1,2..... do
2: repeat
3:
Bn ~ Dirain
4:
V W.
5:
g g M (M²-1)
6:
17 g
7:
until convergence
8:
9:
BR, Dirain
10:
(R <- Bewward(Bn,.ll
11:
for 4/ E (1,2...., L'} do R$ R M (M)-)) T. R
13:
14:
U.. E. V (VD(R.)
15:
+ criteria R Rt, end
M [M--1. [1[1: :]]
17:
end for
18:
end for
already captured by M (Line 13); (3) applies SVD to the residual and retains its top singular vectors the task's core basis (Lines 14-15); (4) concatenates these new bases with M to form M' for future tasks (Line 16).
C ON-THE-FLY SPARSE SKETCHING
C.I REPRESENTATION MATIX CONSTRUCTION AND ON-THE-FLY SPARSE SKETCHING
To build the activation (representation) vector r; for a convolutional layer, treat each receptive-fiele patch (activation patch) that will feed the next kernel
training example: For every image in the mini-batch and every sliding-windon position (u, v). we slice the feature map tensor E IR C into the cuboid patch Pu,v X[: u, + k is the kernel size. By flattening this patch in channel-ma -ajor order, we obtain length-n vector where n k.2 We generate this flattened feature vector over all mini-batches and the spatial dimension of the feature map. Thus, the total number of feature vectors is N = s2. bsz, where is the size of the feature map and bsz the number of samples in the mini-batch. Thus, we construct a feature matrix without sparse sketch, the size of this matrix is (k2 C..) xx (x2 bsz), The resulting representation matrix size for each layer constructed in this way is shown in the Table 10.
+ k). where Cnn the number of channels and
17

---

Published as a conference paper at ICLR 2026
Table 10: The size of the representation matrix for each layer without sketching.
Network
Size of input representation
Memory in Bytes
48 20184, 576 x 14400, 512 x 2500, 1024 x 125,
AlexNet
41 41M46MB
2048 x 125
27 17640, 180 17640, 180 17640, 180 17640,
180 x 17640, 180 x 4410, 360 x 4410,
20 4410, 360 4410, 360 22050,
ResNet18
360 x 6050, 720 x 6050, 40 x 6050,
258.65MB
720 x 12100, 720 x 12100, 720 x 3600,
1440 x 3600,
80 x 3600, 1440 x 3600,
1440 x 3600
768 x 25088, 384 x 25216, 384 x 25216, 384 x 25216,
1536 x 25216, 384 25216, 384 x 25216, 384 25216,
1536 x 25216, 384 x 25216, 384 x 25216, 384 x 25216,
ViT-S
1536 x 25216, 384 25216, 384 x 25216, 384 25216,
1.62GB
1536 x 25216, 384 25216, 384 x 25216, 384 x 25216,
1536 x 25216, 384 25216, 384 x 25216, 384 25216,
1536 x 25216
768 25088, 768 25216, 768 25216, 768 x 25216,
3072 x 25216, 768 25216, 768 x 25216, 768 x 25216.
3072 x 25216, 768 25216, 768 x 25216, 768 25216,
3072 x 25216, 768 x 25216, 768 x 25216, 768 x 25216,
3072 x 25216, 768 25216, 768 25216, 768 25216,
3072 x 25216, 768 x 25216, 768 x 25216, 768 x 25216,
ViT-B/16
6.13GB
3072 x 25216, 768 25216, 768 x 25216, 768 25216.
3072 x 25216, 768 x 25216, 768 x 25216, 768 x 25216.
3072 x 25216, 768 25216, 768 x 25216, 768 x 25216,
3072 x 25216, 768 25216, 768 x 25216, 768 x 25216,
3072 x 25216, 768 25216, x 25216, 768 x 25216,
3072 x 25216
Algorithm 2 outlines the On-the-Fly sparse sketching procedure, where N 32 bas representation vectors arrive sequentially. Each incoming ri-dimensional vector (where generating a random Rademacher sign and target column index uniformly sampled from (1..... n). The signed vector is then accumulated into the corresponding column of the sketch matrix R. Table 11 reports the size of the resulting representation matrix with sparse sketching. For example, in the case of ViT-S, the naive construction of the representation matrix incurs a memory overhead of 1624MB before performing SVD on it. Our method reduces this requirement to 66.37MB, achieving 24.46x saving. Note that this intermediate memory reduction (e.g., GPU RAM usage) and not permanent memory (e.g., Flash memory) reduction.
cC.)) ) is processed by
Figure 5 illustrates RAM usage across the ten-task training sequence on the minilmageNet benchmark using ViT-S. Table 12 reports the total training time, average GPM construction time (in seconds), average Random Access Memory (RAM) usage (in MB), peak RAM usage, and the final performance metrics (ACC and BWT). Without sketching, the average memory overhead is approximately 4533MB. By employing sparse sketching, this can reduced to 3277.09MB .09MB without any degradation in performance. The GPM construction time (which includes the SVD the representation matrix, r-rank approximation, and quantization) can be reduced from 115.06s to 10.60s.
Compared to dense Gaussian random projections, On-the-Fll sparse sketching method offers greater computational and memory efficiency. Each incoming activation vector is processed in O (nnz(R.) time, using only sign flipping and bucketed additions with two light-weigh hash functions, rather than a full OnnN) matrix- vector multiplication. Here, nnz(R.) denotes the number of non-zero columns in R.
18

---

Published as a conference paper at ICLR 2026
Table 11: The size of the representation matrix for each layer with sketching.
Network
Size of input representation
Memory in Bytes
48 x 48, 576 x 576,
AlexNet
512 x 512, 1024 x 125,
3. 3.72MB
2048 125
27 27, 180 x 180, 180 180, 180 180,
180 x 180, 180 x 180, 360 360,
20 20, 360 x 360, 360 360,
ResNet18
360 x 360, 720 x 720, 40 x 40,
34.27MB
720 x 720, 720 x 720, 720 720,
1440 x 1440, 80 x 80, 1440 x 1440,
1440 x 1440
768x768, 384x384, 384x384, 384x384, 1536x 1536,
384x384, 384x384, 384x384, 1536x 1536,
384x384, 384x384, 384x384,
1536x 1536,
VIT-S
66.37MB
384x384, 384x384, 384x384,
1536x 1536, 384x384,
384x384, 384x384,
1536x 1536, 384x384,
384x384,
384x384,
1536x 1536
768 x 768, 768 x 768, 768 x 768, 768 x 768,
3072 3072, 768 768, 768 x 768, 768 768,
768 x 768, 3072 x 3072, 768 x 768, 768 x 768,
768 x 768, 768 768, 3072 3072, 768 768,
768 x 768, 768 x 768, 768 x 768, 3072 x 3072,
768 x 768, 768 768, 768 x 768, 768 768,
ViT-B/16
515.25MB
3072 x 3072, 768 x 768, 768 x 768, 768 x 768,
768 x 768, 3072 x 3072, 768 x 768, 768 768,
768 x 768, 768 x 768. 3072 x 3072, 768 x 768,
768 x 768, 768 768, 768 x 768, 3072 x 3072,
768 x 768, 768 x 768, 768 x 768, 768 x 768,
3072 3072
Empirically and theoretically, sparse sketch maintains &2 norms and leading eigen-directions within Lannson-Lindenstrauss---- bounds, offering subspace quality comparable dense Gaussian projec- tions. Thus, it provides compelling trade-off between accuracy and efficiency, making it especially well-suited for real-time, resource-constraing continual learning settings.
Table 12: Resource usage comparison with and without sketching.
Without sketching
With sketching
Methods
ACC (%)
BWT (%)
ACC (%)
BWT (%)
8QGPM
72.21
-0.54
72.87
-0.34
GPM construction time [s]
115.06
10.60
Total training time [s]
1768
820
Average RAM consumption [MB]
4533.50 MB
3277.09 MB
Peak RAM consumption [MB]
5603.06 MB
3586.50 MB
D MIXED-SCHEME QUANTIZATION AND DE-QUANTIZATION
D. MIXED-SCHEME QUANTIZATION
The CINF quantization scheme performs well when the input distribution is approximately Gaussian. However, it is important to note that the input activation to the very first layer (i.c., the raw image patch) typically exhibits GaussGa distribution. As this effect propagates, the activations in the
19

---

Published as a conference paper at ICLR 2026
Sketching True
5500
Sketching False
5000
Memory (MB)
4500
4000
3500
3000
2500
0.0
0.4
0.6
1.0
Training Progress (%)
Figure 5: Training-time time process RAM usage of 10-split minilmageNet experiment on ViT-S.
early layers also tend to deviate from Gaussianity, particularly during the initial stages of training when model parameters not yet well-formed. This non-Gaussian behavior can result in significant quantization errors when CINF is applied indiscriminately across all layers. To address this concern, we propose mixed-scheme quantization strategy where we apply affine quantization in the first layer by default, and conditionally in the second and third layers based on a normality test. The test combines two statistics: kurtosis x the input vector Mardia (1970) and p-value psw from the Shapiro-Wii Wilk Test Shapiro & Wilk (1965). For deeper layers, where representations tend to become more Gaussian due to the central limit effect, we default to CINF quantization. Formally, given an input matrix U [ui,..., Ur], the mixed quantization function deployed at the second and third layers can be defined as
Q @Attime (u),
k(u) < A psw(u) <
Quant(U[ r)) [quant (u; (u))]= quant(u)
QCINF (u), otherwise.
(17)
Definition of kurtosis and p-value of Shapiro-Wiin Wilk Normality are given below.
Definition D.1 Kurtosis: Kurtosis measures the "tailedness" of the probability distribution. quantifies whether the data distribution has heavier tails (positive kurtosis) or lighter tails (negative kurtosis) compared to Gaussian distribution. The kurtosis of a dataset is given by K = E((X /// 3, where μ the mean and the standard deviation of the dataset.
Definition D.2 Shapiro- Wilk Test of Normality: The p-value obtained from the Shapiro- Wilk test quantiifes the likelihood of the data distribution being Gaussian. Formally, given a null hypothesis Hu (the dataset is Gaussian) and an alternative hypothesis H2 (the dataset is not Gaussian), the p- p-value measures the probability of observing data at least as extreme as the current dataset under the assumption normality. Formally, p-value P(W Wabs Ho) where W is the Shapiro- Wilk test statistic. low p-value (lypically less than a threshold e) suggests rejecting Ho. indicating that the data significantly deviates from a Gaussian. The statistics can be defined
aj ((i)
W
(x: F)
for sample of size with ordered observations T(1) E Din) and sample
20

---

Published as a conference paper at ICLR 2026
((x)
$(x)
15456
1
22
16 eve spoced
even spaced
-1
5.57
123
2
0
NF4 Constmuction
CINF4 Construction
aa)
bb)
Figure 6: (a) NF4 codebook construction process; (b) 4-bit CINF construction process
Mixed-scheme dequantization. In equation equation 11, gradient projection performed in full precision. Therefore, the quantized GPM must first be dequantized to reconstruct the full-precisio projection matrix P PT Recall that each basis vector stored in the GPM is quantized using either the CINF Affine scheme. Each scheme requires distinct set of scalar values for dequantization and projection: (s, μl, A₂) the case of CINF, and (A, 2, A) in the case of Affine. To handle the mixed quantization setting, QGPM separates the basis vectors into groups based their quantization scheme and dequantizes them separately. Formally, for E {Afline, CINF) we find
M - é (aguuntt,MM', St 1 2,,1, O O4_1,)
(18)
P 1 1,a = M A A411 (MC 1,))
(19)
and then compute the final projection matrix as P p P
AAffiie P 1-,,INFF
E CENTERED INLIER NORMAL FLOAT
E CODEBOOK CONSTRUCTION
Here, we provide details that distinguish our CINF code- book from NF4. The original NF4 inserts exact zero value into the codebook to enable lossless quantization of padding other vero-valued elements. This design choice is appropriate for quantizing model parameters or activations, where exact zeros commonly arise due to operations such zero-padding.
Figure 7: Actual CINF4 codebook.
In contrast, our quantization target is the singular vectors Ü. i] obtained from singular value decomposition (SVD), where such zero-generating operations do not exist. Therefore, we omit the inclusion of an exact zero in the codebook. In the following, we present the codebook construction procedures for both NF4 and CINF4.
NF4 Codebook construction
1. Set 8 = ( )
2. Compute evenly spaced probability values P1, .P.... such that P1 6, 1//2,
3. Find their corresponding quantile values under Normal Cumulative Density Function (CDF):
q = + (p:) for 0, I,..,
4. Conpute 9 evenly spaced probability values Ts, Pg. ..., T1e such that rs = 1/2 and 7'6
1-60 8.
21

---

Published as a conference paper at ICLR 2026
5. Find their corresponding quantile values under Normal CDF: q. ( for 9, 10, 16.
6. Normalize the & to the range [-1, I] to get final code: qs = 9./ max; [
CINF4 Codebook construction
Compute 16 evenly spaced probability values P1,22,.. ...,, ...... such that Pi
15.5/16, P16 = 0.5/16.
2. Find their corresponding quantile values under Normal Cumulative Density Function (CDF):
1 + -p:) for 0, 0.......
3. Normalize the q to the range [-1, I] to get final code: qi = Qs/ max; |0:
Figure 6 visualizes the aforementioned process of equal spacing Normal CDF, and Figure shows the actual CINF4 codebook: [-1.000, -0.707, -0.542, -0.416, -0.310, -0.215, -0.127, -0.042, 0.042. 0.127, 0.215, 0.310, 0.416, 0.542, 0.707, 1.000].
F ADDITIONAL EXPERIMENTS
F.1 EXPERIMENTS WITH EMPTY VIT
Table 13: ACC and BWT for 8QGPM and baselines on the empty ViT model
10-split minilmageNet
Methods
ACC (%)
BWT (%)
Memory GPM-FP
1.87
48.47
-3.49
Memory
0.52 MB
GPM-MC
40.64
-15.23
AGEM-MC
23.07
-37.00
DER++MM
26.84
-33.24
ER-MC
25.02
-34.64
FDR-MC
24.36
-34.04
PCAOGD-MC"
20.47
-34.48
8QGPM (Ours)
48.39
-3.59
In the main paper, biases such as CNN, which makes them less data efficient and harder to train from scratch, In the case of the continual learning scenario, we divide the dataset into several chunks to make task. Thus, the amount of each task is extremely insufficient to train the VIT model from scratch. In this section, we investigate the continual learning performance of QGPM over other baselines on the empty VIT model, which is trained from scratch. For the experiment, patch embedding, transformer blocks, multi-head attention with 4 MLP expansion ratio, forming 25 learnable layers. Model parameter size is 4.7MB, and fully occupied GPM size is 7.2MB.
ViT-S with pre-trained weight. Typically, VIT lacks strong inductive
custom ViT model, which has 128
The results of Table 13 validate that QGPM maintains full- cell-precision-eeee performance even when ViT is trained from scratch. Notably, QGPM incorporates fewer basis vectors in this setting, resulting a smaller memory footprint compared to the pretrained ViT case in the main paper. As result, the leheersal-based baseline struggles significantly due the severely constrained replay buffer.
22

---

Published as a conference paper at ICLR 2026
F.2 EXPERIMENTS WITH OTHER DATASETS ON RESNET18
Table 14: The results of continual learning with QGPM on minilmageNet using ResNet18.
10-split CIEAR100
10-split minilmageNet
Methods
ACC
BWT
(%)
BWT
SGD
54.74
-26.79
33.21
-2.992
GPM-FP
53.51
Memory
60.32MB
22,94MB
8QGPM (Ours)
70.46
-1.24
52.83
-1.93
Memory
5.41MB
5.99MB
Table 14 presents the experimental results on 10-split CIFAR- 100 and 10-split minilmageNet using ResNet-18, as reported in the main paper. As shown, QGPM consistently maintains near full-precision performance on various datasets on ResNet18.
F.3
EXPERIMENTS ON AN NLP TASK
Table 15: Continual learning result of QGPM with text classification task.
Bit-width
SGD
Full Precision
8-bit QGPM
ACC (%)
(%C
(W)
(CC(%)
BWT
Performance
75.73
-10.38
77.80
-6.15
77.97
-5.97
Memory
266KB
75KB
While the main paper primarily focuses on vision tasks, in this section, we demonstrate that QGPM can be extended to other domains, such as natural language processing (NLP), We use the Yahoo Answers Topics dataset for topic classification task. Each sample consists of a question title (a short string summarizing the question) and question content, which provides longer, more detailed description. The label an integer range from to 9, representing one of ten coarse-grained categories (e.g., Hcalth, Sports, etc.). The dataset contains 1,400,000 training samples and 60,000 test samples. While text lengths vary across samples, they average around 300 words. For the continual learning setup, split the ten classes into five tasks, each containing two classes.
We use a transformer based lightweight encoder for text classification. It consists of:
1. Token embedding of dimension 128
2. Sinusoidal positional encoding
3. 256 MLP hidden dimension
4. 2 transformer block with multi-head self-attention module
5. Redisual connections around both attention and MLP layers with LayerNorm
The encoder consists of total of 7 learnable layers, and the input activation before each layer is extracted. The fully occupied GPM size and the representation matrix size without sketching for this network are shown in Tables 16 and 17, respectively.
We train the encoder using stochastic gradient descent (SGD) with learning rate scheduling. Across all experiments, the mini-batch size, number of training epochs, and initial learning rate are fixed to 64, 20, and 0,02, respectively. For the GPM parameters, the threshold value én is set to 0.93. For 8-bit quantization, QEA scaling factor a and outlier percentage p are set to 10 and 2, respectively; for 4-bit quantization, they are set to 20 and 3.
Table 15 illustrates that QGPM achieves performance comparable to the full full-precision baseline, while significantly reducing memory overhead. As shown in Table 16, the fully occupied GPM-FP has size of 812 KB, which can be reduced to 228 KB and 143 KB using 8-bit and 4-bit quantization, respectively.
23

---

Published as a conference paper at ICLR 2026
Network
Size maximum M
Memory Bytes
128 x 128. 128 x 128,
Encoder
128 x 128, 256 x 256,
0.812MB
(7 layers)
128 x 128, 128 x 128, 256 X 256
Table 16: The maximum GPM size of the Transformer Encoder (Appendix F.3)
Network
Size of input representation
Memory in Bytes
128 10600, 128 10600,
Encoder
128 x 10600, 256 x 10600,
46.5MB
(7 layers)
128 x 10600, 128 x 10600, 256 x 10600
Table 17: The representation matrix size the Transformer Encoder (Appendix F.3) without sparse sketching
F.4 REHEARSAL BUFFER ADJUSTMENT STRATEGY
To ensure fair comparison under identical memory constraints, adjusted the number of rehearsal samples stored the buffer for each rehearsal-based method (e.g., GEM-MC, DER-MC, FDR-MC, ER-MC) that their total memory usage matched that of QGPM. We computed the per-sample memory footprint (i.c., image resolution number of channels x bit depth) and reduced the number for example, constraining 0.8MB the CIFAR-I00 setup. This strategy aligns with prior works on cemmory-constainee continual learning Zhou et al. (2023); Iscen al. (2020) and allows for fair and controlled evaluation.
of stored samples accordingly
Table 18 compares two strategies for reducing replay buffer memory: lowering image resolution vs. reducing sample count. While both methods perform similarly compared to the performance of 8QGPM for small images like CIFAR1OM 100, resolution reduction degrades performance on larger images like minilmageNet due to severe distribution shifts induced by quantization.
As shown in Table 19, even with more samples (280 and 560) on this resolution- based scheme, the performance worsens, leading to the conclusion that resolution-base quantization is impractical even for moderately sized images (e.g.. minilmageNet).
Table 18: Comparison memory and performance across buffer strategies.
Dataset / Model
Metric
DER+ (Full res Reduced #)
DER+ (Reduced res Full #)
8QGPM
Per image [B]
32
1B
3,072
32
0.5B
1,536
buffer
280
280
CIFARI00/ AlexNet
Buffer mem. size [MB]
0.8203125
0.8203125
0.81
ACC
60.68
65.29
70.7
BWT
18.5
13.2
0.81
image mem. size [B]
21,168
10,584
bulter
93
90 x 2 1.86
minilmageNet/ VIT
Buffer mem. size [MB]
1.968624
1.9686624
0.81
ACC
30.01
26.23 23
48.39
BWT
26.77
-34.06
-3.59
Table 19: Buffer comparison under different per-image precisions (minilmageNett ViT).
Metric
8-bit (21 168 B)
4-bit 584
584 B)
Images in buffer
280
560
Buffer size [MB]
5.92
Ace %
34.10
26.06
BWT [%]
-21.6
10
24

---

g. = (I Po) g. g. Pe) g. Published as a conference paper at ICLR 2026
G ADDITIONAL PROOFS
G.1 PROOF OF THEOREM 3.2
Quantization Error Accumulation: Let M, [V1, V2,.2..... Vm R denote full preci- sion GPM, and E' = [E1. $1,62.... Em]] E Raxm be Gaussian error matrix with each column Ej E R is drawn from (00 In)). We define the quantized GPM incorporating enror as Me = Mo + E gradient g E _R" its orthogonal components with respect to M. and M, are denoted by go and ge. respectively. Then, it holds that ES []] [[E[g, B.]l
[v1 + C1,V2 + C2,..., Vm + Cml R= When naive SGD produces
o² ||gll. implying that the quantizaatin error increases proportionally to the number of error-incororaaiin basis vector in the GPM and quadratically with which quantifies the degree of error.
Proof: Let define projection matrices for M, and Me as
Po Mo M
P. M M
The orthogonal component of the gradient with respect to the subspace spanned by Mo and M. is found as
Hence, the difference between the projected components is
go g. [[T Po) (T- - Pc)lg
(Po P.)g
(Mo Ma -M. M }g
(Mo Ma (Mo + E) (Mo + E)²)g
-M + EM EEE )g.
Since the error matrix Gaussian distributed with each column Ej E R drawn from o3),o2nn)
((MME"' ggl = M, g E[E"]= =0
B(E ]gl MM. .g. EEE = 00
E|E ET = E
of = Σ
Eley eg[ = m o²
j=1
j=1
Thus, the expectation of g.
8. can be found as
Elg.
gel - 0².
Since ||[[]]| < [I]
[| g. 80112] 2 |E Ba]ll2 = m 0². ||g//2-
G.2 PROOF OF THBOREM 3.3
(1 + e) l2 Subspace embedding: A mairix S E Brxk is said to be (1 + e) e)ssusspace embedding for the row space of E TRkxx the En-norm if. for all x E RD,
(1=))]AAx|[ < ||AA|| I [1+))
Subspace embedding of Sparse Sketch: Let S be the N by sparse-embedding matrix constructed by h : [N] > [r] and [N] + {-1,1} be hash functions. Then the i-th column of the sparse-embedding matrix non-zero only in the h(i)-th and this non-zero entry has value
25

---

Published as a conference paper at ICLR 2026
of o (i).. Then with probability 1 - 6 for any rank k matrix A and (g2gg) S is a (1 c) - 12 subspace embedding for the row of A.
Before proving the theorem, we state a few definitions and a helpful lemma.
Definition I: We say C is an c-approximate matrix product of A and B if satisfies
A B - CllF S ||
Definition 2: A distribution D on S RNxx is said to satisfy the (c, 5,,5,,1111 JL. moment property if Vx E RN where ||x||2 = L, EIIIS x 1111 S é'd.
Definition 3: For a scalar random variable X, let | :- El which is L norm on the
space (Pt). Minkowski's inequality gives the triangle inequality: ||X + Y S || 1l + ||
Lemma Let 1 2, (0,1/2), and D on be distribution that satisfies the (e, 1) JL moment property. Then for every pair of matrices A, B with N columns,
PrawD A AB Il
3e| IF
[ 88
Proof of Lemma 1:
We first note that for x, E R4, (S x5'x Sty)
(I|STx] x11 + yl2 - ||ST (x y)1. Thus,
[S S y) (x. x>1k = 2 _II x112 x2) + (1I y - y²) (I|S) (x --)l - ||x -11
1
1
$ 2 (IIIS x11 - -x1ll: + ||||S y12 yēll: + ||||ST (x y) - |x - ylšll))
s 1 (88 + t/ + 4c8 I 3/8
where we first apply the triangle inequality and then apply the JL moment property. From this, we can conclude that for arbitrary x,y, [S x, S ST y) < |x|all] Now since the ij-th entry of AB is given by (A B)) the inner product of the i-th column of A and the j-th column of B, we have that
E||AASS B AB
11lll - (3c)²8
Σ
|A
B
11
S (36) 80
|A B 13)
= (3e))5(IAA)) A / |
(3
where the first line follows from the triangle incquality, and the second inequality is from plugging in the inequality derived previously. Now we plug this into Markov's inequality to get that
E|JAASS B AB
Pr[]AASS B AB
T 1 > ||||I
(3c) A BIC
This will be used prove the final theorem.
Proof of the theorem: We want to show that S is the sparse embedding matrix with at least r -3
rows, S satisfies the (c. 6, 2) -J_L moment property. We need to show that for a unit vector x with
||x||| E[ x| 1)3] s 28.. We do this by expanding get E[|[ST 2E]
x12]
the middle term is and from expansion we can show that [[]| x < 1+3, so [[[[]S 1 <
2. Thus if r > 24g. the (c. -LL moment property hold. Let V be an orthonormal basis for the rows of A. Now since S satisfies the (e, 5, 2) JJ moment property.
Pr Ps V > [|V]] 5
Pr
Pr V SS T V - - 3ck < 6
so, with e f. we get = 0
E columns needed.
26


Reviews:
[{"review_id": "R58bZomnYx", "reviewer": "Reviewer_MtXi", "summary": "This paper proposes Quantized Gradient Projection Memory (QGPM), a memory-efficient extension of Gradient Projection Memory (GPM) for continual learning. Traditional GPM stores orthonormal subspace bases to protect previously learned knowledge by projecting gradients onto a constraint subspace. However, its memory cost scales with model size and the number of tasks.\n\nQGPM compresses the stored subspace bases using vector quantization, then dequantizes them only during projection. The authors analyze the quantization-induced distortion and theoretically show that projection error grows with both the number of stored bases and quantization noise. To mitigate forgetting caused by this distortion, the paper introduces Quantization Error-Aware Projection, which adaptively relaxes the orthogonality constraint based on cosine-similarity loss between original and quantized bases.\n\nExperiments across standard continual learning benchmarks demonstrate that QGPM significantly reduces memory consumption while retaining competitive performance against state-of-the-art CL methods.", "questions": "I don't have any questions.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation for memory efficiency in continual learning by reducing subspace storage cost.\n\n2. Theoretical analysis linking quantization noise to projection error, improving understanding of limitations.\n\n3. Empirical results show that the proposed method keeps performance competitive while significantly reducing memory.", "weaknesses": "1. Computational overhead may increase due to repeated dequantization and projection steps, especially for large models.\n\n2. Insufficient ablation study. It would be helpful to see more analysis on different quantization levels and their impact on performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wBwQ2kFMlU", "reviewer": "Reviewer_cPdv", "summary": "This paper proposes Quantized Gradient Projection Memory, a projection-based continual learning framework that compresses stored gradient subspaces via a distribution-aware quantizer, stabilizes learning with a quantization error–aware projection rule, and reduces construction overhead using on-the-fly sparse sketching, achieving near–full precision accuracy with 4–8× memory savings on standard vision benchmarks under matched memory budgets. The work combines clear algorithmic design, theoretical analyses of quantization-induced drift and sketching guarantees, and thorough experiments that show 8-bit QGPM matches GPM while 4-bit QGPM remains competitive with substantial memory reductions, positioning QGPM as a strong, memory-efficient alternative to rehearsal under privacy constraints.", "questions": "1.Could you comment on the memory scaling of QGPM in a long-sequence setting (e.g., 50+ tasks)? While QGPM compresses bases, the number of bases still grows. Is there a crossover point where a fixed-size rehearsal buffer (like ER) becomes more memory-efficient than the growing QGPM?\n\n2.The QEA factor $\\alpha$ seems critical but requires tuning per setup. How sensitive is the method to this hyperparameter? Is there a more principled way to set $\\alpha$ automatically, perhaps based on the observed mean/max quantization error, to reduce the tuning burden?\n\n3.What percentage of the final QGPM memory footprint (e.g., in the 8QGPM column of Table 1) is consumed by the full-precision outliers stored by CINF? Is this cost consistently negligible across different models and bitwidths?\n\n4.The sparse sketching component appears to be a strict win, reducing SVD time and intermediate memory with no performance loss (Table 12). Is this always the case? Does the $(1 \\pm \\epsilon)$-subspace embedding (Theorem 3.3)  ever introduce approximation errors that interact negatively with the quantization errors, or is it robust in all tested scenarios?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.This work proposes a targeted quantization scheme CINF for GPM bases and a QEA projection rule that adapts orthogonality weights via cosine-error, addressing quantization-induced subspace distortion in a principled way.\n\n2.This work provides theoretical insights into why NF scaling fails under heavy tails, shows quantization-induced drift accumulation, and gives sketching guarantees, with ablations that align with the theory and quantify bitwidth/outlier/QEA effects on ACC and BWT.\n\n3.The results are highly valuable. The ability to achieve nearly full-precision performance with 4x-6x less memory makes GPM a much more viable competitor to rehearsal-based methods, especially in privacy-sensitive domains.", "weaknesses": "1.The core GPM framework still accumulates bases with each new task. While QGPM compresses these bases, the total memory still grows monotonically. The experiments are limited to 5, 10, or 20 tasks. It is unclear how QGPM's memory footprint scales in a true lifelong learning scenario (e.g., 50-100+ tasks) compared to a rehearsal method with a fixed memory buffer. A discussion of this scaling trade-off is missing.\n\n2.The method introduces new hyperparameters that appear critical to performance, particularly the QEA scaling factor $\\alpha$ and the CINF outlier proportion $p$. Table 4 shows that performance is highly dependent on $\\alpha$. Table 9 indicates that $\\alpha$ and $p$ are tuned for each dataset and bitwidth. This raises concerns about the practical tuning cost. A more in-depth sensitivity analysis or a more principled method for setting $\\alpha$ (e.g., adapting it based on observed error statistics) would improve robustness.\n\n3.CINF stores outliers in full precision. The paper mentions this contributes minor overhead, but this cost is never explicitly quantified. If the distributions are truly heavy-tailed, storing 1-3% (from Table 9) of vectors in FP32 could be a non-negligible memory cost that offsets the 4-bit/8-bit quantization gains. A brief analysis of this would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aP2dBIKQaC", "reviewer": "Reviewer_FQfP", "summary": "The paper proposes new techniques to quantize the core bases stored for gradient projection method (GPM).The paper first show that the deviation between ideal and quantized orthogonal updates grows quadratically with the quantization error. The paper addresses this using 2 techniques: a) an enhanced quantization approach that improves over the NFk via centered inlier normal float Q aimed at reducing influence of outliers during Q, b) quantization error aware gradient projection to adaptively relax orthogonality constraints based on estimated deviation from desired update direction.", "questions": "What are the assumptions made on the data distribution? How does it impact the quantization performance when the distribution changes? (especially as the paper argues that the continual data has a heavy tail).\n\nWhy is the codebook kept static? What are the drawbacks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The overall approach to enhance GPM with the new quantization approach as well as the use of sparse sketching for updates have sound reasoning and theoretical bounds as shown in the paper.\n- Basis wise quantization makes sense and should have a higher precision and can help retain near orthogonality after quantization\n- Experimental results and ablation studies have good coverage in terms of datasets, models and baselines.", "weaknesses": "The enhanced quantization over NFk makes sense that it is more robust to outlier values. However, there are a few concerns:\n1. Distributional assumptions: codebook is derived from a standard normal. If actual data distribution as pointed out in the paper deviates (especially has skewness), the quantization can be suboptimal\n2. This is amplified because of the codebook being fixed and not being adaptive to input distribution\n3. Compressing tail could also increase error disproportionately for the NF4; in contrast, the CINF4 codebook construction may over emphasize tails", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0cJaPBMV9h", "reviewer": "Reviewer_8rDA", "summary": "Summary and Strengths\nThis paper presents Quantized Gradient Projection Memory (QGPM) framework that combines basis-wise quantization, quantization-error-aware projection, and sparse sketching for memory-efficient continual learning.", "questions": "See above under weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The core idea of applying quantization to gradient subspace storage is an interesting and novel direction, as it tackles the important issue of memory accumulation in projection-based continual learning methods. The authors’ empirical evaluation is thorough, spanning several benchmarks (CIFAR-100, miniImageNet, 5-Datasets) and architectures (AlexNet, ResNet-18, ViT). The proposed method consistently achieves competitive accuracy under stringent memory budgets, demonstrating that quantizing for gradient projection can indeed mitigate forgetting while reducing storage costs.\nAnother appealing design choice is to retain overloaded (outlier) values in high precision instead of truncating them, which differs from conventional quantization and appears to contribute to robustness  (though, in my view, this intriguing design deserves deeper discussion). Overall, the paper is clearly written and includes extensive experiments supporting the empirical claims.", "weaknesses": "- Simplistic treatment of quantization - While the work proposes a quantization method, its treatment of quantization theory remains rather naive. The paper frequently refers to the scheme as “information-theoretically optimal,” yet the design is essentially a variant of scalar quantization. From rate–distortion theory, vector quantization is provably superior to scalar methods. There is little justification for why this scalar approach is adopted, and the omission of stochastic or vector quantization (or even a discussion thereof) weakens the claim of theoretical optimality.\n\n- Limited insight from the quantization analysis -  The theoretical analysis (e.g., Theorem 3.1) focuses on quantizing an i.i.d. Gaussian source. However, the quantization of Gaussian sources is a well-understood topic, and textbook references (e.g., Lecture Notes on Information Theory by Polyanski and Wu) describe optimal scalar and vector quantizers for such distributions. As such, the purpose of the analysis and the insights drawn from it are not very clear. Clarifying the purpose and novelty of this analysis would improve the paper.\n\n- Disconnect between theoretical analysis and key continual-learning metrics -  A large portion of the paper and appendices is devoted to theoretical derivations related to quantization distortion, yet the core continual-learning metrics, namely, learning performance and memory usage, are analyzed only empirically. Theoretical discussion does not connect quantization distortion to these global measures. If such an analysis is intractable, the paper should explain why; otherwise, incorporating at least approximate analytical links between quantization distortion and forgetting performance would significantly strengthen the contribution.\n\n- Restricted and oversimplified neural network assumptions - Section 3.1 appears to base the analysis on linear layers without bias terms, which limits the generality of the conclusions. In realistic neural networks, nonlinear activations such as ReLU clip all negative activations to zero, fundamentally changing the distribution and thus the quantization distortion behavior. The authors should discuss how such nonlinearities might alter or invalidate the derived results and whether the approach extends beyond this simplified setting.\n\n\n\nOverall,  the paper explores an appealing and practically relevant idea of quantizing gradient subspaces for memory-efficient continual learning, and backs it with solid experiments. However, the theoretical claims are overstated, and the quantization analysis lacks depth and connection to learning outcomes. Addressing these conceptual limitations, clarifying the notion of “information-theoretic optimality,” and broadening the analysis to more realistic neural architectures would be essential for the work to meet the standards of ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)