ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection
Authors: HUIPING ZHUANG
Zhenyu Weng
Hongxin Wei
RENCHUNZI XIE
Kar-Ann Toh
Zhiping Lin
Venue: NeurIPS 2022 Accept (2022)
Keywords: Class incremental learning
absolute memorization
exemplar-free
privacy protection
recursive
large-phase

Abstract: Class-incremental learning (CIL) learns a classification model with training data of different classes arising progressively. Existing CIL either suffers from serious accuracy loss due to catastrophic forgetting, or invades data privacy by revisiting used exemplars. Inspired by learning of linear problems, we propose an analytic class-incremental learning (ACIL) with absolute memorization of past knowledge  while avoiding breaching of data privacy (i.e., without storing historical data). The absolute memorization is demonstrated in the sense that the CIL using ACIL given present data would give identical results to that from its joint-learning counterpart that consumes both present and historical samples. This equality is theoretically validated. The data privacy is ensured by showing that no historical data are involved during the learning process. Empirical validations demonstrate ACIL's competitive accuracy performance with near-identical results for various incremental task settings (e.g., 5-50 phases). This also allows ACIL to outperform the state-of-the-art methods for large-phase scenarios (e.g., 25 and 50 phases).

ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection
Huiping Zhuang', Zhenyu Weng", Hongxin Wei Renchunzi Xie Kar-Ann Toh", Zhiping Lin?
SShinn-Ming Wu School of Intelligent Engineering, South China University of Technology. China Schooo of Electrical and Electronic Engineering, Nanyang Technological University, Singapore SShooo of Computer Science and Engineering, Nanyang Technological University, Singapore Department of Electrical and Electronic Engincering, Yonsei University, Korea 'pzhhang@sccut. edu cn, 2 [zhenyu weng, ezplin @ntu aontuedu..s.. sg .tu..uu..uuu... edu.sg. "katoh@yonsei. ac.
(hongxin001, XIERO002) @e ntu.
Abstract
Class-incremental learning (CIL) learns classification model with training data of different classes arising progressively. Existing CIL either suffers from serious accuracy loss due to catastrophic forgetting, or invades data privacy by revisiting used exemplars. Inspired by linear learning formulations, we propose analytic class-inerementll learning (ACIL) with absolute memorization of past knowledge while avoiding breaching of data privacy (i.e., without storing historical data). The absolute memorization is demonstrated in the sense that class-incrementll learning using ACII given present data would give identical results that from its joint-learning counterpart which consumes both present and historical samples. This equality is theoretically validated. Data privacy is ensured since historical data are involved during the learning process. Empirical validations demonstrate ACIL's competitive accuracy performance with near-identical results for various incremental task settings (e.g., 5-50 phases). This also allows ACIL to outperform the state-of- offtthearr art methods for large-phase scenarios (e.g., 25 and 50 phases).
Introduction
musssincremmntal learning (CIL) [26] [6] trains network phase-by-phase with training data in each phase having distinctive classes. The CIL has received an increasing popularity owing to the need adapt learned models to unseen data classes without needing train from seratch, allowing resource-saving and environmentally-fiee friinddfreddyy machine learning. Developing CIL is a natural call in our dynamic world where data and respective target category task usually available in specific location or time slot. In addition, the CIL intuitively motivated as it resembles real human learning processes where person could continuously adopt knowledge of new object categories on top of the learned information.
Merits of CIL come with costs. The CIL could struggle with the notorious catastrophic forgetting [ rendering the network losing grasp of the learned knowledge when accepting new tasks, which is also known as the task-recency bias. To mitigate the forgetting issue, several branches of CIL methods, such as the bias correction-based [2] and replay-based (or exemplar-based) [16] CII. have been proposed. These CIL. techniques allowed store small number of samples from previous tasks to fight the forgetting of old knowledge. In particular, the replay-based CIL has achieved the tfae--ofthhe-- art performance of revisiting the historical samples, which has brought concerns in terms of data privacy protection.
[12]. However, such competitive results have been obtained the cost
"Coreesponiing author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Data Privacy in CIL. Data privacy is becoming more of value in our interconnected modern world, which naturally applies to CIL problems asking for exemplar-free learning. Note that the "privacy" in CIL (i.e., cannot re-use past exemplars) may be different from the definition of other fields (such as data encryption). The increasing concern of data privacy contradicts many existing CII. techniques, such as the replay-based CII. Several methods from the regularization-bee baase CIL 9 respect privacy as they only impose regularization terms on the loss functions. However, without re- re-accessing the trained samples, their accuracy performance cannot compete with that of the replay-based CIL. Another exemplar-free CIL branch is the generative adversarial network (GAN)-based learning [2I. which preserves privacy by generating historical samples using GANs. This CIL relies heavily on GANs' performance and has not been tested in challenging datasets such as ImageNet [3]
In summary, existing CIL techniques either invade data privacy (e.g.. replay-based CIL) or cannot provide satisfactory accuracy performance (e.g., regularization-based CIL.) In addition, as the forgetting issue persists (though mitigated), CIL's performance experiences a significant degradation as learning phases increase, a pattern shared by many existing CIL. techniques (e.g.. 4 The performance degradation escalates in aarge-hase scenarios- a learning scenario with large number of learning phases for increment (e.g.. 50 learning phases (24)). This has motivated us to find new CIL methods that well tackles the catastrophic forgetting without invading data privacy.
In this paper, we propose an analytic class-incremental learning (ACIL) to handle the issue of forgetting and privacy invasion in CIL. The ACIL is inspired by analytic learning [29] [3), technique that formulates network training into learning of linear stacks. The analytic learning component allows the ACIL to conduct CII in a recursive learning manner that can absolutely memorize the knowledge of every historical sample (i.e., address catastrophic forgetting) while avoiding the breach of data privacy (i.e., without storing any past data). The key contributions are summarized as follows.
We introduce the ACIL, which holds absolute memorization of previous knowledge when accepting new tasks.
The ACIL does not store past samples, thereby achieving data privacy protection, a rare but valuable CIL property.
We provide theoretical validation of ACIL's absolute memorization, showing that the CIL using ACII. given present data provides an identical result to that from its joint-learning counterpart that adopts data from both present and historical phases.
Experiments on benchmark datasets show that the ACIL gives competitive CIL results that do not degrade over increment of data classes during learning phases. In particular, it outperforms the state-of-the-arts by a considerable margin for relatively large-phase scenarios (e.g., 25 or 50 phases).
2
Related Works
2.1 Class-Incremental Learning
Bias correction-based CIL mainly tries to address the task-recency bias. The end-to-end incremental learning [2] reduces the bias by introducing a balance training stage where only an equal number of samples for each class is used. The bias correction (BiC) (23) includes additional trainable layer which aims to correct the bias. The method named LUCIR proposed in [6] fights the bias by changing the softmax layer into cosine normalization one.
Replay-based CIL. stores a small subset of data from the previously accessed tasks to reinforce the network's 's memory of old knowledge. This CIL branch quickly draws attention due to the appealing ability to resist the catastrophic forgetting. For instance, the PODNet 4 adopts an efficient -patial-based distillation loss to reduce forgetting. with focus on the large-phase setting, achieving reasonably good results. The AANets [[] employs a new architecture containing a stable block and plastic block to balance the stability and plasticity. On top of the replay-based CIL, methods exploring exemplar storing techniques [13] are also fruitful. For instance, the reinforced memory management (RMM) [2] seeks dynamic memory management using reinforcement learning. By plugging it onto PODNet and AANets, the RMM attains a state-of the performance.
Regularization-based CIL imposes additional constraints on the loss functions to avoid forgetting. The regularization can imposed on weights by estimating the parameters' importance so relevant weights do not drift significantly. The elastic weight consolidation (EWC) [8] captures the prior

---

importance using an diagonally approximated Fisher information matrix. The EWC is improved by
[T] through finding a better approximation of the Fisher information matrix. The regularization can also be imposed on activations to prevent activation drift, which outperforms its weight-regullarizaton counterpart general. The learning without forgetring (LwF) [9] prevents activations of the old network from drifting while learning new tasks. The less-forgetting learning 7 penalizes the activation difference except the fully-connected layer.
GAN-based CIL replays past samples by generating them using GANs. The deep generative replay 17 generates synthetic samples using an unconditioned GAN. It is later improved by memory replay GAN [22] adopting label-conditions GAN. In general, the GAN-based CIL relies heavily on GAN's 's generative performance, and is only tested on relatively small datasets, such as MNIST.
The bias correction-bann and replay-based CILss allow storing exemplars, leading to privacy invasion. The exemplar-free methods (e.g., regularization-basee and GAN-based CII.) do not give competitive results. Our ACII can memorize historical knowledge without re- re-accessing the data from previous tasks, allowing perform CIL with absolute memorization and privacy reservation.
2.2 Analytic Learning
The analytic learning has been developed to circumvent limitations imposed by pack-propagation (BP), such as gradient vanishing/exploding, divergence during iteration and long training time (i.e., need many epochs). The analytic learning also goes by other names such as psendoinverse learning [5] due to the use of matrix inverse. The analytic learning starts with the shallow learning. One quick example is the radial basis network [15]. which trains the parameters using ceast-squares (LS) estimation after conducting kernel transformation in the first layer. The multilayer analytic learning [8 20 converts the nonlinear network learning into linear segments that can be solved adopting LS techniques in one-epoch training style. For instance, the dense pseudoinverse autoencoder [9] trains stacked autoencoder layer-by-layer by concatenating shallow and deep features using L.S solutions. The analytic learning could experience out-of-memory issue as the weights are learned involving the entire dataset at once. Such a memory issue can be addressed by the bisck-wise recursive Moore-Penrose learning (BRMP) 28 by replacing the joint learning with recursive one. This much resembles the replacement of gradient descent with stochastic gradient descent to reduce memory usage, but differs in that the BRMP can exactly reproduce its joint-leaming result.
The analytic learning and its recursive formulation (e.g., BRMP) brings inspiration to the CIL realm. The BRMP can stream new samples to update the weight without weakening the impact of previous samples. This matches the ACIL's need for the enhanced memorization of previously trained data. By bridging the analytic learning and its recursive formulation, our ACIL can be built to absolutely remember historical samples, without needing to re-access them.
3
The Proposed Method
This section presents the algorithmic details of ACIL, including base training agenda and a CIL agenda. Our presentation of ACIL mainly rooted in CNNs which contain a CNN backbone (feature extractor) followed by fully-coneected network (FCN) layer (classifier) for classification problems. An overview of ACIL is depicted in Figure
3.1
The Base Training Agenda
The base training agenda of ACIL has two stages in a sequential oder, namely a base training via BP and an analytic re-alignment base training (ARaBT), which are illustrated in Figure Ta) [a) and Figure Thb) respectively.
Base Training via BP. The first stage (see Figure aa) of the base training agenda duplicates the conventional BP training on the base dataset. That is, the network is trained with BP-based iteration algorithm (e.g., SGD with momentum) for multiple epochs with an appropriate learning rate scheduler (e.g., step decay scheduler). Let WeNN and WFCN represent the weights for the CNN backbone and the FCN classifier. After the BP-base base training, given an input X, the output of the network is
Y fsoftmax (S (CNN X, WcN)) WFcN)
3

---

Boase training apenda
( Basc trainiin Via BP
(b) Analylic re-aligmmen
Analric
Bose
CNN paramelers
Cseileer
Buse
Dasa
KAnE.
x
(C) Amalytic class incremental learning for phuse
Phose
Itsta
aspamiio
Clasiffrr
Autocomniation
Phase
update
Astevormiation
Phase
Analytic class-incremental learning for phise N
Arolytic Caeditr
x
Trained CNN teature extractor
Batte
Autoxommilaioon
Phase #K-1
update
Autaconelttion
Phase 8'K
Class inceemeceemetttll learning agendda
Figure 1: The ACIL begins with the base training agenda: (a) training network with BP-based iteration method epochs on the base dataset, followed by (b) ARaBT epoch only on the same dataset, which expands the FCN dimension to enhance feature extraction. Subsequently. (c-d) the CIL agenda is conducted in a recursive manner adopting the dataset (train for epoch) the current phase only and correlation matrix (see detinition in 8 encrypted with historical information.
where fCNN (X, WCNN) indicates the CNN backbone output with an X passing through it; faat is flattening operator, reshaping training sample into 1-D vector; fsolmmx is the softmax function.
Analytic Re-alignment Base Training. The second stage of base training (see Figure [b)), the ARaBT, is the key the formulation of ACIL. In this stage, the ARaBT "re-algnss" the network's learning to match the learning dynamics of an analytic learning.
Prior to our development, some definitions related to CIL are introduced. A K-phase CIL indicates that a network is trained for K phases where training data of each phase comes with different classes. Let Dirain {Ximin Y!rii } and Disut {X Y Y!ess) be the training and testing datasets at phase & (k 1,2, ...., K). Xx E RN,, Xxwhxe (e.g., images with shape of x h X e) and YE Ne xdyy (with phase including dy, classes) are stacked input and label (one-hot) tensors. Here Diain (Xbran Yrrain represents the base training set utilized to conduct the ARaBT.
The first step extract the feature matrix (denoted by x-((m)) by feeding the input tensor Xrain through the trained CNN backbone, followed by flattening operation, i.c.,
x(cnn) [enn(SeNNXX... (X.jin WcNN))
where (((mn) E Naxdan Instead of building one FCN layer to map the feature onto the classification terminal, we conduct a feature expansion (FH) process by inserting an additional FCN layer which
expands the feature space into higher one. That is, the feature x ((cnn) expanded to x(fe), as follows
x
(Xtrain WcNN)) Wie) fax (x-(nn)) Wie)
(2)
where ((ee) E No with dite) being the expansion size (with dem I dhee)). fai is an activation function (we adopt ReLU in this paper), and WR. is the FE matrix expanding the CNN-extracted feature. The need for FE process can be justified by the fact that analytic-learning methods require more parameters achieve their maximum performance. For the FE matrix, we determine dite) with a very simple trick by drawing every element from a normal distribution. Such randomization technique has been shown capture useful information for classification problems (e.g., see [ 29
Finally, the expanded feature x((ee) is mapped onto the label matrix Yiain using linear regression procedure via solving
argmin W0
X
W
(0) PCN
+7
W
(0)
(3)
IOX

---

where I|-11p indicates the Frobenius norm, and y regularizes the above objective function. Also, .T is the matrix transpose operator. The optimal solution (3) can be found in
( FCN
(Xx X + yf) x [lell y
(leiT
(1e)
(4)
0
where Wr indicates the estimated FCN weight of the final classifier layer.
TCN
3.2 The Class-Incremental Learning Agenda
With the network learning aligned with the analytic learning (see (4 an analytic learning fashion. To this end, assume that we are given Di D the learning problem in (3 can be extended to
may proceed CIL in
Y train 0
0
0 0
X
2
0
Y train
x
ito)
wilk-1)
(111121
argmin wilk RCN
FCN
+y
FCN
(5)
Yutain
x
fe
k-
where
(e)) =
(annnn train WeNs))Wa
(6)
Note that the stacked label matrix in (5) has a sparse structure due the fact that datasets from different phases are mutually exclusive. The solution to (5) can be written as
R--1
w((k-1)
Σ
x
ToT
+ 71)
(OTT Yo
where w E Ruda x Etsl du, with column size proportional to
Equation 7 gives an LS-based analytical solution for joint learning Dirain 0-k-1: The goal of ACIL is ( given Disan without
to calculate the analytical solution that satisfies at phase & based on FCN
any samples from Disi- Specifically, we aim to obtain (i FCN recursively based on W and
data x Y.train that available only the current learning phase. However, the updated weight
w((R) (k) FCN
must satisfy the joint learning in given Ozk Let
R
(Te) x +
be the regularized autocorrelation matrix (RFAuM) at learning phase k: - 1. Then solution can be summarized the following Theorem.
Theorem 3.1. The FCN weight recursively obtained by
W (k)
W (k-1)
R
(ee X (Te) (e)
(k-1) (k--1)
Rg X- y:
(9)
FCN
is identical to that obtained by at phase k. The REAuM R2 can also be recursively calculated by
R RR R X HoT (I
X (f0) R R
x R
(10)
Proof. See the supplementary materials.
As shown in Theorem 3.1 the proposed ACIL constructs recursive update of the FCN weight matrix without any loss of historical information. One can first conduct the base training agenda on the base dataset (e.g., compute ( and perform CIL afterwards adopting the recursive formulation to
obtain (k) FCN for 0. The computational steps of ACIL is summarized in Algorithm
Absolute Memorization. As observed in Theorem 3.1 the CIL in 9 yields an identical result to that of the joint learning in 7 This allows the ACIL operate with absolute memorization in the sense that the recursive formulation (i.c., the incremental learning) gives the same answer as the one
5

---

Algorithm ACIL
Base training agenda: with Di
CIL agenda:
1. Conventional training with BP on base
for to K (with Di W
and
dataset.
RR-1) do
2. ARaBT: i) Obtain feature matrix with (2; ii)
Obtain feature matrix with 66:
Obtain re-aligned weight W Save RFAuM Ro.
FCN with iii)
ii) Update RFAuM Rş with (10):
iii) Update weight matrix W (K)
FCN with
end for
obtained by its joint analytic learning counterpart. Such absolute memorization differentiates method from the existing CIL techniques that are struggling to fight the forgetting issue. To the best of our knowledge, the ACIL the first CIL that achieves absolute memorization.
Data Privacy Protection. Another benefit of our ACIL lies in privacy protection. Algorithm shows that, during the CIL agenda, historical samples granted. Instead, the R3 is cached to encrypt information for historical samples. However, is impossible to reverse- enginner the process to obtain the original samples based the R& only, avoiding possible breaching of data privacy. This is attractive feature as data privacy has attracted increasing concern in the CII. community.
Although methods in regularization-basee CIL (e.g., LwF) could also protect data privacy, the accuracy performance less ideal. In comparison, as latter shown in the experiments (e.g.. Table D. the proposed ACIL preserves data privacy while achieving very competitive results. In addition, the RFAuM holds a fixed shape (i.e., a square matrix of IR x regardless of the sample size. This takes up less storage room than that of the replay-based CIL.
An Analytic-Learingg Branch of CIL. We may categorize the ACIL into a new branch (i.e., analytic-learminn branch) of CIL. Unlike other branches, the ACIL does not forget any historical information at all. It also attains privacy protection, rare but valuable CIL feature. Even with several appealing features, the ACII is naturally not as powerful as the BP-based joint learning. The ACII is facilitated but also constrained by the fact that freezes the training of CNN weights. As seen in (6). the feature matrix x during the CIL agenda is constructed purely based on a transfer learning w.r.t. the CNN backbone trained on the base dataset. That is, the ACIL extracts the feature of new task classes using a somewhat obsolete feature extractor. This would lead to certain performance drop. However, we would argue that the benefits of ACIL greatly outweigh the potential accuracy loss. Our argument are well supported by the experiments, displaying very competitive results using ACIL.
4 Experiments
We evaluate the proposed ACIL on CIFAR-100, ImageNet-Subset and ImageNet-Full datasets which are benchmark datasets for CIL. We compare the ACIL with several state-of-the CIL techniques,
including LwF [ EWC
[8] Semantic Drift Compensation (SDC)
25
BIC 233, iCaRL [16]
LUCIR [6]. Mnemonics [3], PODNet 4 AANets [ and RMM 12
GAN-based CIL is not
included as it is only tested on less challenging datasets (e.g., MNIST) and its performance relies
heavily on the GAN training.
The LwF adopts distillation-basel loss functions to prevent forgetting. The EWC uses Fisher information matrix. The SDC studies and compensates the semantic drift of features. The LwF, EWC, SDC and the proposed ACIL belong the iCaRL, Mnemonics, PODNet, AANets and RMM, are replay-based methods, requiring storage of past exemplars.
methods. The other methods, i.c., BIC.
4.1 Datasets and Implementation Details
Datasets. CIFAR-100 contains 100 classes of 32 x 32 color images with each class having 500 and 100 images for training and testing respectively. ImageNet-Full has 1000 classes, and 1.3 million

---

Table 1: Comparison of A and F among compared methods. The ACIL adopts dye = 8k, 15k, 15k 1000)) on CIFAR- 100, ImageNet-Sufee Subset and ImageNet- FFul respectively. The ACIL, LWF, EWC and SDC not keep old data while other compared methods adopt the same replay settings (e.g.. I I by reserving exemplars per old class. Results for A(%) are duplicated from (TT) except for the 3-combo method "POD+AANcts+RR
which is copied from the RMM paper [2] (its ImageNet-Full results are not listed due to no ImageNet option in the source code). Results for F()) are cloned from [3].
The strict-memory setting results can be found in
Table the supplementary materials.
Metric Method
Privary
KC5
CIFAR-700
25
50
K
IuugeNersSutort
131
50
K-5
IaugeeNa.Fall
111
25
30
Lel (TPAMI 2018)
89.59
15.58
45.31
53.62
47.51
51.30
46.89
43.14
EWC (PNAS)
34.11
12:33
26.76
SDC (CVPS 2020)
55.46
36.56
62,97
$4.30
70.07
64.96
62.65
58.72
57.12
46.89
LUCIR
(BCCV 2020)
64.83
57.98
74.33
62.48
LUCIR-Meemmnies (CVPR 2020)
63.70
73.30
72.17
71.50
63.12
POD+-AANes (CVP8
71.78
64.85
POU+AANNesRRMM (NeurPPS 2021)
54.12
78.11
66.30
65.95
66.01
74.76
74.13
64.84
LwF (TPAMI 2018)
36
41.66 66
55.32 92
57,00
48.70
47.94
(CaRl (CVPR: 2017)
57.12
36.48
45.84
47
26.03
33.76
18.50
JF3
LUCIR (CVPR 2019)
BIC (CVPR2019)
X
X
31.42
18.70
21
32.50
34.60
46
31.88
27.04
33.48
31.04
35.40
37.98
26.08
25.06
27.29
28.34
30.30
33.17
LUCIR-Meemoiiss (C 2020)
x
11.64
10.50
9.56
10.20
9.85
11.76
13.63
13.45
14.40
ACIL
9.00
9.72
9.28
9.33
3.91
3.40
3.30
3.45
2.75
3.45
3.31
3.40
images for training with 50,000 images for testing. ImageNet- Subset, in particular, is constructed by selecting 100 specific classes from ImageNet- Full based on what defined in 44.
Network Architecture. The architectures for CIL in the experiments are ResNet-32 on CIFAR-100 and ResNet-18 18 on both ImageNet-Full and its subset. These two architectures are commonly adopted for CIL performance comparison. Our ACIL imposes a slight change (i.e., inserts expansion FCN layer), but the CNN backbones are identical to those from the selected ResNet architectures.
Training Details. For conventional BP training in the base training agenda, we train the network using SGD for 160 (90) epochs for ResNet-32 (ResNet-18). The learning rate starts at 0.1 and is divided by 10 at epoch 80 (30) and 120 (60). We adopt momentum of 0.9 and weight decay of x 10 (1 x 10 10-4) with a batch size of 128. The input data are augmented with random cropping, random horizontal flip and normalizing. For fair comparison, this base training setting is identical to that of many CIL methods (e.g., 16 13). ). For the ARaBT and ACIL's incremental learning steps, data augmentation is adopted, and the training ends within only one epoch. The results for the ACIL are measured by the average of 3 runs on an RTX 2080Ti GPÜ workstation. Note that in ACIL, the CNN is only trained during the BP base training phase. After that, the parameters of the CNN backbone are fixed during the incremental phases (i.e, phase #1 to #K) including the ARaBT.
We follow the protocol adopted in 4 The network is first trained (i.e., phase #0) on the base dataset containing half of the full classes from the original dataset. Subsequently. the network gradually learns the remaining classes evenly for K phases (i.e., K K-phase CIL), with the dataset in each phase containing disjoint classes from one another. Most existing methods only report results for K 5, 10, 25. We include K 50 as well to validate ACIL's absolute memorization.
CIL Protocol.
4.2 Evaluation Metric
Two metrics are adopted to evaluate the ACIL. The overall accuracy performance is evaluated by the average incremental accuracy (or average accuracy) Á (%): À k Ex where Ak indicates the average test accuracy of the network incrementally trained at phase k by testing it on Diest The A evaluates the overall performance of CIL algorithms. A higher A score is preferred when evaluating CIL algorithms. The other evaluation metric is the forgetring rate F (%) defined in [3:F= A A/ where A% denotes the average accuracy at phase & by testing it on D The forgetting rate reveals the degree which CII method forgets the base classes. Hence, it is good indicator to evaluate CIL's forgetting issue.
4.3 Result Comparison
We tabulate the average incremental accuracy À and the forgetting rate F from the compared methods in Table [ As shown in the upper panel, overall, the "combo" CIL cebbiques--techniques
7

---

CIFAR-100 100
88
68
LUCI Aiemeniss
(CaML
Lef BWC
100 1A4
AGL
100%
ImageNe Imagetel Subset
82
60-
60
60
15
(hase*(K=5)
(kase#(K=10)
phase#(k=25)
Figure 2: Avg. accuracy w.r.t. phase. The RMM curve is not included as its source code is only applicable for strict-memory settings.
that combine more than one CIL methods-give very competitive A scores. In particular, the PPOD+AANet+RMM"" combo, which incorporates PODNet [4) AANets [I]] into RMM [12], obtains the most competitive results that can be treated as current the state-of-the-art counterpart.
As shown in the upper panel T the ACIL gives a slightly worse average accuracy for 5-phase CIL in general. However, ACIL's performance catches up with those of the state-of-th-arts as K increases, and begins lead for K 25 (i.e., large-phase CII. scenarios). For instance, for 5-phase CII,, the ACIL gives an accuracy of 66.30% on CIFAR-100, which is slightly worse than the results from several combo techniques such as the "POD+AANets" combo (66.31% 1%) by 0.01 % and the "POD+AANets+RMMM combo (68.36%) by 2.06%. However, for 25-phase CIL, the ACIL (with 65.95%) outperforms these combo methods, e.g., outperforming the second best by 1.83% (64. 12% from the SPOD+AANets+RMAA to ACIL's absolute memorization. That is, the accuracy of ACIL remains unchanged for different K values, while other CIL methods experience various levels of forgetting issue that intensifies as K increases. Note that there could be a very mild A degradation from ACIL as K increase (e.g.. 66.30%-665.99 Although theoretically the ACII should give identical results regardless of K, the possible mild drop is likely caused by quantization errors since large indicates more computation rounds hence more quantization operations (e.g., see TABLE VI (28))
combo). Such overtaking pattern is naturally expected owing
This pattern on CIFAR- 100 is quite consistent with those on ImageNet- Subsett and ImageNet-Full. On ImageNet-Full, the ACIL begins outperform the compared methods for K > 10, and leads (with 64.63%) the second best result (63.08% from the 25-phase learning. On ImageNet-Subset the ACIL falls behind the IPOD+AANet+RMMM. comb even for 25 sh-phase learning, but the gap very small (74.59% v.s. 75.01% 1).. The overtaking pattern is further detailed in Figure 2
combo) by 1.55% for
In addition, we report the 50-phase CII. for the proposed ACIL. expected, the average accuracies (i.e., 66.01% %, 13% and 64.35% on CIFAR-100 100, ImageNet-Suuset and ImageNet-Full) are very close to those trained with K = 5, 10 or 25. This allows the ACIL to further outperform the compared methods that are not specializing in large-phase incremental problems. Although the PODNet also aims large-phase problems, its performance cannot compete with ACIL's (57.98% v.s. 66.01% % on CIFAR- 100, and 62.48% v.s. 74.13% on ImageNet-Subset),
Why ACIL Performs Well Conventionally. the analytic learning cannot compete with BP 22 However, if the feature extractor (e.g., CNN layers) is pre-trained with BP with the classifier head designed by analytic learning related techniques, the performance can catch up [14] Such scenario fits well in the CII. procedure in this paper, explaining why our ACIL, performs well.
The expansion size dote from the FE process has huge impact the CIL performance. As plotted in Figure 3 the A on ImageNet- 100 and ImageNet-Füu increases with larger drte)- The performance CIFAR- 100 starts to decline for draa); > 10k, likely because the expansion ratio for ResNet-32 case unreasonably large (e.g., d/ee) dkkmn)) 15k 1.5k/64) /64) compared with that of ResNet- 18 (15k/5) on
ImageNet datascts. As observed in Figure the ACIL on ImageNet datasets should work better with

---

even larger dise)). but our 1 IIB GPU experiences memory leak. Still, the expansion up to d(te) = 15k allows the ACIL to give very competitive results (see Table J.
The forgetting rate F is also presented in the bottom panel of Table 1 Our ACII demonstrates the lowest F scores all the three benchmark datasets. This is further evidence supporting our absolute-memorizztion claim. Note that the absolute memorization does not lead to F = 0. Even healthy joint learning would reduce the performance on the base classes. This can be explained by the example as follows. Let M00 and 100 be the two networks jointly trained on the base 50 classes and the 100 full classes from CIFAR- 100 respectively. Testing Mao on the base dataset would still experience performance loss compared with that obtained by testing M50 on the base dataset, i.e., F > O. Hence, although our ACIL perfectly remembers the pass samples, non-zero forgetting rate still applies when incrementally learning new classes. Nonetheless, the forgetting rate has been shown to be much lower than the existing CIL methods. For instance, for 5-phase learning on ImageNet-Full, the
combo exhibits 13.63% forgetting, but our ACIL only has 2. 2. Yet, the low FF score does not suggest strong resistance for learning new tasks since the average accuracy has been shown to be comparable to state-of-the-ar results (see upper panel of That is, the ACIL bears a relatively good Hability-plasticcy balance.
Table D.
Data Privacy Protection. Apart from the competitive incremental accuracy, the ACIL is in strong support of data privacy. As indicated in Algorithm | the incremental learning surrenders any samples from previous tasks. allowing data privacy across learning phases or platforms. As shown in Table [ prreeeeire CIL (e.g.. LwF, EWC and SDC) suffers much more intensively (e.g.. 45.51% from LwF of 25-phase CII. on CIFAR- 100) than the replay-based -baeed CII. (e.g., 64. (2% from RMM combo of 25-phase CIL on CIFAR-100) Our ACII maintains the privacy while providing comparable or better accuracy performance (e.g.. 65.95% of 25- -hase CIL on CIFAR- Such comfortable balance
would certainly attract attention as we are living in a world that values and protects data privacy.
Imagehet
ImageMet-FFll
14
w/ regularization
FR process
A (%)
13
10)
2
TO
52,99
555
12
x
66.30%
33.0
10
x
4
x
66.25%
x
X
66.23%
51.12%
14
16
13k
154
54
13x
24
1dk
Table 2: Ablation study regarding expansion and
Figure 3: The impact of expansion size dise)-
regularization.
Memory for Storage. The ACIL stores R& instead of exemplars. As an example, for fix-exemplar setting, the memory used by storing R3 (8k) on CIFAR- 100/CO0BO000-0/000 202201111ageeeee I/ImageNet is 8k 8 = 64 million (M) tensor elements, while other methods consume 6. 6.1M MM/301. 00000..1//0000.66 respectively (e.g.. on ImageNet 224 x 224 x x 20 1000 3010.6M) This shows that our method is memory-friendiy to large-shaped image datasets (e.g., ImageNet).
4.4 Ablation Study
In the proposed ACIL,, the FE process governed by due and the regularization controlled by are essential. To show this, we adopt an ablation study by conducting a 5-phase CIL of ResNet-32 (with dita) = 8k) CIFAR- 100 to observe the performance shift w.r.t. these modules. As reported in Table without the FE process (see the first two rows in Table the average incremental accuracy The need for the FE process was enlightened by the fact that the analytic learning is naturally prone to under fiitt-n due to simple linear regression 29. Widening the feature size helps to capture the missing discriminative information.
experiences a sharp drop (c.g.. 66.30% 30%0~52.9994
The regularization factor 7, on the other hand, plays an important role but behaves robustly during the CIL experiments. As shown rows 2-5 in Table the ACIL performs robustly for considerably wide range of y values (e.g., 10 10--.. However, it would be unwise to remove the regularization as the ACIL could suffer from strong accuracy reduction without its support (c.g., 100%%+5..1299

---

4.5 Potential Positive and Negative Societal Impacts
It allows researchers to avoid breaching user privacy while pushing forward their learning methods. From this angle, our method gives positive societal impact by protecting privacy. We do not foresee obvious societal impacts. However, our ACII does rely certainly on the CNN feature extractor in a similar way of transfer learning. In this case, there might be various follow-up research aiming to improve the extraction power by many trials and errors, leading to certain electricity pressure consumed by GPU operations.
main characteristic of ACIL is
5 Conclusion
In this paper, we have presented an analytic valuable features (i.e., the absolute memorization and the data privacy protection) for addressing several existing limitations of class-incremental learning. The analytic learning has been incorporated key component to conduct incremental learning of new tasks in recursive manner. Such recursive learning style allows the ACIL to have absolute memorization. That is, the incremental learning of ACIL given present data would produce identical results to that of joint learning which accesses both present and historical data, a property that has been theoretically validated. The recursive formulation has also the merit of not storing any samples from historical tasks, thus avoiding the breach of data privacy. Experiments have been conducted to validate our claims. Overall, our ACIL gives very competitive accuracy results, In particular, outperforms the state-of-the-ard methods for large-phase scenarios (c.g., incremental learning with 50 phases).
learning (ACIL) which bears two
6
Acknowledgment
We thank the anonymous reviewers for their very constructive comments for improving this manuscript. This work was supported in part by the Science and Engineering Research Council, Agency of Science, Technology and Research, Singapore, through the National Robotics Program under Grant 1922500054.
References
[1] Eden Belouadah, Adrian Popescu, and Ioannis Kanellos. A comprehensive study of class incremental learning algorithms for visual tasks. Neural Networks, 135:38-54, 2021.
[2] Francisco M. Castro. Manuel Marin-Jimenet Nicolas Guil, Cordelia Schmid. and Karteck Alahari. End-to-end incremental learning. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[4] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Msiio-ECCO 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XX 16, pages 86-102. Springer, 2020.
[5] Ping Guo, Michael R Lyu, and NE Mastorakis. Pseudoinverse learning algorithm for feedforward neural networks. Advances in Neural Networks and Applications, pages 321-326, 2001.
[6] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[7] Heechul Jung. Jeongwoo Ju, Minju Jung, and Junmo Kim. Less-forgetting learning in deep neural networks. arXiv preprint /607. 002.22,022 2016.
[8] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barne Barwinska, et al.
10

---

Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 11<(13):3511-3522, 2017.
[9] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):9955-2947, 2018.
[10] Xialei Liu, Mare Masana, Luis Herranz, Joost Van de Weijer, Antonio M. López, and Andrew D. Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Puttern Recognition (ICPR), pages 2262-2268, 2018.
[11] Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class- incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1544-2555
June 2021.
[12] Yaoyao Liu, Bernt Schiele, and Qianru Sun. Rmm: Reinforeed memory management for class-incremental learning. Advances in Neural Information Processing Systems, 34, 2021.
[13] Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi- class incremental learning without forgetting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[14] Cheng-Yare Yawng Low, Jaewoo Park, and Andrew Beng-Jin Teoh. Stacking-based deep neural network: Deep analytic network for pattern classification. IEEE Transactions on Cybernetics, 50(12):5021-5044 2020.
[15] J. Park and I. W. Sandberg, Universal approximation using basis-function networks. Neural Computation, 3(2):246-577, 1991.
[16] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), July 2017.
[17] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page 2994-3003, 2017.
[18] Kar-Ann Toh. Learning from the kernel and the range space. In the Proceedings of the 17th 2018 IEEE Conference on Computer and Information Science, pages 417-422, IEEE, June 2018.
[19] Jue Wang, Ping Guo, and Yanjun Li. Densepilae: feature reuse pseudoinverse learning algorithm for deep stacked autoencoder. Complex & Intelligent Systems, pages 1-11, 2021.
[20] X. Wang. T. Zhang, and R. Wang. Noniterative deep learning: Incorporating restricted boltzmann machine into multilayer random weight neural networks. IEEE Transactions on Systems, Man, and Cyhernetics: Systems, 49(7): 299-13088 1308, 2019.
[21] Chenshe Wu, Luis Herranz, Xialei Liu, Yaxing Wang, Joost van Weijer, and Bogdan Raducanu. Memory replay gans: learning to generate images from new categories without forgetting. In Conference on Neural Information Processing Systems (NIPS), 2018.
[22] Chenshen Wu, Luis Herranz, Xialei Liu, Yaxing Wang, Joost van de Weijer, and Bogdan Raducanu. Memory replay gans: Learning to generate images from new categories without forgetting. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 5966-5976, 2018
[23] Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEFF//VF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[24] Shipeng Yan, Jiangwei Xie, and Xuming He. DER: Dynamically expandable representation for class incremental learning. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3014-3023, June 2021.

---

[25] Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-ines caass-incrementll learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[26] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C.-C. Jay Kuo. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020.
[27] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452-1466 2018.
[28] Huiping Zhuang, Zhiping Lin, and Kar-Ann Toh. Blockwise recursive Moore-Penrose inverse for network learning. IEEE Transuctions Systems, Man, and Cybernetics: Systems, pages 1-14, 2021.
[29] Huiping Zhuang, Zhiping Lin, and Kar-Ann Toh. Correlation projection for analytic learning of classification network. Neural Processing Letters, pages 1-22, 2021.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged include justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Will be
included.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] See the Supplementary material.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [No] Will release the code and instructions shortly.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] Results are reported by the average of 3 runs.
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e-g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use ResNet and have cited the paper.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] All datasets publicly accessible.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] All datasets are publicly accessible with no personally identifiable information or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "p3p3FU5WIYb", "reviewer": "Reviewer_xNZm", "summary": "This paper aims to tackle the problem of catastrophic forgetting without revisiting historical exemplars so that data privacy is preserved. To do this,  an analytic class-incremental learning (ACIL) algorithm is proposed with absolute memorization of past knowledge. Empirical results demonstrated competitive accuracy performance that does not degrade over increment of data classes during learning phases. . ACIL also outperformed the state-of-the-art methods for large-phase scenarios.", "questions": "1. Can you add descriptions of the benchmarking algorithms?\n2. Why not evaluate the GAN-based methods on these large datasets?\n3. Are the CNNs trained separately for each phase?", "limitations": "Not applicable.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - Section 3 clearly described the proposed algorithm. Even readers with no background in analytic learning and CIL can easily understand.\n - The idea of using analytic learning in CIL is novel and sound.\n - Has empirical evaluations demonstrating the power of ACIL.\n\nWeaknesses:\n - Should explain the benchmarking algorithms in the empirical evaluations, e.g. what is LwF.\n - Should include more privacy-preserving baselines in the experiments. Currently, only LwF is used. How about the GAN-based methods mentioned in Section 2? Why not evaluate the GAN-based methods on these large datasets?\n - Should clarify how CNN was trained (separately for each phase or it is one CNN for every phase?)\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YK1A_6ik83", "reviewer": "Reviewer_UFfv", "summary": "In this paper, the authors propose a method to learn a perfect FC classifier for CIL without using exemplars. Instead, they need to store autocorrelation matrixes in the memory. They provide extensive experiments to show the effectiveness of the proposed method.", "questions": "See the \"weaknesses\" part. ", "limitations": "It would be better to add a section to discuss the potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- This paper is well-written and easy to follow.\n\n- The proposed method achieves impressive results without using the exemplars.\n\n- Extensive experiment results on multiple datasets are provided.\n\n### Weaknesses\n\nI have reviewed this paper as an ICML submission. During the review, I indicate the following weaknesses:\n\n- In the abstract and introduction, the authors claimed they fully address the catastrophic forgetting problem. However, the results (e.g., in Table 1) of ACIL are still much lower than the joint training results. I think it is because CNN is fixed. Therefore, the authors should adjust their presentation and remove the claim of “fully addressing the catastrophic forgetting”.\n\n- The proposed method needs to store autocorrelation matrixes in the memory. Thus, the author should also compare the memory usage between existing methods and the proposed method. The authors need to answer whether the proposed method requires more memory compared to saving exemplars.\n\n- The proposed method uses a fixed CNN network after the first phase. So it is not possible to update the low-level image features with the new data in the following phases. If we observe some new classes that are very different from the first phase data, this method might not perform well. Maybe the authors should include some experiments to evaluate the above situation.\n\n- There are some small typos in the paper. For example, the authors write “\nMerits of ACL come with costs” in Line 014. However, the abbreviation “ACL” never appears before. I think it might be “CIL”. In Figure 1, the word “clasasifier” is incorrect.\n\n- The authors should move the proof of Theorem 3.1 to the supplementary.\n\nThe above weaknesses have been addressed in the current version. Thus, I have no more questions and recommend acceptance. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5UD1ZcHDrOU", "reviewer": "Reviewer_B6wm", "summary": "The paper suggests a new CIL (Class-incremental learning) algorithm. The proposed algorithm outperforms existing CIL algorithms in the sense that it is both reasonably accurate, and does not expose past examples to privacy breaches. Previous algorithms could have only achieved one of these desired properties.", "questions": "Is the conjecture that the mild degradation mentioned in row 263 could be explained by quantization errors supported by the reference cited there? What is the reasonable damage that such errors usually cause?", "limitations": "Yes.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The key contribution of an algorithm which is both accurate and private seems novel and significant. It is also well explained and motivated. \n2. A thorough literature review is conducted, assisting the reader to understand the paper's contribution.\n3. The limitations of the algorithm (especially for small K values) are well explained.\n\nWeaknesses:\n1. It could have been interesting to analyze why for small K values ACIL does not outperform state-of-the-art, and perhaps even to conjecture whether there exists a private algorithm that outperforms the state-of-the-art for any value of K.\n2. For the inexperienced reader, it is not obvious that the mild degradation mentioned in row 263 could really be explained by quantization errors, as he/she may not be familiar with the plausible rate of such kind of errors. Also, it might have been interesting to discuss what could happen when K is extremely large (in case that it is indeed an interesting scenario, and if it is not, it might be worth explaining why). As far as the (inexperienced) reader knows, the degradation might be catastrophically high in this case.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept