Online Learned Continual Compression with Stacked Quantization Modules
Authors: Lucas Caccia
Eugene Belilovsky
Massimo Caccia
Joelle Pineau
Venue:  (2020)
Keywords: continual learning
lifelong learning

Abstract: We introduce and study the problem of Online Continual Compression, where one attempts to learn to compress and store a representative dataset from a non i.i.d data stream, while only observing each sample once. This problem is highly relevant for downstream online continual learning tasks, as well as standard learning methods under resource constrained data collection. We propose a new architecture which stacks Quantization Modules (SQM), consisting of a series of discrete autoencoders, each equipped with their own memory. Every added module is trained to reconstruct the latent space of the previous module using fewer bits, allowing the learned representation to become more compact as training progresses. This modularity has several advantages: 1) moderate compressions are quickly available early in training, which is crucial for remembering the early tasks, 2) as more data needs to be stored, earlier data becomes more compressed, freeing memory, 3) unlike previous methods, our approach does not require pretraining, even on challenging datasets. We show several potential applications of this method. We first replace the episodic memory used in Experience Replay with SQM, leading to significant gains on standard continual learning benchmarks using a fixed memory budget. We then apply our method to compressing larger images like those from Imagenet, and show that it is also effective with other modalities, such as LiDAR data.

Under review as a conference paper at ICLR 2020
ONLINE LEARNED CONTINUAL COMPRESSION WITH STACKED QUANTIZATION MODULES
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce and study the problem of Online Continual Compression, where attempts to learn to compress and store a representative dataset from a non i.i.d data stream, while only observing each sample once. This problem is highly rele- vant for downstream online continual learning tasks, as well as standard learning methods under resource constrained data collection. To address this we propose a new architecture which Stacks Quantization Modules (SQM), consisting of a series of discrete autoencoders, each equipped with their own memory. Every added module is trained to reconstruct the latent space of the previous module using fewer bits, allowing the learned representation to become more compact as training progresses. This modularity has several advantages: 1) moderate com- pressions are quickly available early in training, which is crucial for remembering the early tasks, 2) more data needs to be stored, earlier data becomes more compressed, freeing memory, 3) unlike previous methods, our approach does nor require pretraining, even on challenging datasets. We show several potential ap- plications of this method. We first replace the episodic memory used in Experi- ence Replay with SQM, leading significant gains on standard continual learning benchmarks using fixed memory budget. We then apply our method to online compression of larger images like those from Imagenet, and show that also effective with other modalities, such as LiDAR data.
INTRODUCTION
Interest in machine learning in recent years has been fueled by the plethora of data being generated on a regular basis. Effectively storing and using this data critical for many applications, especially those involving continual learning. In general, compression techniques can greatly improve data storage capacity, and, if done well, reduce the memory and compute usage in downstream machine learning tasks (Gueguen et al., 2018; Oyallon et al., 2018). Thus, learned compression has become topic of great interest (Theis et al., 2017: Ballé et al., 2016; Johnston et al., 2018). Yet its application in reducing the size of datasets bound for machine learning applications has been limited.
This work focuses on the following familiar setting: new training data arrives continuously for a learning algorithm to exploit, however this data might not be iid, and furthermore there is insuf- ficient storage capacity to preserve all the data uncompressed. We may want to train classifiers, reinforcement learning policies, other models continuously from this data, or use samples ran- domly drawn from it at a later point for a downstream task. For example, an autonomous vehicle (with bounded memory) collects large amounts of high-ddmensiioal training data (video, 3D lidar) in non-stationary environment (e.g. changing traffic patterns), and overtime applies an ML algo- rithm improve its behavior using this data. This data might be transferred at a later point for use in downstream supervised learning. Current standard learned compression algorithms, e.g. Torfason al. (2018), 1)), are not well designed to deal with this case, as their convergence speed is too slow to be usable in an online setting.
In the field of continual/ifelong learning (Thrun & Mitchell, 1995), which has for now largely focused on classification, approaches based on storing memories for later use have emerged some of the most effective in online settings (Lopez-Pazz et al., 2017; Aljundi et al., 2018; Chaudhry et al.; 2019; Aljundi et al., 2019). These memories can be stored is, via generative model (Shin et al., 2017). Then, they can either used for rehearsal (Chaudhry et al., 2019; Aljundi et al., 2019) or for constrained optimization (Lopez-Paz et al., 2017; Chaudhry et al.; Aljundi al., 2018).

---

Under review as a conference paper at ICLR 2020
Indeed many continual learning applications would be nearly solved with replay approaches if one could afford to store all samples. These approaches are however inherently limited by the amount of data that can be stored'.
Learning a generative model to compress the previous data stream thus seems like an appealing idea. However, learning generative models, particularly in the online (possibly non-stationary) set- ting, continues to be challenging, and can greatly increase the complexity of the continual learning task. Furthermore, such models are susceptible catastrophic forgetting Aljundi et al. (2019). An alternate approach is to simply learn a compressed representation of the data; this is typically faster and more stable than learning to generate the whole data distribution. While the learned com- pression may itself exhibit forgetting and representation drift, causing challenges for continual and online cases, learned compression method that can learn continuously and online would allow the storing of far larger amount of samples for replay.
In this work we investigate the use of quantized autoencoders, specifically the VQ-VAE framework (van den Oord et al., 2017), observing that these can learn continuously and online with minimal forgetting, particularly when augmented with their own internal rehearsal mechanisms. We propose multi-level stacked model that allows the compressor to adaptively store samples at different com- pression scales, based on the amount of data, storage capacity, and effectiveness of the model in compressing samples. Furthermore, the learned compressed representation allows multiple contin- ual learning models to be trained from the same data.
The main contributions in this work are follows: (a) we introduce and highlight the importance of the online continual learned compression problem; (b) we demonstrate how Multi-level VQ- VAE, combined with internal replay, can effectively learn compressed representations of online data, (c) we show online learned compression can yield state-of-the-art performance in standard online continual image classification benchmarks.
RELATED WORK
Learned compression has been recently studied for the specific case of image compression. Work by Theis et al. (2017); Ballé et al. (2016); Johnston et al. (2018) have shown learned compressions can outperform standard compression algorithms like JPEG. Some of these methods however are challenging train, thus in our work we focus on the VQ-VAF approach den Oord et al. (2017); Razavi et al. (2019) which allows us to address online continual learning settings and permit a multi- level storage.
Continual Learning research currently focuses on overcoming catastrophic forgetting (CF) in the supervised learning setting, with some limited work in the generative modeling and reinforcement learning settings. Most continual learning methods can be grouped into three major families.
Some algorithms dynamically change the model's ' architecture to incorporate learning from each task separately. Popular methods are Rusu et al. (2016), Li & Hoiem (2018) and Fernando et al. (2017). Although these methods can perform well in practice, their introduction of task-specific weights requires growing compute and memory costs which are problematic for the online setting. Another set of techniques employ regularization to constrain weights updates in the hope of maintaining knowledge from previous tasks. Notable methods in this class include (Kirkpatrick et al., 2017; Huszár, 2017; Zenke et al., 2017; Nguyen al., 2017; Chaudhry et al., 2018). This set of approaches is inefficient in the online setting Chaudhry et al. (2019).
The last family of methods encapsulates all that have a mechanism store information about the previous data distributions. This memory then serves as a tool for the continual learner to rehearse previous tasks. The simplest instantiation of this method is keep and sample from buffer of old data to retrain the model after every update (Chaudhry et al., 2019). This approach is widely used in RL where is known as Experience Replay (ER) (Lin, 1993). Another method, known as Gen- erative Replay (GR) (Shin et al., 2017), uses generative modeling to store past tasks distributions. The continual learner then trains on generated samples to alleviate CF. Other notable examples are Gradient Episodic Memory (GEM) (Lopez-Paz et al., 2017 301 iCarl (Rebuffi et al., 2017), and Max- imally Interfered Retrieval (MIR) Aljundi et al. (2019), as well as (Aljundi et al., 2018; Hu et al.,
"This is typically not feasible for on- device machine learning. And while many ML services currently run on the cloud, the move towards increasingly higher data privacy standards is likely to push many MI. algorithms locally on device.
2

---

Under review as a conference paper at ICLR 2020
2018). Most closely related to work Riemer et al. (2017) consider compressing memories for use in the continual classification task. They also employ a discrete latent variable model but with the Gumbel approximation which shows to be far less effective than our approach. Most notably separate offiin iid pre-training step for the learned compression is required in order to surpass the ER baseline, distinctly different from the online continual compression consider.
Lidar compression is considered in Tu et al. (2019) and Caccia et al. (2018). Both approaches similar projection from 3D (x, M, z) coordinates to 2D cylindrical coordinates, and leverage deep generative models to compress the data. However, neither method was designed to account for potential distribution shift, nor for online learning. In this work we show that reusing this 2D projection in conjunction with out model allows us to mitigate the two issues above for lidar data.
Algorithm 1: SQM Replay
Algorithm 2: AdaptiveCompress
Input: Learning rates Wetas Car Initialize: Memory M; fae for 1..T do
Input:
datapoint SQM with modulcs, distortion threshold deh
Initialize: Memory M; 00
Fetch data from current task for Bine Dr
2 8 Compute hidden rep. for all blocks
for m E 1..N do
3 ENCODE(SQM,
B t- Bine
Iterate over blocks, from most compressed to least for T...1 do
ift 1 then
& Petch data from buffer Bre ~ SAMPLE(M)
& Fetch latent indices and hidden rep. computed in forward pass argmin, 2g ARGMIN(module.)
< (Bice B..)
11
end
12
Train Compressor Network 0gen +- ADAM(Leq, B, (tac)
8
8 Decode to output space for E i...1 do 2g DEECDDEmmbbbl,, =q)
13
14
if > then UPDATEBUFFERRERCM Ouc)
10
& Test reconstruction error if MSE(2, x,) < duh then return argmin,,
15
&Svv current indices ces ADDTOMEMORY(M, Bine, Oae)
end
17
end
8 If compression is good enough, return original image return .r,
end
19 end
METHODOLOOGY
In this section we outline our approach to the online continual compression problem. Our learned compression network consists of a set of Multi-resolltion VQ- (Q-VAE blocks. These blocks only com- municate information forward and are not learned jointly unlike the architecture presented in Razavi al. (2019). Memories are compressed using an adaptive scheme that controls what resolution the sample is stored, and therefore how compressed the sample should be. Furthermore memories can be revisited and further compressed as the learned compression module improves. Finally a rehearsal phase that utilizes the stored memory is used minimize forgetting and update representations stored in the memory.
3.1 PROBLEM SETTING
We consider the problem setting where stream of samples x D, arrives from different distri- butions D, changing over time t = 1. T. We have fixed storage capacity of C' bytes where we would like to store the most representative information from all data distributions D Dr. There is notably a trade-off in quality of information versus samples stored. We propose to use learned compression model, and most crucially, this model must also be stored within the C bytes, to en- code and decode the data samples. Another critical requirement is that at anytime t the content of the storage (data and/or compression model) be usable for downstream applications. A key challenge is that the learned compression module will change over time, while still need to be able to decode the memories in storage.
The high level training of the online learned compression is described in Algo 1. Random memories are decoded and used to train the current compression module, at the same time this also allows us re-encode those memories using the update weights. The approach also incorporates form of
3

---

Under review as a conference paper at ICLR 2020
replay to reduce forgetting and address drift, whereby the memories' quality degrade away from their original values. In the rest of this section we discuss the architecture, objective and storage we propose.
3.2 VECTOR QUANTIZED VAE
Variational Autoencoders (VAE) (Kingma & Welling, 2013) consist of two parts: the encoder network parameterizes the posterior distribu- tion g<>/z) and the decoder network p(z(z) aims to reconstruct the original input from the inferred latent variables 2. In standard VAE the prior and posterior distributions are usually Gaussian with diagonal covariance. On the other hand Vector Quantized Autoencoders (VQ-AEE) use a discrete latent representation instead (van den Oord et al., 2017). This model additionally keeps an embedding table E E RRXD consisting of K vectors of size D. Given an input (c.g. an RGB image), the en- coder first encodes it as HI, WA tensor, where Hn, and W,, denote the height and width of the latent representation. Then, every D dimensional vector goes through a discretiza- tion bottleneck using nearest-neighbor lookup on the embedding table. Specifically, Fij arg minge E (z)nc()); - ell2. The output of the discretization step is then fed through the decoder. The gradient of this non-differentiable step approximated using the straight-through estimator. A key property to notice that to reconstruct the input, only the H₁ x Wn. in- dices are required, thus yielding very powerful compression (van den Oord et al., 2017). The full VQ-VAE objective, Leg is given in van den Oord et (2017).
Block
Ancoder
deooder
Black
enoodar
NEC
Figure Stacked Quantization Modules. Each level uses its own loss and maintains its own replay buffer. Dotted lines indicate gradient is stopped in the backprop
3.3 STACKED QUANTIZATION MODULES
To ensure adaptivity of the compression model, we adopt Stacked Quantization Modules (SQM). Each module contains a VQ-AE and corresponding index buffer of adaptive capacity. Its put is the 2qq (f-1) from the previous layer. A full diagram of the Stacked Quantization Modules (SQM) given in Figure 1. Each module reconstructs its input from latent representation zy, where BITS 2 < BITS(2((-)) Here BITS denotes the memory bits used by the encoding. The compression rate at a given block is given by
H x W X C' x log, (256)
No His X Whi log, (K:)
Thus the compression rate is controlled by: K;, the number of embeddings in the codebooks of block i, the spatial dimension of the latent rep (H,s. Wn.) and the number of codebooks N
VQVAE-2 (Razavi et al., 2019) also uses multi-scale hierarchical organization, where unlike SQM the top level models global information such as shape, while the bottom level, conditioned on the top one, models local information. While this architecture tailored for generative modeling. it is less attractive for compression, as both the bottom and top quantized representations must be stored for high quality reconstructions.
Notably unlike VQVAE-2 (Razavi et al., 2019), each module is learned greedily without backward communication between modules using the current estimate of =q
((-1) similar Belilovsky et al.

---

Under review as a conference paper at ICLR 2020
(2019); Nøkland & Eidnes (2019). This formulation is important for allowing the modules each converge as quickly as possible at their respective resolution. In other words a subsequent block is not required to build representationn which account for all levels of compression, thus minimizing interference across resolutions. This rapid convergence is particularly important for the case of online continual learning.
3.4 MULTI-LEVEL STORAGE
Our aim is to store the maximum number of samples in the allotted C' C'bytes of storage, while assuring their quality, and our ability to reconstruct them. The SQM allows us implement Multi-Level storage, wherein each module stores samples at its respective scale. Samples are stored at different levels based the compressors' current ability to compress them. When replay occurs, sample may be able to propogate into a lower level (and thus permit more samples to enter the storage). This process is summarized in Algorithm 2. In practice, both the SQM training and the sample recompression are done in a single forward pass, allowing for fast training.
Such an approach particularly helpful in the online continual learning setting and allows knowl- edge retention before the compressor network has learned valid representations. Note that as per Algorithm 2, samples can be completely uncompressed until the first module is able to effectively encode them. This can be crucial in some cases, if the compressor has not yet converged, to avoid storing poorly compressed representations. Further taking into account that compression difficulty is not the same for all datapoints, this allows use of more capacity for harder data, and fewer for data.
We also note, since we maintain stored samples at each module and the modules are decoupled, that such an approach allows to distribute training the individual modules in parallel and in an asynchronous manner (Belilovsky et al., 2019).
3.5 MULTI-LEVEL RESERVOIR SAMPLING
Reservoir Sampling (RS) is a critical component of selecting a representative set of samples con- tinual classification Chaudhry et al. (2019). Its popularity is due to two reasons. First, it gives in expectation a sample representative of the whole data stream. Second, it is a conceptually sim- ple algorithm, easy to implement, and gives strong empirical results. Indeed more sophisticated approaches can often be outperformed by basic RS in Chaudhry et al. (2019).
With this motivation, need to adapt RS to multi-level storage settings. There few challenges. First, in our model the capacity in terms of samples is not fixed. RS adds a new point with prob. buffer capacity
p puints seen far However in practice, the buffer capacity actually increases as the compressor network gets better. We therefore estimate it using the current capacity at the time of buffer addition. In the same vein, one could argue that in practice the compressor network's s sample capacity is actually bigger than the amount of samples it is currently storing since old samples were added when the buffer was less performant, However, since the points get re-compressed during rehearsal, this issue is mostly resolved.
Sample selection for deletion and sampling also more complex than in standard RS. While would be more advantagcous remove datapoints from the least compressed representation level, doing so introduces bias; some classes may be harder to compress, and removing them more frequently would result in more forgetting of said class. To address this issue, we perform all buffer deletion (and insertion) agnostic to the level of the sample. When memory must be freed, samples are deleted from every block such that their relative memory consumption does not change.
3.6 PUTTING IT ALL TOGETHER ONLINE COMPRESSION SQM
The online continual compression is overall summarized in Algorithm 1. Incoming data is added to memory as described in Algorithm 4 and along with randomly sampled data from storage used to update the current SQM model. The randomly sampled data also updates its representation in storage as per Algorithm 3. Algorithm 2 describes the multilevel compression scheme. Note that Algorithm 1 can be run its continual classification) which would use the same batch order as well as independently during a data collection phase.
concurrently with downstream application (e.g. online
5

---

Under review as a conference paper at ICLR 2020
Algorithm 4: AddTToMemory
Algorithm 3: UpdateBufferRep
Input: Memory M with capacity C (bytes), sample i
Input:
Memory M. Autoencoder AE with L
Nreg = BYTES(
levels, data D, distortion threshold de.
capacity = max( Nreg, NUM SAMPLES (M))
for E do
@Frcbability of adding
hidz, blocksa = AdaptComp,ess(r, AE, dea)
add ~ B SAMPLE SEEN &Bernoulli
espiell
Delete Old Repr.
DELETE(M[=)
if add then
hidz, blockia = AdapiCompressíz, AE, da,)
Add new one ADD(M, hids)
while BITS(hid,) FREE SPACE(M) do
DELETE RANDOM(M)
end
end
end
EXPERIMENTS
We evaluate the efficacy of the proposed methods on suite of canonical and new experiments. In Section 4.1 we present results standard supervised continual learning benchmarks on CIFAR- 10. In Section 4.2 we evaluate other downstream tasks such as standard iid training applied on the storage at the end of online continual compression. For this evaluation we consider larger images from Imagenet, as well as on lidar data.
4.1 ONLINE CONTINUAL CLASSIFICATION
Although CL has been studied in generative modeling (Ramapuram et al., 2017; 7; Lesort et al., 2018; Zhai et al., 2019; Lesort et al., 2019) and reinforcement learning (Kirkpatrick al., 2017: Fernando et al., 2017; Riemer et al., 2018), supervised learning is still the standard for evaluation of new methods. Thus, we focus on the online continual classification of images for which our approach can provide complement to experience replay. In this setting, new task consists of new image classes that the classifier must learn, while not forgetting the previous ones. The model is only allowed one pass through the data (Lopez-Paz et al., 2017; Chaudhry et al.; Aljundi et al., 2019; Chaudhry et al., 2019). The online compression here takes the role of replay buffer in replay based methods such Chaudhry et al. (2019); Aljundi et al. (2019). In short, apply Algorithm 1, with an additional online classifier being updated at line 13.
Here consider the more challenging continual classification setting often referred to using shared-head (Aljundi et al., 2019; Farquhar Gal, 2018; Aljundi et al., 20.8). Here the model is not informed of the task (and thereby the subset of classes within it) test time. This is in contrast to other (less realistic) CL classification scenarios where the task, and therefore subset of classes, is provided explicitly the learner (Farquhar & Gal, 2018; Aljundi al., 2019).
For this set of experiments, we primarily report accuracy, i.e. T Ei= Rr,S, and forgetting, i.e.
T T T-1 El max(R..) Ry,, with R TTT representing the accuracy matrix where R2y is the test classification accuracy on task j when task is completed.
Baselines basic baseline for continual supervised learning is Experience Replay (ER). It con- sists of storing old data in a buffer to replay old memories. Due to its very simple nature, this baseline was often omitted in continual learning papers. However, recent research made it clear that it is a critical bascline to consider, and in some settings is actually state-of-the-an (Chaudhry et al., 2019; Aljundi et al., 2019; Rolnick al., 2018). SQM can be viewed as an add-on to ER that incorporates online continual compression. In addition we consider the following baselines. iid online (upper-bound) trains the model with single-pass through the data on the same set of sam- ples, but sampled iid. iid offline (upper-bound) evaluates the model using multiple passes through the data, sampled iid. We use 5 epochs in all the experiments for this baseline. fine-tuning trains continuously upon arrival of new tasks without any forgetting avoidance strategy. iCarl (Rebuffi et al., 2017) incrementally classifies using a nearest neighbor algorithm, and prevents catastrophic forgetting by using stored samples. GEM (Lopez-Paz al., 2017) uses stored samples to avoid

---

Under review as a conference paper at ICLR 2020
Accuracy (1)
Forgetting (1)
M = 20
= 50
M 20
M = 50
iid online
60.8 + 1.0
60.8
N/A
N/A
iid offline
79.2 0.4
7..2
N/A
N/A
GEM (Lopez-Paz al., 2017)
1..8+1.1
1..1
73.5 1.7
70.7 + 4.5
iCarl (5 iter) (Rebuffi et al., 2017)
2.2
1.6
2.4
10.61111
fine-tuning
0.3
1.34
8.74
0.7
ER
1.2
3.71
3..4 2.4
ER-MIR (Aljundi et al., 2019)
2.1
4..0
2.0
30.2 2.3
(Riemer al., 2018)
2.0
2..
2.
67.2 - 3.9
SQM (ours)
39.9 1 0.8
46.2 0.8
1.1
42.8 + 1.3
Table 1: Shared head results on disjoint CIFAR-10. Total memory per class M measured in sample memory size. We report (a) Accuracy, (b) Forgetting (lower is better).
increasing the loss previous task through constrained optimization. It has been shown to be strong baseline in the online setting. It gives similar results to the recent A-GEM Chaudhry et al.. ER-MIR (Aljundi al., 2019) controls the sampling of the replays bias sampling towards sam- ples that will be forgotten. We note that the ER-MIR critera is orthogonal to SQM, and both can be applied jointly. We leave this as future work.
We evaluate with the standard CIFAR-IO split (Aljundi et al., 2018), where 5 tasks are presented sequentially, each adding two new classes. Evaluations are shown in Table 1. Due to our improved storage of previous data, we observe sigmificant improvements over the other methods and baselines at various memory sizes. This is despite the drifting representation and decoder model. We can contrast SQM's performance with ER's 's to understand the net impact of our compression scheme. Specifically, SQM improves over ER by 12.4% and 13. 1% in the M=20 and M=50 case, highlighting the effectiveness of online compression. Our approach only lags ER-MIR in forgetting in the M=50 setting. However, this method is orthogonal to ours and could thus be used jointly.
We also implemented the baseline of Riemer et al. (2018) that uses discrete autoencoder with gumbel softmax (Jang et al., 2016). To compare directly to the other authors setup we benchmarked implementation on the Incremental CIFAR-100 multi-head experiment Lopez-Paz et al. (2017). 7). By using our (more sophisticated) architecture, we were able to get 60.3 vs the reported 43.7 using buffer of size 200. While SQM performs similarly in this setting (65.1), its compressed represen- tations incur significantly less drift compared to Riemer et al. (2018). In the single head setting, this distortion becomes much more problematic as the classifier learns to assign old labels blurry images, leading to poor performance. We show an example of this drift in Fig. 4 in the Appendix.
The CIFAR-10 dataset has low resolution (3 32 32) and uses lot of data per task (10K samples). These two characteristics might leave the online compression problem easier than in a real-life scenario. Specifically, if the first tasks are long enough and the compression rate is not too large, the model can quickly converge and thus not incur too much representation drift. Indeed, we found that using a single module worked best for this task, as adding more modules was not worth the additional storage cost. For these reasons, we study the adaptive instantiation of our proposed method (A-SQM) in more challenging settings presented in the next section.
4.2 OFFLINE EVALUATION ON LARGER IMAGES AND LIDAR
Besides the standard continual classification setup, we propose several other evaluations to deter- mine the effectiveness of the stored data and compression module after learning online compression.
Offiine training on Imagenet We compare the effectiveness of the stored memories of SQM after certain amount of online continual compression. We do this by training in standard iid way an offline classification model using only reconstructions obtained from the storage sampled after online continual compression has progressed for a period of time. In each case we would have the same sized storage available. We remind the reader that simply having more stored memories does not amount to better performance their quality may be severely degraded and affected by drift.
We use the mini-imagener dataset, but resized to 128 x 128, larger than the typical size used as we aim emphasize the utility of this methods for larger inputs. Online continual compression arrives
7

---

Under review as a conference paper at ICLR 2020
Accuracy
RS
5.2
2 Module A-SQM (ours)
19.8
Ablate Modular Training
12.7
Ablate Adaptive Compression
10.8
Ablate 2nd Module
9.61
Ablate 2nd Module & Adaptive compression
12.0
Table 2: Offline training evaluation of storage from online continual compression. We see clear
gain over a standard Reservoir sampling approach. We then ablate each component of our proposal showing each component is important. Note storage used in each experiment is identical (including accounting for model sizes).
Imageneet buffer sample proportion
1.0
uncarr preseed
0.004
Blocko
Block]
Avg. valid loss Aug. dift cam
0.002
qumbel
0.000
Lider bulfer semple proportion
Reconstructio Loss
1.0
0.003
0.004
0.003
0.0003
0.0000
10
20
30
50
Tasks
Tasks
Figure 2: Left: Distribution of buffered samples for the Imagenet (top) and Lidar (bottom) ex- periments. We highlight that SQM's adaptive buffer is flexible enough to choose different sample proportions depending on the scenario. Right Top: Drift in the buffered lidar representations. Right Bottom: Blockwise validation loss for lidar evaluation
using the standard pplitmminmaagenet from Chaudhry al. (2019), which yields 20 different tasks and 100 classes total. After all samples have been seen and stored as best possible in storage size by Algorithm I, we train Resnet18 model (similar to the one used Chaudhry et al. (2019) adjusted for larger input size) using the stored samples. We train with SGD and learning rate of 0.1, with early stopping using a validation set. The storage size C' is equivalent to 1000 uncompressed samples. Results of this evaluation are shown in Table 2.
Using this evaluation we first compare a standard reservoir sampling approach on uncompressed data to 2 module A-SQM using the same size storage. We observe that performance is drastically increased using the compressed samples. We then use this to perform a series of ablations to demon- strate each component of our proposal is important. First we ablate the modular training, meaning the model is trained end-to-end instead of greedily, which is unable to perform well online. Indeed, we observe that the convergence speed is slower, resulting in fewer samples being stored for the early tasks. Secondly we consider not using the adaptive compression scheme described in Sec 3.4, thus samples are compressed at the bottom level. This greatly decreases performance, for two reasons: the compressions stored early on are very poor since the model has not converged yet, and the model cannot allocate more capacity to more challenging samples.. We then consider remov- ing the 2nd Module showing that multiple levels aid in performance. We additionally illustrate the change the amount of samples stored at each level by adaptive storage in Figure 2.
LIDAR Range data can be very large and storage inefficient, it is also often collected by vehicles in potentially changing environments. Efficient storage can be important for having more represen- tative data in downstream applications. Here we show qualitatively several examples of applying this. For this experiment we use the Kitti Dataset (Geiger et al., 2013), which contains 61 LiDAR scan recordings, each belonging to either the "residential", "road", "city" environments. The online compression is presented one by one with scans from each environment, we present all the record- ings from one environment, before moving on to another. The data is processed as in Caccia et al. (2018), where points from the same elevation angle are sorted in increasing order of azimuth angle along the same row. This yield a 2D grid, making it compatible with the same architecture used in the previous experiments.
8

---

Under review as a conference paper at ICLR 2020
Figure 3: Top row real uncompressed test lidar scans. Rows two, three and four recon- structions with compression rate 8, 16, and 32x. Regions of interest are highlighted in red. In the left column,
note all compression levels retain the essential information, albeit some far way obstacles are slightly deformed. In the right column, a thin obstacle is present in the first two levels, but absent in the third level.
We first show qualitative results in Figure 3, using a Block SQM, observe that we are able to effectively reconstruct the LiDAR samples. and that we easily tradeoff quality with compression. We note that reconstruction quality has been previously linked to performance on downstream tasks such SLAM Zhang& & Singh (2014). Note SQM can also be adapted to use task relevant criteria besides MSgoa 2 line 9)
We now proceed with quantitative evaluation of the LIDAR compression. We compare SQM with a 2-Block version of the Gumbel Autoencoder used in Riemer et al. (2017). We note that this baseline is stronger than in the original paper. We used our more sophisticated architecture, along with adaptive compression, which turned out to be essential to make the baseline competitive. In the challenging online continual compression setting, it is crucial that the stored representations remain valid as the model is updated. In Figure 2. We first look at representation drift incurred in the buffered samples. That is, after every task we decode every compressed representation and monitor the distortion with the true lidar scan (Fig. 2). The performance gap is quite stricking: SQM operates with drift values similar to the distortion observed test time, throughout the whole 50 training task sequence. On the other hand, the representations learned from the Gumbel AE quickly stray away from their original and become unusable. We then analyse the stackability of blocks in SQM. For lidar datasets such (Geiger et al., 2013), there is a high variance in the complexity of scans, therefore having multiple compression levels enables better storage allocation. In Fig 2, we observe that the difference in recontruction loss between blocks in SQM is small and behavde which is not the case for the Gumbel baseline.
CONCLUSION
We have introduced online continual compression. We have shown how replay combined with novel Multi-level quantization module can allow for the compression model to be learned online and yield large decodable dataset despite representation and model drift. We have shown effectiveness of this online compression approach on standard continual classification benchmarks, as well as for compressing larger images and lidar data. We believe future work can consider dealing with temporal correlations for video and reinforcement learning tasks, as well as improved prioritization of samples for storage.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Online continual learning with no task boundaries. In arXiv, 2018.
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. In Advances in Neural Information Processing (NeurIPS), 2019.
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimization of nonlinear trans- form codes for perceptual quality. In 2016 Picture Coding Symposium (PCS), pp. 1-5. IEEE, 2016.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Decoupled greedy learning of cnns. arXiv preprint 190.::9900084 2019.
Lucas Caccia, Herke van Hoof, Aaron Courville, and Joelle Pineau. Deep generative modeling of lidar data. arXiv preprint arXiv:1812.011 2018.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In ICLR 2019.
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Rieman- nian walk for incremental learning: Understanding forgetting and intransigence. arXiv preprint arXiv: 10112, 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic mem- ories. arXiv preprint arXiv: 1902. 10486, 2019.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint arXiv:1805.09733 2018
Chrisantha Fernando, Dylan Banarse, Charles Blundell. Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint
2017.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, pp. 0278364913491297,, 2013
Lionel Gueguen, Alex Sergeev, Ben Kadlec, Rosanne Liu, and Jason Yosinski. Faster neural net- works straight from jpeg. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp- 3944. Curran Associates, Inc., 2018. URL /stp://papede mips. ctl.p..cc/ppprr//// 7649- faster neural networks straight frottffrom- jpeg.pde
Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma. Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. 2018.
Ferene Huszár. On quadratic penalties in elastic weight consolidation. arXiv:1712.03887, 2017.
arXiv preprint
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzation with gumbel-softmax. arXiv preprint arXiv:161 1.01 1..1 144, 2016.
Nick Johnston, Damien Vincent, David Minnen, Michele Covell, Saurabh Singh, Troy Chinen, Sung Jin Hwang, Joel Shor, and George Toderici. Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3385-4393, 2018
Diederik Kingma and Max Welling. 1r1iv:1312.61 2013.
Auto-encoding variational bayes. arXiv preprint
10

---

Under review as a conference paper at ICLR 2020
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, pp. 20161 1835, 2017.
Timothée Lesort, Hugo Duseré, Michael Garcia-Ortiz, Andrei Stoian, and David Filliat. Generative models from the perspective of continual learning, arXiv preprint arXi::18/2.09111 2018.
Timothée Lesort, Alexander Gepperth, Andrei Stoian, and David Filliat. Marginal replay vs condi- tional replay for continual learning. In International Conference on Artificial Neural Networks, pp. 466-480. Springer, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-2447, 2018.
Long-Ji Lin. Reinforcement learning for robots using neural networks. Technical report, Carnegie- Mellon Univ Pittsburgh PA School Computer Science, 1993.
David Lopez-Paz et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Cuong V Nguyen, Yingzhen Li. Thang D Bui, and Richard Turner. Variational continual learning. arXiv preprint 1rXiv:1710.10620 2017.
Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals, arXiv preprint arXiv: 1901. 106666, 2019.
Edouard Oyallon, Eugene Belilovsky, Sergey Zagoruyko, and Michal Valko. Compressing the input for cnns with the first-order scattering transform. In The Luropean Conference on Computer Vision (ECCV), September 2018.
Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Lifelong generative modeling. arXiv preprint orXiv:/705.000
2017.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint arXiv:/9066 /906.0046, 2019.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proc. CVPR, 2017.
Matthew Riemer, Michele Franceschini, and Tim Klinger. Generation and consolidation of rec- ollections for efficient deep lifelong learning. CoRR, abs/1711.06761, 2017. URL http:
//arxiv... wrw./////aoo. abs aalbs/1111 /1711. 06111.6666.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint arXiv: 1910, 2018.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Greg Wayne. Experience replay for continual learning, 2018
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.044 144,0,6606.0467,, 04671, 2016.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. arXiy preprint urXiv:1703.003: 2017.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15(1-2):25-46, 1995.
11

---

Under review as a conference paper at ICLR 2020
Róbert Torfason, Fabian Mentzer, Eirikur Ágústsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Towards image understanding from deep compression without decoding. In International Conference Learning Representations, 2018. / httpp:/s///ppeeveie....
net/ mett/forme Forum? d=HKXWCMORN.
Chenxi Tu, Eijiro Takeuchi, Alexander Carballo, and Kazuya Takeda. Point cloud compression for 3d lidar sensor using recurrent neural network with residual blocks. In 2019 International Conference on Robotics and Automation (ICRA). pp. 3274-3280. IEEE. 2019.
Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neurul Information Processing Systems, pp. 6306-6315, 2017.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. arXiv preprint 1703.40000042000 2017
Mengyao Zhai, Lei Chen, Fung Tung, Jiawei He, Megha Nawhal, and Greg Mori. Lifelong gan: Continual learning for conditional image generation. ArXiv, abs/1907.10107, 2019.
Ji Zhang and Sanjiv Singh. Loam: Lidar odometry and mapping in real-time. In Robotics: Science and Systems, volume pp. 2014.
12

---

Under review as a conference paper at ICLR 2020
A FURTHER DETAILS OF EXPERIMENTS
We include here further details regarding the models used in the Imagenet and CIFAR-10 experi- ments. For all our experiments, we set D the size of the embedding table equal to 100. For CIFAR- 10, we use a block SQM, latent size (16 x 16 x 1) where the last index represents the number of codebooks. The codebook here contains 128 embeddings, giving a compression factor of 13.7. For Imagenet, we use 2 block SQM. The first block has a latent size (32 x 32 x 2), and the second block has latent size (32 32 1). All the codebooks have 128 embeddings, giving compression factors of 27.4 and 54.9 respectively.
B VISUAL COMPARISONS
Below we show an example of the image quality of our approach compared to Riemer et al. (2018).
Figure 4: Bottom row: random buffer reconstructions using Riemer et al. (2018). Middle row: random buffer reconstructions using SQM. Top row: corresponding original image. Columns ordered w.r.t their entry time in the buffer, from oldest to newest. All samples above were obtained from the disjoint CIFAR-10 10 task, and are 12x smaller than their original image.
13


Reviews:
[{"review_id": "rclwaBjUar", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am not familiar with the generative model and continual learning. Thus, I can only give my review based on the authors writing and other reviewers' comments. \n- The paper proposes a new problem setup as \"online continual compression\".\n- The paper gives a combination of many existing techniques to address the new problem. (I agree with Review #4)\n\nI think the presentation and the organization of this paper should be improved in order to properly place their contributions in the literature. \n- Since the authors try to promote a new problem set up with a solution containing little technical breakthroughs, I suggest the authors put more space on motivating the application and showing its impotence. Currently, their presentation focuses too much on methodology parts. \n- Thus, it is better to organize the paper as an application from LiDAR and convince reviewers why their method is good for such an application (Review #1 also thinks the presentation is poor).\n- If the authors insist on keeping their paper as a  methodology one, at least one more experiment (not the synthetic one on ImageNet) from real applications are needed (same as Review #2).\n\nOverall, I think the paper is not ready for being published. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJlpS_iRYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the problem of continual learning with limited memory storage. Specifically, the training data is arrived sequentially (might not be i.i.d.) for a model to exploit and there is not enough storage capacity to keep all the data without compression. This problem is important in many real-world applications with massive amount of data collected. The authors propose an approach named Stacked Quantization Modules to compress the data so that they can be stored efficiently. Each module is an auto-encoder with quantized latent representations. Several aspects including the communication between these stacked modules, and which level will a specific sample be compressed at, are taken into account in the algorithm design. In the experiments, the authors show some quantitative evaluations on CIFAR10 and ImageNet that the proposed method surpass several baseline methods. A qualitative visualization of LiDAR data reconstruction is also demonstrated. Overall I think the paper is tackling an interesting problem with an effective and novel solution. \n\nI have a few concerns that I wish the authors could help to clarify. First, in the VQ-VAE, each image is quantized to be H*W*D, where each D-dimensional vector is represented by the index of the nearest neighbor in the embedding table of each module. I checked the paper but could not find a place that discuss how this embedding table comes from. It is pre-defined with some pattern or is it learnt somehow? \n\nWhat is the latent space size of each module when trained on CIFAR10 and ImageNet? \n\nThe experiments on ImageNet only select 100 classes out of the 1000 classes. Would this method extends to large-scale datasets? How would the form of the tasks (in case of number of classes per task) affect the results? \n\nThere seems to be some typos. For example, the end of the first paragraph of Sec. 4.1 mentioned \"line 13\" of Alg. 4, which is not referred correctly as Alg. 4 only has 10 lines. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkgU6JAW5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presented a Stacked Quantization Modules (SQM) for the problem of Online Continual Compression, based on the VQ-VAE framework by van den Oord et al. (2017). Experiments were conducted on online continual image classification benchmarks to show the effectiveness of the proposed SQM. In general, the novelty of the paper is a little bit limited and the writing of the paper is not very easy to follow.\n\n- The SQM was constructed by stacking the known VQ-VAE. It is unclear why the stacking works for online continual compression. How many stacks should be used? What are the yellow rectangle parts in Figure 1?\n\n- What are the relationship between Alg 1-4 ? More explanations or discussions are necessary.\n\n- In Section 3.1, \"The high level training of the online learned compression is described in Alg. 4.\". It is very confused. I can't see the related content in Alg.4.\n\n- In Section 4.1, \"In short, we apply Algorithm 4, with an additional online classifier being updated at line 13.\" I don't understand it. I cannot see line 13 in Algorithm 4, because there is only 10 lines in Algorithm 4.\n\n- In Section 3.3, BITS(.) needs definition.\n\n- In Section 4.1, \"Here we consider the more challenging shared-head setting, where the model is not informed of the task (and thereby the subset of classes) at test time. This is in contrast to other (less realistic) CL classification scenarios where the task, and therefore subset of classes, is provided explicitly to the learner Farquhar & Gal (2018).\" It is very difficult to understand what the above experimental settings are.\n\n- For Figure 3, the textures or lines of the bottom reconstructed one are not so smoothed or straight as the top one.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "H1g1GUrr9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The study tackled the problem of limited storage for ever-growing data for a long-term learning scenario. The authors proposed to stack Quantization Modules while separating them during training to obtain an online compression system that has multiple resolutions, different memory horizons, and reduced catastrophic forgetting. They also proposed a modified reservoir sampling to accommodate this architecture. \n\nThe idea is very simple yet interesting, the paper is a good read, and the results seem promising. The ablation study is well designed but not discussed enough. Additionally, the experiments cannot support the idea well since it is on a very special setting, the LiDAR experiment is missing quantitative evaluation, and different tasks (such as text classification, or visual tracking with only one labeled sample) might introduce different difficulties in this online learning setting. I recommend a weak accept for this paper to encourage the idea. \n\nTherefore, I would recommend the authors to explore other tasks and see if their idea applies to different domains and tasks. Also, a quantitative evaluation for the LiDAR experiment with enough details and some explanation of the inner dynamics of the system during learning seems essential. \n\nThe paper could enjoy a pass of proofreading and typesetting (especially please pay attention to the correct use of \\cite{} and \\citep{}). Algorithm 1 is not mentioned in the body of the manuscript.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byl22PIRnr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work contributes to introducing a problem called Online Continual Compression. This problem requires to avoid catastrophic forgetting and learn in an online way. Generative methods should be one of the popular ways to do continual learning. This work’s model can be categorized into this clue since it also aims to save samples from old tasks by learning a generative model. In this way, the generator plays a similar role Experience Replay (ER) (here is called Generative Replay). The main core of this work should be the stacked quantization modules (SQM) which can be regarded as a hierarchical variant of the VQ-VAE model. In their SQM, hidden encodings z_q^i will be encoded and its input is z_q^{i-1} which is from previous layer. \n\nThis works covers related works very well. However, there are some questions I am really concerned:\n1)\tAbout the studied problem “Online Continual Compression”, what’s the difference between “online” and “continual”? In continual learning, tasks will be learned sequentially, right? If so, continual learning should run in an online learning way. \n2)\tThe motivation of the hierarchy in this work is unclear. What I mean is that the hierarchical model should be expected to capture higher-level semantic features. But in this work, the index outputs z_q^{i-1} is encoded by its subsequent layer. It seems a bit weird since the z_q^{i-1} is not an image and its elements are index values. So what is the higher-level semantic information? By the way, it seems that there is an error in the model figure 1. The last MSE from Block 1 should be connected to the block before decoder 1 in Block 1, rather than the reconstructed one from decoder 1. Therefore, I strongly suggest authors give more insights and clarify the motivation of hierarchy. Writings in the METHODOLOGY part is unclear. More details about the SQM model should be described in a mathematical way.  \n3)\tAnother question about the details of generative replay. How do you do the replay? Details about this can’t be found in this work? In Alg.1, what is the \\theta? Is the \\theta_{ae} at line 14 of Alg.1 wrong? It should be \\theta_{gen}, right? \n4)\tYou use the data-stream technique reservoir sampling to add and update the memory buffer (alg. 4). Will it lead to some information loss? Can we just update memory without reservoir sampling? Please give more insights about this.\n5)\tHow to find the distortion threshold d_th in Alg.2? \n6)  the part of ablation studies is good. But I suggest authors should consider a baseline with the same proposed framework but using a single-layer VQVAE with the same memory capacity as the hierarchical models. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new problem setup as "online continual compression". The proposed idea is a combination of existing techniques and very simple, though interesting. Parts of the algorithm are not clear, and the hierarchy is not well-motivated. Experimental results seem promising but not convincing enough, since it is on a very special setting, the LiDAR experiment is missing quantitative evaluation, and different tasks might introduce different difficulties in this online learning setting. The ablation study is well designed but not discussed enough.
Decision: Reject