On Overcompression in Continual Semantic Segmentation
Authors: Maciej Kowalski
Thomas L Lee
Amos Storkey
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Continual Learning
Class-Incremental Semantic Segmentation
Information Bottleneck
overcompression
dropout

Abstract: Class-Incremental Semantic Segmentation (CISS) is an emerging challenge of Continual Learning (CL) in Computer Vision. In addition to the well-known issue of catastrophic forgetting, CISS suffers from the semantic drift of the background class, further increasing forgetting. Existing attempts aim to solve this using pseudo-labelling, knowledge distillation or model freezing. We argue and demonstrate that frozen or rigid models suffer from poor expressibility due to overcompression. We improve on these methods by focusing on the offline training process and the expressiveness of the learnt representations. Beyond the characterisation and demonstration of this issue in terms of the Information Bottleneck principle, we show the benefit of two practical measures: (1) using shared but wider convolution modules before final classifiers to improve scaling for new, continual tasks; (2) introducing dropout into the encoder-decoder architecture to improve regularisation and decrease the overcompression of information in the representation space. We improve the IoU on the 15-1 and 10-1 scenarios by over 2% and 3% respectively while maintaining a smaller memory and MAdds footprint. Last, we propose a new benchmark setting that lies closer to the nature of lifelong learning to drive the development of more realistic and valuable architectures in the future.

On Overcompression in Continual Semantic Segmentation
Anonymous Author(s)
Affiliation Address email
Abstract
Inasemnncremtttll Semantic Segmentation (CISS) is an emerging challenge Continual Learning (CL) in Computer Vision. In addition the well-known issue of catastrophic forgetting, CISS suffers from the semantic drift of the background class, further increasing forgetting. Existing attempts aim solve this using pseudo- labelling, knowledge distillation or model freezing. We argue and demonstrate that frozen or rigid models suffer from poor expressibility due to overcompression. We improve on these methods by focusing on the offline training process and the expressiveness of the learnt representations. Beyond the characterisation and demonstration of this issue in terms of the Information Bottleneck principle, we show the benefit of two practical measures: (1) using shared but wider convolution modules before final classifiers to improve scaling for new, continual tasks; (2) introducing dropout into the encoder-decoder architecture improve regularisation and decrease the overcompression of information in the representation space. We improve the IoU on the 15-1 and 10-1 scenarios by over 2% and 3% respectively while maintaining smaller memory and MAdds footprint. Last, we propose new benchmark setting that lies closer to the nature of lifelong learning to drive the development of more realistic and valuable architectures in the future.
10
11
12
14
16
Introduction
18
Continual Learning (CL) aims to address the shortcoming of standard supervised learning that requires large quantities of labelled data. It pursues a more natural, human-likk ability to quickly and continuously learn from small exposures to training data. The learner experiences stream of tasks whose relatedness is not known beforehand 4 and has the potential to quickly learn new task by re-using past experiences. Clearly, have prevent the model from naively storing all experiences. This constraint enforces the need to efficiently compress the knowledge to modest size and limit the computation used recreate representation for a task hand. Recent work Continual Learning 26 focuses on the numerous ways to combat forgetting using replay methods, continually changing the models' architecture to adapt to new knowledge or distillation methods.
20
21
22
24
25
27
28
Innseenncremetal Semantic Segmentation (CISS) is recently recognised problem that, on top of forgetting, deals with the unique issue of background shift [2] of the unknown pixels (see Section
30
2. Existing work focuses on the efficient transfer of existing knowledge between the continual steps while acquiring enough information work with the tasks at hand. The prevalent approach using rigid encoders that aim to maintain initial features throughout training while focusing on attempts to learn new classes. As result, the most successful approaches tend to fair well with initial classes while struggling with the ones learned in the online fashion.
31
32
34
Most problems, however, tend to also have offline phase where the model is initialised and learns the initial data and tasks. Mirzadeh et al. [27] show that wide convolutional models outperform
36
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

37
standard ResNet-based encoders T7 in Continual Learning, proving that the choice of the initial architecture is fundamental to combating catastrophic forgetting. Further inspired by the Information Bottleneck principle [35] we claim that we should put just as much effort into the training of the offline phase as we put into the online phase for best results.
38
40
41
1.1
Deep neural networks and overcompression
42
When developing models for CL. it is important to develop from pre-trained models that have many diverse features. Adding additional filters to existing model is resource intensive, and provides training challenges. Therefore, it is important to start with an extensive feature set that we can adapt to the different scenarios they appear. However, in initial offline training the limited data, the many unimportant features are pulled closer to existing relevant filters, and such excessive compression means that we might remove features that are crucial for learning prediction of one of the future classes.
44
46
47
48
The Information Bottleneck principle (IB) B5 can partially explain and support the above difficulty of training Continual Learning models with gradient- based optimisation. The IB interpretation of learn- ing suggests that hidden layers in neural networks 53
50
52
54 learn to maximally compress the characteristics in the input data that are irrelevant to the task hand, and where the higher level layers learn to compress first and the the lower level layers as training pro- gresses. On the other hand, the uncertainty about the future requirements posed by CL requires us to maintain as many high-quality features pos- sible. Thus, in our best interest to maintain diverse features while limiting overcompression in Continual Learning.
SSUL
DCSS
Joint SSUL
DCSS
56
57
Number learned
58
59
Figure 1: IoU and Joint upper-bounds com- parison of the proposed DCSS model and SSUL. K in the 15-1 scenario. DCSS achieves better online scores despite being worse in of- fline training, suggesting that we can improve CL performance by increasing the expressive- ness of the learnt representations.
60
61
62
63
64
Although the importance of the Information Bottle- neck theory has been questioned by some papers 31I. it serves as an interesting point of view on the learning phases of neural networks and can be noticed empirically. We know that wider models better manage catastrophic forgetting (27) proving that passing information through the bottleneck, which inherently imposes compression, can have adverse effects on future tasks. Therefore, we use two simple approaches to verify if the reduction of overeompression can give us attainable gains in CL, particular in CISS:
65
66
87
88
69
70
71
72
1. Increase the width of the last layer to reduce the bottleneck effect and increase the amount of information for each subsequent continual task;
73
2. Introduce dropout layer between the pre-trained encoder and decoder to introduce uncertainty into the model, encouraging a more robust representation that is later reused by the future steps.
75
77 By developing Dropout Continual Semantic Segmentation (DCSS), which incorporates only these two simple additions while improving on previous works by sizeable margin (Figure [ demonstrate that improving the representation provided by the initial network has a significant impact on overall results.
79
2
Related work
82
2.1 Catastrophic forgetting
83 Catastrophic forgetting 30 24
is a phenomenon observed primarily in Continual Learning where 84 the earlier learned concepts are forgotten while incorporating the more recent samples. Forgetting appears when previously- representations are degraded by more recent exposures, a typical
86 case in all SGD-based algorithms. Solutions proposed to address this issue can be grouped into the following categories:

---

88
Rehearsal learning In rehearsal learning, we utilise the fact that, although we have lost access to the original training data from the past, we are usually allowed maintain the data in different 90 form. 3 store exemplar images in small memory buffer that can be replayed to simulate previous
89
91
data distribution. Compressed features in the form of embeddings [S] 20 can be used as an efficient form of memory that can also have advantages in terms of privacy. Finally, we can use generative strategies [32] that can efficiently store and reconstruct past experiences to enable the replay of raw images. [2 uses a brain-inspired dual-memory system where the new memories are consolidated
92
93
94
95
from recent memory to a long-term memory using a generative model, similar to mechanisms that occur during sleep.
97
Adaptive architecture Other approaches focus on the adaptability of the model architectures. To integrate new classes and tasks, can extend the architecture using feature extractors with domain- specific trainable layers [24] [9.. Model freezing can help with performance degradation during fine-tuning of the new models for new tasks. It is also possible to adapt the networks without explicitly adding new modules. [12 dynamically re-arranges existing sub-networks, each specialised in
98
89
100
101
102
one specific task, account for new knowledge. Moreover, continually changing data distributions can be accounted for by explicitly correcting the classifier drift [ 38
103
104
Knowledge distillation Knowledge distillation methods consider the use of sacchersttueen approach. A knowledgeable teacher model trained on past inaccessible experiences can inform the current model of the past. Therefore, distillation aims at constraining the model to prevent forgetting as it changes to adopt new data. There are several ways to constrain the model, with the most salient
106
107
108
methods being applied to the weights (22). gradients
25 or output probabilities [2 29].
109
2.2
Information Bottleneck principle
110
Tishby et al. the Information Bottleneck principle [35] claim that modern Deep Neural Networks undergo two phases of learning. During the generalisation phase, the model learns an internal feature representation to extract high-level information, used for final prediction. In the compression phase, unused and noisy features that prevent robust adaptation to the data are stripped away and removed from the feature space, ultimately improving useful fcatures' signal quality. Tishby et al. measured the mutual information / [T; and (TT; Y). which quantify the hidden layer's 's information about 116 the input and the output, where is the input, Y is the output and T is the internal representation. They showed that the amount of information about the output [TT; Y) steadily increases until high number of epochs is experienced, and the model starts overfitting. The information about the input TT; raises initially but quickly starts decreasing. This decrease can be considered as the compression phase because we remove information from the input that does not contribute much to the output. Compression accelerates rapidly when the number of training epochs is high.
112
114
118
119
120
121
122
2.3
Class-incremental semantic segmentation
128
2.3.1
Problem definition and notation
124 We follow the definition of van de Ven et al. (36 who use three distinct scenarios for Continual Learning: Task-Incremental, Dommain-Incrementa and maasemnncremttall learning. This work focuses on the Class-Incremental setting initially formulated by Cermelli et al. (2] where we split the set of semantic segmentation classes into f = 1,..., incremental tasks, with & 1 being the offline step. The model learns each disjoint set of classes incrementally while trying not to forget the previous steps. At each step a new task / arrives with a training dataset D, that consists of pairs (x*
125
126
127
128
129
y') ). where x' E N denotes an input image of N pixels, and y² E (C* U {cf))) denotes the
130
131
corresponding moound-truth pixel labels, where C³ is the set of classes that are seen task t and cj is the background class for that task.
132
133
At test time, the semantic segmentation model fg is required to predict whether a pixel belongs to class learned so far, E clit-1 u or the true background class cb. However, the labels of an image from a task f only contains classes from C1, not classes seen in past or future tasks. Thus, during training, the background label of, is also assigned to the pixels of potential objects that belong to past classes Cli--1 and the future classes C c++1:T (Equation
135
3

---

= = UC LUC--lu UCt
(I)
past
future
138
This counter-intuitive behaviour of the background class' labels causes the problem of background shift [2]. on top of the catastrophic forgetting found in CL. Each pixel with background label can be a class from the past, future or the actual background. This introduces unwanted noise into the model 140
139
141
because the same object can be labelled differently, depending on the step t. Once the learning of task
1 by the model S is done, the prediction for pixel i of an input image at test time is obtained by
y
arg min (1,c(a) ccy14
for all the classes learned so far in Cl The performance is measured by the Intersection-over- Union (loU) metric for the three splits of Cli,. Cermelli et al. [2] introduce two different settings for Ineremnnl Semantic Segmentation: Disjoint and Overlapped. Both cases consider training examples where only the background class cf and classes in C' are used as labels. In Disjoint, only
143
144
145
146
147
the old and present classes can appear, while in Overlapped pixels can belong to any old, current and future classes. Thus, following the previous works, we also focus only on the Overlapped setting it is more challenging and closer to real conditions.
149
150
2.3.2
Existing work
151
Class-Incremental Semantic Segmentation is a relatively new problem, with few works attempting to address it Cermelli et al. in MiB [2] first defined the issue of background shift, problem unique to Continual Semantic Segmentation. To alleviate the issue, they proposed to adapt the loss function such that it takes into consideration potential previous classes Cl:f-1 that could be included in cf They sum the output probabilitiess of C lit and C to match the available label cf that contains them when computing the cross-entropy loss. Conversely, they sum the probabilities of C and rg'-1 for knowledge distillation to match the output of fō-!. The summation of class probabilities to modify the cross- cross-entropy and distillation losses is a novel approach in semantic segmentation, but it makes hard to learn the underlying probability distribution for the classes at each pixel.
152
153
155
156
158
160
OPOP g extended the knowledge distillation on output probabilities from with Localised Pooled Distillation (9) a form of heavy attention transfer that prevents from forgetting at the cost of decreased flexibility. Additionally for task f, the target labels y are augmented with generated pseudo-labels
162
163
with a model fe-1 from a previous step. This yields the target semantic maps ÿ containiing background class c/. pseudo labels f-1 and current labels y²
23
184
165
SSUL
3 finds success with further increase of model's rigidity using model freezing. Instead of fine-tuning the whole model at task f, all parameters from fø ft-1 are being frozen in fø preventing
166
any changes to the prediction of classes Cl:1. Moreover, SSUL uses saliency maps generated by a separate, off-the-shelff salient object detector [99 to predict region of interest from the background which helps to differentiate the difference between true background and background that may contain past future class, which they label using the unknown class. Similarly to pseudo-labels, the saliency map is added to the labels y and predicted by the model during training. The unknown class
167
169
171
C, and the background class cs predictions are combined at test time, representing the unclassified pixels. Taking advantage of the foreground prediction, SSUL performs weight transfer from the unknown class to the new classifier d!-1 + o. at the beginning of each continual step.
173
174
175
3
DCSS model design
178
3.1
Wider classifiers with shared representation
177
Overcompression can hurt the learning of future tasks. Our goal is to maintain as many features as possible for future use. One simple way to achieve this is to use wider networks that are no longer bottleneck to the information signal. To verify our hypothesis we decide to increase the number of channels the last layer of the frozen, offline representation. In this way, we should have more information available for the continual steps.
178
180

---

Online
3k3 Conv 258 chennels
1xt Conv
ResNet RosNet encoder
ASPP
3x3 Conv 258 chennels
Txt Conv
3x3 Conv 258 channals
1xt Conv
Frozen
Figure 2: High-level architecture of SSUL trained in the online phase.
Each step has its own x 3 convolution block that
Online
Drapout p
ASPP
Res
Conv
Frozen
Figure 3: High-level architecture of our DCSS model. One HEAD module trained in the offline step and frozen. Dropout layer is placed right after the pre-trained encoder to increase uncertainty in the learnt decoder.
182
In the V3 architecture S used for this project the final 1 x classifier is preceded by the ASPP module and one 3 x convolution. The last 3 convolution important to aggregate and mix multi-scale features produced by the ASPP module. We will refer to this 3 x 3 convolution layer as HEAD for readability reasons. We increase the size of HEAD output channels from 256, as in the original DecpLab to 2048 channels. Hence, more detailed feature space is available for the classifiers to exploit. Recent papers tackling CISS, including MiB, PLOP and SSUL, replicate the HEAD module for each step. Therefore, each set of classes C has its own HEAD and final classifier, as seen in Figure 2 However, the HEAD layer a costly operation that has over 0.5M parameters with 256 channels and over 4.5M parameters when using 2048 channels. Therefore, prevent the large parameter accumulation we use only a single HEAD layer, frozen after offline training seephaisee-epprrbble
185
185
187
189
191
182
and add only 1 X convolutions online steps 1. Additionally, convolution in the shared HEAD to produce the same activation map using a cheaper operation [ with fewer parameters and MAdds. In work, we have used intermediate width multiplier of 4 and 4 convolution groups. Standard shared HEAD has almost twice the parameters of the depthwise- separable implementation (4.71M 2.68M). The high-level architecture comparison can be found in
193
195
196
197
Figures and
The reason why at step = 1 the DCSS is larger than standard DeepLabV3, despite both being trained in an offline fashion, is the requirement of having a separate classifier for the background class c and the unknown class cu. which means that already have three HEAD modules at 1, including the contrast, increasing the size of the HEAD to produce 2048 channels with separable convolution in DCSS, compared to 256 channels in SSUL, yields significantly smaller model the long run (Figure while achieving better results than SSUL.
199
one for classes C t-
201
203
3.1.1
Dropout source of uncertainty
205
randomly sets activations of a layer to 0 with probability p. effectively disabling their contribution to the calculation of the output and introducing uncertainty that should hinder overcompression. Since the placement of the Dropout is essential, we follow Spilsbury et al. 33 and place the Dropout layer in between the encoder and decoder modules. The idea behind this decision
Dropout
34
207

---

SSUL
SSUL DCSS
Parameters (M)
76-
DCSS
DeepLabV3
Multiply-Aod A-Add (B)
DeepLabv3
0.5
60.5
363
Number of classes
Number of classes
Figure 4: Resource comparison in the 15-1 scenario. SSUL adds significantly more trainable parameters each step, surpassing DCSS size and complexity in just 2 steps.
209
is that we want to prevent interference in the pre-trained encoders while maximising the effect on the decoder.
210
211
We test two popular Dropout approaches CNNs: standard Dropout and channel Dropout, where the whole convolutional channel can be dropped with probabilityy p. Moreover, use ScheduledDropout that linearly increases the dropout probability from 0 to p during training. During the exploration phase (generalisation), smaller values of p help with feature accumulation. Heavier regularisation is more useful during the exploitation phase (compression), thus requiring higher values of p. In the end, obtain similar benefits while improving learning convergence. Last, we suggest the removal of the Dropout after offline training. Limiting the access to features in online training will only restrain the model from using them while offering few benefits in generalisation since we freeze the encoder and are unlikely to learn new features.
213
33
215
217
218
219
4
Experiments
220
221
The code for this project has been based on the implementations of PLOP papers and SSUL that use the DeepLabV3 architecture (5). To compare the results of DCSS to we also use ResNet101 while keeping in mind that a smaller encoder would ratio. A standard learning rate of 0.01 has been used, with the value decreased to 0.001 for the backbone when using pre-trained weights (5). We use poly learning rate scheduler, as proposed by the DeepLabV3 authors. The same data augmentation has been used as in SSUL. We use batch size of 24 for all experiments compared to the size of 32 used by SSUL, as we found more success with smaller values, especially for the continual steps. We use binary cross-entropy loss with sigmoid activation function, as in SSUL. Pseudo-labels are added to labels in the continual steps, with an entropy-based threshold of T = 0.9. In CISS problem we train each step for 50 epochs. The experiments were run 4 RTX 2080Ti GPUs using the torch. disstriuuee library 7 to work around the limited GPU memory. Each experiment, including the offline step, takes approximately 2-4 hours, depending on the scenario. In the case where the offline step can be reused, the training usually is decreased to 1-2 hours.
Implementation details
222
223
previous work 2 9 3
224
have a better performance-ton
225
226
227
228
230
232
234
235
The experiments are evaluated on the PASCAL VOC 2012 10 semantic segmentation dataset with 20 foreground object classes and one common background class. The dataset augmented with the extra contour annotations [14] of the PASCAL VOC 2011 dataset. Following [9) we split the training set and used 80% for training and 20% for validation, with
PASCAL VOC 2012
237
239
the testing set left untouched for model evaluation. To help distinguish the background containing potential future classes from the true background, SSUL [3] proposed the of off-the-shell
241
salient object detector [19 to predict a region of interest. Thus, we follow their implementation and extend the labels of the PASCAL VOC training set to include the additional foreground class.
'The scheduler has been called "poly" by the authors of DeepLabV3, even though the learning rate is not polynomial (37]

---

Table I: Main results for the CISS problem. DCSS previous works in new classes (middle columns) in all scenarios and achieve improvements in combined scores in the more difficult scenarios with multiple tasks.
VOC 10-1 (11 tasks)
VOC 15-1 (6 tasks)
VOC 3 (6 tasks)
VOC 19-1 (2 tasks)
VOC
15-5 (2 tasks)
Method
0-10
11-20
all
0:15
16-20
all
0-5
6-20
all
0-19
20
all
0-15
15-20
all
ILT 26
7.15
3.67
5.50
8.75
7.99
8.56
22.51
31.66
29.04
67.75
10.88
65.05
67.08
39.23
60.45
MiB
12.25
13.09
12.65
34.22
13.50 29.29
57.10
42.56 46.71
71.43
23.59
69.15
76.37
49.97
70.08
PL.P
44.03
15.51
30.45
65.12
21.11
54.64
17.48
19.16
18.68
75.35
37.35
73.54
75.73
51.71
70.09
SSUL 3
71.31
45.98
59.25
77.31
36.59
67.61
71.17
45.38
52.75
77.73
29.68
75.44
77.82
50.10
71.22
DCSS (ours)
73.34
50.20
62.32
77.66
42.69
69.33
68.10
48,83
54.34
77.22
36,85
75.30
77.49
51.49
71.30
Joint (V3)
78.41
76.35
77.43
79.77
72.35 77.43
76.91
77.63
77.43
77.51
77.04
77.43
79.77
72.35
77.43
Joint (DCSS)
77.80
76.58
77.22
78.77
72.25
77.22
76.29
77.61
77.22
77.30
75.60
77.22
78.77
72.25
77.22
243
Evaluation metrics The difficulty of the Class- Innremennrr challenge also depends on the number of steps and the number of new classes. Thus, our model evaluated on five different scenarios generated from the PASCAL VOC, each with a varying level of difficulty: 10-1 (11 tasks), 15-1 (6 tasks), 5-3 (6 tasks), 19-1 (2 tasks) and 15-5 (2 tasks). The numbers in each scenario define the number of classes introduced at each step. For example, 5-3 (6 tasks) means learning 5 base classes
244
246
247
248
at the offline step & = 1, followed by incremental steps & E { 6} introducing 3 new classes at each step. yielding 6 training steps covering all 20 classes for PASCAL VOC. We report the results as (IoU) for three categories of classes: old (offline), new (online) and all classes combined (offline online). This split helps to distinguish the difference between forgetting and issue with obtaining the knowledge for the new tasks.
250
mean antersection-over-nnit
252
4.1
Experimental results
254
In Table | we observe that DCSS consistently outperforms other models while having a simpler and more extendable architecture that is also easier to train (Figure B). the most popular 15-1 scenario, DCSS achieves 1.72% improvement over SSUL. The biggest overall gain of over 3% can be found in 10-1 scenario that has a larger number of continual steps. Most importantly. in all scenarios we have improved the mean score of the new, continual classes, up to 6% in the case of 15-1. This gain can be surprising result, considering that we are only training a single 2048 2048-dimensn dimensional vector at measures used in CISS effectively prevent models from adding new features to the representation and, thus, is enough to simply learn a linear mapping of existing features during the continual phase.
255
256
257
258
259
260
each step. We conclude that the current counter-forgetion
261
263
To compare our results to the ones produced by previous works, we have to constrain ourselves to model of a similar learning capacity. Figure I proves that we increase Continual Learning perfor- mance despite the decreased overall performance of DCSS in offline learning, further signifying the importance of our contributions to online learning. What is most important, all of this is achieved with a smaller and easier to train model. In 15-1 DCSS has over 2M parameters less than SSUL. (Figure while in the extreme case of 10-1 the model is actually smaller by over 5M parameters and the difference grows linearly with each additional step.
Drapouttd
264
012
Dropout2d
285
286
267
268
269
612
08
e2 Dropout
271
272
273
Figure 5: IoU comparison of Dropout in 15-1 scenario. We notice that a small amount of un- certainty produced by the Dropout can increase the loU of the model.
275
276
Figure 5) shows improvement over the range of the Dropout probabilities p compared no Dropout
277
278
(p 0). Simply by adding the Dropout layer during the offline training we have increased the performance during the online steps, yielding a more accurate model on average. We have found
279
more success with the standard dropout (Dropout d) rather than the channel dropout (Dropout2d),
281
despite literature stating the reverse for CNNs [ [8] The results follow our intuition, which
suggests that dropping the whole channels removes feature across the whole image, whereas
7

---

Table 2: Experimental results on the proposed RCISS protocol. Both SSUL and DCSS decrease their performance in new tasks, although the scores lie surprisingly close to the original scores. This suggests that more training cannot extract missing features from the frozen model.
VOC 10-1 (11 fasks)
VOC 15-1 (6 fasks)
VOC 5-3 (6 tasks)
VOC 15-5 (2 tasks)
Method
Epochs
0-10
11-20
all
0-15
16-20
all
0-5
6-20
all
0-15
15-20
all
SSUL B
50
71.31
45.98
59.25
77.31
36.59
67.61
71.17
45.38
52.75
77.82
50.10
71.22
DCSS
50
73.34
50.20
62.32
77.66
42.69
69.33
68.10
48.83
54.34
77.49
51.49
71.30
SSUL. 8
73.23
38.10
56.50
77.34
32.44
66.65
69.90
27.52
39.63
76.68
47.64
69.76
DCSS
73.78
43.26
59.24
77.16
37.46
67.71
68.44
38.84
47.30
77.46
45.60
69.88
283
standard dropout can work inter-channel, which is vital in pixel-level tasks. Removing whole channels can prevent the propagation of features, while removing specific neurons can decrease the reliance on more salient parts of the image and promote more holistic attention.
285
286
4.1.1
New protocols and evaluation
287 We have seen the performance of DCSS on the tasks proposed initially by [2]. DCSS offers a better IoU score while being a generally cheaper model to train in the long run. We argue, however, that the evaluation protocol has an unrealistic assumption about the ability to store the continual training data
289
and use it multiple times over the step f. This is unnatural setting that promotes irrelevant models. Continual Learning models should aim to efficiently use the available data and aim for an approach
291
292
similar few-shot learning, where only few examples for each class are available. Therefore, we
293
also evaluate our DCSS model on the proposed Restricted Class- Incremental Semantic Segmentation protocol (RCISS), where train each continual step for only one epoch instead of the 50 in SSUL. This scenario lies closer to the lifelong learning setting by Chaudhry et al. (4]
284
285
In the experiments on RCISS we remove learning rate scheduling and use fix learning rate of 0.01. Moreover, we reduce the batch size in the continual phase from 24 to 4, effectively increasing the number of backpropagation steps in the hope that, while being noisy, it will better utilise the limited exposure to data. Table 2 shows the result of our experiments on four scenarios in RCISS, similar to previous scenarios in standard CISS. We notice that both models struggle more with learning multiple classes at once (5-3), although DCSS manages to outperform SSUL in the continual classes by a considerable margin. However, notice that the results for epoch are still similar to the ones with
297
299
301
302
303
full 50 epochs of training. We find this result interesting as it signifies the difficulty of extracting performance in continual learning even with extensive training.
304
Discussion and conclusions
305
308
We showed that the current approaches to CISS problem put too much importance on the continual phase of learning. The small difference between RCISS and CISS shows that there is limit to the amount of information that can be obtained by simply learning new tasks with a continual manner while having significant constraints on the model flexibility to prevent catastrophic forgetting. The balance between rigidity and flexibility is hard achieve, and most recent works found success with
307
308
309
310
the focus on the former. With heavily constrained or frozen representations we are unable to adapt to new tasks or classes. Thus, assuming that the potential cost of adding features is much higher than removing them, we take a step back to consider what can be done to maintain potentially useful features for the future that would be pruned otherwise. We show that with a few simple changes to the offline training protocol we can propagate more information to future learning steps, lessening the need for complex online methods.
312
313
314
316
317
We adapted the idea of wide continual networks 27 for model freezing. Wider architecture of the decoder can help to maintain more features for CL; surprisingly, even though we are essentially learning linear mapping of the embedding from the frozen model we improve the IoU in CISS. This
318
319
320
further suggests that having a rich representation outweighs most attempts at learning new features the benefit of being easily scalable and far simpler to train. The simple addition of a single Dropout layer further improved the loU, surpassing all current approaches based on the DeepLab model.
using additional trainable layers while having
322
8

---

324
Despite that, our work aims to show not the benefits of the proposed methods but rather the general lack of consideration of overcompression in Continual Learning, Information Bottleneck principle suggests that deep learning models have generalisation and compression phase. Although useful in offline learning. compression is counterproductive in Continual Learning where we don't t know the future requirements. We believe that most short-term gains in CL could be achieved by focusing
325
327
328
329
on training with future's uncertainty in mind. Further gains could be achieved with a different approach to learning that will increase the generalisation. Recent trends tend to remove the reliance on supervised training for a more contextualised signal using self-supervised learning in the form of representation learning (13), where more holistic type of learning must appear to capture the whole context. Altogether, these approaches should allow for encoders that are less spsk-specific, providing the required flexibility in Continual Learning.
331
contrastive models [66 6 or
333
5.1
Future work
336
For subsequent work on the topic of CISS, we think a good idea would be to look at expanding our findings to different architectures. For example, SATS [28]. used Transformer- measee encoder and self attentien transfer to achieve state-of-thh-ar results on CISS. So, while we expect our encoder- agnostic findings to maintain their validity for other architectures, it would be interesting to explore the topic of overcompression of Transformer models in the context of CL., Additionally, the research community is in dire need of dedicated Continual Semantic Segmentation datasets, which need to
338
340
342
emphasize the issue of the background shift problem and introduce temporal and spatial locality of the data, which should spur new interest in this topic.
343
Societal impacts
344
This paper touches upon the topic of Continual Learning which greatly increases the move towards more private and efficient training. Our contribution emphasizes the need for optimal offline training procedures that have immense impact the future utility during continual phase, improving the ease of training and, in turn, decreasing the required compute and data. We do not see any foreseeable negative impacts of this work while noticing the positive impacts of more efficient CL models that should reduce the computational burden in the short term and allow for more intelligent agents in the long term.
346
348
350
352 References
[1] Eden Belouadah and Adrian Popescu. ScalL: Classifier Weights Scaling for Class Incremental Learning, 2020. arXiv:2001.05755.
354
355
[2] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulò, Elisa Ricci, and Barbara Ca- puto. Modeling the Background for Incremental Learning Semantic Segmentation, 2020. arXiv:2002.00718.
356
[3] Sungmin Cha, Beomyoung Kim, Youngjoon Yoo, and Taesup Moon. SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning, 2021. arXiv:2106.1 15562..
359
360
361
[4] Arslan Chaudhry, Marc" Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient Lifelong Learning with A-GEM. In ICLR, 2019.
383
[5] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking Atrous Convolution for Semantic Image Segmentation. 2017. #fXiv:1706.05587
384
365
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations, 2020. arXiv:2002.05709.
386
367
[7] R. Collobert, K. Kavukcuoglu, and C. Farabet. Torch7: A Matlab-like Environment for Machine
388
Learning. In BigLearn, NIPS Workshop, 2011.
369
[8] Elliot J. Crowley, Gavin Gray, and Amos Storkey. Moonshine: Distilling with Cheap Convolu- tions, 2017. 1.X.v:1711.02613.

---

371
[9] Arthur Douillard, Yifu Chen, Amaud Dapogny, and Matthieu Cord. PLOP:
Learning without
372
Forgetting for Continual Semantic Segmentation, 2021. urXiv:2011.11323
[10]
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98-138) January 2015.
374
376
[1]] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. PathNet: Evolution Channels Gradient Descent in Super Neural Networks. 2017. arXiv:1701.08734.
378
[12]
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual Learning via Neural Pruning. 03 2019. arXiv:1903.04477
380
[13]
Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep Multimodal Representation Learning: A Survey. IEEE Access, 63394, 2019.
382
[14] B. Hariharan, P. Arbeláez, I., Bourdev, S. Maji, and J. Malik. Semantic contours from inverse detectors. In / International Conference on Computer Vision, 2011.
384
385
[15] Tyler I. Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. RE- MIND Your Neural Network to Prevent Catastrophic Forgetting, In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
386
387
388
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning, 2019, 1..072
389
390
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, 2015. arXiv:1512.03385.
391
392
[18] Mingjie He, Jie Zhang, Shiguang Shan, Xiao Liu, Zhongqin Wu, and Xilin Chen. Locality- Aware Channel- Wise Dropout for Occluded Face Recognition. IEEE Transactions on Image Processing, 31:788-798, 2022.
393
395
[19] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip H. S. Torr. Deeply Supervised Salient Object Detection with Short Connections. IEEE Transactions Puttern Analysis and Machine Intelligence, 41 apr 2019.
397
[20] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-Efficient Incremental Learning Through Feature Adaptation. 2020. arXiv:2004.00713.
399
[21] Ronald Kemker and Christopher Kanan. FearNet: Insiirrd Model for Incremental Learning. In International Conference on Learning Representations, 2018.
401
[22]
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks, 2016. arXiv:1612.00770
403
404
405
406
[23] Dong-Hyun Lee. Pseudo-Label: : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks. ICML 2013 Workshop Challenges in Representation Learning (WREPL), 07 2013.
407
408
409
[24] Zhizhong Li and Derek Hoiem. Learning without Forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-2947, 2018.
410
[25] David Lopez-Pazz and Marc' aurelio Ranzato. Gradient Episodic Memory for continual learning. June 2017. arXiv: W.Xiv:1706.088 708408
412
[26] Umberto Michieli and Pietro Zanuttigh. Incremental Learning Techniques for Semantic Seg- mentation. In International Conference on Computer Vision (ICCV), Workshop on Transferring and Adapting Source Knowledge in Computer Vision (TASK-CV), 2019. 2019
414
10

---

418
[27] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin. Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Architecture Matters in Continual Learning, 2022.
417
arXiv:2202.00275.
[28] Yiqiao Qiu, Yixing Shen, Zhuohao Sun, Yanchong Zheng, Xiaobin Chang, Weishi Zheng, and Ruixuan Wang. SATS: Self-Attention Transfer for Continual Semantic Segmentation, 2022. arXiv:220300766..
420
[29] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental Classifier and Representation Learning, 2016. arXiv:1611.077
423
424
[30]
Anthony V. Robins. Catastrophic Forgetting, Rehearsal and Pseudorchearsal. Connect. Sci., 7:123-144 146, 1995.
[31] Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. On the Information Bottleneck Theory of Deep Learning. In International Conference Learning Representations, 2018.
427
[32] Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual Learning with Deep Generative Replay. In Proceedings of the 31 International Conference on Neural Information Processing Systems, 2017.
430
431
432
[33] Thomas Spilsbury and Paavo Camps. Don't ignore Dropout in Fully Convolutional Networks,
2019. arXiv:1908.0911
[34] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(56): 1900)):9999999999 1958, 2014.
435
436
[35] Naftali Tishby and Noga Zaslavsky. Deep Learning and the Information Bottleneck Principle, 2015. utXiv:1503.02400
438
439
[36]
Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning, 2019. arXiv:1904.07734.
[37] Yanzhao Wu, Ling Liu, Juhyun Bae, Ka-Ho Chow, Arun Iyengar, Calton Pu, Wenqi Wei, Lei Yu, and Qi Zhang. Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks, 2019. Stti:1990800477.
442
(38] Yue WI, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large Scale Incremental Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 374-382, 2019.
445
446
447
[39] Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong Learning with Dynamically Expandable Networks. In International Conference on Learning Representations, 2018.
448
419
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
459
contributions and scope? [Yes]
454
(b) Did you describe the limitations of your work? [Yes]
455
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
457
458
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...

---

462
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
463
464
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Error bars are not reported because we did not have enough compute resources. The results report an average of random seeds for each experiment.
487
468
489
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
471
472
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
474
(b) Did you mention the license of the assets? [Yes] MIT License
475
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
476
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
478
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
479
480
5.
If you used crowdsourcing or conducted research with human subjects...
481
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
484
485
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
486
12


Reviews:
[{"review_id": "rfBZHDc0SE9", "reviewer": "Reviewer_tdRz", "summary": "In this work, the authors state that overcompression is a vital problem for Continual Learning and introduce two simple improvements for Class-Incremental Semantic Segmentation (CISS) problem. The first improvement is widening the network layers before the final classification layer to improve feature quality. The second improvement is adding a dropout layer after the encoder during the offline phase of the model training to create more robust features for further CL steps.", "questions": "1) lines 208-210: 'The idea behind this decision is that we want to prevent interference in the pre-trained encoders while maximising the effect on the\ndecoder.' Explain in detail this sentence, please. What has interference prevention in common with a frozen encoder, and what effect should be maximized on the decoder?\n2) Why is DCSS worse in offline training (Fig. 1) while having more parameters? \n3) What are the SD of results presented in tables 1-2?\n4) How does the proposed evaluation scheme differ from a standard online learning setting, where the model is allowed to see each data sample once?\n5) What are the limitations of the proposed improvements?", "limitations": "Limitations of the proposed improvements were not described. The authors mentioned general positive societal impact of efficient CL training.\n", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe authors focus on an important topic of overcompression, which has a negative impact on continual learning model training. Also, They show an increase in performance in specific setups using two proposed improvements. A possible novelty is using dropout layer only during the offline training phase of CL model to prevent overcompression.\n\nWeaknesses:\nOriginality:\nThe paper lacks originality. The authors propose to use previously introduced improvements for a particular problem setting. The proposed 'new' evaluation framework is also not original. Likely it is a kind of a standard online learning setup.\n\nQuality:\nOne of the main claims (line 8, lines 78-80) that 'we can improve CL performance by increasing the feature expressiveness of the learnt representations' is not directly supported with evidence within the work. The authors do not use any metrics to show improvement in the representations. Without additional ablation study, it is not clear\nhow parts of the solution impact the overall results.\n\nThere are multiple minor improvements or ideas that are not strongly connected. For example, the evaluation scheme proposal is not directly connected to other presented ideas. The authors state that they did not have enough computing resources to provide error bars for reported results. \n\nClarity:\nThe work is not self-contained. For example, the authors mention the background shift problem multiple times. They state in line 153 that this is a unique problem of Continual Semantic Segmentation, but they do not explain the essence of this problem. It is unclear how different parts of the related work section relate to the proposed solution.\n\nThe results in bold in Table 2 are not the best in columns. This is misleading. Some words are missing in the Table 1 caption.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LW0O1nuXiCy", "reviewer": "Reviewer_Wub6", "summary": "- The paper considers the problem of continual learning on semantic segmentation.\n\n- The authors propose two simple ideas to improve the architecture:\n    - using shared but wider convolution modules\n    - introducing dropout into the encoder-decoder architecture \n\n- Experiments have been conducted on the PASCAL VOC dataset, outperforming previous approaches.", "questions": "- If the over compression is a problem,  what will a more powerful architecture will perform, as the authors pointed out in the future work.", "limitations": "- limitation has been included in the paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- strength\n    - the paper is well-written, easy to follow.\n\n- weakness\n    - writing-wise, the paper discusses a lot on the information bottleneck, I don't know how relevant it is, as the further proposed solutions are more like empirical tricks.\n    - I think the considered scenario does not reflect what happens in practise, ``Class-Incremental Semantic Segmentation: Disjoint and Overlapped. Both cases consider training examples where only the background class c_b^t and classes in C^t are used as labels.'' if we do have labels for task (t-1), why are we not allowed to use them ? if for efficiency, why can't use only a portion of them, like replay.\n    - the proposed solutions, although simple, does not give too much insight, more like some tricks.\n    - experiment on PASCAL is not good enough, if the idea is to continuously learn new task, or in fact learning to segment new categories, it would make sense to test at least on COCO, with 80 classes, or even LVIS, with over 1000.\n\nOverall, I don't think this paper should be published on NeurIPS.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "78y1VyNOZHe", "reviewer": "Reviewer_TqWg", "summary": "The authors firstly reveal the overcompression of features causes model degradation when training the backbone of the continual segmentation model. They try to explain the reason from the aspect of the information bottleneck principle. Then they retrain the backbone combined with the extra Dropout layer and train the classifier with a greater number of features than SSUL. The experimental results show the effectiveness of their proposed methods.", "questions": "1. As the statement, high-quality features are required to solve the CL tasks related to the model structure. How to explain or define the high-quality or low-quality features? Why can a wider network extract the higher-quality features?\n2. The number of output channels is different from the original Deeplab V3, what datasets use to pre-train the model? How to pre-train the backbone model? Is it initialized with the DeepLabV3 pre-trained parameters? \n3. A wider network structure largely increases the number of model parameters which may enhance the model memory and manage the catastrophic forgetting problem. Obviously, the increasing number of channels means a higher capability of the model. More channels mean more features but do not mean the overcompression problem can be tackled. So, why can the increasing number of output channels solve the overcompression model? The ablation study does not compare the result from SSUL head modules with the same number of channels. If SSUL model has more channels, what will happen?\n4. During the model design phase, most models adopt dropout layers to enhance the feature extraction. So as your model, why can dropout layers solve the overcompression problem?  What is the difference between the dropout layer from your and others? As your mentioned, all layers suffer from the overcompression problem. Why is the dropout layer only inserted before the ASPP module? What if the dropout layer inserts in all the convolutional layers?\n", "limitations": "Prior knowledge about the SSUL model needs to be listed in more detail. Some experimental settings should be clearly described especially your proposed module or methods.\nFor Question 3, conduct some ablation study in exploring the same number of channels in the SSUL Head layer. \nFor Question 4, conduct experiments on some dropout layers add after the other convolutional layers.\nExperimental results partially show the effectiveness of your proposed methods. Try to explain in theory.\n", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This proposed method is based on SSUL but the introduction about SSUL is not detailed enough which may lead to confusion and misunderstanding. From the whole paper, it seems that a strong backbone can significantly improve the accuracy of CISS while the classifier just needs to classify features. The authors reveal the overcompression of features cause model degradation problem when training the backbone and some tricks for alleviating the problems. While their proposed methods try to extract more so-called robust or high-quality features for prediction, these methods can not solve the overcompression problem. Experimental results partially prove the effectiveness but lack theoretical proof and ablation study. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "76Mp7njk-le", "reviewer": "Reviewer_vqZ3", "summary": "This paper proposes to tackle continual semantic segmentation by addressing the overcompression issue in model learning. In order to improve the generalization ability of representation to upcoming tasks, the authors draw inspiration from previous studies in the field and introduce wider convolution for final feature extraction as well as apply dropout to the output of the feature encoder.  The proposed method achieves good performance on the Pascal VOC dataset.\n", "questions": "- How about placing the Dropout and the wider convolution layer in different locations of the network?\n- How does the proposed method perform under the \"Disjoint\" setting?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n+ This paper is clear and easy to read.\n+ The proposed method is simple and efficient.\n+ The proposed method achieves good performance on PASCAL VOC dataset.\n\nCons:\n- The contribution of this paper is limited. Previous works have demonstrated the effectiveness of wide convolution layers[a] and Dropout[b] for continual learning problems. This paper basically applies the techniques to the semantic segmentation task, which can be hardly considered as a significant contribution.\n\n- The authors try to interrupt the effectiveness with the Information Bottleneck principle, yet either theoretical or empirical explanations are missing. Only the improvements in final accuracy are not sufficient to convince the reader about the paper's completeness. By the way, the term \"generalisation phase\" sounds wired for explaining the Information Bottleneck principle in Sec.2.2.\n\n- Experiments are only conducted on the Pascal VOC dataset, while comparisons on ADE20K are missing. Since the proposed method is designed with a fixed feature extractor, it looks hard to generalize in situations the base class set is small and the new task set is large.\n\nIn summary, this paper presents a simple framework for continual semantic segmentation. Yet the contributions are insignificant, and the experiments are insufficient. \n\n[a] Mirzadeh et. al., Architecture Matters in Continual Learning.\n[b] Mirzadeh et. al., Dropout as an Implicit Gating Mechanism For Continual Learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject