ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation
Authors: Youngmin Oh
Donghyeon Baek
Bumsub Ham
Venue: NeurIPS 2022 Accept (2022)
Keywords: continual learning
incremental learning
semantic segmentation

Abstract: We address the problem of incremental semantic segmentation (ISS) recognizing novel object/stuff categories continually without forgetting previous ones that have been learned. The catastrophic forgetting problem is particularly severe in ISS, since pixel-level ground-truth labels are available only for the novel categories at training time. To address the problem, regularization-based methods exploit probability calibration techniques to learn semantic information from unlabeled pixels. While such techniques are effective, there is still a lack of theoretical understanding of them. Replay-based methods propose to memorize a small set of images for previous categories. They achieve state-of-the-art performance at the cost of large memory footprint. We propose in this paper a novel ISS method, dubbed ALIFE, that provides a better compromise between accuracy and efficiency. To this end, we first show an in-depth analysis on the calibration techniques to better understand the effects on ISS. Based on this, we then introduce an adaptive logit regularizer (ALI) that enables our model to better learn new categories, while retaining knowledge for previous ones. We also present a feature replay scheme that memorizes features, instead of images directly, in order to reduce memory requirements significantly. Since a feature extractor is changed continually, memorized features should also be updated at every incremental stage. To handle this, we introduce category-specific rotation matrices updating the features for each category separately. We demonstrate the effectiveness of our approach with extensive experiments on standard ISS benchmarks, and show that our method achieves a better trade-off in terms of accuracy and efficiency.

ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation
Donghyeon Baek School of Electrical and Electronic Engineering, Yonsei University https://cvlas yonsei. ac pc..k//projsttsALLPP /ALIFE
Youngmin Oh
Bumsub Ham"
Abstract
We address the problem of incremental semantic segmentation (ISS) recognizing novel object/stuff categories continually without forgetting previous ones that have been learned. The catastrophic forgetting problem is particularly severe in ISS, since pixel-level ground-truth labels are available only for the novel categories at training time. To address the problem, regularization-based methods exploit probability calibration techniques to learn semantic information from unlabeled pixels. While such techniques understanding of them. Replay- based methods propose to memorize a small set of images for previous categories. They achieve state-offie off-the-ar performance the cost of large memory footprint. We propose in this paper a novel ISS method, dubbed ALIFE, that provides better compromise between accuracy and efficiency. To this end, we first show an in-depth analysis on the calibration techniques to better understand the effects on ISS. Based on this, we then introduce an adaptive logit regularizer (ALI) that enables our model to better learn new categories, while retaining knowledge for previous ones. We also present feature replay scheme that memorizes features, instead of images directly, in order to reduce memory requirements significantly. Since a feature extractor changed continually, memorized features should also be updated at every incremental stage. To handle this, we introduce category-specifie rotation matrices updating the features for each category separately. We demonstrate the effectiveness of our approach with extensive experiments on standard ISS benchmarks, and show that our method achieves better trade-off in terms of accuracy and efficiency.
effective, there is still lack of theoretical
Introduction
Humans capable of learning new concepts continually, while preserving or even improving previously acquired knowledge. Artificial neural networks are, however, prone to forget the knowledge they have learned if being trained with samples for new object/scene categories alone. The reason for this problem, so-called catastrophic forgetting [11, 27]. is that parameters of neural networks change abruptly to handle new categories without accessing training samples for previous categories. A straightforward way to alleviate the problem is to re-train a model with training examples for entire categories observed so far, which is however computationally demanding.
Incremental learning is an alternative approach to learning new categories continuously without re- re-training on the entire dataset. While many methods have been proposed for incremental classifica- tion [4, 21, 23, 25, 33, 36], few attempts explore incremental semantic segmentation (ISS), where training images for new categories are partially labeled reduce the cost for manual annotation. That is, pixels for new categories are labeled only, while remaining ones are marked as unknown. The unknown regions should be considered separately, since they could contain previous categories along with ones that would be seen in the future.
Cooresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

35
66-
HiB
25-
20
25
60-
HD
SDA
PLOP
64-
65
x19 reduuction
SSUL-M
SSLA.
R.OP
SDR
15-
15
5SJLN
Durn
10-
10
61
60-
CF+RD
CER
(F+00 CCE-CCD
Nemory
Hemory' (HB)
(a) IoU scores on ADE20K [40].
(b) hlou-memory comparison on PASCAL VOC [10].
Figure I: (a) Quantitative comparison of intersection-of-union (IoU) scores on ADE20K. Each model learns 50 novel categories after learning 100 categories. To be specific, there is a total of 5 incremental stages, and each model learns 10 new categories at every incremental stage. mloU: an average IoU score for 150 categories. hloU: harmonic mean between two average IoU scores for previous (100) and novel (50) categories. (b) Quantitative comparison of hIoU and memory requirement on PASCAL VOC. (From left to right) Each method learns I and 5 novel categories after learning 20 and 16 categories, respectively. Ours-M memorizes IK features for each previous category. For RECALL [26] and SSUL-M [3], we consider the size of memory required to store images only, and discard that for corresponding labels or saliency maps. Best viewed in color.
Current ISS methods be categorized into regulaiization-aase [2, 9, 30] and replay-based ap- proaches [3, 26]. Regularization-ban hased methods typically exploit knowledge distillation (KD) [15] to preserve the discriminative ability of ISS models for previous categories. In particular, the seminal work of [2] alleviates the semantic shift of unknown regions. Specifically, it introduces calibrated cross-entropy (CCE) and calibrated KD (CKD) terms learn semantic information from unknown regions. Although CCE and CKD terms have shown the effectiveness on ISS methods [2, 9, 30], there still lack of theoretical understanding. Replay-based based methods propose to use a replay buffer consisting of web-crawles or previously seen images. The replay buffer provides rich information storing the knowledge for previous categories, but it incurs large memory footprint, particularly for the task of semantic segmentation that typically adopts high-resolution images.
In this paper, present novel ISS method, dubbed ALIFE, that alleviates catastrophic forgetting and 2) reduces memory requirements. For the first aspect, we analyze gradients of CCE and CKD terms for better understanding the effectiveness on catastrophic forgetting, and make the following observations: (1) For unknown regions, CCE reduces logit values of new categories, which is crucial for preventing overfitting the new categories. However, always raises logit values of all previous categories, without considering whether predictions for those regions are correct or not, which lessens the discriminative power for previous categories. (2) CKD makes it difficult
distinguish new categories from a background one. Motivated by these observations, we introduce an adaptive logit regularizer (AI.)) that enables better learning new categories and alleviating catastrophic forgetting for previous ones (Fig. 1(a)). For the second aspect, propose to exploit latent features for replaying, reducing memory footprint and avoiding data privacy issues (e.g., in medical imaging [35]). In contrast to the replay buffer storing images directly, our approach to using the features updates them at every incremental stage, as a feature extractor is also updated continually at training time. Specifically, exploit category-specific rotation matrices using the Cayley transform. Rotating latent features is computationally efficient, while maintaining the relations between the features. We demonstrate that exploiting ALI with memorizing features achieves better trade-off in terms of accuracy and efficiency (Fig. 1(b)). Our main contributions are summarized as follows:
We show an in-depth analysis of probability calibration methods widely used for ISS [2, 30]. and introduce ALI that enables our model to better learn new categories, while maintaining the knowledge for previous categories.
We present a novel replay strategy using category- rotation matrices, which helps to alleviate catastrophic forgetting for previous categories with much less memory requirements than replaying raw images.
Extensive experiments demonstrate the effectiveness of our approach to using ALI and replaying features with rotation matrices. We set new state of the art on standard ISS benchmarks [10, 40].

---

2 Related work
Many incremental learning approaches have been introduced to preserve knowledge for previous categories, while recognizing new ones [28]. They can be categorized into mask-incremental and class- incremental methods. Task-ineremental methods [21, 23, 25] treat learning new set of categories as a new task. With a perfect task identifier, they classify each task separately at test time. The perfect identifier is however often not available practice [32]. On the contrary, class-incremental methods [4, 8, 33, 36] attempt to recognize all categories observed so far at once without the task identifier, being more practical. the following, we describe class-incremental methods pertinent to ours.
Image classification. Incremental methods typically adopt a KD technique [15] in order to encour- age current model to imitate softmax probabilities [1, 33] or intermediate representations [8. 18] obtained from a previous one, retaining the discriminability for previous categories. Another line of works focuses on preventing classifier from overfitting to new categories, by correcting biased weights of a classifier with a post-processing [36, 39] or normalizing a classifier [18]. Similarly, our ALI is beneficial to alleviating the overfitting problem and maintaining the discriminative power for previous categories, but differs in that it adaptively regularizes logit values during training. In addition, we exploit latent features for replaying, while all the aforementioned methods rely on an image replay strategy that entails practical issues such as data privacy and large memory requirements. To avoid these problems, recent works employ a pseudo replay scheme that synthesizes images [20, 31. 34] or latent features [38] of previous categories. However, they often require generator [12], which should also be trained incrementally. Instead of using raw images or synthesizing features, the work of [19] proposes to store latent features. It trains feature adaptation network (FAN) mapping the latent features into a new feature space. The FAN might be sub-optimal, since the same network applies to all features without differentiating categories. Adopting individual FANs for each category might address the problem, but this requires high computational cost and large memory footprint. Dif- ferently, we propose to train category-specific rotation matrices. This is more efficient and accurate, as a rotation matrix is light-weight and features of each category are updated separately.
Semantic segmentation. A relatively few methods address the task of ISS, which can further be classified into regularization-bar
and replay-based approaches. The first approaches [2, 9, 30] focus on preserving knowledge for previous categories without using an experience replay. For example, MiB [2] proposes to consider the semantic shift of a background category in ISS, and introduces CCE and CKD terms using probability calibration techniques. While the calibration techniques are intuitive and effective, there are still lack of theoretical explanations. We provide a gradient analysis on CCE and CKD, and point out that CCE and CKD could disturb preserving knowledge for previous categories and discriminating new categories from the background category, respectively. SDR [30] and PLOP [9] propose to further regularize a current model in latent feature space. Specifically, SDR minimizes distances between features of the same category [6]. while PLOP distills intermediate features using a multi-scale strip pooling technique [8. 17]. The second approaches [3. 26] rely on an image replay strategy. RECALL [26] exploits web-crawled images with pseudo labels at each incremental stage, while SSUL [3] maintains a small set of images for previous categories, together with corresponding ground-truth masks. Both methods show ofate-o--the-ard art results at the cost of large memory footprint. Our approach differs in two aspects: (1) We exploit latent features for replaying, which reduces the size of required memory significantly; (2) Our method updates a feature extractor continually, and thus it is more flexible than RECALL and SSUL freezing the extractor.
3 Method
In this section, we first describe the ISS task briefly (Sec. 3.1), and then present our approach consisting of three steps at every incremental stage. Specifically, we introduce ALI based on an in-depth analysis for CCE and CKD to train a ISS model (Sec. 3.2) and present feature replay scheme using category -specfffc rotation matrices (Sec.3.3) to fine-tune a classifier using memorized features (Sec. 3.4). Note that the last two steps optional. Please refer to the supplementary material for detailed derivations, more analysis, and a pseudo code of our approach.
3.1 Problem statement
Following the common practice 12, 9, 26, 30], we consider a series of T learning stages. Each stage t has its own dataset, D { u)) yz' of size n', where and y an image and
3

---

corresponding ground-truth mask, respectively. For incremental stages (i.e., t 1), we have two disjoint sets, prev and Chow for previously learned and novel categories, respectively. Note that ground-truth masks are labeled only for the categories of C Formally, we denote by y(p) ground-truth label at position p. and define labeled regions follows:
Riew
U Re:
(I)
CE Chow
where R. is set of locations labeled as a category c, that is, R. = (pl v(P) = c)-
Our goal is to train a model that recognizes all categories observed so far, Call the union of the disjoint sets, i.e., Ca C Crr U Chew. In detail, our model consists of feature extractor & and a classifier w. The feature extractor takes an image and outputs a convolutional feature map for prediction, o² position p. A logit value for a category c is then computed by the dot product between feature and a classifier weight for the category c, w RD, as follows:
+ f', where we denote by fP) IR_ D-dimensional feature vector at
(p) w (p),,
where we omit bias terms of the classifier for brevity.
3.2 Step
In the first step. initialize network weights of model at the stage i with those for the previous stage - 1, and train a feature extractor and a classifier of the current model with a corresponding dataset D A simple way to mitigate catastrophic forgetting for ISS is employing cross- cross-entropy (CE) and KD [15] terms. To this end, we compute probabilities for a category c at position as follows:
(p)
e=)(p)
(p) Exerg ehhpp) CE C.
and (p) Enery
Cl
prev:
(t((p)
which are obtained by applying the softmax function to logit values across Ch,, and C Crryy. respectively. The CE and KD terms for ISS are then defined follows:
LCE (P) log log (p); p E RRew and LKD(P)=
-Pk (p) Vp. (4)
REO
prev
where c* = vp). The CE term is defined for the labeled regions RD Reew only, indicating that logit val- ues of previous categories always decrease. Exploiting the CE term alone is thus prone to overfitting to new categories and leads to catastrophic forgetting. The KD term addresses this problem by transfer- ring the discriminative power of a previous model, trained to classify previous categories, into the cur- rent one. Note that prey = Ca and the KD term computes softmax probabilities across previous categories without considering new ones. Instead of exploiting CE and KD terms, the seminal work [2] proposes to use CCE and CKD, widely adopted in ISS [2, 9, 30]. In the following, we analyze gradi- ents of CCE and CKD to better understand the influences on ISS, and introduce ALI to train model.
CCE. The CCE loss [2] additionally computes a probability for unlabeled regions by summing the probabilities over all previous categories follows:
Table 1: Gradients of CCE w.r.t =!- Note that the optimization process is carried out by gradient descent.
LCCR(p) {
pog (p); E R
Conditions
Gradients
(5)
log P'ce (p), o Rie
p C Riew
v(p)
c v(p)
P
where Pice (p) = -EEco EkEC pee (p). This marginal difference from the vanilla CE term brings significant improvement on
Cpew
Ri
c C Cpeen
pb q.
ISS. To analyze the reason behind the improvement, we summarize in Table gradients of LOCE w.r.t z The first two rows show that CCE updates logit values for the labeled regions in the same way as in LCE. We can see from last two rows that CCE also provides gradients for the unlabeled regions. Specifically, the third row shows that CCE reduces logit value for new categories by the corresponding probability P'c, if p R This is reasonable, since all features on those regions do

---

not at least belong to new categories. CCE thus helps to avoid the overfitting problem, alleviating catastrophic forgetting. Considering the fact that q. is always larger than p': (See the supplementary material), we can see from the last row that CCE raises logit values for all previous categories by d. - 2 in the unlabeled regions. This prevents the logit values of previous categories from continuing to decrease in the unlabeled regions, which is however effective only when predictions of the current model are correct. Otherwise, when predictions are incorrect, CCE rather raises logit values of the wrongly predicted categories, aggravating the incorrect predictions.
CKD. Assuming that categories of new are likely to be labeled as a background one in the previous stage, the CKD loss [2] is defined as follows:
LP) = -Phy" (p) loggeca(p))
Σ (p)
Vp,
(6)
where P'xa (p) = Ike (bg)UC (P)) For previous categories except the background, this term enables transferring knowledge from pk= to pk directly. Similarly, vanilla KD term in Eq. (4) also encourages a current model to output probabilities similar pk- for k E Cp but it distills the knowledge from pkz - into o without considering logit values for new categories. To compare KD and CKD, we compute gradients w.r.t = for E Cree as follows:
@LKD(P) = (p) - (p) and @LCKD(P) = P'((P) - Pe (p). dz((p)
(7)
@z((p)
Here we provide two cases to explain the reason why CKD better transfers the knowledge for previous categories except the background: (1) When o and p-- are equal, the gradient of KD w.n.t a is zero, indicating that 2 remains the same. However, note that p': is lower than p-- in this case, since p' is always lower than g' This suggests that 2 should rather increase in order that p follows the target probability p-- The gradient of CKD w.r.t z has negative value in the same case (i.e., g' = p' ' suggesting that 2 increases by gradient descent. (2) When the current model already provides the same probability as the target one (i.e., p'. = 11-1), it is unnecessary to adjust z The gradient of KD w.r.t =6 has a positive value, as q: is larger than p (or equivalently pf-1 in this case). This indicates that ef decreases by gradient descent, and p.. in turn becomes lower than p--1. On the other hand, CKD maintains z the same, since its gradient w.r. o is zero. These examples describe the effectiveness of CKD. On the contrary, CKD also has negative effects on ISS, Let us suppose the gradients of CKD w.r.t o for c E {bg} U U 'ew as follows:
OLCKD(P) = (p'aa(P) - -hi (p) (2)(p)
Pe (p)
(8)
Pixa (p)
The problem of CKD lies in how it transfers the knowledge from Pby (-1 into phg- In particular, CKD makes P'kda to imitate pig instead of directly distilling from Pbg Phg- For example, when plad is lower than phoy the gradient of CKD in Eq. (8) has negative value. Thus, logit values for background and new categories increase by gradient descent. This in turn raises prob- abilities for background and new categories all together, disturbing discriminating the new cat- egories from the background at training time. In case of Peka Pig t-1 CKD reduces the logit values for background and new categories, which is however problematic when pha < phyy Note that the logit value for the background category 2igg should rather increase in this case.
ALL We have shown that CCE alleviates catastrophic for- getting, while CKD better guides transferring the knowledge of a previous model than KD. To avoid the negative effects of CCE and CKD, we present in Table 2 new form of gradients w.r.t 2f. The first row is identical to the third one of Table 1 that alleviates catastrophic forgetting. The second row similar to the gradients of CKD Eq. (7) that better capture knowledge for previous categories. Note that CKD excludes the background category in Eq. (7), since it assumes that new categories of the current stage (C% are marked as the background one at the previous stage. Differently, ours enables computing gradients for all
Table 2: Gradients of ALI w.r.t 2/.
Conditions
Gradicnts
E C
P&C. Ri.
Ci
P.
5

---

previous categories including the background one (See the second row), since it does not require the assumption. Accordingly, this form of gradients incorporates the advantages of CCE and CKD, while discarding the negative effects. Integrating the gradients in Table 2 w.r.t z we define ALI as follows:
LALI(D) = log
Σ RECan
(li(p)
E Ph (D)=()) P R Riow AEO
The first term applies the log-sum-ex) function to logit values across C.ll- approximating the maximum logit value over Call- The second term computes weighted average of logit values for Oprev with probabilities of a previous model. In this context, ALI can be viewed as minimizing the difference between the maximum logit value and the weighted average adaptively. That is, reduces the maximum logit value, while raising the logit values for previous categories. Note that ALI does not require computing either Poee in CCE Peka in CKD [2].
Training. To train our model, we use CE and ALI terms for labeled and unlabeled regions, respec- tively. We also apply vanilla KD term for labeled regions to further regularize our model. An overall objective for the first step is defined as follows:
(p) (pE(p) AALILALI(D) + AKD (p) I p R
(10)
WheredALt and ARD are balance parameters. We denote by L[:] an indicator function whose output is 1 if the argument is true, and 0 otherwise.
3.3
Step 2
After training our model, we first extract features of new categories order to replay them in subsequent stages. Then, compensate distribution shift of memorized features, which extracted in the previous stage f - 1, before using them to finettune a classifier in the third step. To be specific, exploit category-specific rotation matrices to update memorized features of each category separately. In the following, we provide detailed descriptions for memorizing features and training matrices. Note that we freeze both feature extractor of and classifier w' for the second step.
Memorizing features. We extract features for prediction (i.e., f`) and store them to replay in subsequent stages. Specifically, given input images containing of new categories, feature extractor first produces feature maps. For each image, features for each category are then averaged using a ground-truth mask. This process is repeated until the number of features for each category reaches a preset number S. That is, we memorize S' features for each category. We provide the pseudo code in the supplementary material.
Training matrices. Memorized features, extracted in the previous stage f 1, are not compatible with a current classifier wt. To address this, the work of [19] in image classification proposes to exploit two-layer perceptrons, called FAN, where the number of parameters is roughly 32 Two main limitations of FAN are as follows: (1) updates all features without considering categories. Adopting separate FANs for each category alleviates this issue, but is computationally expensive. (2) FAN ignores the relations between memorized features. The structural information in the feature space is crucial for the generalization ability of classifiers. To address these problems, we propose to train category
rotation matrices update features of each category separately. The rotation transform enables maintaining the relations between features within the same category, while the number of parameters for each matrix 0.5(D2 D) only (See the supplementary material). To this end, we first define skew- niew-symmetric matrix S, of size D x D for the category E C% Cpeev A rotation matrix for the category is then defined using the Cayley transform as follows:
R - (I
(11)
where indicates an identity matrix. Learning the parameters for the matrices is challenging, since training samples of D' are labeled only for C% To handle this, extract features f from previous feature extractor øi-1, and compute a correlation score as follows:
s
(s(s)
v (p)=
ReLU
(12)
(p)]
[m,(())]

---

where we denote by ms(s) E IR the s-th item in memorized features of the category This allows identify features associated with previous categories C prev since ft and m,(s) (s) share the same feature space. We then apply the softmax function over the correlation score as follows:
(p((
(p(
(13)
Σp eTve(P)
where is temperature controlling the sharpness of de. Note that both features f and ft at position p encode the same semantic information. Using this fact, we define prototypes of the category for previous and current stages, r/- and r respectively, by computing weighted average each feature map, as follows:
r = r
-oCo (p), re (<DD
(14)
We can match the prototypes for each category via the corresponding rotation matrix R Namely, each matrix R, rotates previous prototype rd-1 to align it with current one r To train the matrix R we define an objective function as follows:
La2 LPID + (1- AROT)
(15)
where denote by LFID and LrEG fidelity and regularization terms, respectively, balanced by the parameter AROT. The fidelity term maximizes cosine similarity as follows:
(1
Pr [elll
LFID
Σ
(16)
|rr"
OE Creev
where fc Rere- This encourages the matrices R. to align fe with r The regularization term enforces the rotated prototypes fg to be compatible with the current classifier w' to better guide the alignment process. To this end, compute CE loss using the softmax classifier w² follows:
eformnt
LREG
Σ
-
log
Every efenn
(17)
CECL
3.4 Step
In the third step, we first update memorized features of each category follows:
me(s) Ro)m(s).
(18)
The updated features along with training samples of D are then used to fine-tune a classifier wi with the following objective:
Lsa(p) (FL AALI AMEM
(19)
We denote by LPI. and MMEM focal loss (FI.) [24] and CE terms, respectively, defined as follows:
v(p)
p E R Rew
(FL(p) = =-(1
-1--1pppp)))) - p'(p)) logp'(()) d=
(20)
argmax RECY pre
Pk
(p), p R
and
S
cmmo(s)-uf
Σ
LMEM =
log
(21)
ΣkeOY (
CECH
prey
where we a 2 by default. Following [3. 26, 30], we mark unlabeled regions in the training samples of D as predictions obtained from previous model on-the-fly. Note that we freeze a feature extractor and the rotation matrices for fine-tuning the classifier.
4 Experiments
In this section, we present quantitative comparison between our method and the state of the art, and show ablation studies. More results including qualitative comparisons can be found in the supplementary material.

---

Table 3: Quantitative results on ADE20K [40] terms of IoU scores. SSUL-M [3] uses a replay buffer that consists of 300 previously seen images together with corresponding ground-trun labels. Numbers in bold are the best performance, while underlined ones are the second best. We show standard deviations in parentheses. Numbers for other methods [2, 29] are taken from SSUL. f: Results are obtained by running the source codes provided by the authors.
100-5001)
50-10022
100-50(5)
Methods
mloU baxk mlo
mloU
hloU
mlol/
mioU
mloU
blo
buse mioU
mloU
hioU
Without memorized images features
ILT [29]
18.29
14.40
17,00
16.11
3.53
12.85
9.70
5.54
0.08
1.31
0.49
0.15
MiB 12]
40.52
17.17
32.79
24.12
45.57
21.01
29.31
28.76
38.21
11.12
29.24
17:23
PL.OP [9]
23.42
21.31
29.56
31,92
(0.02)
(0.15)
(0.06)
(0.15)
(0.03)
(0.08)
(0.06]
[0.08]
(0.04)
(0.03)
(0.04)
(0.03)
SSUL [3]
41.28
18.02
33.58
25.09
48.38
20.15
29.56
28.45
40.20
18.75
33.10
25.57
ALIFE
42.18
23.07
35.86
29.83
48.98
25. 69
33.56
33,70
41.02
22.76
34.98
29.28
(0.08)
(0.53)
(0.12)
(0.41)
(0.12)
(0.20)
(0.16)
(0.19)
(0.23)
(0.55)
(0.34)
(0.51)
With memorized images or features
SSUL [3]
42.79
17:54
34.37
24.88
49.12
20.10
29.77
28.53
42.86
17.66
34.46
25.01
ALIFE-M
30.28
48.99
26.15
33.87
41.17
(0.05)
(0.45)
(0.12)
(0.36)
(0007)
(0.10)
(0.09)
(0.10)
(0.21)
(0.22)
(0.21)
(0.23)
4.1
Implementation details
Dataset and evaluation. We evaluate our method on standard ISS benchmarks (PASCAL VOC [10] and ADE20K [40]. PASCAL VOC provides 10, 582 training [13] and 1, 449 validation samples with 20 object and one background categories, while ADE20K consists of 20, 210 and 2, 000 samples for training and validation, respectively, with 150 object/stuff categories. There are three incremental scenarios for each dataset, where we denote by each scenario A- A-B(C). A, B and C' indicate the number of categories at base stage, the total number of novel categories, and the number of incremental stages, respectively. We follow the same scenarios as in [2, 3, 9, 26, 30), unless otherwise specified. Following the common practice [3, 9], we focus on an overlapped setting, where unlabeled regions could contain either previous or future categories. For evaluation, we report IoU scores on the validation set for each dataset, and do not exploit test-time augmentation or dense CRF techniques [22]. We denote by mloUbase. mloUnew. and mIoU the mean IoU scores over base, new, and all categories, respectively. Computing an IoU score over all categories (i.e. mloU) is typical, but we have found that this does not reflect IoU scores for new categories well. To address this, in the evaluation protocol of zero-shot learning methods (e.g., [37]). we propose to use the harmonic mean (hloU) of mloUbase and mIoUnew new For all experiments, we report scores averaged over runs (Le. different random seeds).
Training. Following [2, 3, 9, 26, 30], we adopt DeepLab V3 [5] with ResNet- 101 [14]. ResNet- is initialized with pre-trained weights for classification on ImageNet [7]. We use the SGD optimizer with an initial learning rate set to 1e-2 and 1e-3 for base and incremental stages, respectively. Deepl-ab-Ve V3 is trained for 30 and 60 epochs at a base stage (1 = 1) on PASCAL VOC [10] and ADE20K [40], respectively. For each incremental stage (1 > 1), we perform cross-validation to choose the number of epochs on PASCAL VOC, while fixing it to 60 on ADE20K. We train rotation matrices for 10 epochs using the Adam optimizer with an initial learning rate of 1e-3, and fix a preset number S and temperature value to 1, 000 and 10 for all experiments. We fine-tune classifier for I epoch using the SGD optimizer with an initial learning rate of le-3. For all experiments, we adjust the learning rate by the poly schedule. We provide detailed description of hyperparameter settings in the supplementary material.
4.2 Comparison with the state of the art
ADE20K. We compare in Table 3 our approach with state-of-the-as methods, including MiB [2], PLOP [9], and SSUL [3]. Note that RECALL [26] is not designed to handle stuff categories, and results on ADE20K [40] are not available. From this table, we have three findings as follows: (1) Our approach exploiting the first step only, denoted by ALIFE, already outperforms all other methods in terms of both mloU and hloU scores by significant margins for all scenarios. This validates the effectiveness of our approach without memorizing features. In particular, we can that ALIFE even
8

---

Table 4: Quantitative results on PASCAL VOC [10] terms of IoU scores. SSUL-M [3] memorizes
100 images in total, while RECALL [26] uses 500 images for each previous category. Note that both
SSUL. and SSUL-M also require an off-the-shelf saliency detector [16] on PASCAL VOC. Numbers in bold are the best performance, while underlined ones are the second best. We show standard
deviations in parentheses. Numbers for other methods are taken from corresponding papers. f:
Results are obtained with the source codes provided by the authors.
20-1(1)
16-5(1)
16-5(5)
Methods
mloU
mloUp new
mloU
hloU
mlo0) Lase
mloUpew
mloU
hlo
mloU
mioUnee
mloU
MoU
Without memorized images or features
EWC [21]
26.90
14.00
26.30
18.42
24.30
35.50
27.10
28.85
0.30
4.30
1.30
0.56
Lw1-MC [23]
64.40
13.30
22.05
35.00
52.30
8.40
6,90
7.26
ILT [29]
67.75
10.88
65.05
18.75
67.08
39.23
60.45
49.51
8.75
7.99
8.56
8.35
MiB
70.42
17.70
67.91
28. 25
76.68
49.03
70.09
59.81
37.98
12.28
31,86
18.56
(0.13)
(1.89)
(0.17)
(2.44)
(0.11)
(0.27)
(0.14)
(0.23)
(0.72)
(0.19)
(0.57)
(0.25)
SDR [30]
71.30
23.40
69.00
35.24
76.30
50.20
70.10
60.56
47.30
14,70
39,50
22.43
PI.OPT [9]
75.89
34,90
73.94
47.81
76.37
49.55
69.98
60.10
64.51
19.93
53.90
30.45
(0.21)
[0.93]
(0.24)
(0.91)
(0.14)
(0.29)
(0.18)
(0.26)
(0.13)
(0.20)
(0.06)
(0.21)
SSUL|3]
77.73
29.68
75.44
42.96
77.82
50,10
71.22
60.96
77.31
36.59
67.61
49.67
ALIFE
76.61
49.36
75.31
60.03
77. 18
52.52
71,31
62.50
64,44
34,91
57.41
45.29
(0:52)
(1.01)
(0
(0.45)
(0.52)
(0.41)
memorized images
SSUL []
78.83
49.76
61.01
78.40
73.02
65.20
78.33
49.01
60.30
RECALL [26]
68.10
55.30
68.60
61
67.70
54.30
65,60
60.26
67.80
50.90
64.80
58.15
ALIFE-M
76.72
52.29
75.56
62.19
77.66
55,27
72.33
64.57
66.09
38.81
59.59
48.89
(0.57)
(0.62)
(0.55)
(0:49)
(0.36)
(0.95)
(0.21)
(0.39)
(0.64)
(1.86)
(0.93)
(1.64)
outperforms SSUL-M [3] that memorizes 300 images along with ground-truth labels for replaying. A plausible reason is that SSUL freezes a feature extractor, limiting the flexibility to deal with new categories. (2) ALIFE shows substantial IoU gains over MiB [2] using CCE and CKD for all scenarios. This verifies that both CCE and CKD are not always helpful for ISS. ALI is free from the limitations of CCE and CKD, and allows our model to better learn new categories without forgetting previous ones. (3) Our approach memorizing features, denoted by ALIFE-M. improves the performance over ALIFE in terms of all metrics for all scenarios. Note that SSUL-M even performs worse than SSUL for 100-50(1) and 100-50(6) cases. Considering that we rely on at least 9 times less memory requirements than SSUL-M for 100-50(1) I) and 50- 50-100(2) cases, the gains from memorizing features remarkable compared to those of SSUL-M over SSUL.
PASCAL VOC. We show in Table 4 quantitative results on PASCAL VOC [10]. Note that comparison of SSUL [3] (SSUL-M) and other methods including ours is not fair, since it additionally exploits an off-the-shelf saliency detector [16] that brings significant improvements on PASCAL VOC. From this table, we can see that ALIFE outperforms all other methods in terms of hloU scores for 20-1(1 1(1) and 16-5(1) scenarios, further demonstrating the effectiveness of our approach without memorizing features. We can also see that ALIFE-M gives substantial gains over ALIFE in terms of all metrics for all scenarios. In particular, the largest mloU and hloU gains of 2. 18% and 3.60%, respectively, are reported on 16-5(5). This confirms once again that our feature replay scheme is effective even for the most challenging scenario on PASCAL VOC. SSUL-M [3] and RECALL [26] largely outperform ours only for 16-5(5), but they require at least 604 and 15 times more memory footprint than ALIFE-M, respectively. Moreover, RECALL has difficulty for handling stuff categories, and SSUL performs poorly on ADE20K [40], where the saliency detector is not applicable. Our approach is versatile in that it is free to handle stuff categories without performance degradation on ADE20K (See Table 3).
4.3 Discussion
16-/11
20-1013
The ordering of categories. To show the robust- ness of our approach to different category orderings, we generate two random category orderings for 16- 5(1) and 20-1(1) cases, and report the mean and stan- dard deviation of hloU scores over three orderings: one alphabetical and two random orderings. We com- pare in Fig. 2 our approach without memorizing fea-
Figure 2: Comparison of average hloU scores over three different category orderings on PASCAL VOC [10].

---

tures, denoted by ALIFE, to MiB [2] and PLOP [9]. Results of MiB and PLOP are obtained by running the source codes provided by the authors. We can clearly see that ALIFE produces better results than other methods on both 16-5(1) and 20-1(1) cases.
Ablation study on the first step. We show in Ta- ble 5 an ablation analysis on different loss terms of our approach without memorizing features (Eq. (10)). The first row shows that using the CE term alone per- forms poorly due to catastrophic forgetting. We can see from the second row that ALI alleviates the catas- trophic forgetting problem remarkably. The third row shows that additionally using the KD) term for labeled regions on top of CE and ALI terms further gives IoU gains. On the other hand, we can see from the fourth row that applying the KD term for unlabeled regions rather degrades the performance. Since our ALI already encourages current model to imitate knowledge of a previous one for unlabeled regions, we conjecture that additionally applying the KD term for those regions is redundant. The last row shows that using the KD term for all regions is beneficial to improving the performance slightly.
Table 5: Comparison of IoU scores using dif- ferent loss terms of our approach on 16-5(1) of PASCAL VOC [10]. Labeled, Unlabeled, and All indicate that KD is applied for labeled, unlabeled, and all regions, respectively.
RD
Labated
Uniahaled
19:51
50:15
60:03
17.18
52.53
71.31
62.88
15.65
19:75
56:11
19:57
Ablation study on the third step. We report in Ta- ble 6 IoU scores for different losses of our method in the third step (Eq. (19)). From the first four rows, we can see that 1) using CE or FL terms alone degrade the performance. This is because training samples of a current dataset mainly contain new categories, causing the class imbalance between previous and new categories; 2) FL provides better results than CE; 3) The pseudo labeling strategy works favorably for both CE and FI., mitigating the imbalance. From the last three rows, we can see that 1) replaying 1K fea- tures for each previous category lessens the influence
Table 6: Comparison of IoU scores using dif- ferent loss terms of our approach on 16-5(1) of PASCAL VOC [10]. Labeled: CE or FL is applied only for labeled regions. All": To apply CE or FL for all regions, we mark un- labeled regions as predictions of previous model on-the-fly.
MEDI eddt
56.37
05:31
36.51
10:48
75:66
7735
13:30
11:44
of the class imbalance problem significantly and 2) ALI is also helpful to alleviate the imbalance problem, further boosting the performance.
Limitation. Our approach to memorizing features for an experience replay reduces memory re- quirements significantly and avoids data privacy issues. Nonetheless, it requires more memory than other methods [2, 20] that do not adopt the experience replay. Since our approach to using ALI without memorizing features shows state-of-the-art results standard ISS benchmarks [10, 40], we believe that ALI could give useful insights for developing ISS methods that do not rely on the replay strategy. It is also worth noting that all existing methods including ours assume that newly incoming samples are clean and reliable. However, in practice, training samples of a new task might be biased and unreliable. This raises new concerns: 1) addressing noisy samples during training and 2) memorizing clean samples only. Handling these potential risks would also be an interesting future direction for ISS.
Conclusion
We have introduced a new ISS method, ALIFE, that alleviates catastrophic forgetting and reduces memory requirements for an experience replay. First, we have presented a gradient analysis of CCE and CKD for better understanding the effects on catastrophic forgetting, and have proposed ALI incorporating the merits of CCE and CKD. Second, we have proposed a feature replay scheme using the Cayley transform that requires less memory footprint than memorizing raw images. Finally, have shown that ALIFE sets a new state of the art on standard ISS benchmarks.
Acknowledgments and Disclosure of Funding
This work was supported by Institute of Information & Communications Technology Planing & Evaluation (ITTP) grant funded by the Korea government (MSIT) NN..RS-2022-001-1 Develop- ment of Fundamental Technology and Integrated Solution for Gex--rennoaooon Automatic Artificial Intelligence System) and the Yonsei Signature Research Cluster Program of 2022 (2022-22-0002),
10

---

References
[1] Francisco M Castro, Manuel Marín-Jiménezz Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning, In ECCV, 2018
[2] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Modeling the background for incremental learning in semantic segmentation. In CVPR, 2020.
[3] Sungmin Cha, YoungJoon Yoo, Taesup Moon, et al. SSUL: Semantic segmentation with unknown label for exemplar-based class-incremental learning. NeurIPS, 2021.
[4] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, 2018.
[5] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv, 2017.
[6] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In CVPR, 2005.
[7] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
[8] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. PODNet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, 2020.
[9] Arthur Douillard, Yifu Chen, Arnaud Dapogny, and Matthieu Cord. PLOP: Learning without forgetting for continual semantic segmentation. In CVPR, 2021.
[10] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. IJCV, 88(2)3003-338, 2010.
[11] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4): 123-135, 1999.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
[13] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV, 2011.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. In NeurlPS Workshop, 2014.
[16] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip HS Torr. Deeply supervised salient object detection with short connections. In CVPR, 2017.
[17] Qibin Hou, Li Zhang, Ming-Ming Cheng, and Jiashi Feng. Strip Pooling: Rethinking spatial pooling for scene parsing. In CVPR, 2020.
[18] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In CVPR, 2019.
[19] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficient incremental learning through feature adaptation. In ECCV, 2020.
[20] Ronald Kemker and Christopher Kanan. FearNet: Brain-inspired model for incremental learning. ICLR, 2018
[21] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barriikk,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):352133222 2017.
11

---

[22] Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. NeurIPS, 2011.
[23] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE TPAMI, (0M1))/94040222293552 2017.
[24] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[25] David Lopez-Pat Paz and Marc'Aurelic Ranzato. Gradient episodic memory for continual learning. NeurlPS, 2017.
[26] Andrea Maracani, Umberto Michieli, Marco Toldo, and Pietro Zanuttigh. RECALL: Replay- based continual learning in semantic segmentation. ICCV, 2021.
[27] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[28] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The sicalttyy-llticcityy dilemma: In- vestigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4:504, 2013.
[29] Umberto Michieli and Pietro Zanuttigh. Incremental learning techniques for semantic segmen- tation. In ICCV Workshop, 2019.
[30] Umberto Michieli and Pietro Zanuttigh. Continual semantic segmentation via repulsion- attraction of sparse and disentangled latent representations. In CVPR, 2021.
[31] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning remember: A synaptic plasticity driven framework for continual learning. In CVPR, 2019.
[32] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. GDumb: A simple approach that questions our progress in continual learning. In ECCV, 2020.
[33] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental classifier and representation learning. In CVPR, 2017.
[34] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. NeurlPS, 2017.
[35] Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In ACM SIGSAC CCS, 2015.
[36] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, 2019.
[37] Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In CVPR, 2017.
[38] Ye Xiang, Ying Fu, Pan Ji, and Hua Huang. Incremental learning using conditional adversarial networks. ICCV, 2019.
[39] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In CVPR, 2020.
[40] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] We describe in Sec. 3.2 the assumption of CKD. Our ALI does not require any assumptions.
(b) Did you include complete proofs of all theoretical results? [Yes] We provide detailed derivations in the supplementary material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We describe implementation details in Sec. 4.1. For details, please refer to the supplementary material.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] For all experiments, we report averaged scores over runs. For example, report standard deviations in Tables 3 and 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We use 2 NVIDIA TITAN RTX GPUs for all experiments. Please refer the supplementary material for details.
4. If you using existing assets (e.g., code, data, models) or curating/reelasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We train DeepLab-
V3 [5] on PASCAL VOC [10] and ADE20K [10].
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "jqF4J6VXohk", "reviewer": "Reviewer_PwVw", "summary": "This paper describes an approach to Incremental Semantic Segmentation (ISS) of images. The authors provide a detailed and critical analysis of the Calibrated Cross Entropy (CCE) and Calibrated Knowledge Distillation (CKD) losses from the MiB approach [2], and from this analysis derive a new calibrated loss (called ALI) that combines the stability benefits of CCE with the forward transfer benefits of CKD. The authors additionally propose a three-stage incremental learning approach that first fits to a new task using ALI and (uncalibrated) CE and KD losses, then fits a rotation matrix to align memorized features with the new feature space, and finally fine-tunes the new task classifier on the current task samples and the newly rotated and memorized features from previous tasks. An extensive experimental evaluation is given on PASCAL VOC and ADE20K.\n", "questions": "+ **Improvement of S2 over S1**: on ADE20K the improvement of adding replay (and rotation and fine-tuning) is very marginal compared to that seen on PASCAL. This is also evident in the poor improvement (or even deterioration as noted in the paper) of SSUL-M with respect to SSUL. Do you have an explanation for this? Why would replay perform so poorly in some cases compared to others?\n\n+ **mIoU scores over all**: I found the discussion of the drawbacks of the mIoU over *all* categories a bit hard to parse. Are the columns in Tables 3-4 the mIoU over all categories, as commonly reported in the literature? This should maybe be made more clear to facilitate comparison with existing work.\n", "limitations": "The authors include a thoughtful discussion of technical limitations in the Supplementary Material. I think something could and should be said about potential risks of incremental learning as well -- for example, a discussion of the dangers of biases becoming *baked in* to incrementally-learned representations, and the difficulty of eliminating them. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n+ **Clarity**: The paper is generally very well-written and the main contributions are clear and understandable. The paper was a pleasure to read and I am convinced it adds something interesting to the discussion on incremental learning. \n+ **Motivation and technical development**: Similarly, the main technical motivations and developments are very clearly described, and the gradient analysis of CCE and CKD (as well as the proposed ALI) losses are quite compelling. The proposed approach is described in detail and I feel like the results presented would be easy to reproduce from the technical descriptions in the paper.\n+ **Experimental evaluation**: The experimental evaluation is also quite convincing -- although see *Weaknesses* and questions below for some issues. The proposed approach demonstrates clear advantages over the state-of-the-art, both in terms of accuracy and memory efficiency. \n\n# Weaknesses\n\n+ **Steps**: Though the paper is well-written, there are quite a few moving parts and I feel that the generic use of \"Step 1\" and \"Step 2\" does some disservice to putting all of the pieces together into a coherent, global picture of the proposed approach. The approach seems more clearly articulated in **three** stages (names improvised): New Task Acquisition (currently Stage 1), Drift Compensation (via rotation), and maybe Current Task Fine-tuning.\n  \n+ **Detailed comparison with CKD + CCE**: This seems like a very important comparison as one of the main novelty claims is improvement over MiB which proposed these calibrated losses. The only direct comparison with these seems to be in Fig. 1, and thus gets kind of forgotten when arriving at the experimental evaluation. Moreover, the terminology is somewhat unclear: the proposed approach, I think, should be CE+KD+ALI (eq. 10).\n\n+ **Ablations**: The feature alignment rotation is not ablated, moreover it would be interesting to see the ablations on ADE20K in attempt to understand why replay adds so little.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ipm2baEcZnO", "reviewer": "Reviewer_Rpor", "summary": "The catastrophic forgetting problem is particularly severe in ISS, since pixel-level ground-truth labels are available only for the novel categories at training time.\nTo address the problem, regularization-based methods exploit probability calibration techniques to learn semantic information from unlabeled pixels. While such techniques are effective, there is still lack of theoretical support. Replay-based methods propose to memorize a small set of images for previous categories. They achieve state-of-the-art performance at the cost of large memory footprint. The author propose in this paper a novel ISS method, dubbed ALIFE, that provides a better compromise between accuracy and efficiency. To this end, it first  show an in-depth analysis on the calibration techniques to better understand the effects on ISS. Based on this, it then introduce an adaptive logit regularizer (ALI) that enables our model to better learn new categories, while retaining knowledge for previous ones. It also present a feature replay scheme that memorizes features, instead of images directly, in order to reduce memory requirements significantly. Since a feature extractor is changed continually, memorized features should also be updated at every incremental stage. To handle this, it introduce category specific rotation matrices updating the features for each category separately. The paper demonstrate the effectiveness of our approach with extensive experiments on standard ISS benchmarks.", "questions": "questions are put in Strengths and Weakness.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths of this paper are as follows:\n1. The paper shows an in-depth analysis of probability calibration methods widely used for ISS, and introduces ALI that enables our model to better learn new categories, while maintaining the knowledge for previous categories.\n2. The paper present a novel replay strategy using category-specific rotation matrices, which helps to alleviate catastrophic forgetting for previous categories with much less memory requirements than replaying raw images.\n3. Extensive experiments demonstrate the effectiveness of our approach to using ALI and replaying  features with rotation matrices. It set a new state of the art on standard ISS benchmarks.\n\nWeakness of this paper are as follows:\n1. The proposed method  requires more memory than other similar methods [1, 4, 9] that do not adopt the experience replay.\n2. The proposed method could not achieve the best performance on benchmark in Table 4, though the author claims that the proposed method need far less memory. But is it possible to increate the memory to achieve the best performance?\n3. in line 137, the classifier weight is learnable, right? or it's fixed.\n4. in line 147, how does the labeled regions R_new mean? is any region of image labelled and other regions not labelled? why there is labelled region and unlabelled regions? is it because the labelled regions are previous categories indicated region?\n5. in line 264, in the fine-tuning step, is the rotation matrices fixed or still learnable?\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Dcf21AkZoK8", "reviewer": "Reviewer_eeB2", "summary": "The paper addresses the question of incrementally adding new categories in a semantic segmentation problem. This is a variant of the more standard class incremental learning problem where data are annotated with the new categories only in each incremental session and may contain unannotated regions from seen or unseen categories.\n\nThe proposed approach makes two contributions: definition of a new term in the loss for regularization that takes into consideration the various types of data, and alignment of the memorized features from previous sessions to the new features using a category-specific rotation estimated from a criterion combining discriminating capacity and similarity with previous features. The approach is evaluated on two standard benchmarks in this field (PASCAL VOC and ADE20k).\n", "questions": "- It was not clear where the performances of competing approaches (SSUL, PLOP, etc.) come from. Did you re-implement the methods or take the number from the papers? Are the backbones identical?\n\n- How can it be justified that the group of rotations, which are global isometries, is a good set of transformations ensuring feature alignment? How do the features behave when the number of incremental stages is large (I suspect that new features not only isometrically shift but may also be rearranged in a non-uniform way)? Maybe a T-SNE plot would give some idea about what is happening during sessions.\n", "limitations": "Non applicable", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Good and thorough discussion of loss terms in order to handle the various types of data and annotation state.  \n- Nice proposition to incrementally adapt memorized features using global alignment by rotations.\nThe ablation study, experiments and result analysis justify the approach on standard benchmarks.\n\nWeaknesses\n- Incremental work. Evolution of classical concepts of incremental learning: knowledge distillation and data replay.\n- The maximal number of incremental stages is limited to 5: other papers (SSUL [3]) use more stages (until 11). Since the number of incremental steps is a critical parameter of incremental learning, in general, it would be fair to assess the approach on similar grounds.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7fqdGm1-8fQ", "reviewer": "Reviewer_hRHc", "summary": "The paper proposes a method for class-incremental learning in semantic segmentation. First, in a theoretical analysis of the gradients induced by commonly used loss functions are analyzed and several shortcomings are pointed out. As a way to mitigate these problem, a new knowledge distillation loss dubbed adaptive logit regularizer (ALI) and a feature replay strategy are proposed. Experimental evaluation shows the superiority of ALI over previous (non-replay-based) approaches in several experimental settings and competitive performance compared to other replay-based approaches, while having a much lower memory footprint.", "questions": "1. Eq. 9: I am not quite sure how this equation is derived from Table 2. Could the authors elaborate?\n2. Have you varied the random seed between the three experiments that you average over?\n3. Did you determine all method hyperparameters by cross-validation, in particular during the mentioned grid search mentioned in Appendix B?\n", "limitations": "Limitations have been discussed to some degree in the supplementary. However, I feel the hyperparameter sensitivity might be an additional limitation.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Strong motivation of the method in the introduction and through the theoretical analysis of previously used loss functions. \n\n2. Well-written related work section, which is informative in putting the paper into the larger context of incremental semantic segmentation. The novelty of the method is clearly stated and discussed w.r.t. other works.\n\n3. Section 3.2 provides an overall very clear and informative mathematical description and analysis of de-facto standard loss functions in incremental semantic segmentation.\n\n4. The experimental results are strong compared to previous methods and appear to be SOTA for non-replay-based methods.\n\nWeaknesses:\n\n1. CKD analysis: In the analysis of the CKD loss, it is not always clear to me, what the desired gradient behavior is vs. the observed behavior. Specifically, it would help, if on page 5, ll. 180-196, this would be clearly stated. Afterwards, it is also not completely clear to me how ALI solves these issues. This could be added in page 5, ll. 197-208 with reference to the desired behavior in the previously mentioned text part.\n\n2. Loss application of ALI: It is not quite clear to me, why you use the vanilla CE and KD losses in Eq. 10 and also why you apply them specifically in these regions. I think this should be motivated in Section 3.2 and supported by an ablation study later on. In particular an ablation showing the experimental comparison with the commonly used CKD and CCE losses would help to better understand the benefits of ALI. Otherwise, also other factors such as better chosen hyperparameters could be the reason for the observed improvements through ALI.\n\n3. Hyperparameter sensitivity. The method seems to contain a lot of hyperparameters, which even have to be determined for every incremental stage separately (cf. Table B in the appendix). I think this might limit the usability of the method, as it is computationally very expensive to run a grid search for each new incremental step. Maybe the authors could elaborate on this point?\n\n4. Experimental setting: The approaches SSUL and RECALL compare on Pascal-VOC on the 10-1 setting and show particularly strong results in this setting. I think it would be better to evaluate this replay-based approach also in the most challenging settings introduced by other replay-based methods in incremental semantic segmentation.\n \n5. Hyperparameter ablation studies: The influence of the different hyperparameters of the method is not shown. Table B of the appendix shows that different hyperparameters are chosen for each setting, so it would be good to know the influence of these hyperparameters on the final performance.\n\nOverall, I really like the approach taken in this work. I also feel, the contributions contain a lot of value to the incremental and continual learning community. I would like to raise my rating, if my concerns are addressed satisfactory in the rebuttal.\n\nMinor comments and suggestions:\n\n6. Page 2, ll. 44-45: I think the CCE and CKD losses are indeed theoretically motivated. Rather than saying there is lack of theoretical support, I would rather rephrase to saying that the conducted theoretical analysis points out shortcomings which are addressed in this work.\n\n7. Page 4, ll. 146: There might be a superscript t missing in the symbol R_new.\n \n8. Page 4, ll. 162-166: Although it might appear trivial, it would be helpful to remind the reader that optimization is carried out in negative direction of the gradients shown in Table 1.\n \n9. It would help a lot to state the dimensions of the features used for replay, mentioned on page 6, ll. 223-225 and in Eq. 12.\n \n10. Eq. 12: The correlation score and the item index s have the same symbol, which is a bit confusing. Also, is the dot operator a scalar product between two feature vectors? Maybe this could be clarified.\n\n11. I think the bold-face highlighting in Table 3 and 4 is a bit misleading. I would propose to compare replay-based methods and non-replay-based methods. In this sense, you would still have SOTA results for non-replay-based methods and competitive results among replay-based methods. But it seems unfair to compare Ours-S2 with, e.g., MiB or PLOP.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept