ADAPT: Attentive Self-Distillation and Dual-Decoder Prediction Fusion for Continual Panoptic Segmentation
Authors: Ze Yang
Shichao Dong
Ruibo Li
Nan Song
Guosheng Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Continual Learning
Panoptic Segmentation
Knowledge Distillation
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Panoptic segmentation, which unifies semantic and instance segmentation into a single task, has witnessed considerable success on predefined tasks. However, traditional methods tend to struggle with catastrophic forgetting and poor generalization when learning from a continuous stream of new tasks. While continual learning aims to mitigate these challenges, our study reveals that existing continual panoptic segmentation (CPS) methods often suffer from efficiency or scalability issues. To address these limitations, we propose an efficient adaptation framework that incorporates attentive self-distillation and dual-decoder prediction fusion to efficiently preserve prior knowledge while facilitating model generalization. Specifically, we freeze the majority of model weights, enabling a shared forward pass between the teacher and student models during distillation. Attentive self-distillation then adaptively distills useful knowledge from the old classes without being distracted from non-object regions, which effectively enhances knowledge retention. Additionally, query-level fusion (QLF) is devised to seamlessly integrate the output of the dual decoders without incurring scale inconsistency. Our method achieves state-of-the-art performance on ADE20K and COCO benchmarks. Code is available at https://github.com/Ze-Yang/ADAPT.

Published as a conference paper at ICLR 2025
ADAPT: ATTENTIVE SELF-DISTILLATION AND QUAL-DECODER PREDICTION FUSION FOR CON- TINUAL PANOPTIC SEGMENTATION
Ze Yang, Shichao Dong, Ruibo Li, Nan Song & Guosheng Lin"
College of Computing and Data Science Nanyang Technological University 50 Nanyang Ave, Singapore 639798 { [ze001, , shichao001, ruibo001, nan001 (eee1eeeee.. www..nuu.eeeee ed. sg, gslin@ntu.edr.c
ABSTRACT
Panoptic segmentation, which unifies semantic and instance segmentation into a single task, has witnessed considerable success predefined tasks. However, traditional methods tend to struggle with catastrophic forgetting and poor gener- alization when learning from continuous stream of new tasks. While continual learning aims to mitigate these challenges, our study reveals that existing continual panoptic segmentation (CPS) methods often suffer from efficiency scalability issues. To address these limitations, we propose an efficient adaptation framework that incorporates attentive self-distillation and dual-decoder prediction fusion to efficiently preserve prior knowledge while facilitating model generalization. Specifically, we freeze the majority of model weights, enabling a shared forward pass between the teacher and student models during distillation. Attentive self- distillation then adaptively distills useful knowledge from the old classes with- out being distracted from non-object regions, which effectively enhances knowl- edge retention. Additionally, query-level fusion (QLF) is devised to seamlessly integrate the output of the dual decoders without incurring scale inconsistency, Our method achieves state-of ttate-of-the-art performance on ADE20K and COCO bench- marks. Code is available at petttps://iiiiie com Ze Yang/angggaaa.aa /AAAPT
INTRODUCTION
Panoptic segmentation Kirillov al. (019h a Xiong et al. 2019); Wang et al. 2020 Cheng al. (2000); (Strudel et al. (2021): Cheng et al. (2021 2022) is a fundamental computer vision task that provides a comprehensive understanding of visual scenes Li et al. (2024): Dong et (2024) by uni- fying semantic and instance segmentation. It predicts semantic masks for stuff maases--mmrrhouus background regions without distinct object boundaries, such as sky, road, and grass- and instance masks for thing classes, which are countable objects with well-defined boundaries like people, cars, and animals. This holistic task is crucial for various applications. including autonomous driv- ing Cordts et al. (2016), robotics Ros et al. (2015), and image editing (Aksoy et al. (2018).
In real-world scenarios, models are often required to continually adapt to new classes and evolving data distributions Yang et al. (2020) (022). To this end, continual learning (CL) has been developed to equip models with the capability to learn from sequence of tasks while maintaining previously acquired knowledge. A key challenge in CL is known as catastrophic forgetling, where the model gradually loses information learned from previous tasks during new training stages. This issue has been extensively studied in classification Li & Hoiem (2017); Rebuffi et al. (2017). object detec- tion Shhelkov et al. (2017): ); [iu et al. (2023) and semantic segmentation Cermelli et al. (2020); Douillard et al. (2021) Yang al. 2023
Despite this progress, continual panoptic segmentation (CPS) remains an under-investigated area with existing approaches exhibiting notable limitations. Concretely, CoMFormer Cermelli et al. 2023) finetunes the entire model on new tasks while employing knowledge distillation (KD) to
Guosheng Lin is the corresponding author.

---

Published as a conference paper at ICLR 2025
mitigate forgetting. However, this approach presents three potential risks. First, finetuning the entire model grants excessive plasticity, increasing the risk of overfitting. Second, although the final outputs are constrained by the distillation loss, the substantial plasticity in the intermediate layers can still lead to catastrophic forgetting. Lastly, updating the entire model using KD necessitates separate forward passes for the teacher and student models, significantly raising computational costs. Conversely, ECLIPSE Kim et al. (2024) seeks to preserve base performance by freezing most model weights. This strategy inevitably constrains the model's capacity to generalize new tasks due to restricted plasticity. Moreover, ECLIPSE continuously introduces additional learnable query features and embeddings for each new task, which may cause scalability issues as the number of tasks grows.
In this paper, we propose novel approach for CPS that effectively balances base knowledge reten- tion and learning new tasks. Building on Mask2Former Cheng et al. (2022), we freeze the image encoder and pixel decoder to preserve base knowledge and enhance efficincy.. Notably, this weight- freezing design allows for shared forward pass between the teacher and student models, thereby significantly reducing computational overhead. By selectively finetuning only the cross-attention layers and feed-forward networks in the transformer decoder, our adaptation strategy optimally balances between plasticity (learning new information) and rigidity (retaining old knowledge). In addition, the computational overhead of our approach remains nearly constant, regardless of the number of continual learning steps.
To further alleviate forgetting, develop attentive self-distillation mechanism, inspired by focal loss Lin et al. (2017). Existing KD methods Cermelli et al. :DuulaDD et al. (2021) Yang al. (2023) often treat all entities (pixels or instances) uniformly, leading to an overemphasis on dominant background regions that provide little benefit in preserving prior knowledge. In contrast, our approach adaptively ne-weights the contribution of each entity in the distillation loss based on the background confidence predicted by the teacher model. This modulated distillation can effectively concentrate on learning informative entities, enhancing the retention of previous knowledge.
Douillard
Finally, recognizing that KD can still accumulate errors over successive learning steps, we propose a dual-decoder prediction fusion paradigm, where we retain the transformer decoder trained on the base dataset for subsequent inference. Specifically, we utilize this fixed decoder to predict base classes while the adapted decoder predicts novel classes learned up to the current step. To com- bine the predictions from these two decoders, we devise robust query-level fusion (QLF) strategy to avoid the scale inconsistency issue that occurs in another naive solution - probability levell fu- sion (PLF). Extensive experiments on ADE20k and COCO benchmarks showcase that method outperforms state-of-the-aar approaches in CPS.
In summary, approach, ADAPT, offers efficient and scalable solution CPS. Our contribu- tions can be summarized as follows:
We present an efficcent adaptation strategy that freezes the image encoder and pixel de- coder, enabling shared forward pass between the teacher and student models during dis- tillation, significantly reducing computational overhead.
We develop an atlentive self-distillation loss, which emphasizes informative entities (or queries) and down-weights less useful ones based on background confidence, improving the preservation of prior knowledge.
We devise a (QLF), which combines the outputs of the base and adapted decoders without relying on probability fusion, effectively preventing the scale inconsistency issue present in PLF
prediction fusion mechanism, dubbed as query-level fusion
Our method outperforms current CPS approaches, achieving superior results ADE20k and COCO benchmarks.
2 RELATED WORKS
2.1 PANOPTIC SEGMENTATION
Panoptic segmentation, introduced by Kinillov et al. (2019b), aims to unify semantic and instance segmentation, with Panoptic FPN Kinillov et al. (209aa) extending this concept using feature pyra-
2

---

Published as a conference paper at ICLR 2025
mid network for both stuff and shing classes. UPSNet Xiong al. (2019) improved efficiency by introducing a learnable panoptic head, while Axial-DeepLab Wang et al. (2020) leveraged axial at- tention enhance spatial feature representation. panoptic segmentation using pixel-level classification in a fully convolutional framework. Trans- formers further advanced the field with MaskFormer [heng et al. (202T), I which introduced mask classification, simplifying segmentation. Mask2Former(Cheng et al. (2022) extended this by provid- ing generalized, transformer-based framework for universal segmentation tasks, achieving state- of-the-an art performance. For fair comparison, we adopt the Mask2Forme framework, consistent with prior works on continual panoptic segmentation Cermelli al. (2023): Kim al. (2024).
Cheng al. (2020) streamlined
2.2 CONTINUAL PANOPTIC SEGMENTATION
Continual panoptic segmentation remains a relatively under-explored field, with initial efforts fo- cusing addressing the challenge of catastrophic forgetting. Building upon the transformer-basee architecture Mask2Former Cheng et al. (2022). JoMFormer/Cermelle et al. (2023) employs knowl- edge distillation and mask-based model. However, this approach yields suboptimal base class performance due to error accumulation over continual learning steps. ECLIPSE Kim al. (2024) improves base class retention by freezing the majority of model weights and progressively introduces learnable prompt features to handle new tasks. While this strategy improves rigidity, it limits the model's plasticity and raises scalability issues additional prompts are required for each task. To resolve these limitations, we propose novel approach that effectively preserves base knowledge while maintaining strong generalization ability, without incurring scalability issues.
to mitigate forgetting when finetuning the entire
2.3 SELF DISTILLATION
Unlike traditional knowledge distillation HariilaaatooHiitoon al. (2015), which typically employs larger teacher models to guide smaller student models, self-distillation leverages the same architecture to transfer knowledge gained in carlier stages improve performance later stages. This technique has been applied across various domains to mitigate forgetting, including classification Li & Hoiem
2017 : Rebufhi et al. (2017); p: Zhang al.] (2021), object detection (Shmelkov et al. (2017); Liu
2023 semantic segmentation Cermelli et al. (2020); Douillard et al. (2021) 1): Yang al. (0233),
panoptic segmentation Cermelli et al. (2023), and image generation [Song et al. (2024). However, most of these methods require separate forward passes for the teacher and student models, resulting in considerable computational overhead. To overcome this limitation, we propose to share a single forward pass for the majority of modules between the teacher and student models. By freezing the model up to the pixel decoder and adapting only the final decoder, our approach achieves a double cenefit-improving efficiency and retaining base knowledge.
3 METHOD
3.1 PRELIMINARY
Problem Setting. Panoptic segmentation is a challenging task that aims unify semantic and instance segmentation. It predicts semantic masks for 'stuff" classes (amorphous, background regions without distinet object boundaries, e.g., sky, road, grass) and instance masks for 'thing' classes (countable object classes with distinct boundaries, e.g., people, cars, animals). In the con- text of continual learning, the objective is to train a model across a sequence of tasks (or steps) t= 1,,......, T. input image x contains at least one new-class mask label, i.e., |gy| A O. Importantly, at step 4, only the current class labels yj E Ct are available, while the labels from previous steps co:111 or future steps C++1:T are inaccessible, even though these classes may still appear in the input image. This scenario is known as the overlapped setting Cermelli et al. (020)). Following the convention in continual learning, the class subsets across tasks are disjoint. A special class label 0 represents "no ob ject" detected for given query, akin to the "background" class in traditional segmentation methods Chen et al. (2017). Once all learning steps are completed, the model is evaluated on the full set of classes C encountered throughout training.
At each step f, a new training set D' = {(z,, ys 1)) is available, where each
3

---

Published as a conference paper at ICLR 2025
Dowelex
Desoder Miser
Transfornc
Pool Doccder
Brecckt
Mjsc
Alcetiio SeFOrsittionn
Combi Segrants
Cuurnies
Transtomerr
Docoder
For rilro roo only
Predicton hader
pastigrant, MAY,
Figure 1: Overview the proposed dual-decoder framework for continual panoptic segmentation. During training, attentive self-distillation is applied between M and M During inference, the fixed base decoder Mirams predicts base classes, while the adapted decoder Minnn handles novel classes. We apply query-level fusion (QLF) to effectively combine the outputs of these two decoders and preventing scale mismatches. More details can be found in Sec. 3.2 and Sec. 3.3
Model Architecture. We adopt Mask2Former Cheng et al. (2022), widely used transformer- based model for universal segmentation tasks. Given an input image x, the image encoder Menc first extracts low-resolution features. These features are then progressively upsampled by the pixel decoder Mpixel to generate high-resolution per-pixel embeddings. The transformer decoder Muans
utilizes a of learnable object queries Q = { N 1=11 which interact with the feature pyramid output by the pixel decoder via cross-attention. For each query şi, the decoder outputs a binary mask M; E with associated class probabilities p (i, c) E IRNN+11 where // and W denote the spatial dimensions of the input image, $ is the stride and N₂ the number of classes. This architecture offers two key advantages: (1) compared to per-pixel classification/ Long et al. (015)), mask classification enables better differentiation between objects, making it ideal for panoptic segmentation, and (2) its DETR-style Striiin Carion al. (2020) end-to-end set prediction eliminates the need for cumbersome two-stage process He et al. (2017). resulting in a more efficient and streamlined framework.
R _
x W.
3.2 ADAPTATION STRATEGY
Continual panoptic segmentation has been scarcely explored. The existing method, CoM- Former Cermelli et al. (2023), finetunes the entire model on the new sequence of tasks, which often leads to significant degradation in the performance of base classes. To mitigate this issue, Kim et al. Kim al. (2024) freeze most of the model's weights, allowing only the newly introduced learnable queries (or prompts) to be updated. While this mechanism helps maintain base class per- formance (rigidity), it compromises the model's ability to generalize to new tasks (plasticity). In this work, we freeze the image encoder Menc and pixel decoder Mpixel for efficiency and systemat- ically investigate the effects of fine-tuning different components of the transformer decoder Miranss which consists of L transformer blocks each with self-attention layers (SA.) cross-attention layers {_ and feed-forward networks (F We observe that freezing the image en- coder Mene and pixel decoder Mpixel significantly reduces catastrophic forgetting while preserving considerable generalization capacity. Additionally, it enables the feasibility of our self-distillation design, which relies on the shared and frozen weights. For the transformer decoder M we fine- tune only the cross-attention layers and the feed-forward networks (FFN), striking an ideal balance between rigidity and plasticity, along with high efficiency. We initialize the learnable queries Q' from Q' and follow Cermelli et al. (023) for classifier initialization..
3.3 ATTENTIVE DIL--DISTILAAION
Even though only the transformer decoder is updated during continual learning, the forgetting issue persists as the previous class labels CU:t-1 are no longer available. To mitigate this issue, we resort to knowledge distillation, which penalizes the discrepancies between the predicted class probabilities of the teacher model T and student model S given the same input. Following the prior method Yang eet al. (2023) in continual semantic segmentation, we opt to establish correct class correspondences for knowledge distillation by modifying the old model outputs. Notably, the natural bias toward
4

---

Published as a conference paper at ICLR 2025
newly learned classes in continual learning results in false negatives (FN) for base classes and false positives (FP) for novel classes, degrading both base and novel class performance. Consequently, the key objective is to mitigate this bias using the knowledge from the previous classes Co:t-1 To this end, we apply knowledge distillation only to the queries B that are not matched with any new classes C in the bipartite matching Carion et al. (2020), which can be formulated as:
0 kd (z, y)
Σ Σ iEB CECO)U(()
1' (1, log (',,'
[
where pt (i. e) refers to the probability of class for query of predicted by Mirans and ( is the probability p-- c) predicted by Minss expanded with zero probability for the added novel classes e C², formally as:
0
if
1r (1,)) (i,c) (p) otherwise.
(2)
However, this distillation loss treats all queries equally, despite the fact that over 95% of queries contain no ob ject. This can cause the model to overemphasize these dominant no ob ject regions, which contribute little to preserving previously learned knowledge. To address this issue, propose adaptively re-weighting the contribution of each query within the distillation loss. In- spired by focal loss Lin et al. 20.7).. we incorporate a modulating term a(1 - p. (1, 0)) into the distillation loss, resulting in our adaptive weighted version:
Ka (x. v)
1 B
Ca(1-
P'- (i,
Σ CEC
(i.)
(3)
EEB
where y 2 O is tunable focusing parameter and 0 is a scaling factor. When y 1, this modulating term down-weights queries that are predicted as no object with high confidence (it-'(i, 0) - 1) by the old model. The parameter y controls the shape of the re-weighting curve. Specifically, the larger value of y extends the flat range of the curve and down-weights more no object queries with lower confidence. In addition, the scaling factor a allows the modulating term to be greater than one, making it possible to emphasize the contribution of object queries (p' (i. 0) - 0). When y 0, the proposed adaptive re- re-weighted KD loss reduces the standard weightteig version. We empirically observe that = 3 and a = 4 yields the best results.
(i--1(i,
Discussion. We highlight the distinction of our adaptive re-weighted self-distillation in three as- pects. First, thanks to our model freezing and adaptive decoder design, the teacher model and student model share the same weights up to the pixel decoder Mpiieel Unlike traditional knowledge distillation (KD) based methods Cermelli et al. (2023); Douillard et al. (202I), which require sepa- rate forward passes for the teacher model and student model, our approach performs only a single forward pass except for the final dual decoders M and M manifesting higher efficiency. Second, in contrast to Kim et al. (2024), which encounters scalability issues due to the continual in- troduction of new prompt features and embeddings for each new task, our design remains invariant to the number of tasks T or the number of classes Ct| within each task t. allowing for greater scal- ability. Third, distinct to (Cemmelli et al. (2023), which applies unbiased KD Cermelli et al. (2020) to all queries, our approach selectively applies modulated KD only to queries that are not matched with any new class, thereby maintaining old knowledge more effectively and efficiently.
3.4 DEAL-DECODER PREDICTION FUSION
Although knowledge distillation has been adopted to alleviate catastrophic forgetting, it tends to accumulate errors over successive continual learning steps, devastating old knowledge learned in the early stage. This challenge motivates us to retain the transformer decoder Mi trained on the base dataset DO as it contains precise knowledge for decoding useful information from the feature pyramid output by the pixel decoder Mpizeet In particular, only the weights of the cross- attention layers and the feed-forward networks need to be stored, since the self-attention layers are fixed and shared across all learning steps. Notably, the retained base decoder does not participate in the training phase, resulting in no additional computational overhead. During inference, the base decoder M is used to predict masks for the base classes CO, while the adapted decoder M handles the new classes & learned up to the current step t. Following Cheng et al. (2022); Cermelli et al. (2023), the Softmax activation function to calculate the final class scores.
5

---

Published as a conference paper at ICLR 2025
Af
Ai
Af
hood
Spiiee
A
Probabilty-level Fusion (PLF)
Basiggoudd
Query-ievll Fusion (QLF)
Figure 2: Illustration of Probability-Leee Fusion (PLF) and Query-Lever Fusion (QLF). Each query predicts a segment with associated class scores. The query square and corresponding segment edge are bolded in the same color (red or purple) indicate their correspondence.
Probability- fusion. As the transformer decoders M and MMd Maans are responsible for predict- ing base and novel classes respectively, an effective mechanism is needed to combine their outputs for the final predictions. One straightforward method, which refer to as probability leveel fusion (PLF), is to replace the base-class c Co probability scores př.c for each query predicted by M with the corresponding scores ph,, predicted by M For brevity, here we omit x in p (i, e) and use the subscript version pl,e- After obtaining the combined class scores for each query, we need to determine their associated binary masks. Specifically, the corresponding mask MT for the i-th query 4 is derived by selecting between the masks M and Mf, which are generated by the base decoder M and the adapted decoder M respectively. The selection process is determined by the combined probability score p¹, and is defined as follows:
[ph if (pi otherwise,
M if E C M
M
(4)
otherwise.
Here, 2; argmax, Pi.c. denotes the class with the highest combined probability score. This mechanism ensures that masks predicted by the base decoder are used for base classes (c E C.), while masks from the adapted decoder are utilized for novel classes E C2:t). However, this approach can lead to inconsistencies between the probability scales of base and novel classes, as they are not optimized on the same data distribution. We observe that base-class probabilities tend to have larger values than novel-class probabilities, which can lead to confusion between base and novel classes. For instance, shown in Fig. 2 the novel class "airplane" misclassified as the base class "sky" due to this discrepancy in probability-level fusion.
Query-level fusion. light of the inconsistency issue in PLF, we propose query-level fusion (QLF) as a simple yet effective alternative. As illustrated in Fig. [2] we discard the base-class probabili- ties cc (²) predicted by Min. so that the adapted decoder only makes predictions for novel classes. Conversely, M mans is solely responsible for predicting the base classes. In this way, the "airplane", though with high confidence in the base class "sky", be correctly suppressed by the "no object" class with even higher confidence. Otherwise, the "airplane" will be misclas- sified "sky", occurred in PLF (See left side of Fig. 2 During query-level fusion, the more accurate sky mask with a finer boundary, predicted by the base decoder, supersedes the less precise version generated by the adapted decoder, successfully eliminating error accumulation. This pro- cess is achieved using the standard panoptic inference procedure (see Algorithm [ Appendix) in Fark2Former Cheng et al. 2022
EXPERIMENTS
4.1 DATASET AND EVALUATION METRICS
We validate the effectiveness of our approach on ADE20K Zhou et al. (2017) and COCO Lin et al. 2014) benchmarks. ADE20K consists of 25,574 training images and 2,000 validation images. This

---

Published as a conference paper at ICLR 2025
dataset includes a total of 150 classes, divided into 100 "thing" classes and 50 "stuff" classes. Unlike COCO Lin et al. (2014), with an average of 7.7 instances and 3.5 classes per image, and Pascal al. (2010) with 2.3 instances and 1.4 classes per image, ADE20K averages 19.5 instances and 9.9 classes per image. The complexity of scenes in ADE20K makes it a challenging benchmark for continual panoptic segmentation. COCO consists of 1 118,287 training images and 5,000 validation images with 133 classes, We leave the COCO results in the Appendix. We adopt the conventional Panoptic Quality (PQ) Kirillov al. (2019b) metrie used in panoptic segmentation, which is defined as the product of Recognition Quality (RQ) and Segmentation Quality (SQ):
VOC
TPI
X Cmaeee ( loU(p,g)
PQ = RQ x SQ
TP F 1F
(PP]
Recognition Quality (RQ)
Segmentation Quality (SQ)
where loU(p. 9) is the Intersection- Union between a predicted segment p and ground truth segment g. TP. FP, and FN represent true positives, false positives, and false negatives, respec- tively. RQ measures how well the model detects and classifies objects, while SQ evaluates how well the predieted segments match the ground truth in terms of shape and overlap.
4.2 BASELINES AND CONTINUAL PROTOCOLS
We adhere to the continual panoptic segmentation protocol from ECLIPSE et al. (2024). evalu- ating both 100- (n 5, 10, 50) and 50- n (n 10, 20, 50) scenarios. For instance, 100-12 -10 refers to base training with 100 classes, followed by 5 incremental steps, each introducing 10 new classes. Note that smaller n results in more continual steps, making the scenario more challeng- ing. We re-implement the representative methods MiB Cermelli et al. (2020) and PLOP Douil-] lard et al. (2021), originally developed for continual semantic segmentation, as competitive base- lines. Additionally, we include the state-of-the-arr methods CoMFormer Cermelli et al. (2023) and ECLIPSE Kim et al. (2024), which are specifically designed for continual panoptic segmentation, in our evaluation. Alongside these, we also evaluate the simplest baseline, FT (Fine-Tuning) Tuning), which finetunes the base model on new tasks without any specific continual learning design. All exper- iments are conducted under the overlap setting, following the latest state-of-the-art approach Kim et al. (024).. After the final step T, we evaluate performance on base classes (C"), novel classes and the full of classes (C":1", denoted as all).
(C
4.3 IMPLEMENTATION DETAILS
Following state-of-the-ar methods Cermelli et al. 2023); Kim et al. (024)) we adopt the Mask2Former Cheng et al. (2022) model with an output stride of 4, using ResNet-50 He et al. (2016) as the backbone, unless otherwise stated. To ensure fairness, we follow the same training hyperparameters as our competitors Cermelli et (2033); Kim et al. (2024), with the exception of using a higher learning rate, which leads to faster convergence and slightly improved performance. For all settings, we report Panoptic Quality (PQ) results on the standard validation set. Experiments conducted using two NVIDIA RTX 6000 Ada GPUs on ADE20K and four COCO. For more details, please refer A.5
4.4 PERFORMANCE
As shown in Table [ the FT baseline fails retain previous knowledge due the absence of anti-forgetting mechanisms. This leads to catastrophic forgetting, not only of base class knowl- edge but also of the new classes learned in earlier steps, resulting in poor overall performance on novel classes. MiB Cermelli et al. (2020) and PLOP Douillard et al. (202T) demonstrate significant improvements over FT, thanks to unbiased knowledge distillation and multi-scale local POD distil- lation. CoMFormer CCermell et al.) (203) improves performance further, particularly in the 100-n settings, surpassing PLOP by +5.5 PQ on base classes in the 1C0-10 scenario. ECLIPSE Kim et al. (2024) mitigates the forgetting issue by only updating newly introduced prompts, which brings improvements on base classes, such as a +5.2 PQ gain over CoMFormer in the 100-10 setting. Our approach, FDAS, consistently surpasses all state-of-the-art methods across all scenarios. In the 100-10 setting, FDAS outperforms CoMFormer and ECLIPSE on both base (+6.2/+1.0 PQ) and novel (+9.2/49.8 PQ) classes, achieving 42.2 PQ base classes and 26.3 PQ on novel classes. In
7

---

Published as a conference paper at ICLR 2025
100-5 (11 tasks)
100-10 (6 tasks)
100-50 tasks)
Method
I-100
101-150
1-100
101-150
1-100
101-150
FTf
0.0
0.0
2.9
0.0
25.8
MiB Cermelli
2020
24.0
6.5
18.1
27.1
10.0
21.4
35.1
19.3
29.8
PLOPT Dounaar et 2021
28.1
15.7
24.0
30.5
17.5
26.1
41.0
26.6
36.2
CoMFormen Cermell or a. 2023
34.4
15.9
36.0
17.1
29.7
41.1
27.7
36.7
ECLIPSED
Kim et 2024
41.1
16.6
32.9
41.4
18.8
33.9
41.7
23.5
35.6
ECLIPSE
Kim al. 2024
39.5
31.2
41.2
33.0
39.9
21.1
33.6
ADAFT
42.3
19.8
34,8
42.2
26.3
36.9
42.5
27.8
37.6
joint
43.4
32.9
39.9
43.4
32.9
39.9
43.4
32.9
39.9
(a)
50-10 (11 tasks)
50-20 (6 tasks)
50-50 (3 tasks)
Method
I-50
51-150
all
1-50
51-150
all
1-50
51-150
all
FTi
0.0
1.7
1.1
0.0
4.4
2.9
0.0
12.0
8.1
MiB: Cermelli
2020
34.9
16.8
38.8
10.9
20.2
42.4
15.5
24.4
PLOPT Douilian el 2021
39.9
15.0
23.3
43.9
16.2
25.4
45.8
18.7
27.7
CoMFormer
Cermelll
2023
38.5
15.6
23.2
42.7
17.2
25.7
45.0
19.3
27.9
ECIJIPSF (PPSS//Kin Kim ara
45.9
17.3
26.8
46.4
19.6
28.6
46.0
20.7
29.2
ECLIPSE
Kim et al. 2024
46.4
16.8
26.7
47.1
17.7
27.5
46.0
18.6
27.7
ADAPT (OUIS)
49.5
21.3
30.7
49.7
26.2
34.0
49.8
28.7
35.7
joint
50.5
34.6
39.9
50.5
34.6
39.9
50.5
34.6
39.9
(b)
Table 1: Continual panoptic segmentation results (PQ) on the ADE20K Zhou et al. (0117) bench- mark, with the number of base classes (C)) (a) 100 and (b) 50. All methods are based upon the Mask2Former framework Cheng et al. (2022). The joint setting indicates that all classes are trained simultaneously in an offline manner. and f indicate results are taken from Cermelli al. (2023) and Kim et al. (024), respectively. denotes results reproduced using the official code.
more challenging 50-10 scenario, FDAS exceeds CoMFormer and ECLIPSE by +11.0/+5..7 PQ on base classes and +3.1/44. PQ novel classes, respectively. Beyond its strong performance, FDAS offers a computational advantage by utilizing a single-forward self-distillation design, unlike MîB, PLOP, and CoMFormer, which require double forward passes. Compared to ECLIPSE, our FDAS balances strong base performance with enhanced generalization capacity, made possible by the complementary fixed and adaptive decoders combined with self-distillation.
4.5
ABLATION STUDY
Unless otherwise stated, we conducted ablation studies on the multi-step ADE20K 100-10 setting.
4.5.1
OVERALL STUDY
Building upon the baseline CoMFormerCermelll et al. (2023). Table[[highligee the complementary contributions of each proposed component, demonstrating their collective impact on the overall performance improvements. Our adaptation strategy improves both base and novel performance, showing enhanced plasticity and rigidity in balance. Incorporating attentive self-distillation further boosts novel class performance since the inherent bias toward new classes is alleviated. As expected, prediction fusion substantially enhances knowledge retention, bridging the base performance gap with joint training. Finally, combining all three components yields the best results, demonstrating that the proposed components are complementary to each other. In summary, our approach offers well-balanced and efficient solution for continual panoptic segmentation.
We fixed the weights of the image encoder and pixel decoder for two key reasons. First, finetuning these two components does not enhance the model's generalization capacity, yet it substantially increases the risk of overfitting and computational cost, making it an inefficient trade-off. In contrast, freezing the image encoder and pixel decoder helps alleviate the forgetting issue since their weights preserved retain base knowledge. Interestingly, comparing the first two rows of Table 2 the transformer decoder itself (ablated in the following paragraph) demonstrates sufficient learning capability generalize to new tasks. Second, this design allows large portion of weights to be shared between the teacher model and the student model during knowledge distillation, which we refer eeff-distillation.
8

---

Published as a conference paper at ICLR 2025
Aduptation Strategy
Attentive Self-
Predition Fusion
100-10 (6 tasks)
Distillation
1-100
101-150
all
36.0
17.1
29.7
38.3
20.1
32.3
37.2
22.6
32.4
42.4
35.8
42.2
26.3
36.9
Table 2: Effect of the proposed components. Without attentive self-distillation, unbiased KD
serves as the baseline. When the adaptation strategy is omitted, weight-share self-distillation be- comes infeasible, and the entire old model is used the teacher model.
Self-att (SA)
Cross-att (CA)
Trainable Params
100-10 (6 tasks)
FFN
1-100
101-150
all
0.29M
42.0
5.3
29.7
9.75M
42.3
23.6
36.1
v
2.66M
42.5
16.6
33.9
V
2.66M
42.6
34.1
12.12M
42.1
22.5
35.6
12.12M
42.2
26.3
36.9
5.03M
42.3
18.0
34.2
14.50M
42.4
24.3
36.4
Table 3: Impact of fine-tuning different components of the transformer decoder. FFN, Self-att
(SA), and Cross-att (CA) refer to the feed-forward network, self-attention layers, and cross-attention layers, respectively.
4.5.2 ADAPTATION STRATEGY
Based on this, investigate the impact of fine-tuning different components within the transformer decoder, presented in Table Finetuning the FFN alone yields strong results (23.6 PQ) on novel classes, while pross-attention (CA) or self-attention (SA) alone shows limited generalization ability. It highlights the critical role of FFN in enhancing model generalization. The best performance is achieved when both FFN and CA are finetuned together, reaching 26.3 PQ on novel classes and 36.9 PQ overall. In contrast, finetuning all components or combining FFN with SA does not lead to sub- stantial gains, indicating that the combination of FFN and CA offers the most effective improvement in generalization without overfitting.
4.5.3 ATTENTIVE SELF- SILL-DISSILAAION
As shown in Table the effect of the attentive self-distillation is influenced by the modulation factors a and y. When y = 0, the attentive ceff-distillation loss degrades to the uniform version, which results in poorer performance novel classes (c.g., PQ drops to 17.1 and 15.7 for 2 and = 4, respectively) regardless of different values a. This can be attributed to the overwhelming influence of the dominant informative old class objects. As y increases, the attentive self-distillation can effectively focus on informative regions (old classes Cú:t-1), resulting improved novel performance. This because, without retention of old knowledge, the base classes will be misclassified as novel classes due to the inherent bias in continual learning, leading to false positive predictions for novel classes and hence lowering PQ. In contrast, these mistakes can be effectively corrected by our attentive self-distillation with a concentration on informative old classes. Specifically, the combination of = 4 and 7 yields the highest overall PQ of 36.9, striking balance between retaining base knowledge (42.2 PQ) and learning novel classes (26.3 PQ). This demonstrates the effectiveness of re-weighting the contribution of queries based preserving previous knowledge while efficiently learning new information.
ob jejctt regions, which diverts the distillation process away from
ob object confidence, allowing the model to concentrate
4.5.4 DUAL-DECODER PREDICTION FUSION
As shown in Table 5 without any fusion strategy, the model struggles to retain base knowledge due to accumulated errors multiple learning steps. Introducing probability-level fusion (PLF), where base-class probabilities are replaced by those from the base decoder, significantly boosts base performance to 43.1 PQ. However, PLF negatively affects novel classes, reducing PQ from

---

Published as a conference paper at ICLR 2025
100-10 (6 tasks)
100-10 10 (6 tasks)
OK
1-100
101-150
all
7
1-100
10I-150
all
42.6
17.5
34.2
41.9
24.3
36.0
Fusion Strategy
100-10 (6 tasks)
2
0
42.6
17.1
34.1
42.1
23.9
36.0
1-100
101-150
all
4
0
42.6
15.7
33.6
42.4
25.0
36.6
42.7
31.9
42.5
22.8
Without
37.2
22.6
32.4
Probability-Iovel
43.1
20.8
35.7
42.2
36.1
22.2
Query-level
42.2
26.3
36.9
2
42.3
23.8
36.2
2
42.1
23.5
35.9
42.5
22.4
35.8
42.2
26.3
36.9
10
42.6
20.1
35.1
10
42.5
23.3
36.1
Table 5: Effect of dual-decoder pre-
diction fusion.
Table 4: Effect of and y in the attentive self-distillation
loss.
Training #hters
Inference
100-10 (6 tasks)
Method
#Params
Time
FL.OPs
GPU Mem.
Time
1-100
101-150 all
CoMFormer CoMFormerCermeill Cermelli ct al. 2023
44.38M
4k
3.36 hrs
97.46G
4010 MB
43,9 ms
36.0
17.1
29.7
ECLIPSE Kim et al 2024
0.55M
16k
3.48 hrs
99.27G
5407 MB
68.7 ms
41.2
16.5
33.0
ADAPT (Ours)
12.12M
4k
2.27 hrs
105.76G
7127 MB
48.3 ms
42.2
26.3
36.9
Table 6: Efficiency comparison on both training and inference phases. The training time is reported using two NVIDIA RTX 6000 Ada GPUs. FLOPs, GPU memory usage, and time cost calculated on a per-image, per-device basis during inference with the same device setup.
22.6 to 20.8, due scale inconsistency between the base and novel class probabilities. In contrast, query-level fusion (QLF) resolves this inconsistency by eliminating the need for probability fusion, resulting more balanced performance across both base and novel classes and yielding the highest overall PQ of 36.9.
4.5.5
EFFICIENCY
Table highlights the efficiency of our approach. During incremental training, ADAPT requires only 2.27 hours for five training steps, representing reduction of 34.8% and 32.4% in training time compared CoMFormer Cermelli al. (2023) and ECLIPSE Kim et (2024), respectively. The reduced training time can be attributed to ADAPT's frozen- and shared-weight design between the teacher and student models, which eliminates the need for dual-model forward passes, a key re- quirement for knowledge distillation in CoMFormer. Additionally, our efficient adaptation strategy enables fourfold reduction in the number of training iterations each step, decreasing from 16k in ECLIPSE Kim et al.) 2024) to 4k in our approach. This substantial reduction contributes to the overall decrease in training time. terms of inference, our method incurs modest 6.5% increase in FLOPs compared to ECLIPSE. However, this is accompanied by substantial performance gains, particularly in the novel PQ, which improves from 16.5 to 26.3 remarkable 59.4% increase. Sim- observed when compared to CoMFormer. Furthermore, the inference speed of ADAPT remains comparable to that of CoMFormer, highlighting the efficiency and practicality of our approach. In contrast, ECLIPSE demonstrates slower inference speeds, likely attributable its unoptimized implementation.
ilar improvements
5 CONCLUSION
In this paper, we propose novel approach for continual panoptic segmentation that effectively balances the retention of base knowledge and learning new tasks. Concretely, we present an efficient adaptation strategy that freezes the image encoder and pixel decoder, allowing a shared forward pass between the teacher and student models, significantly saving computational costs. Upon this baseline, we introduce an attentive self-distillation loss, which emphasizes informative queries and down-weights non-object regions during distillation, to effectively enhance knowledge retention. Additionally, we devise a query-level fusion mechanism to combine the predictions from the dual- decoders. It successfully avoids the scale inconsistency issue occurred in probability- fusion. Our method, ADAPT, demonstrates superior performance on ADE20k and COCO benchmarks, showcasing its effectiveness and scalability in continual learning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021). This research is also supported by the MoE AcRF Tier 1 grant (RG14/22).
REFERENCES
Yagiz Aksoy, Tae-Hyun Oh, Sylvain Paris, Mare Pollefeys. and Wojciech Matusik. Semantic soft segmentation. ACM Transactions Graphics (TOG), 37(4):1-13, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), pp. 213 229, 2020.
Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Mod- cling the background for incremental learning in semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9233-9242, 2020.
Fabio Cermelli, Matthieu Cord, and Arthur Douillard. Comformer: Continual learning in semantic and panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3010-3020, 2023.
Liang-Chich Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 055807 2017.
Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chicl Chen. Panoptic-deeplab. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455, 2020.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34: 18644-1777 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedingy of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1290-1299, 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Tobias Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3213-3223. IEEE, 2016.
Shichao Dong, Ze Yang, and Guosheng Lin. 3d-virtfusion: Synthetic 3d data augmentation through generative diffusion models and controllable editing. arXiv preprint arXiv:2408. 13788, 2024.
Arthur Douillard, Yifu Chen, Arnaud Dapogny, and Matthiu Cord. Plop: Learning without for- getting for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4040-4050, 2021.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision (IJCV), pp. 303-338, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2961-2969, 2017.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: /003.02500.25.. 2(7), 2015.
11

---

Published as a conference paper at ICLR 2025
Beomyoung Kim, Joonsang Yu, and Sung Ju Hwang. Eclipse: Efficient continual learning in panop- tic segmentation with visual prompt tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3346-3356, 2024.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6408, 2019a.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic seg- mentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9404-9413, 2019b.
Ruibo Li, Chi Zhang, Zhe Wang. Chunhua Shen, and Guosheng Lin. Self-supervised 3d scene flow estimation and motion prediction using local rigidity prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Zhizhong Li and Derek Hoiem. Learning without forgetting, IEEE T-PAMI, 40(12):2935-2944 2017.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. European confer- ence on computer vision (ECCV), pp. pp. 140-755,70 2014.
Tsung-Y. Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2980-2988 2988, 2017.
Yaoyao Liu, Bernt Schiele, Andrea Vedaldi, and Christian Rupprecht Continual detection trans- former for incremental object detection. In Proceedingy of the TEEF/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 23799-23808 2023.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl:
Incremental classifier and representation learning. In CVPR, 2017.
German Ros, Angel D. Sappa, and Joan Serrat. Vision-based offline-online perception paradigm for autonomous driving. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, pp. 231-238. IEEE, 2015.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detec- tors without catastrophic forgetting. In ICCV, 2017.
Nan Song, Xiaofeng Yang, Ze Yang, and Guosheng Lin. Towards lifelong few-shot customization of text-to-image diffusion. arXiv preprint arXiv:2411.05544, 2024.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12662-272,00 2021.
Huiyu Wang. Yukun Zhu, Brian Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Axial- deeplab: Stand-alone axial-attention for panoptic segmentation. In European Conference on Com- puter Vision, pp. 108-126. Springer, 2020.
Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8818-88 8826, 2019.
Ze Yang. Yali Wang, Xianyu Chen, Jianzhuang Liu, and Yu Qiao. Context-transfommer Tackling object confusion for few-shot detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 2266--12600, 2020.
12

---

Published as a conference paper at ICLR 2025
Ze Yang, Chi Zhang, Ruibo Li, Yi Xu, and Guosheng Lin. Efficient few-shot object detection via knowledge inheritance. IEEE Transactions on Image Processing, 32:321-334, 2022.
Ze Yang, Ruibo Li, Evan Ling, Chi Zhang, Yiming Wang, Dezhao Huang, Keng Teck Ma, Minhoe Hur, and Guosheng Lin. Label-guided knowledge distillation for continual semantic segmentation on 2d images and 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18601-18612, 2023.
Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(8):4388- 4403, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 CONTINUAL PANOPTIC SEGMENTATION ON THE COCO BENCHMARK
We evaluate approach the COCO panoptic segmentation benchmark Lin al. (2014), which consists of 1 118, 118,287 training images and 5,000 validation images distributed across 133 classes. In alignment with with|Kim Kim al. (2024), we use 83 base classes and incrementally add 50 additional classes. It is important to note that the class order in COCO panoptic segmentation follows a sequence of "things" and "stuff"". We report results based on a randomly shuffled class order as proposed by Kim et al. (2024), given by the following sequence:
[1, 3, 10, 47, 58, 9, 88, 16, 126, 120, 17, 129, 35, 119, 59, 57, 54, 90, 75, 38, 80, 48, 131,
56, 95, 25, 2. 68, 110, 32, 29, 7, 52, 83, 102, 84, 73, 45, 117, 87, 46
118, 34, 61, 19, 13, 111, 63, 98, 130, 66, 78, 92, 33, 86, 127, 104, 64, 49, 36, 6, 91, 50,
112, 65, 132, 92, 27, 122, 22, 51, 85, 125, 28, 89, 70, 62,
108, 125, 123, 39,
20, 40, 41, 114, 128, 74, 18, 99, 100, ED, 30, 124, 69, 37, 13, 23, 116, 55, 26, 121, 71, 67,
106, 132, 42, 107, 105, 109, 103, TE, 94, 15, 78, 53, 21, 96, 72, 112, 44, 4).
We compare our method with two baseline approaches, PLOP Douillard et al. (202I) and CoM- Former Cermelli et al. (2023), with the ResNet-50 backbone network under the overlap setting. As illustrated in Table 7 our approach outperforms these baselines in terms of all class performance.
83-5 (11 tasks)
83-10 (6 tasks)
Method
1-83
84-133
all
1-83
84-133
all
CoMFormer ECLIPSE Kim of al. 2024
CerfoelllCCCCllllii Cormelli et al. 2023
33.5
21.9
29.1
38.3
30.7
35.5
44.2
18.7
34.6
44.9
21.3
36.0
ADAPT (Ours)
45.3
23.2
37.0
39.3
Joint
51.1
52.1
51.5
51.1
Table 7: Continual panoptic segmentation results (PQ) on the COCO Lin et al. (014)) benchmark with the number of base classes set to 83 under the overlap setting.
CaMFarmer
ECLISSE
ADAPT (Ours)
35
30
20
15
base
novel
all
Figure 3: Continual panoptic segmentation result (PQ) distributions for randomly shuffled class orders under the ADE20K Zhou et al. (2017) 100-10 scenario.
A.2 IMPACT OF CLASS ORDER SHUFFLING ON PERFORMANCE
We examine the robustness of our method with respect to randomly shuffled class orders. We con- duct experiments on the ADE20K 100-10 scenario, utilizing the 10 randomly shuffled class orders proposed by Kim et al. (204)) for consistency. The PQ distributions are presented Figure 3 through boxplots We note that the results reported in Kim et al. (2024) affected by evaluation
14

---

Published as a conference paper at ICLR 2025
100-5 (11 tasks)
100-10 (6 tasks)
100-50 (2 tasks)
Method
I-100
101-150
al
1-100
10I-150
all
1-100
101-150
all
MiBt Cermelli al. PLOPT Doutllad
2020
20.5
4.3
15.1
27.7
7.1
20.8
33.7
10.5
26.0
PLOPtDoutJan el 2021
19.2
8.8
15.8
28.9
10.6
22.8
34,8
12.4
27.4
CoMFormer;
Cermeln et al. 2023
20.1
8.2
16.1
29.7
10.3
23.3
34.7
13.2
27.6
ECLIPSE: Kim et al. ADAPT (Ours)
2024
34.4
8.9
25.9
34.4
10.2
26.4
35.2
13.3
27.9
36.6
9.3
27.5
36.5
11.7
28.3
36.6
18.0
30.4
Table 8: Continual Panoptic Segmentation results (PQ) on ADE20K Zhou et al. (2017) under the disjoint setting. All approaches are based on the same framework Mask2Former Cheng et (2022)
with the ResNet-50 et al. (2016) backbone. & indicate results are taken from Kim et al. (0024).
bug, where performance was consistently averaged over the default class order regardless of shuf- fling. After correcting this issue, we observe that the results deviate from those originally reported in Kim et al. (2024). Notably, our method, ADAPT, exhibits strong resilience to various shuffled class orders, consistently outperforming alternative approaches.
A.3 CONTINUAL PANOPTIC SEGMENTATION THE DISJOINT SETTING
The pioncering work of Cermelli et al. 2020) introduced two distinct settings for continual learn- ing: disjoin/ and overlap. Since the overlap setting is generally considered more challenging and realistic, focused primarily it in main paper. In this section, present experimental sults for continual panoptic segmentation on ADE20K Zhou et al. (0017) under the disjoint setting. The results, summarized in Table 8 highlight the superior performance of ADAPT over existing methods in continual panoptic segmentation.
CoMFommr
ECL IISE
ADAPT (Ours)
Groud--nni
Figure 4: Qualitative visualization for continual panoptic segmentation under the ADE20K Zhou et al. (017) 100-10 scenario.
15

---

Published as a conference paper at ICLR 2025
A.4 QUALITATIVE VISUALIZATIONS
We provide qualitative visualizations in Fig. to to demonstrate the superior rigidity and plasticity of approach. Specifically, ADAPT effectively preserves base class knowledge, such cabi net and door (first column), while also generalizing well to novel classes, including fan (second column), tent (third column), trade name (last column), and trash can (last column).
A.5 MORE IMPLEMENTATION DETAILS
Following state-of-the-art methods Cermelli et al. (2023); Kim et al. (2024), we adopt the Mask2Former Cheng et al. (2022) model with an output stride of 4, using ResNet-50 He et al. (016) the backbone, unless otherwise stated. To ensure fairness, hyperparameters as our competitors Cermelli et (2023); Kim et al. (2024), with the exception of using a higher learning rate, which leads to faster convergence and slightly improved performance. The initial learning rate is set to 10 for all steps in the 100-n settings. In the 50-n settings, we use 2 10-4 for the incremental steps (f 0). while maintaining 10 during base training (f 0).
follow the same training
We train the network for 160k iterations during base training, and for 400 iterations per class in all subsequent steps. The batch size consistently set to 16 across all settings. We use the AdamW optimizer Loshchilov & Hutter (2018) with the same weight decay values as in in(nnnn et al. (2022). For all settings, we report Panoptic Quality (PQ) results on the standard validation set. Experiments are conducted using two NVIDIA RTX 6000 Ada GPUs on ADE20K and four on COCO.
16

---

Published as a conference paper at ICLR 2025
Algorithm Panoptic Inference with Mask2Former
Require: query.class_scores Tensor of shape quu--uueries num lum_cassse++]]
Require: query-masks: Tensor of shape [num-queries, height, width]
Require: num classes: Number of classes
Require:
threshold: Confidence threshold for selecting valid queries
Require:
threshold:: Minimum overlap threshold for Mon-Maximum Suppression (NMS)
Ensure: panoptic.seg: Final panoptic segmentation mask
Ensure: segmentssiinf oo Information the detected segments
1: Initialize panoptic_seg zeros (hight,, width)
2: Initialize segments info + [
3: Initialize current segment id 0
4:
5: scores, labels <- max(sofmaxxqqurryeaass scoress
6: query-masks + sigmoid (query qquery.maks mmakss)
7: valid (labels + background) & (scores > threshold)
8: valid_scores &- scores valid]
9:
query masks[valid]
10:
score_masks +- valid_scores x vulid__ueer..masks
inum_valid_queries height, width] [height, width]
12: class < argmax(score__maa) dim=0)
13:
14: for each query in valid queries do
15:
pred + scores[ mask baask_area-aa area (class id masks s)mmu
16:
17:
original aarea + (valid-query) masks|i] qusiii > 0.5)
18:
mask +- (classid imsssss i) & (valid_qqery) maalid_query_maskk mawey_maass] 0.5)
19:
if mask_area > O and original arrea > and mask.sum > 0 then
if mask_area, wtt.e/arrr//riiiiindee originallaree
mherlap-threshod then current segment id <- current. segment. id +
21:
panoptic.sev seg mask]] current wurrent.segment.nn segments in fo.apeed
23:
24:
25:
"id": current-segmentt.e "category.id": pledsclass
26:
27:
28:
29:
end if
30: end
end for
32:
return panoptic.seg, segments.ingge in
17


Reviews:
[{"review_id": "w7UwxGZclR", "reviewer": "Reviewer_diCs", "summary": "This paper addresses the challenges in continual panoptic segmentation (CPS). In order to tackle catastrophic forgetting, the authors propose a dual-decoder Mask2Former framework combined with attentive self-distillation (called ADAPT) to efficiently retain past knowledge and generalize to new classes.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The reviewer found the paper to be well-written and easy to understand.\n\n- The reviewer likes the use of a dual-decoder framework that allows for knowledge retention with minimal computational costs.\n\n- The paper also presents a solid self-distillation mechanism focuses on informative regions by down-weighting non-object areas, making knowledge retention more targeted. The reviewer finds the query-level fusion (QLF) strategy to eliminate the scale mismatch issues commonly found in probability-level fusion to be interesting.", "weaknesses": "- Although the reviewer likes the idea of freezing the encoder as it reduces computational load and helps retain base knowledge, it may restrict the model's ability to adapt fully to new classes over very long sequences of tasks, particularly as the diversity or complexity of new classes increases. The authors can comment on this.\n\n- On a similar note, while the authors report results with ResNet-50 backbone, it’s unclear to the reviewer how the method scales with larger models or higher-resolution images. The freezing and distillation mechanisms might still impose computational and memory limitations in more demanding setups.\n\n- The author's might benefit from observing the effect of their method on open-vocab evaluation settings. Further, it may be interesting to evaluate on standard settings as well. For example, on COCO dataset with the number of classes growing in the number of tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nq27XLNW5H", "reviewer": "Reviewer_c4nN", "summary": "The paper proposes a new method for continual panoptic segmentation. This task requires model to be able to adapt to new data and semantic classes, while maintaining the capabilities acquired from previous training stages. A key challenge in this task is catastrophic forgetting, which can result in significant performance degradation, especially for classes from earlier stages that are underrepresented in later stages. The method relies on mask transformer panoptic model and adresses the catastrophic forgetting by careful fine-tuning of only parts of the initial weights. Additionally, it enforces prediction consistency w.r.t. to earlier model instances while emphasizing the loss for informative queries. Finally, the predictions of the initial and the latest model are ensembled to maintain the balance between base and novel classes. The experiments are conducted on ADE20k panoptic dataset following continual learning setups from previous work.", "questions": "Recently, vision-language models such as CLIP caused significant improvements in open-vocabulary panoptic segmentation (e.g. [1])? Would such design solve some technical challenges in continual panoptic segmentation (e.g. adding new classes)? Perhaps this approach trained in a regular way could represent another baseline for CPS?\n\n[1] Yu, Q., He, J., Deng, X., Shen, X., & Chen, L. C. (2023). Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36, 32215-32234.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method achieves state-of-the-art performance in continual panoptic segmentation on ADE20k dataset.\n\nThe method is computationally more efficient than related work. Due to the freezing of the backbone and the pixel decoder, teachers inference is consisted of only the transformer decoder. \n\nThe ablation study reveals positive effects of the proposed contributions on the overall performance.", "weaknesses": "Paragraph describing query-level fusion should be better written. Implementation details or some equations might improve clarity of this part.\n\nPresentation of the ablation and validation experiments could be of higher quality. I dont understand the necessity of subsection 4.5. The first table referenced in text is actually table 5, which makes it a bit confusing. \n\nThe advantages regarding the computational efficiency are not supported with any experimental results or analysis. Some table comparing training time or FLOPs in a single training iteration with the literature would make this more convincing. Similar analysis would be beneficial for the test-time inference as well. Dual-decoder prediction fusion obviously causes some computational overhead, and this is not measured in any way.\n\nDual decoder prediction fusion heavily relies on the assumption that most of the data and classes were available in the initial phase. What if this is not the case? Is a setup where most of the data and classes become available in some intermediate step possible in real applications? Perhaps this should be discussed. \n\nThe technical novelty of the presented contributions is limited. Self-distillation and weight freezing have already been considered in continual panoptic segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WibeJixEyl", "reviewer": "Reviewer_Bxb3", "summary": "This paper aims to resolve the efficiency or scalability issues in continual panoptic segmentation methods. A dual-decoder framework is proposed that incorporates attentive self-distillation and prediction fusion to preserve prior knowledge while facilitating model generalization. The majority of pixel decoder's weights are fixed and shared between the teacher and student models. Thus a single forward pass for efficient knowledge distillation can be achieved. Moreover, an attentive self-distillation is introduced to distill useful knowledge from the old classes without distracting from non-object regions. Additionally, a query-level fusion is introduced. to seamlessly integrate the outputs. Experimental evaluation is conducted on ADE20K dataset.", "questions": "1. [Line 057-058] Is the \"raising computational costs\" only occur during training? If so, does the computational cost of inference remain same? [Line 060-063] Why continuously introducing additional learnable query features and embeddings for new task is not good? Why this strategy \"constrains the model's capacity to generalize to new tasks due to restricted plasticity\"? Overall, the reviewer believes that if there are no theoretical or numerical experimental evidence, it would be better to do not judge methods from other areas.\n2. [Line 138-139] Since there are many claims that \"most of these methods require separate forward passes for the teacher and student models, resulting in considerable computational overhead.\", please give clear comparison and experimental results to support.\n3. Why the experiments are only conducted on one benchmark? At least two datasets should be involved to validate the generalization ability of the proposed method.\n4. It is recommended to adjust the table size and position distribution.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Clear figure illustration for methods.\n2. Interesting idea for probability-level  fusion and query-level fusion.", "weaknesses": "Weaknesses: \n\n1. Lack of evidence for Computational Costs and Generalization\n\n2. Need for Experimental Comparison on Computational Overhead\n\n3. Limited Experimental Validation\n\n4. Need for Further Improvement in Table Presentation\n\nThese concerns collectively emphasize the need for additional theoretical or experimental evidence and broader validation of the proposed method's claims and generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oB9VIOQCho", "reviewer": "Reviewer_fZFv", "summary": "This paper introduces a novel approach for continual panoptic segmentation, addressing efficiency and scalability challenges faced by existing methods (e.g., CoMFormer’s computational limitations and ECLIPSE’s scalability issues). The authors propose an adaptation strategy that minimizes parameter updates by freezing the majority of model parameters, selectively fine-tuning cross-attention and feedforward layers in the transformer decoder. This is coupled with an attentive self-distillation mechanism to balance plasticity and rigidity effectively. Additionally, a dual-decoder prediction fusion strategy at the query level further enhances model performance. The proposed method, termed ADAPT, is empirically evaluated on the ADE20K dataset, showing improved performance in terms of both plasticity and rigidity over existing approaches.", "questions": "Please check the weaknesses.\n\n## Justification.\n\nOverall, this paper makes significant contributions to the field of continual panoptic segmentation, demonstrating state-of-the-art results in both plasticity and rigidity. However, several areas require additional refinement to meet the high standards expected at ICLR, including more rigorous analysis, qualitative insights, and additional experimental details.\n\n## Recommendation\n\nAlthough my initial recommendation is 5: marginally below the acceptance threshold, my real rating is the Borderline.\nMy final decision will depend on the authors' rebuttal and any revisions addressing the outlined weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## 1. Strong Motivation and Meaningful Contributions\n\n- The paper is well-motivated, addressing key limitations in current methodologies: (1) CoMFormer’s limited rigidity and computational inefficiency, and (2) ECLIPSE’s restricted plasticity and scalability challenges.\n- The proposed solution addresses these limitations effectively. Specifically, the selective fine-tuning of cross-attention and feedforward layers is a meaningful contribution that demonstrates the improved balance between model plasticity and rigidity.\n\n\n## 2. Impressive Experimental Results\n\n- ADAPT achieves state-of-the-art performance, significantly outperforming existing methods in continual panoptic segmentation tasks. \n- The experimental results demonstrate the effectiveness of the proposed solution, with a notable performance margin over alternative approaches.", "weaknesses": "# Weaknesses\n\n## 1. Limited Insight into Mechanism Behind Parameter Freezing (L.209)\n\n- While cross-attention layers and feedforward networks are activated, self-attention layers are frozen, as detailed in L.209. \n- However, the authors provide only empirical results (Table 2) without an in-depth analysis or hypothesis to support this choice. \n- A theoretical justification would strengthen the contribution.\n\n## 2. Lack of Qualitative Analysis\n\n- The paper would benefit from qualitative analysis to complement the quantitative results. \n- Such insights would enhance the reader’s understanding and provide explicit visual support for the model’s performance.\n\n## 3. Absence of Comparative Analysis on Training Efficiency\n\n- Given that efficiency is one of the claimed contributions, a comparative analysis on training efficiency, such as training iteration throughput or GPU memory usage, would strengthen the paper. \n- A comparison between ADAPT, CoMFormer, and ECLIPSE on these training efficiency metrics would provide a fuller view of ADAPT’s performance.\n\n## 4. Insufficient Details in Figure 2\n\n- Figure 2 lacks clarity, particularly regarding color representations (e.g., which color represents a query or a specific class, such as sky). \n- It is also unclear which instances are misclassified in L.315-23. \n- Section 3.4 does not clarify (despite having 8 queries in the query-level fusion,) why the probability distribution includes only 4 scores.\n- More detailed descriptions of the example base class set, new class set, and the query and color representations would aid comprehension.\n\n## 5. Ambiguity in Baseline Setting (Table 5)\n\n- Table 5 lacks clarity regarding the baseline setting. \n- The baseline results without any additional components appear identical to CoMFormer’s results. \n- Clarification on whether this is a coincidence or an intentional baseline setting using CoMFormer would help clarify the impact of each proposed component.\n\n## 6. Probability Inconsistency Analysis\n\n- It is unclear which activation function was used to obtain class probabilities. \n- Mask2Former and CoMFormer use softmax, whereas ECLIPSE utilizes sigmoid. \n- If softmax was used, probability inconsistency may arise due to its relative scoring.\n- But, If sigmoid was used, probability inconsistency may reduced.\n-  An analysis comparing softmax and sigmoid’s impact on class probability consistency would be beneficial.\n\n\n# Additional Comments\n\n## A. Experiments in the Disjoint Setting\n\n- While the overlap setting is standard in continual segmentation research, additional results under the disjoint setting would provide further insight and help fellow researchers contextualize the findings.\n\n\n## B. Expanded Experiments\n\n- Additional evaluations on (1) the COCO panoptic segmentation benchmark, and (2) experiments with shuffled class ordering, as seen in PLOP and ECLIPSE, would offer robustness analysis on ADAPT’s handling of class ordering, enhancing its utility for future research.\n\n\n## C. Clarification on Reported Performance Metrics\n\n- In Table 1, performance scores for FT, MiB, PLOP, and CoMFormer are cited from ECLIPSE, while ECLIPSE’s score is reproduced. \n- This may cause confusion, as it’s unclear why certain scores are excerpted versus reproduced. \n- Given ADAPT’s strong performance across metrics, it may be beneficial to rely on previously reported scores for consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)