Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images

Authors: bo  yuan
danpei  zhao
zhuoran  liu
wentao  li
tian 2024 li
  con
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-04 13:52:09

Abstract: Continual learning (CL) breaks off the one-way training manner and enables a model to adapt to new data, semantics and tasks continuously. However, current CL methods mainly focus on single tasks. Besides, CL models are plagued by catastrophic forgetting and semantic drift since the lack of old data, which often occurs in remote-sensing interpretation due to the intricate fine-grained semantics. In this paper, we propose Continual Panoptic Perception (CPP), a unified continual learning model that leverages multi-task joint learning covering pixel-level classification, instance-level segmentation and image-level perception for universal interpretation in remote sensing images. Concretely, we propose a collaborative cross-modal encoder (CCE) to extract the input image features, which supports pixel classification and caption generation synchronously. To inherit the knowledge from the old model without exemplar memory, we propose a task-interactive knowledge distillation (TKD) method, which leverages cross-modal optimization and task-asymmetric pseudo-labeling (TPL) to alleviate catastrophic forgetting. Furthermore, we also propose a joint optimization mechanism to achieve end-to-end multi-modal panoptic perception. Experimental results on the fine-grained panoptic perception dataset validate the effectiveness of the proposed model, and also prove that joint optimization can boost sub-task CL efficiency with over 13% relative improvement on PQ.

Main paper: 
 introduction: 
 Continual learning (CL) enables a model to continuously acquire knowledge in a sequential manner. Currently, many CL methods are designed for specific single tasks, which cover natural language processing [27] , computer vision [28, 58, 65] , etc. However, the complex and practical applications urge the model to have the capacity for CL in multi-task learning (MTL). For example, the continuously incremental data in remote sensing usually requires the model to have the ability to continually interpretation on new data, semantics and tasks.Over the past decade, CL has been intensively concerned since it can break through the typical one-off training schema and enable the model to evolve with continuous data. However, CL encounters two main challenges including catastrophic forgetting and semantic drift [56, 65] . These problems occur when the parameter updates result in the loss of previously learned knowledge, leading to prediction chaos and model degradation. Traditionally, the popular fully-supervised methods conduct a complete re-training on the incremental data that may result in an analogous issue akin to Alzheimer's disease, where the model tends to forget its previously learned knowledge due to parameter changes [28] . However, current CL approaches face challenges in the trade-off between preserving old knowledge and learning new ones. A range of researches [3, 4, 9] propose to retrospect known knowledge including sample selection as exemplar memory [43, 50] . However, these replay-based methods usually bring extra memory costs and arouse privacy concerns. Another kind that does not rely on old data utilizes transfer learning manners like knowledge distillation to inherit the capability of the old model [46, 53] . In remote-sensing images (RSIs), the complex semantic relations and fine-grained semantic classes make CL extremely challenging. And there is still a lack of exploration on multi-task and multi-modal CL.As seen in Fig. 1 (a), typical single-task CL approaches only support separate training on different tasks, which limits the CL capacity on complex practical scenarios. In this paper, we propose a CL architecture for panoptic perception [64] namely Continual Panoramic Perception (CPP). As illustrated in Fig. 1 (b), CPP enables multi-task CL within a single model that shares the same image encoder for multi-modal interpretation. Particularly, CPP consists of a collaborative cross-modal encoder (CCE), a task-interactive knowledge distillation (TKD) module and a task-asymmetric pseudolabeling (TPL) mechanism. Concretely, the CCE module extracts image features, which are projected to mask embeddings and text embeddings synchronously for the corresponding decoder branch. To alleviate catastrophic forgetting, the proposed TKD module utilizes multi-modal features to address semantic drift to boost distillation efficiency. While TPL integrates the pseudo label to improve the label confidence for CL steps. Fig. 1(c ) illustrates the proposed CPP training pattern that synchronously proceeds segmentation and captioning tasks within an integrated model. After each CL step, new semantics are learned while achieving compatibility with the old knowledge.The main contributions of this paper are summarized as follows:• We propose continual panoptic perception, a novel architecture supporting multi-task continual learning covering pixel-level classification, instance-level segmentation and image-level captioning. • We present a task-interactive knowledge distillation method, utilizing cross-task knowledge inheritance and task-asymmetric pseudo-labeling to reconcile stability and plasticity. • The proposed method achieves an end-to-end continual learning manner on remote-sensing panoptic perception dataset, and also proves the feasibility of joint optimization across multi-modal CL tasks. 
related work 2.1 continual image segmentation: 
 Continual learning originates from as early as [34] and has been explored in various fields, including computer vision [65] , natural language processing [60] , remote-sensing [33] , etc. With respect to image segmentation, the main challenges are catastrophic forgetting and semantic drift, which arise from the absence of old data and parameter updates [8, 22] . According to whether relying on old data [56] , the CL methods can be divided into replay-based methods [7, 25, 29, 50, 58, 63] and exemplar-free methods [2, 6, 17, 35, 39, 44, 46, 53, 59, 65] . The former involves storing a portion of past training data or features as exemplar memory, which brings extra memory costs and raises privacy concerns. The latter usually utilizes knowledge distillation to inherit the capability of the old model. Considering reducing reliance on annotations, few-shot CL methods are also deeply explored in [24, 40, 47, 66] . However, current CL methods are mainly designed for single tasks, the multi-task continual learning has not been deeply studied.Image captioning [20] describes an image with words or sentences to meet human-like semantic understanding. Depending on the network architecture, there are main three branches including heterogeneous-architecture-based, attention-based, and pre-trainingbased methods. Encoder-decoder architectures normally include a CNN-based visual encoder, an RNN-based language decoder, and a cross-modal attention block [19, 54] . Attention-based methods [1, 23] introduce semantic attention mechanisms to model contextual information among areas in an image. While some recent methods witness large performance boosting via large-scale pre-training [10, 21] and promoting [52] . Continual image captioning allows generating captions over a series of new tasks coming with new semantics [15, 37] . Considering the rich semantics in RSIs, the investigation of continual image caption is a valuable and promising task.Multi-task learning (MTL) aims to jointly learn multiple related prediction tasks with shared information across tasks [45, 61, 62] . Existing MTL studies mainly focus on feature-based and parameterbased methods. The former assumes that different tasks share a common feature representation. In deep learning, it learns a common feature representation for multiple tasks by sharing feature layers in a similar architecture [16, 48] . While parameters-based methods leverage parameter sharing [14, 41] across tasks, which mainly include hard sharing [5] and soft sharing [55] . The main challenges of MTL are the negative optimization and the seesaw phenomenon. Although MTL has been explored in incremental learning on classification [26] , object detection [31] and image segmentation [49, 57] , which still remains single-task interpretation. However, the multi-objective continual MTL still has a long way ahead. In this paper, we are committed to proving the feasibility of multi-task continual panoptic perception.1 t M - t M mask GT caption GT ⊙ refine ˆt m 1 t m - t m loss 1 t - m Q 1 t - t Q t m Q t t Q m Q t Q ❉ Frozen Learnable 
method: 
 As seen in Fig. 2 , the proposed network consists of a Collaborative Cross-modal Encoder (CCE) for multi-modal feature extraction, a Task-interactive Knowledge Distillation (TKD) for model inheritance and a Task-asymmetric Pseudo-labeling (TPL) approach for retrieving old knowledge, respectively. In the proposed architecture, we aim to utilize a shared encoder for multi-task including dense prediction and image captioning. The former can be seen as a segmentation task and the latter is a global semantic understanding task. Thus we utilize a unified Transformer architecture to cope with cross-modal feature extraction.As illustrated in Fig. 2 , considering an image 𝑥 ∈ R 𝐶 ×𝐻 ×𝑊 , we firstly apply a model-agnostic feature extractor to extract image features. Concretely, the image encoder 𝐸 produces a downsampled feature F ∈ R 𝑁 ×𝐻 ′ ×𝑊 ′ . The image feature F derived from CCE can be utilized for both mask prediction and caption generation. Concretely, the encoded image feature from CCE is:EQUATIONwhere 𝐸 (•) is the encoding process. Θ(•) is a linear mapping.Whereafter the decoder is decoupled to a Transformer for mask prediction, one-to-one mappings between the mask predictions and the ground truth are generated. And a Transformer for word prediction. For pixel-level predication, the F is firstly generated to 𝑁 mask embeddings via a Transformer decoder with 𝑁 query input. We regard both instance and semantic segmentation as mask classification problems and handle them with a Transformer-based architecture. Concretely, assuming N learnable queries 𝑄 ∈ R 𝐶 𝑞 ×𝑁 , F is used as keys (K) and value (V). A standard Transformer decoder is used to update 𝑄. To make full use of shared features, the query is projected into mask embeddings Q m ∈ R 𝑁 ×𝐶 𝑒 in the mask generation branch, and the text embedding Q t for captioning. Specifically for the captioning task, a standard Transformer decoder consisting of multiple decoder layers is used for sentence prediction, each having a masked self-attention mechanism, a crossattention mechanism, and a feed-forward neural network. The selfattention mechanism can tackle the long-range context, which is beneficial to both segmentation and captioning tasks.To inherit the capacity from the old model in the absence of old data, we propose a task-interactive knowledge distillation method under the circumstances of only the previous model 𝑀 𝑡 -1 and the incremental data D 𝑡 can be accessed. For fine-grained CL tasks, it faces misclassification challenges while the old classes and future classes are mixed in bg class during CL steps. There are proofs showing the latent domain gap between the cross-modal data could lead to a learning ambiguity during knowledge distillation [42] . We propose a cross-task contrastive distillation method based on taskinteractive guidance. Firstly, using Q 𝑡 -1 m and Q 𝑡 m indicate the mask embeddings, while Q 𝑡 -1 t and Q 𝑡 t for text embeddings. Concretely, the TKD module consists of Intermediate Contrastive Distillation (ICD) and Cross-guided Instance Distillation (CID). Intermediate Contrastive Distillation. Since the segmentation task and the captioning task are both derived from the same features F produced by CCE, the ICD is conducted on both tasks. To alleviate the classifier confusion caused by semantic drift, we propose a contrastive distillation across Q m and Q t . Inspired by [65] , we argue that using embedding from the old model is more confident to reduce prediction error since catastrophic forgetting. Concretely, the optimization objective contains global output logits and classwise contrastive learning between old and new classes. Thus the constraint for ICD is:EQUATIONwhere 𝑑 (•) is a similarity measure to constraint the representation consistency between 𝑀 𝑡 -1 and 𝑀 𝑡 . 𝑓 𝑡 -1 𝑎 , 𝑓 𝑡 𝑝 and 𝑓 𝑡 𝑛 represent the anchor, positive and negative embeddings, respectively. 𝑓 𝑡 -1 𝑘 indicates the embedding belonging to k-th class from 𝑀 𝑡 -1 . 𝑓 𝑡 𝑖 and 𝑓 𝑡 𝑗 indicate the corresponding embeddings from 𝑀 𝑡 . Cross-guided Instance Distillation. In the RSIs, the large amount of instances and complex distribution increase the probability of semantic fusion. Thus the instance segmentation requires finegrained distillation. Directly distilling the feature map without considering foreground regions would result in training with a large amount of background information, leading to insufficient learning of important foreground regions and poor distillation performance. Thus we propose a distillation method emphasizing the interaction classification and localization, respectively. Since Chen et al. [12] propose a cross-modal instance distillation, we adapt it to multitask CL scenarios. The quality score 𝑞 𝑟 serves as an indicator to guide the student on which teacher's predictions should be paid more weight.𝑞 𝑟 = (𝑐 𝑟 ) 𝛾 × 𝐼𝑜𝑈 (𝑚 𝑡 -1 𝑟 , m𝑡 𝑟 ) (1-𝛾 ) (4)Aerial view of a runway featuring five aircraft, including a TU-160 and four TU-95s.：FA-181 t C - t C ：TU-95 new C old C t step ⊙ Pixel-wise fusion Text concatenation bg t step GT t MThe image shows an aerial view of a runway with three aircraft parked on t he tarmac, i ncl uding three TU-95s ：FA-18 where 𝛾 indicates the weight of classification and segmentation. 𝑐 𝑟 indicates the predictive classification results of instance 𝑟 . 𝑚 𝑡 -1 and m𝑡 are the segmentation results of 𝑀 𝑡 -1 and 𝑀 𝑡 , respectively. It is noted that m𝑡 𝑟 is the student output from the teacher decoder to restrain the model inheritance [51] . For fine-grained tasks, the recognition branch should be assigned a higher weight to alleviate misclassification due to large intra-class variance. Thus a constraint between 𝑀 𝑡 -1 and 𝑀 𝑡 is proposed to optimize the instance segmentation task via a guided instance distillation loss.1 t C - t C ：TU-95The CID objective is defined as:EQUATIONwhere 𝑦 𝑟 indicates the predicted classification result of instance 𝑟 . Thus the TKD objective is the combination of ICD and CID:EQUATIONSince only the incremental classes are labeled, we propose an exemplar-free pseudo-labeling method with asymmetric task reliance, i.e., more emphasis on confident predictions. As seen in Fig. 3 , we observe the captioning task focuses more on the global context. While the segmentation tasks tend to cause pixel misclassification within a single intact target area. Using Y 𝑡 -1 𝑐𝑎𝑝 indicates captioning output from the 𝑀 𝑡 -1 . Thus the segmentation annotation ȳ𝑡 𝑖 at 𝑡 step for pixel 𝑖 is defined as:EQUATIONwhere ỹ𝑡 -1 𝑖 is the pseudo label for pixel 𝑖 generated from 𝑀 𝑡 -1 , 𝑝 𝑖 is the predictive probability of pixel 𝑖 at 𝑡 step. 𝑥 𝑖 is the 𝑖-th pixel in image 𝑥. 𝑦 𝑡 𝑖 is the incremental annotation for pixel 𝑖 at 𝑡 step. 𝑐 𝑏 indicates the unknown background class. The asymmetry manifests itself by threshold or more rely on captioning predictions. Since the learned and future classes are mixed in 𝑐 𝑏 at each CL step, the relative scoring of softmax could lead to significant semantic chaos and catastrophic forgetting during CL training steps. Instead, we take independent Sigmoid with binary cross-entropy loss to cope with the variable class number during CL steps and alleviate the interference from variable background semantics.Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images MM '24, October 28-November 1, 2024, Melbourne, VIC, Australia.For the captioning task, every sentence starts with "START" and ends with "END" keywords. The label at 𝑡 step is concatenated with the pseudo label before "END". Thus the captioning label for 𝑡 step is:EQUATION)where ỹ𝑡 -1 𝑐𝑎𝑝 indicates the predicted captioning from 𝑀 𝑡 -1 and 𝑦 𝑡 𝑐𝑎𝑝 is the readily available label at 𝑡 step. ⊕ indicates the concatenation operation.The EQUATIONwhere L 𝑚 is formed by focal loss [30] and dice loss [36] :EQUATIONwhere 𝜂 1 and 𝜂 2 are the weighting component. 𝑐 𝑔𝑡 𝑖 is the label for class 𝑐. And the captioning loss is formed by a cross-entropy loss:EQUATIONwhere 𝑝 𝑐𝑎𝑝 is the word prediction probability from the caption module and ȳ𝑐𝑎𝑝 represents the annotation. Thus the integrated objective is defined as:EQUATION)where 𝜆 is the weight of captioning tasks. Specifically, the distillation loss is only applied at CL steps. 
experiments 4.1 datasets and protocols: 
 Datasets: FineGrip [64] is a multi-task remote-sensing dataset that includes 2649 images, with 12054 fine-grained instance segmentation masks belonging to 20 foreground classes, 7599 background semantic masks covering 5 classes, and 13245 fine-grained sentence description annotations. The sample number in the training set is much smaller than that in the validation set, which provides challenging but practical scenes for CL under limited data conditions. Note that all stuff classes are trained at the initial step since they are covered in most samples. We evaluate our model on 20-5 (2 steps), 15-5 (3 steps) and 15-2 (6 steps), respectively. Protocols: There are mainly two different CL settings: disjoint and overlapped. In both settings, only the current classes 𝐶 𝑡 are labeled and others are set as background. In the former, images at 𝑡 step only contain 𝐶 0:𝑡 -1 ∪ 𝐶 𝑡 ∪ 𝐶 𝑏𝑔 . While the latter contains 𝐶 0:𝑡 -1 ∪ 𝐶 𝑡 ∪ 𝐶 𝑡 +1:𝑇 ∪ 𝐶 𝑏𝑔 , which is more realistic and challenging.In this study, we focus on overlapped setting. We also report two baselines for reference, i.e., fine-tuning on 𝐶 𝑡 , and training on all classes offline. The former is the lower bound and the latter can be regarded as the upper bound of this task.We use MaskFormer [13] with ResNet-50 [18] as the base encoder to extract image features. For all experiments, the initial learning rate is 0.01 and decayed by a poly policy. The implementation is based on PyTorch 1.10 with CUDA 12.3 and all experiments are conducted on a workstation with four NVIDIA RTX 4090 GPUs. For all CL steps, the training epoch is set to 90. The hyper-parameters are set as 𝜆 = 2.0 according to the analysis in Sec. 4.4.1, 𝜂 1 = 20.0 and 𝜂 2 = 1.0. We compute the panoptic quality (PQ), segmentation quality (SQ) and recognition quality (RQ) to measure the segmentation efficiency.While the Bilingual Evaluation Understudy (BLEU) [38] is used to evaluate the generated text. The implementation is based on MMDetection [11] .To comprehensively evaluate the CPP performance on old and new classes, we compute the segmentation performance and captioning performance at the initial step and the final step, respectively. The proposed model is tested on two aspects following [56] : multistep with few-class (MSFC) and few-step with multi-class (FSMC).Particularly, FSMC emphasizes the ability to learn new knowledge (plasticity) since many new classes are adapted in a single step.In contrast, MSFC underlines the ability of anti-forgetting on old knowledge (stability) because many CL steps are conducted. The proposed model is evaluated from multi-modal CL performance and cross-task CL correlation, which are the key problems beyond the pioneer single-task CL approaches.As seen in Table 1 , we first evaluate the PQ for old, new and all classes after all CL steps, respectively. Compared to naive fine-tuning approaches, freezing encoder after the initial step can help alleviate catastrophic forgetting. The proposed CPP method achieves superior anti-forgetting on old classes and plasticity on new classes simultaneously. The captioning performance is evaluated from two aspects: The performance at the initial step and after all CL steps. The former is the initial learned result with limited partial classes. While the latter indicates the matching accuracy of the predicted words after all CL steps. On the one hand, the captioning capacity improves since the new semantics are learned to meet with an intact understanding of the image. On the other hand, the BLEU scores declined after CL steps due to catastrophic forgetting. The results in Table 1 also prove that the joint multi-task optimization in CL tasks should boost sub-task capacity. Compared to the segmentation method [13] in the proposed CL setting, the proposed CPP achieves 3.33%, 3.68% and 2.76% PQ improvements on all classes on 20-5, 15-5 and 15-2 tasks, respectively. While compared to fine-tuning method with freezing enoder, CPP achieves overall improvements across multi-modal tasks. Concretely, for 15-2 task, the long-step learning brings severe catastrophic forgetting. As a comparison, CPP maintains relative higher performance on both segmentation task and captioning task, which demonstrates the effectiveness of the proposed multi-modal continual learning architecture.Table 1 : Quantitative performance on FineGrip dataset. We evaluate the segmentation task with PQ (%). 𝐶 𝑜 , 𝐶 𝑛 and 𝐶 𝑎 are the performance for old classes, new classes and all classes after all CL steps, respectively. The captioning performance is evaluated by BLEU scores (beamsize=5), which are reported before (B 𝑏 ) and after (B 𝑎 ) all CL steps. † indicates we re-implement the method to CL tasks. FE indicates freezing encoder manner. The aerial image shows four aircrafts lined up on the runway, consisting of four TU-160s.The aerial image shows six aircrafts lined up on the runway, consisting of four TU-160s and two Tu-95s.The aerial image shows a group of airplanes, with a total of 6 aircraft, consisting of four TU-160 bombers and two TU-95 bombers.Captioning Before Captioning After Captioning GT nine FA-18s Image Incre. Annotation Pred. Before Pred. AfterThe aerial image shows an airfield with nine F-16s.The aerial image shows an airfield with eight FA-18s.The image features an aerial view of a military airfield, showcasing a row of nine FA-18 jets parked in a line.Captioning Before Captioning After Captioning GT ImageSegmentation GT Pred. Before Pred. AfterThe aerial image shows four aircrafts lined up on the runway, consisting of four TU-160s.The aerial image shows six aircrafts lined up on the runway, consisting of four TU-160s and two Tu-95s.The aerial image shows a group of airplanes, with a total of 6 aircraft, consisting of four TU-160 bombers and two TU-95 bombers.Captioning Before Captioning After Captioning GTSegmentation GT Pred. Before Pred. AfterThe aerial image shows an airfield with nine F-16s.The aerial image shows an airfield with eight FA-18s.The image features an aerial view of a military airfield, showcasing a row of nine FA-18 jets parked in a line.Captioning Before Captioning After Captioning GT Fig. 4 depicts the CPP results on 15-5 and 10-5 tasks, respectively. It is noted from two aspects. On the one hand, the unknown semantics can be ignored at the previous steps, as seen in the first row in Fig. 4 before CL training. On the other hand, semantic chaos occurs when semantic instances are misclassified. As illustrated in the second row in Fig. 4 , the foreground instances are located but misclassified to false class labels. While after the CL training steps by CPP, the semantic chaos is alleviated by distinguishing old and new classes. Meanwhile the captioning results are also replenished by taking into account both new and old semantics. To reveal the contribution of each module in the proposed method, we respectively disclose the corresponding modules as seen in Table 3 . The proposed TKD combines segmentation flow and captioning flow to perform KD across different tasks. To disclose the effectiveness of TKD, the ICD and CID are separately and jointly verified, respectively. Compared to the fine-tuning method, ICD achieves significant improvement, proving the anti-forgetting effectiveness of leveraging intermediate features constraints between 𝑀 𝑡 -1 and 𝑀 𝑡 . While CID emphasizes the consistency of output distribution at CL steps to align the ability from the old model. On the other hand, the synergy of ICD and CID proves the homogeneity in the proposed CPP for multi-modal CL task. For example, the PQ on 𝐶 𝑎 achieves 2.17% higher than ICD only and 4.78% higher than CID only on 20-5 task. While compared to the fixed-threshold-based pseudo-labeling method, the proposed TPL utilizes cross-task reliance to improve the confidence of the generated pseudo labels and achieves higher performance on all three CL scenarios. Note that when not adopting TPL, we apply a confidence-based pseudo-labeling with a fixed threshold with Γ = 0.7.As defined in Eqn. 12, CPP utilizes a multi-task learning approach within a unified architecture. The weights assigned to multi-modal tasks significantly affect the efficiency of CL. As shown in Table 4 , the performance of CPP achieves the highest PQ since the captioning task is weighted as 𝜆 = 2.0, demonstrating the cross-modal boosting in CPP tasks. backbones including ResNet-50, ResNet-101 and Swin-T [32] are explored. Quantitatively, the model with ResNet-101 achieves higher PQ, SQ and RQ than that with ResNet-50, which proves a stronger backbone can achieve higher anti-forgetting performance and compatibility on new classes. It also shows that Transformer-based models achieve better recognition performance since the RQ score is significantly higher than two CNN-based models. However, CNNbased models achieve higher segmentation performance as the SQ 4.4.4 Impact of pseudo-labeling. Pseudo-labeling is an effective way to alleviate catastrophic forgetting due to the lack of old data and annotation. We compare the proposed TPL with the typical single-task confidence-based method with a fixed threshold, in which we set Γ = 0.5 as a convention and Γ = 0.7 following [17, 65] . As seen in Table 5 , the proposed TPL achieves 1.56% PQ improvement on 𝐶 𝑜 and 0.92% superiority on 𝐶 𝑎 to the fixed confidence threshold setting (Γ = 0.7), which indicates the cross-task interactive can improve the pseudo-labeling effectiveness. However, the decline in 𝐶 𝑛 shows the conflict between retaining the old knowledge and bias on the new knowledge.To reveal the robustness to class learning orders of the proposed method, we perform experiments on 15-5 task with five different class orders including the ascending order and four random orders on thing classes as follows. In Fine-Grip [64] dataset, 𝐶 21-25 indicates the background stuff classes, which serves as the partial base classes in the CPP experiments. The results shown in Table 6 indicate the PQ of 𝐶 1:10&21-25 and 𝐶 11:20 on five different orders. Note that 𝐶 21-25 are background stuff classes that are learned at the initial step. The average PQ and standard variance are reported. It reveals that the class incremental orders have an evident impact on CPP performance. For example, the average PQ on 𝐶 1:10&21:25 varies sharply. It proves the critical challenge of catastrophic forgetting in multi-task CL. On the other hand, the performance in all classes after all CL steps proves the learning stability of CPP.  
conclusion and discussion: 
 In this paper, we propose a continual panoptic perception (CPP) method that enables multi-task continual learning for multi-modal interpretation in remote-sensing images. Aiming to achieve CPP within a single intact architecture, the proposed architecture contains a shared encoder for multi-modal tasks, with cross-task distillation for solid knowledge inheritance. It is also proved that crossmodal task-interactive learning achieves mutual enhancement on sub-tasks. Experiments on the fine-grained panoptic perception dataset validate the effectiveness of the proposed method. However, CPP encounters pendent challenges including intricate cross-modal feature relevance, unbalanced knowledge proportion and semantic chaos caused by catastrophic forgetting. Our future work will focus on improving joint optimization efficiency and developing more robust and interpretable CPP approaches. 
 
 Appendix:  


Decision: This paper proposes the Continual Panoptic Perception (CPP) model, a novel continual learning framework aimed at enhancing multi-modal incremental interpretation of remote sensing images. 

After rebuttal and discussions, this paper receives two positive ratings and one negative rating. After carefully  reviewing the paper and all the reviewers' comments, The AC agrees to accept the paper and strongly recommends incorporating the content from the rebuttal into the final version.

Reviews: review: paper_summary: paper_summary: This paper proposed a unified continual learning model called CPP, that leverages multi-task joint learning covering pixel-level classification, instance-level segmentation and image-level perception for universal interpretation in remote sensing images. Concretely, they proposed a collaborative cross-modal encoder (CCE) to extract the input image features, which supports pixel classification and caption generation synchronously. To inherit the knowledge from the old model without exemplar memory, they proposed a task-interactive knowledge distillation (TKD) method, which leverages cross-modal optimization and task-asymmetric pseudo-labeling (TPL) to alleviate catastrophic forgetting. Furthermore, they also proposed a joint optimization mechanism to achieve end-to-end multi-modal panoptic perception.
strength_weakness: strength_weakness: The Figures in the manuscript are exquisite.
limitations: limitations: 1.	The manuscript could benefit from a more comprehensive comparative analysis with state-of-the-art approaches. Specifically, it lacks a detailed discussion on how the proposed CPP model performs relative to existing models in handling catastrophic forgetting and semantic drift, critical challenges in continual learning.
2.	In lines 739-740 of the manuscript, the authors should elaborate further on why the threshold is fixed at 0.7 when TPL is not used.
3.     In Table 1, the segmentation task with SQ and RQ also needs to be evaluated and added.
paper_summary: paper_summary: This paper proposes a new multi-task learning architecture called CPP, which can solve the limitation of single task in typical CL. It can implement pixel-level classification, instance-level segmentation and image-level perception tasks in one model, and perform multimodal interpretation of remote sensing images. CPP also solves the common catastrophic forgetting and semantic drift problems in CL to a certain extent by using CCE, TKD and TPL, especially when old data is not available. The paper also conducts sufficient experiments on the proposed innovations.
strength_weakness: strength_weakness: 1. Unlike the typical single-task CL method, the CPP framework proposed in this paper can implement multiple tasks in one model, covering pixel-level classification, instance-level segmentation and image-level perception at the same time.
2. The proposed TKD method effectively reduces the problem of catastrophic forgetting and improves the model's ability to retain previously learned knowledge by using cross-modal features for distillation.
3. The method proposed in the paper realizes end-to-end learning in a multi-task setting and simplifies the training process.
5. The experiments in the paper are very sufficient, and a relatively comprehensive ablation experiment is conducted on each parameter.
limitations: limitations: 1. The data set selected in the paper is relatively simple, and the model's ability to be verified on multiple extensive data sets is not verified
2. The paper does not discuss the computational complexity of the method. The simultaneous processing of multiple tasks and multiple modalities may lead to high computational and memory requirements
3. The legend in Fig. 5 blocks part of the main figure, and even in conjunction with the content of Section 4.4.2 of the main text, this figure is not easy to understand.
paper_summary: paper_summary: The paper introduces the Continual Panoptic Perception (CPP) model, aiming to address challenges in continual learning. CPP comprises a Collaborative Cross-modal Encoder (CCE) for pixel-level classification and caption generation, Task-interactive Knowledge Distillation (TKD) to mitigate forgetting, and Task-asymmetric Pseudo-labeling (TPL) for knowledge retrieval. Experimental results validate the model's effectiveness across various protocols.
strength_weakness: strength_weakness: 1. The proposed CPP model integrates a shared Encoder and modules for multi-task performance, including classification, segmentation, and captioning, showcasing innovative and logical design.
2. The efficacy of CCE and TKD is convincingly demonstrated in experiments, highlighting substantial performance gains with CPP.
3. This paper is well-written and easy to follow, which also shows its accessibility and readability.
limitations: limitations: 1. I wonder which similarity measure the authors using in Equantion (3), is it cosine similarity or others? Clarification is needed on this similarity measure and its impact on results, as well as potential alternatives.
2. In the proposed Cross-guided Instance Distillation (CID) module, the authors utilize IoU to compute the quality score. Could you confirm whether this refers to the mean IoU (mIoU), which averages across different classes, or the class-agnostic IoU calculation?
3. Equation (4) references "c_i" without prior explanation, requiring inclusion of its definition for clarity.
4. The "PQ" used in the Abstract are encouraged to be replaced by the full name "Panoptic Quality" introduced in the main text, and here may cause some doubt without any explanation or the full name. Simultaneously, PQ, SQ and RQ are suggested to be illustrated and elaborated.