Domain Shared and Specific Prompt Learning for Incremental Monocular Depth Estimation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 02:50:55

Abstract: Incremental monocular depth estimation aims to continuously learn from new domains while maintaining their performance on old
domains. The catastrophic forgetting problem is the key challenge when the model adapts the dynamic scene variations. Previous
methods usually address this forgetting problem by storing raw samples from the old domain, allowing the model to review the
knowledge of the old domain. However, due to the concerns of data privacy and security, our objective is to tackle the incremental
monocular depth estimation problem in more stringent scenarios without the need for replaying samples. In this paper, we attribute
the cross-domain catastrophic forgetting to the domain distribution shifts and continuous variations of depth space. To this end,
we propose Domain Shared and Specific Prompt Learning (DSSP) for incremental monocular depth estimation. In detail, to alleviate
the domain distribution shift, complementary domain prompt are designed to learn the domain-shared and domain-specific knowledge which are optimized by the inter-domain alignment and intra-domain orthogonal loss. To mitigate the depth space variations, we first introduce a pre-trained model to generate the domain-shared depth space. Then, we design $S^2$-Adapter that quantizes
depth space variations with scale&shift matrices and converts the domain-shared depth space to domain-specific depth space. Our
method achieves state-of-the-art performance under various scenarios such as different depth ranges, virtual and real, different
weather conditions, and the few-shot incremental learning setting
on 12 datasets. We will release the source codes and pre-trained
models.

Main paper: 
 introduction: 
 Monocular depth estimation aims to generate dense depth maps from a single RGB image, which is an important scene perception technique. In practice, new domains substantially emerge with environmental variations, such as weather and illumination conditions. Incremental monocular depth estimation intends to facilitate continuous learning and adaptation of the model in such dynamic environments while maintaining its depth estimation performance in the old domain.In incremental monocular depth estimation, the catastrophic forgetting problem is the key challenge that is caused by domain distribution shifts and depth space variations. 1) Each domain has a unique data distribution corresponding to the environmental factors, which leads to severe domain gaps among them. Consequently, when the model adapts to a new domain, the shift of feature distribution leads to the phenomenon of forgetting old knowledge. 2) Different domains usually have different depth ranges and spatial structures, so the depth space exhibits scale and shift variations across domains. When the model adapts to these variations on new domains, it may result in the forgetting of old knowledge.To address the aforementioned problems, some approaches [28, 29] attempt to reduce the domain gaps by employing multi-domain learning across as many domains as possible and alleviate the influence of the depth space variations by predicting relative depth maps. However, exhaustively covering all domains in the real world is impractical, and retraining the model from scratch consumes plenty of time and computational resources. Other methods [6, 11, 40] try to continuously fine-tune the model and ease the forgetting problem by maintaining a replay buffer involving samples from old domains which enables the model to review them while learning new knowledge. Such as LL-MonoDepth [11] stores 500 samples in each domain for reviewing old knowledge while employing multiple predictors to adapt the depth space variations on different domains. However, storing samples may not always be feasible, especially for scenarios where long-term storage of data is not permitted owing to privacy or data use legislation [34] . Therefore, we aim to address the incremental monocular depth estimation in a more strict protocol without the need for replay samples.Recently, pre-trained models embed knowledge into enormous parameters, and this empowers them with a better capability of mitigating forgetting. Researchers leverage prompt learning with pre-trained models to learn task-specific knowledge in the new domain. Inspired by this, we introduce learnable prompts to capture and retain domain-specific knowledge, which represents domain distribution information through several parameters, thereby saving storage costs and avoiding concerns of data privacy and security. Furthermore, prompts can also learn unified feature representations across domains to reduce domain gaps and enhance the generalization capability of the model.Therefore, we propose Domain Shared and Specific Prompt Learning (DSSP) for incremental monocular depth estimation, which designs complementary domain prompts and Scale&Shift Adapters (𝑆 2 -Adapter) to mitigate domain distribution shifts and depth space variations. Specifically, we first design domain-shared prompts to continually learn across all domains and prevent knowledge forgetting by domain shift through inter-domain alignment constraint, while we design domain-specific prompts solely to learn in the corresponding domain to enhance the adaptability of the model and retain knowledge in their parameters. Additionally, we introduce an intra-domain orthogonal loss to constrain these two types of prompts to focus on the unified inter-domain representation and individual domain distribution respectively. Both two types of prompts are maintained in the prompt bank. Therefore, we leverage the frozen pre-trained model with the complementary domain prompts to predict a relative depth map. Furthermore, to capture the depth space variations and convert the relative depth into metric depth, we devise the lightweight 𝑆 2 -Adapter which precisely learns the map between the relative depth and metric depth through quantifying depth space variations into scale&shift matrices. As illustrated in Figure 1 , in contrast to previous replay-based methods, DSSP generates complementary domain prompts by learning domain-shared prompts and domain-specific prompts, facilitating the learning of new knowledge while retaining previous knowledge without data privacy and security concerns.During the inference phase, we predict the domain of the test image according to the domain-specific prompt. Then, we generate complementary domain prompt and subsequently inject them into the pre-trained model to predict the relative depth map. Next, the 𝑆 2 -Adapter is selected by the domain identity to restore the depth space transformation and predict the final depth. Experiments on 12 datasets demonstrate our DSSP outperforms the SOTA methods.The main contribution of our work can be summarized as follows:• We propose domain shared and specific prompt learning to alleviate the catastrophic forgetting problem of incremental monocular depth estimation without data privacy and security concerns. • We devise a 𝑆 2 -Adapter which quantifies depth space variations into scale&shift matrices to capture the map between relative depth and metric depth and mitigate the influences of depth space variations.• Extensive experiments on multiple domain incremental sequences show our method achieves superior incremental monocular depth estimation performance. 
related work 2.1 monocular depth estimation: 
 As a technique for scene perception, monocular depth estimation aims to generate per-pixel depth maps for scenes based on a single RGB image. Previous learnable methods achieve satisfactory performance through supervised [1, 9, 12, 14, 15, 19, 21, 27] , selfsupervised [10, 20, 24, 39, 42, 51] and unsupervised learning [36, 49, 50, 52] on single domain. However, due to the lack of robustness and generalization, models usually exhibit poor performance in new domains when they are applied in the real world. Some works tend to collect a large scale of data samples across various domains and learn a domain-invariant model [29, 37, 43, 46, 47] . However, it is impossible to cover all domains in practical applications. Therefore, when a new domain emerges, the model has to combine the data of the new domain with all old domains and be retrained from scratch, which is time-consuming. Meanwhile, the large number of parameters in these pre-trained models makes training them from scratch expensive in computing resources. To address the above problem, researchers devote numerous efforts to exploring incremental monocular depth estimation methods, so that the model can continuously adapt to the new domain without forgetting the knowledge of the old domain. Recently, some researchers try to store a part of samples of old domains and maintain a replay buffer that allows the model to access when fine-tuning on the new domain. Although the model alleviates the catastrophic forgetting problem by reviewing these replay samples, it is not practical to store data for a long time due to data privacy and security concerns in many real-world scenarios.Incremental Learning is a typical setting to continuously train a single model on non-stationary data distributions. The major challenge is the catastrophic forgetting problem [26] , where adaptation to a new distribution usually results in a much-reduced ability to capture the old distribution. Inspired by the complementary learning systems (CLS) theory [18, 25] , researchers devote numerous efforts to facilitating the model to learn from continuous data streams, such as storing some training samples for each of the previous tasks and reusing them while learning a new task (replay methods) [2, 3, 5, 23, 30, 41, 48] ; imposing extra regularization instead of storing training samples (regularization-based methods), such as LwF [22] using a knowledge distillation loss on previous tasks and EWC [17] enforcing an additional loss term to alleviate changing on the weights important for previous tasks; fixing trained parameters on old tasks and employ extra network branches for training a new task (parameter isolation methods) [16, 31, 33, 44] . These methods are empirically demonstrated useful for image recognition [16, 17, 41] , while they mainly focus on the class-incremental problem. Directly transferring these methods to incremental monocular depth estimation may face two challenges, domain distribution shifts and depth space variations.  
mythology: 
 In this section, we first introduce the paradigm of incremental monocular depth estimation tasks. Then, we overview our pipeline during both the training and inference phases. Next, we describe the complementary domain prompt learning strategy. Furthermore, we detail the 𝑆 2 -Adapter. Finally, we elaborate on inferring the domain identity of the test images for the generation of complementary domain prompt and the selection of 𝑆 2 -Adapter.In this task, the data of different domains are incrementally obtained. We define a sequence D with 𝑆 domains,D = {D 1 , • • • , D 𝑆 }. In the 𝑠-th domain D 𝑠 = {(𝑥 𝑠 𝑖 , 𝑦 𝑠 𝑖 )} 𝑁 𝑠 𝑖=1, there are 𝑁 𝑠 images 𝑥 𝑠 𝑖 ∈ X 𝑠 and its corresponding depth map 𝑦 𝑠 𝑖 ∈ Y 𝑠 . Given an image 𝑥 from arbitrary domains, our goal is to train a single model 𝑓 𝜃 𝑠 : X 𝑠 → Y 𝑠 parameterized by 𝜃 𝑠 to predict the depth map ŷ denotes as ŷ = 𝑓 𝜃 (𝑥). Generally, the model 𝑓 𝜃 𝑠 is optimized by,L = L 𝑑 (𝑦, ŷ) + R (1)where L 𝑑 is the depth estimation loss to learn the knowledge on the new domain, and R is a regularization loss for preserving the knowledge from old domains. Our method tackles the more challenging incremental monocular depth estimation which data from previous domains are not be seen anymore.In incremental monocular estimation, the catastrophic forgetting problem is caused by continuous domain distribution shifts and depth space variations. To this end, we propose DSSP, which is composed of complementary domain prompt and 𝑆 2 -Adapter. As illustrated in Figure 2 , we maintain a prompt bank containing learnable prompts throughout all incremental stages. The complementary domain prompt, along with image embedding, is fed into the frozen pre-trained depth estimation model and then yields a relative depth map. We froze the pre-trained model to prevent it from the influenced by domain distribution variations. Subsequently, following adjustments in scale and shift by our 𝑆 2 -Adapter, the relative depth map is transformed into the final metric depth map. In the inference phase, we first infer the domain identity of the input image, thereby generating the complementary domain prompt and selecting the 𝑆 2 -Adapter based on the domain identity. Finally, based on the domain distribution knowledge within the prompt and the depth space rule within the 𝑆 2 -Adapter, we can continuously estimate the depth map accurately with less forgetting. During all incremental learning stages, only the complementary domain prompt and the 𝑆 2 -Adapter are trainable, which only cost less than 1M parameter per domain.To address the issue of forgetting in incremental monocular depth estimation, previous methods store raw samples from the old domain and input them alongside samples from the new domain into the model. These approaches allow the model to review knowledge from the old domain while learning from the new domain. However, in many real-world scenarios, data from the old domain may not be retained and accessed in the long term due to privacy and security concerns. Therefore, instead of directly storing raw samples of each domain, we introduce learnable prompts to capture and store domain knowledge. These learnable prompts can explore the individual features of each domain, enhancing the adaptability of pre-trained models. Meanwhile, these prompts can also mine inter-domain common knowledge through long-term learning to improve the cross-domain generalization of the model. Overall, the prompts can not only learn the characteristics of each domain but also model generic information across domains, thereby mitigating the catastrophic forgetting problem without data privacy and security concerns.Based on the aforementioned analysis, we devise complementary domain prompt learning, which consists of domain-shared prompt and domain-specific prompt to learn both domain-general knowledge and domain-specific knowledge for each domain respectively. Given a pre-trained model with 𝑀 transformer layers in which the embedding dimension of the hidden layer is 𝑑, we learn 𝑀 domainshared prompt P 𝑠ℎ𝑎𝑟𝑒𝑑 and domain-specific prompt P 𝑠𝑝𝑒𝑐𝑖 𝑓 𝑖𝑐 :EQUATIONEQUATIONwhere 𝑙 𝑝 is the length of prompts. Then, we generate the complementary domain promptP 𝑐𝑜𝑚𝑝 = {𝑝 𝑖 ∈ R 2 * 𝑙 𝑝 ×𝑑 |0 ≤ 𝑖 ≤ 𝑀 -1}by:EQUATIONwhere [•, •] denotes the concatenation along the length dimension. Next, we inject the complementary domain prompt P 𝑐𝑜𝑚𝑝 into the first transformer layer 𝐿 1 of the pre-trained model alongside the image embedding 𝑒 which is extracted by the input image 𝐼 :EQUATIONSubsequently, the rest complementary domain prompt is injected into the pre-trained model with each hidden embedding feature:EQUATIONwhere 𝑐𝑙𝑠 𝑖 is the class token, and ℎ 𝑖 ∈ R 𝑑 (1 ≤ 𝑖 ≤ 𝑀 -1) is the hidden embedding feature output from the layer 𝐿 𝑖 .During training, the domain-shared prompt continuously learns domain generalized knowledge across all domains. In order to prevent the domain-shared prompt from forgetting information about old domains, we introduce an inter-domain alignment constraint between the domain-shared prompt at current stages and the previous incremental stage 𝑃 pre :EQUATIONL IDA enhances the ability to learn unified cross-domain feature representations of domain-shared prompt by matching the distribution of them across domains, reducing knowledge loss caused by domain distribution shiftS, and enabling the model to better leverage knowledge from old domains for learning in new domains. Meanwhile, we introduce an intra-domain orthogonal constraint between the domain-shared prompt and domain-specific prompt to guide them to focus respectively on the learning of general feature representation and individual domain distribution:EQUATION)Supervision under L IDO constraint, domain-specific prompt can focus on learning the unique domain distribution information of each domain to facilitate the model in adapting to new domains and preserve this domain characteristic information incorporated into the prompt bank for subsequent inference. Finally, we can predict the relative depth map d 𝑟𝑒𝑙 by the frozen pre-trained model 𝑓 with the learnable complementary domain prompt P 𝑐𝑜𝑚𝑝 :EQUATION3.4 𝑆 2 -Adapter Furthermore, to convert the relative depth map into metric depth map, it is necessary to capture the depth space variations across domains. Here, we decouple the depth space variations into two factors: depth scale and depth shift. The depth scale factor controls the scaling relationship from the relative depth map to the metric depth map, while the depth shift factor indicates the shift between each pixel to the accurate depth value. These two factors are widely used in multi-domain learning to obtain a unified depth representation by metric depth d and stabilize the multi-domain learning process: EQUATIONHowever, as 𝑠 ( d) and 𝑡 ( d) are unavailable, a direct thought is to estimate them by learning a global scale factor 𝑤 and shift factor 𝑏:EQUATIONFurthermore, due to the potential inaccuracies in the relative depth maps predicted by pre-trained models, a global scale factor and shift factor may not be enough to capture the depth space variations. Therefore, we design a lightweight 𝑆 2 -Adapter which consists of a scale adapter and a shift adapter to learn pixel-wise scale matrix and shift matrix respectively. In practice, we implement each adapter by two convolutional layers with 1 × 1 kernel size, followed by the formulation to obtain the final metric depth map:EQUATIONEQUATIONEQUATIONwhere 𝑊 𝑖,𝑗 means the 𝑗-th weight of the 𝑖-th adapter. We employ 𝑅𝑒𝐿𝑈 activation function to ensure that 𝑾 𝑠𝑐𝑎𝑙𝑒 is positive, while 𝑾 𝑠ℎ𝑖 𝑓 𝑡 is not subject to this constraint. Finally, we can predict the metric depth map through the 𝑆 2 -Adapter:EQUATIONAnd due to the various depth scales in different domains, we use the Scale-Invariant Loss [1] to learn depth knowledge:EQUATIONwhere 𝑔 𝑖 = log d 𝑖 -log d𝑖 and d 𝑖 denotes the ground truth depth map, and 𝑇 denotes the number of pixels with valid depth values. We empirically set the 𝛼 and 𝜆 to 10 and 0.15 respectively. Finally, the overall loss L on each domain is:EQUATIONwhere the hyper-parameter 𝛽 and 𝛾 is empirically set to 1 and 10 respectively.In the inference phase, we devise complementary domain prompt generation to analyze the content of input images to select the corresponding domain-specific prompt and 𝑆 2 -adapter.In detail, we first extract image features 𝑓 𝑖𝑚𝑎𝑔𝑒 ∈ R ℎ×𝑤×𝑑 via a query function that is implemented by the frozen pre-trained model. Then we flatten 𝑓 𝑖𝑚𝑎𝑔𝑒 and all domain-specific prompt respectively to get 𝑓 ′ 𝑖𝑚𝑎𝑔𝑒 ∈ R ℎ * 𝑤 * 𝑑 and 𝑓 ′𝑖 𝑝𝑟𝑜𝑚𝑝𝑡 ∈ R 𝑁 * 𝑙 * 𝑑 to compute the inner product between these features and get the domain identity 𝑘 by:EQUATIONConsequently, based on this domain identity 𝑘, we can generate the complementary domain prompt and inject it into the model. Additionally, we can select the 𝑆 2 -Adapter based on the domain identity 𝑘. Thus, the corresponding complementary domain prompt and the 𝑆 2 -Adapter are accurately selected to guide the model for depth predictions. 
experiments 4.1 experimental setup: 
 Datasets. We evaluate our method on 12 benchmark datasets, including 2 indoor and 10 outdoor datasets with different characteristics. The details of them are given in Table 1 , and the data augmentation strategies are described in the supplementary material. Implementation details. We implement experiments on PyTorch and train with RTX 3090 GPU with the AdamW optimizer for 10 epochs in each domain. We use a batch size of 8 and a onecycle policy schedule for adjusting the learning rate with a max learning rate of 0.0000161. We use the dpt-hybrid-based model from work [29] as our backbone. We evaluate our method with 5 metrics: relative mean absolute error (AbsRel); Root Mean Squared Error (RMSE); threshold accuracy 𝛿 1.25 ; Average metrics and Forgetting metrics. The Average Metric represents the generalization capability of the model and is calculated by the mean value of the RMSE metric across all domains in the final incremental session. The Forgetting Metric assesses the ability of the model against forgetting and represents the difference between the RMSE metric of the first domain after all training stages and its value after the completion of the first stage of training. [38] Outdoor 0~80 85k/1k vKITTI_v2 [4] Synthetic 0~80 12k/0.6k Cityscapes [7] Outdoor 0~200 2.9k/0.5k CS_Foggy [32] Foggy 0~200 8.9k/1.5k CS_Rainy [13] Rainy 0~200 9.5k/1.1k DAOD [45] Outdoor 0~120 174k/7.7kD_Sunny [45] Sunny 0~120 0.4k/0.1k D_Cloudy [45] Cloudy 0~120 0.4k/0.1k D_Foggy [45] Foggy 0~120 0.4k/0.1k D_Rainy [45] Rainy 0~120 0.4k/0.1kIncremental domain sequences. We design 4 incremental domain sequences to comprehensively validate the performance of the model:(1) NYU_v2→ScanNet→KITTI. In this sequence, the depth range varies cross-domain (10m→6m→80m). Therefore, this sequence is primarily employed to assess the adapting capacity of models to depth space variations.(2) vKITTI_v2→KITTI→CityScapes. The primary focus of this sequence is to assess the model's capacity to adapt to image style variations, such as from virtual to real scenes. (3) CityScapes→CS_Foggy→CS_Rainy. In this sequence, the weather conditions in each domain are different. Therefore, training under this sequence can verify the capability of models to mitigate domain distribution shifts. (4) DAOD→D_Sunny→D_Cloudy→D_Foggy→D_Rainy.Due to the extreme imbalance of the sample numbers across domains in this sequence, this sequence can test the model's adaptability and resistance to forgetting in long-term fewshot scenarios.Comparisons methods. We compare DSSP against our baseline, the classical incremental learning method (EWC [17] ), and the SOTA incremental monocular depth estimation method (LL-MonoDepth [11] ) under the above four incremental domain sequences. To evaluate the performance of our baseline, we train it through continuous fine-tuning (FT) and joint-domain training (JDT). We implement the classical incremental learning method EWC [17] based on our backbone. To reproduce the results of the LL-MonoDepth, we retain 500 replay samples for it in each domain, consistent with the original paper. Notably, due to the insufficient number of training samples under other weather conditions in DAOD dataset, we randomly retain 10% of the training samples for replaying in these domains.Comparisons under different depth ranges and image styles.As illustrated in Table 2 , DSSP achieves superior performance under all incremental domain sequences. Specifically, in the first sequence, compared to the SOTA methods, DSSP exhibits a significant improvement in the Average metric, demonstrating that our model effectively mitigates the influence of the depth space variations through 𝑆 2 -Adapters. In the second sequence, benefiting from our complementary domain prompt, DSSP is capable of learning unified feature representations across domains while also acquiring individual domain distributions for each domain. Therefore DSSP can maintain superior performance in both Average and Forgetting metrics. In the third sequence, none of the methods exhibited any occurrence of forgetting (i.e., forgetting metrics are less than 0). This indicates that under this scenario, the domain shift induced by weather variations is not as much as in the former two sequences, thereby enabling satisfactory performance through direct fine-tuning (FT). Figure 4 illustrates a comparison between our DSSP and other methods across all domains and shows the final results. It can be observed that nearly all methods produce satisfactory depth map predictions in the last domain (KITTI). Notably, benefiting from complementary domain prompt, which alleviates the effects of domain distribution shifts, DSSP maintains extraordinary performance in all domains. Meanwhile, owing to the inclination of this incremental sequence to validate the model's adaptability across varying depth ranges, our model, facilitated by 𝑆 2 -Adapter, exhibits the capacity to adapt depth space variations. For instance, within the first two indoor datasets, our model exhibits a relatively accurate measure of the depth of distant objects. The qualitative results under the other three additional incremental settings are provided in the supplementary material.Comparisons under the long-term few-shot scenario. Here, we conduct experiments on long-term few-shot scenarios, where the model is required to adapt 5 domains with different weathers. And the training samples of new domains are only 400. Figure 3 describes the performance curve of different methods during training. As one can observe, DSSP is more stable and achieves better performance in each session. As illustrated in Table 3 , DSSP achieves outstanding performance compared with other methods. Compared with EWC, DSSP obtains an improvement of 2.569 in terms of average RMSE over all domains. We also notice that DSSP is slightly inferior to EWC in terms of forgetting rate. Because EWC unfrozes the large-scale pre-trained model with additional regularization. Such a paradigm consumes plenty of time and computational resources and may destroy the pre-trained knowledge. Compared with the SOTA incremental depth estimation method LL-MonoDepth, DSSP obtains significant improvements in terms of Average and Forgetting metrics. The superior performance can be attributed to two 𝑆 2 -Adapter. Then, we access the performance of our DSSP without the 𝑆 2 -Adapter. As shown in Table 4 (w/o 𝑆 2 -Adapter), the absence of the 𝑆 2 -Adapter leads to the lack of capability of adapting the depth space variations thereby producing the severe forgetting problem.Regularization Loss. We further validate the effects of the constraints imposed on domain-shared prompt and domain-specific prompt. As demonstrated in Table 4 (w/o L IDO , w/o L IDA and w/o all R ), the absence of either constraint leads to a decrease in the model's generalization ability and resilience to forgetting. Specifically, the lack of inter-domain alignment constraint for domainshared prompt hinders the learning of unified feature representations across domains, thereby failing to ease domain gaps, ultimately resulting in severe forgetting phenomena. Similarly, the absence of intra-domain orthogonality constraint for both types of prompts induces knowledge interference during learning, preventing domain-specific prompt from acquiring individual domain distribution information, thereby impeding the model's adaptation to new domains and weakening its generalization capability.The length of the Prompt. In Figure 5 , we validate the effect of prompt length on "NYU-v2 → ScanNet → KITTI" task. As one can observe, the best Average is achieved when the prompt length is 300. The best Forgetting is achieved when the prompt length is 250. For considerations of storage and efficiency, we adopt a default prompt length of 100.Then we study the prediction accuracy of domain identity by our method. As shown in Figure 6 , our method achieves good performance under the "NYU_v2, ScanNet, KITTI" sequences which demonstrates that our domain-specific prompt can learn the individual domain distribution information. Meanwhile, due to the small number of samples in the DAOD incremental sequences, our method can accurately predict the majority of domains to which the samples belong. Due to varying degrees of inter-domain gaps, different incremental domain orders may result in diverse results. Consequently, we evaluate the performance of our method across all sequences on three datasets: NYU_v2, ScanNet, and KITTI. The performance of our method under different sequences on other datasets and detailed results of metrics are presented in the supplementary material. As shown in Figure 7 , due to the substantial disparity in depth range between the KITTI dataset and other indoor datasets, training a model solely on KITTI and subsequently on other datasets leads to significant performance degradation (a)(f). However, when the model has been pre-trained on any indoor dataset, its performance on the KITTI dataset remains stable or even improves (b)(d). 
conclusion: 
 In this paper, we develop the DSSP for incremental monocular estimation without data privacy and security concerns, which designs complementary domain prompt and 𝑆 2 -Adapter to mitigate domain distribution shifts and depth space variations. Specifically, we first generate the complementary domain prompt by learning domain shared and specific prompt which are supervised by the inter-domain alignment and intra-domain orthogonal constraint to facilitate the model to predict a unified depth representation. Then, we design a lightweight 𝑆 2 -Adapter that quantizes the depth variations into scaling and shifting, enhancing the ability of the model to capture the mapping from unified depth representations to metric depth space and eventually predict the metric depth map. Extensive experiments on 12 public datasets under four different incremental scenarios demonstrate the superiority of DSSP. We design four incremental domain sequences to provide a comprehensive benchmark for incremental monocular depth estimation. In the future, we will validate the forgetting issue and the transfer ability of our DSSP on larger depth estimation pre-trained models.  
 
 Appendix:  


Decision: This paper presented a method for incremental monocular depth estimation, aiming to address the problem of catastrophic forgetting when adapting to new domains. Specifically, a domain shared and specific prompt learning (DSSP) approach was proposed to achieve the goal. The paper received comments from three reviewers. The main strengths of this work are: the extensive experimental analysis, and the writing. On the other hand, there are some concerns raised by the reviewers, including: insufficient details about the methodology and some statements, the motivation of particular structure designs, missing explanation of some weak performance, and concerns about the domain-shared and domain-specific design justification.

Overall, most concerns were addressed during the rebuttal, though there are still some remain. Considering the remaining concerns are relatively minor and can be addressed with a minor revision, the AC would like to recommend Accept, but strongly suggest the authors include the supporting evidence for those raised concerns in the final version by carefully reviewing again the comments from all the reviewers.

Reviews: review: paper_summary: paper_summary: The main content of this paper is the research on incremental monocular depth estimation, which aims to solve the catastrophic forgetting problem that models face when adapting to new domains. The paper puts forward two main contributions:
Domain Shared and Specific Prompt Learning (DSSP) : A new incremental monadic depth estimation method is proposed, which mitigated the problems of domain distribution shifts and depth space variations by designing complementary domain prompts. This approach solves data privacy and security concerns by eliminating the need to store original samples of old domains
S2-Adapter: An adapter is designed to quantify the variation of depth space, which converts domain shared depth space into domain specific depth space by scaling and shifting matrix. This adapter enhances the ability of the model to capture mapping from a unified depth representation to a metric depth space, thereby improving the predictive accuracy of the final metric depth map.
And the paper has completed enough experiments to prove its effectiveness.
strength_weakness: strength_weakness: The experimental Settings of the paper are sufficient. Extensive experiments were carried out on 12 datasets covering different scenarios, such as different depth ranges, virtual and real environments, and different weather conditions, which demonstrated the robustness and applicability of the method. Four incremental domain sequences are designed to comprehensively verify the performance of the model. This comprehensive evaluation strategy helps to demonstrate the effectiveness of the method in different situations.
In addition, the technical details described in the article are sufficient, and detailed mathematical formulas and algorithm descriptions help readers understand the working principle of the method and can reproduce the experimental results.
limitations: limitations: 1, the paper uses Prompt Bank to implement the existing domain review. However, in the acquisition of existing knowledge, common methods include Latent Representation-based RAG, etc., which can further explain the motivation of using this structure.
2. In Section 3 MYTHOLOGY, the specific details about the network are not explained, including the size of input and output, the number of prompt, dimension and so on.
paper_summary: paper_summary: The paper proposes a method called Domain Shared and Specific Prompt Learning (DSSP) for incremental monocular depth estimation, aiming to continuously learn from new domains while maintaining performance on old domains. The method addresses the catastrophic forgetting problem caused by domain distribution shifts and depth space variations. DSSP uses complementary domain prompts and a Scale&Shift Adapter (S2-Adapter) to mitigate these issues without requiring replay samples, thus preserving data privacy and security. The paper demonstrates that DSSP achieves state-of-the-art performance across various scenarios, including different depth ranges, virtual and real environments, weather conditions, and few-shot incremental learning on 12 datasets.
strength_weakness: strength_weakness: - The paper includes extensive experiments across multiple datasets and scenarios, demonstrating the effectiveness of DSSP. 

- The paper is well-organized and clearly written, with detailed descriptions of the methodology, experimental setup, and results. Figures and tables are used effectively to illustrate the concepts and findings.
limitations: limitations: - The paper mentioned that the DSSP can reduce storage costs. However, this assertion requires further validation with results.
- You should present the results of progressively adding each component of the method and displaying the associated results at each step until the final method is achieved. This differs from the ablation study presented in Table 4.
- Why can the pre-trained model with M transformer layers obtain M domain-shared / -specific prompts? You should provide more details.
- The proposed method involves multiple components, including domain-shared and specific prompts, S2-Adapter, and various loss functions, which might increase the complexity of both training and testing. Further discussion regarding training and inference times would be beneficial.
paper_summary: paper_summary: This paper proposes to solve the incremental monocular depth estimation problem via prompt learning  without replaying samples. The proposed Domain Shared and Specific Prompt Learning (DSSP) adopts complementary domain prompts to mitigate domain distribution shifts and depth space variations. The prompt learning helps to learn task-specific knowledge in the new domain. The proposed method is verified under various scenarios under different conditions.
strength_weakness: strength_weakness: 1. The writing of the paper is clear and easy to follow, with few issues (detailed in the weaknesses section).

2. This paper is theoretically well-motivated and the results are impressive compared to other attempts.
limitations: limitations: 1. What is the prompt bank size and how the prompt bank is updated and will the memory mechanism brings in large computation cost? 

2. The data of different domains are incrementally obtained. Specifically, when obtaining data from different domains, is it considered to assign different weights? Will such a design affect the performance in certain domains?

3. What is the purpose to use metric depth and will it brings more befenit than relative depth? 

4. How much improvement can Shared and Specific Prompts each bring, and is there any related ablation study?

5. The method in this paper significantly improves performance on some datasets, such as Cityscapes, but shows weaker improvement on datasets like NYU and KITTI. How can this phenomenon be explained, and does it suggest that the method might overfit?