MTSNet: Joint Feature Adaptation and Enhancement for Text-Guided Multi-view Martian Terrain Segmentation

Authors: yang  fang
xuefeng  rao
xinbo  gao
weisheng  li
zijian  min
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 05:54:43

Abstract: Martian terrain segmentation plays a crucial role in autonomous navigation and safe driving of Mars rovers as well as global analysis of Martian geological landforms. However, most deep learning-based segmentation models cannot effectively handle the challenges of highly unstructured and unbalanced terrain distribution on the Martian surface, thus leading to inadequate adaptability and generalization ability. In this paper, we propose a novel multi-view Martian Terrain Segmentation framework (MTSNet) by developing an efficient Martian Terrain text-Guided Segment Anything Model (MTG-SAM) and combining it with a tailored Local Terrain Feature Enhancement Network (LTEN) to capture intricate terrain details. Specifically, the proposed MTG-SAM is equipped with a Terrain Context attention Adapter Module (TCAM) to efficiently and effectively unleashing the model adaptability and transferability on Mars-specific terrain distribution. 
Then, a Local Terrain Feature Enhancement Network (LTEN) is designated to compensate for the limitations of MTG-SAM in capturing the fine-grained local terrain features of Mars surface. Afterwards, a simple yet efficient Gated Fusion Module (GFM) is introduced to dynamically merge the global contextual features from MTG-SAM encoder and the local refined features from LTEN module for comprehensive terrain feature learning. Moreover, the proposed MTSNet enables terrain-specific text as prompts resolving the efficiency issue of existing methods that require costly annotation of bounding boxes or foreground points. Experimental results on AI4Mars and ConeQuest datasets demonstrate that our proposed MTSNet can effectively learns the unique Martian terrain feature distribution and achieves state-of-the-art performance on multi-view terrain segmentation from both the perspectives of the Mars rover and the satellite remote sensing. Code is available at https://github.com/raoxuefeng/mtsnet.

Main paper: 
 introduction: 
 Since the 1960s, humans have conducted multiple Mars exploration missions. In recent years, with the advancement of deep space exploration technology, Mars exploration has once again become a popular topic. As relevant exploration missions progress, the increasing Mars data has greatly promoted the application of deep learning in Mars exploration [3, 12, 31, 33] . In Mars exploration, Martian terrain semantic segmentation plays a crucial role as the foundation for autonomous navigation and safe driving of Mars rovers [13] , and can also assist satellites in precisely analyzing the global geological landforms of Mars, providing support and decision-making basis for Mars exploration programs.Currently, semantic segmentation technology has made significant progress in fields such as autonomous driving [4, 7] , humancomputer interaction [37] , and medical image analysis [11, 20] . However, these methods will encounter unprecedented challenges when applied to the extremely unstructured environments like the Martian surface [30] . The characteristics of Martian landforms are highly unstructured, lacking obvious structural features with regular geometric shapes or texture patterns on the Martian surface [10] . These factors make it infeasible to directly apply existing semantic segmentation models trained on common datasets to the task of learning Martian terrain features. Moreover, the images collected by Mars rover and orbiting satellite [27] at different locations and perspectives have distinct shapes and large differences in imaging scale, further increasing the difficulty of accurate terrain semantic segmentation with current semantic segmentation methods.In recent years, the emergence of foundation vision models has attracted widespread attention [5, 45] . These models, based on deep learning architectures and trained on large-scale and diverse image datasets, have acquired powerful feature extraction and zero-shot generalization capabilities. This means that even for unknown scenarios, after appropriate adjustments or transfer learning, they can still adapt to the semantic knowledge of new scenarios [38] . The recently emerging Segment Anything Model (SAM) [19] , as a foundation model for prompt-based segmentation tasks, can segment objects of interest given a semantic prompt in the form of bounding boxes, points, or masks [38] . Several recent works have proposed strategies to reduce the computation costs of SAM. FastSAM [42] develops a CNN-based architecture to segment all objects in an image for efficiency improvement. MobileSAM [44] presents a decoupled distillation for obtaining a lightweight image encoder of SAM. EfficientSAM [48] leverages masked image pretraining, which learns to reconstruct features from SAM image encoder for effective visual representation learning. Although SAM has achieved excellent segmentation performance in general scenarios, due to the significant difference in data distribution between the source domain and the target domain, its performance still falls into a local optimum when applied to specific distribution data of the Martian surface environment [2, 5] . Besides, SAM and most of its variants are usually only capable of using prompts in the form of points, bounding boxes, etc., for semantic segmentation, overlooking the great potential of text prompts, which can greatly reduce annotation costs and speed up the training process.To address the above issues, we propose MTSNet framework shown in Figure 1 . While retaining the SAM model and inspired by various Adapter approaches [8, 16, 35] , we present a novel Terrain Context Attention Adapter Module (TCAM) to enhance the model's adaptability and generalization capability for Martian images through fine-tuning strategy. Meanwhile, we construct a Local Terrain Feature Enhancement Network (LTEN), which captures local details in Martian terrain through stacks of efficient convolution operators and multi-scale attentions, compensating for the global contextual terrain features from MTG-SAM encoder and thus generating refiner segmentation results. To better fuse the contextual features from MTG-SAM and the local features from LTEN, we design a Gated Fusion Module (GFM) to dynamically fuse them through a gating mechanism to realize adaptive and learnable feature fusion process. Finally, we introduce a text prompt encoder enabling the model to perform efficient terrain segmentation by simple terrain text prompts, such as soil, sand, and etc., resolving the difficulty of using bounding boxes or foreground points as prompts in the complex and unstructured Martian environment. In summary, our contributions are as follows:(1) To the best of our knowledge, our proposed MTSNet is the first to present and address the multi-view Martian Terrain Segmentation task with only text prompts and report superior performance on two representative benchmarks with significant flexibility and efficiency.(2) We propose an efficient MTG-SAM encoder equipped with a lightweight Terrain Contextual Attention Adapter Module (TCAM), which is able to fine-tune the SAM model with extremely few adapter parameters, allowing it to quickly and adequately adapt to the target domain data. (3) We design a lightweight terrain fine-grained feature learning network with stacks of convolution operators and multiscale attention blocks, which can well compensate for the limitation of the MTG-SAM encoder in terrain detail capture. (4) For better feature fusion, we designed a simple yet efficient gated fusion module (GFM) to dynamically fuse contextual and fine-grained features through an adaptive and learnable gating mechanism, which exhibits superior effectiveness. 
related work 2.1 martian terrain segmentation: 
 Martian terrain segmentation is a special and important subset of semantic segmentation, with the goal of segmenting different terrains from diverse Martian images. Whether it is the analysis of Martian landforms and terrain by satellites or tasks such as autonomous obstacle avoidance, path planning, and terrain traversability judgment for Mars rovers [13] , all are profoundly influenced by Martian terrain segmentation. In the research of Martian terrain segmentation, Liu et al. [24] proposed a Hybrid Attention Semantic Segmentation (HASS) network that combines global intra-class and local inter-class pixel attention mechanisms. Swan et al. [40] collected a Martian terrain segmentation dataset called AI4Mars and evaluated its performance using DeepLabV3 [1] . Zhang et al. [46] proposed a semi-supervised learning dataset called S5Mars dedicated to Mars images and introduced a semi-supervised learning framework for Martian image semantic segmentation. However, current methods for Martian terrain segmentation generally target processing Martian images from a specific viewpoint and cannot be applied simultaneously to semantic segmentation of terrain from multiple viewpoints, such as those of Mars rovers or satellites. Instead, our proposed MTSNet aims to fully exploits the powerful generalization ability of visual foundation model and the flexibility and efficiency of text prompt-based fine-tuning mechanism, enabling the model to simultaneously model the diversity of Mars terrain from different perspectives within a unified framework, with excellent multi-modal learning capabilities.Recently, the Segment Anything Model (SAM) [19] has attracted widespread attention as a foundation model for prompt-based image segmentation tasks. It can segment images based on flexible and diverse prompts, such as bounding boxes, masks, and points. However, SAM's performance is unsatisfactory for some domainsspecific data, such as medical images or images of Mars surface. This can be attributed to the fact that the data used to train SAM is primarily sourced from general images on the earth, and the imaging mechanism or structural distribution of these source data are very different from the target domain data [2, 5, 29] . Nonetheless, some works attempt to break through this difficulty in transfer learning, and have successfully applied SAM to domains-specific segmentation tasks by setting tailored learnable modules and fine-tuning mechanisms [2, 21, 25] . Ma et al. [25] proposed MedSAM, which only trains the SAM decoder while freezing the encoder parameters. Chen et al. [2] proposed SAM-Adapter, which incorporates domain-specific information and visual prompts into the segmentation network using a simple yet effective adapter technique. Hu et al. [21] proposed AutoSAM that keeps the encoder frozen but adds an independent CNN-based trainable prediction head. However, these methods rely on bounding boxes or foreground points as prompts, requiring costly annotations on images, which limits their application to tasks where data annotation is scarce. Considering the scarcity and re-annotation cost of Mars terrain dataset, we present a novel extended SAM model, referred to as MTG-SAM, that can accept text as prompt input, thus only the terrain text of interest need to be provided, and the model can predict the corresponding terrain segmentation mask, which is crucial for the safe landing and autonomous driving of the Mars rover. 
overview: 
 As shown in Figure 1 , our proposed multi-view Martian Terrain Segmentation framework (MTSNet) consist of Martian Terrain text-Guided Segment Anything Model (MTG-SAM) encoder equipped with Terrain Contextual attention Adapter Module (TCAM), a Local Terrain Feature Enhancement Network (LTEN), a Gated Fusion Module (GFM), a text prompt encoder, and a mask decoder by [18] . As illustrated, given an original Martian Terrain image and a text prompt describing the terrain of interest as input, MTSNet can predict the corresponding terrain segmentation mask.In original SAM, the image encoder possesses the majority of model parameters, and full model training would incur a high computational cost. Therefore, to integrate Martian terrain knowledge into it at a lower cost, we adopt the Adapter technique [16, 35] . Specifically, during the training process, we froze all parameters of the original image encoder and inserted a lightweight trainable Terrain Context Attention Adapter Module (TCAM) into each Transformer block. The TCAM comprises merely 7.64% of the trainable parameters of the original image encoder and consists of channel-spatial attention to make the image encoder adapt to the domain knowledge of Martian terrains, as shown in Figure 1 . In the TCAM, there is first a channel-spatial attention module that learns terrain features from the channel and spatial dimensions.In the channel dimension, we separately use global max and global average pooling to compress the input feature 𝑋 with shape 𝐶 × 𝐻 × 𝑊 into shape 𝐶 × 1 × 1, then pass them through a shared MLP, respectively. The two pooled channel features are then added together, and channel-wise weights are obtained via a sigmoid function and multiplied with the input features. The channel attention process is shown as𝑌 𝑐 = 𝛼 𝑐 ⊙ 𝑋 𝛼 𝑐 = 𝜎 𝑀𝐿𝑃 𝑃 𝑎𝑣𝑔 (𝑋 ) + 𝑀𝐿𝑃 (𝑃 𝑚𝑎𝑥 (𝑋 )) (1)where 𝑋 represents the original input features, 𝑌 𝑐 represents the output of channel attention, 𝛼 𝑐 denotes the channel attention scores, 𝜎 represents the sigmoid function, 𝑃 𝑎𝑣𝑔 and 𝑃 𝑚𝑎𝑥 denote the global average pooling and global max pooling, respectively.In the spatial dimension, as shown in Eq. 2, we similarly use two pooling operations to compress the 𝑌 𝑐 maps into 1 × 𝐻 × 𝑊 , concatenate them along the channel dimension, and recover the channel dimension through convolution. We then use a sigmoid function 𝜎 to obtain spatial location weights.𝑌 𝑠 = 𝛼 𝑠 ⊙ 𝑌 𝑐 𝛼 𝑠 = 𝜎 𝑓 𝑃 𝑎𝑣𝑔 (𝑌 𝑐 ) ; 𝑃 𝑚𝑎𝑥 (𝑌 𝑐 ) (2)where 𝑌 𝑠 denotes the output of spatial attention, 𝛼 𝑠 denotes the spatial attention scores, and 𝑓 denotes the convolution operation.After the channel-spatial attention module, we further employ convolutional layers to downsample the spatial resolution of the feature maps by a factor of two, and then use transposed convolution to restore the resolution while maintaining the same number of channels as the input, and finally add it to the input feature 𝑋 , which can be described as followsEQUATIONwhere 𝑌 represents the output of TCAM, 𝑓 𝑇 represents the transposed convolution. The Martian surface contains rich geomorphological features, which exhibit unique geometric and textural characteristics. These local features play a crucial role in accurate terrain recognition and contribute to generating more refined segmentation results. However, global context encoder may fail to fully capture these subtle differences. Therefore, we propose a Local Terrain Feature Enhancement Network (LTEN) to enhance the learning of local refinements. Tis composed of the stacks of multiple Local Terrain Enhancement (LTE) blocks. The structure of each LTE block is shown in Figure 2 . First, we utilize 3x3 convolutional layers to extract image features while maintaining the image size. After the convolution operation, we use average pooling for downsampling with a scaling factor of 2. Finally, to fuse important terrain features from different spatial locations and scales, we apply an Efficient Multi-scale Attention module (EMA) [26] to improve the model's ability to recognize and distinguish various complex terrain structures. Conventional feature fusion methods such as Concatenation, Addition, and Multiplication directly fuse features without considering the relative importance of the fused features, potentially including redundant information [22] . To address this issue, when fusing local terrain features and global context features, we propose a Gated Fusion Module (GFM) as shown in Figure 3 . It can filter out redundant information for each channel and location during fusion process, enabling dynamic and adaptive feature fusion. The formula for GFM is as followsEQUATIONwhere 𝐹 𝐿 represents the LTEN features obtained from LTEN, 𝐹 𝑆 represents the global context features obtained from the MTG-SAM encoder, and 𝐺 denotes the gating signal. It is obtained by linearly transforming the LTEN features and MTG-SAM global features with weights 𝑊 1 and 𝑊 2 , respectively, and then applying a sigmoid function.As mentioned above, the current applications of SAM use bounding boxes and points as segmentation prompts. However, in Martian terrain semantic segmentation, the complex structures of Martian terrains make it difficult to effectively distinguish different terrains, and providing prompt annotations requires considerable domain expertise. Not only is this costly, it is also not feasible in practice.We believe text prompts can be utilized for terrain segmentation. To this end, we propose a text prompt encoder on top of the original prompt encoder, enabling the model to perform segmentation using text prompts. As shown in Figure 1 , in the text prompt encoder, we first encode the text using text encoder of pretrained CLIP [34] model, which remains frozen during our training process. Since CLIP was not trained on terrain terminology, the generated text embedding have representation limitations. To address this issue, we introduce a trainable text affine layer (TAL) that takes the text embedding from CLIP text encoder as input and transforms them into terrain terminology relevant representations. The relevant operation is as followsEQUATIONwhere 𝐸 represents the text embedding from CLIP, 𝑦 denotes the transformed representations, and 𝑊 𝑇 and 𝑏 are the weight and bias of the text affine layer, respectively. By appending the terrain terminology relevant text embedding to the MTG-SAM prompt encoder, the prompt encoder is capable of using text prompts for Martian terrain segmentation 
experiments 4.1 datasets and evaluation metrics: 
 We conduct experiments on two Mars datasets to demonstrate the effectiveness and superiority of our proposed method, namely the AI4Mars dataset [40] and the ConeQuest dataset [32] . The AI4Mars dataset [40] is the first publicly available largescale annotated Mars dataset, with images acquired from the Navigation Cameras (NAVCAM) and Mast Cameras (Mastcam) onboard the Curiosity rover. It consists of 16,386 annotated images, among which the training set includes 16,064 annotated images, and the test set contains 322 annotated images. The pixels in each image are classified into four terrain classes: soil, bedrock, sand and big rock. In our study, we focus solely on terrain segmentation. Therefore, we compute the Dice scores of each class and the Intersection over Union (IoU) Metric for comparison, which are defined as followsEQUATIONwhere 𝐺 represents the ground-truth segmentation mask and 𝑆 represents the predicted segmentation mask. The ConeQuest dataset [32] is the first publicly available expert-annotated dataset specifically designed for the task of recognizing conical landforms on the Martian surface. This dataset encompasses high-resolution remote sensing images covering three prominent regions on Mars-Isidis Planitia (IP), Acidalia Planitia (AP), and Hypanis (HP)-with over 13,000 samples, each accompanied by detailed metadata. There are two benchmark tasks on ConeQuest dataset: Spatial Generalization and Cone-size Generalization, designated to evaluate a model's performance in unseen regions and on cones of different sizes, respectively [32] . In our study, we follow the evaluation metrics adopted by the Spatial Generalization benchmark task (BM-1) proposed in the ConeQuest dataset [32] , including Mask Intersection over Union (Mask IoU or MI), Pixel Intersection over Union (Pixel IoU or PI), Pixel Accuracy (PA), Pixel Precision (PP), and Pixel Recall (PR), to assess the consistency and accuracy of our MTSNet in conical landforms segmentation in the perspective of satellite remote sensing, and we compare our proposed MTSNet with the benchmarked methods.In our experiments, we initialize the image encoder with a pretrained SAM ViT-base model from [18] . We perform data augmentation on each training image, including random rotation up to 10 degrees and random saturation and brightness transformations. For the AI4Mars dataset, we randomly select 2,400 images from the 16,064 training images as the validation set, while the remaining images are used for training. For the ConeQuest dataset, we follow the official split for the training, validation, and test sets. Across all datasets, we resize the images to 512x512 resolution during training, using a batch size of 8, and train on an RTX 3090 GPU with approximately 14GB of memory required. During training, we employ the AdamW optimizer and fix the initial learning rate at 1e-4.For the AI4Mars dataset, the experimental results are shown in Table 1 , where we compute the average Dice scores and IoU scores across the four classes in the test dataset. We compare our method with classical convolutional neural network-based segmentation methods (UNet [36] , UNext [43] , DeepLabV3 [1] ) and Transformerbased methods (TransUNet [9] , MedT [41] ). Additionally, We also use original SAM with text prompt without any training (called SAM-ZS), and retrain SAM-Adapter [2] and AdaptiveSAM [29] on two datasets for fair comparison.As shown in Table 1 , general semantic segmentation methods perform poorly on Mars terrain segmentation. Among the convolutional neural network-based methods, UNet performs moderately on the "bedrock" and "soil" classes, but excellently on the "big rock" class with a Dice score of 98.45, though its overall performance is mediocre. While UNext shows some improvement on "bedrock" and "soil", it severely fails on the "big rock" class. The performance of DeepLabV3 is between UNet and UNext. As for the Transformerbased methods, MedT and TransUNet adopt Transformer encoders, requiring more data to generate good embeddings. Due to the small scale and class imbalance of the AI4Mars dataset, their performance is unsatisfactory. Moreover, we can see that in the zero-shot case, i.e., SAM-ZS, it is inapplicable to the Mars terrain segmentation task, while SAM-Adapter and AdaptiveSAM, which employ the Adapter based fine-tuning technique, exhibit significant performance improvements, outperforming conventional segmentation methods.Even using only TCAM, our method can achieve the highest scores in most classes compared to other methods, particularly excelling in recognizing "sand" and "big rock". When equipped with LTEN module, the performance is significantly boosted, reaching an average IoU score of 77.37%, increased by 5.71% and an average Dice score of 80.36%, increased by 5.68%, respectively.Figure 4 illustrates the visual comparison of our method with SAM-Adapter and AdaptiveSAM. It can be observed that the results of our method are closest to the ground truth annotations, with occasional partial prediction errors. In contrast, the results of SAM-Adapter and AdaptiveSAM exhibit over-prediction or underprediction of terrain features, as well as blurred boundaries in terrain prediction. Using only TCAM in our method, the aforementioned issues are significantly improved. With the introduction of LTEN, it can be seen that the predictions become even more aligned with the ground truth. In addition, MTSNet runs at about 16 frames per second in our experimental environment, which can basically meet the real-time requirements. Although the CNN-based methods have lower computational complexity and faster inference speed, its performance is too weak compared to MTSNet and cannot solve the Martian terrain segmentation task well. Compared with other ViT-based models such as MedT, MTSNet achieves significant advantages in performance and time efficiency. In future work, we plan to explore more efficient methods such as FastSAM [42] , EfficientSAM [48] , etc. to further improve computational efficiency. From top to bottom, the images of four terrain classes with corresponding text prompts, i.e. "sand", "soil", "big rock", and "bedrock", respectively, as illustrated on the left side of image. On the ConeQuest dataset, we trained on samples from three different regions: Isidis Planitia (IP), Acidalia Planitia (AP), and Hypanis (HP), and separately tested on samples from each region. We compare MTSNet with the official benchmarked models including UNet [36] , FPN [23] , DeepLabV3 [1] , and MA-Net [6] , while also including SAM-Adapter and AdaptiveSAM for comprehensive comparison.For the Isidis Planitia (IP) region, the experimental results are shown in Table 2 . UNet, FPN, DeepLabV3, and MA-Net exhibit varying performances, with FPN achieving relatively high levels in Mask IoU and Pixel IoU. AdaptiveSAM and SAM-Adapter perform relatively moderately, and our method also achieve moderate improvements in Pixel IoU and Pixel Accuracy. We speculate that the mediocre performance of MTSNet is mainly due to 1) the sample and region imbalance of IP region data and 2) the ViT's inherent inability to learn from extremely imbalanced data [49] . The IP region data contains 91% positive samples and only 9% negative samples, which is extremely imbalanced. We believe that proper feature fusion [28] , semi-supervised feature learning [39] and tailor-made re-balancing loss functions may alleviate the model's performance degradation on such data.In the Acidalia Planitia (AP) region, as shown in Table 3 , benchmarked models like UNet fluctuates in performance, but FPN maintains a good segmentation accuracy. AdaptiveSAM and SAM-Adapter still perform sub-optimally. On the contrary, our method achieves significant improvements across several metrics, attaining the best scores of 53.21 in Mask IoU, increased by 3.26% and 54.93 in Pixel IoU, increased by 0.88%, respectively. For the Hypanis (HP) region in Table 4 , we observe that all methods encounter greater challenges. Compared to other regions, UNet, FPN, DeepLabV3, and MA-Net exhibited declines across various metrics. AdaptiveSAM and SAM-Adapter are more limited in this region. Notably, although our method also faces substantial challenges, it demonstrates adaptability and performance improvement, achieving the best results in Mask IoU, Pixel IoU, Pixel Accuracy, and Pixel Recall, improving Pixel IoU by 2.89% and Pixel Recall by 6.78%.Figure 5 presents the comparison of the results from our method and other methods on three regions. In the Isidis Planitia region (first row) and the Acidalia Planitia region (second row), for cases where multiple conic landforms exist, other methods exhibit serious missed detection problem, whereas our method ensures no omissions, despite some errors in details. In the Hypanis region (third and fourth rows), compared to other methods, our method does not suffer from recognition errors, while also achieving higher segmentation accuracy.Through the above experimental results, we can see that benchmarked models like UNet, FPN, DeepLabV3, and MA-Net exhibit varying degrees of performance differences on conical landforms, performing well only in some regions. However, in other regions, when handling unseen distribution features, their generalization capabilities are significantly limited. AdaptiveSAM and SAM-Adapter perform moderately, with metrics noticeably lower than other models. In contrast, our method demonstrates a breakthrough in generalization ability, achieving more outstanding results in the spatial generalization task. To investigate the influence of different model components, we conducted ablation experiments on the proposed model using the AI4Mars dataset, with the following details.Effectiveness of the text affine layer (TAL) : Table 5 shows the performance comparison between the model with TAL and the model without TAL. The results demonstrate that incorporating the TAL as a text domain adapter significantly improves the segmentation performance across various terrain types. Specifically, with the TAL, we observed an average increase of 0.87% in Dice scores and 1.00% in IoU scores. These results validate our approach of [15] , Coordinate Attention [14] , Efficient Multiscale Attention (EMA), as well as applying spatial attention and channel attention separately. It should be noted that here we focus on evaluating the impact of different attention mechanisms on the TCAM module, without additionally introducing LTEN. The experimental results in Table 7 show that although various attention mechanisms improve the model's performance to some extent, our channel-spatial attention achieves the optimal results. Specifically, compared to PSA, our method significantly improves the Dice scores and IoU scores across all classes and overall. Compared to Coordinate Attention or EMA, although there are some differences in individual classes, our method has a certain advantage in overall performance. Moreover, applying spatial attention and channel attention separately yields inferior overall performance compared to channel-spatial attention, indicating that handling complex Mars terrain features requires simultaneously considering both spatial position information and cross-channel dependencies. Figure 6 presents a visualized comparison among different attention mechanisms in TCAM. We generate the heatmap in Figure 6 from the output of the last Transformer block's TCAM. It can be observed that for "sand" and "big rock", the channel-spatial attention more effectively captures the salient features. When dealing with "soil" and "bedrock", the Coordinate Attention, PSA, and channelspatial attention all exhibit better focusing capabilities. Overall, the channel-spatial attention appears more balanced, demonstrating relatively better performance and stability under various terrain conditions.Effectiveness of EMA in LTE Block: In LTEN, to enhance the learning capability of local terrain features, we incorporated Efficient Multi-scale Attention (EMA) after each LTE Block. To validate the effectiveness of EMA, we designed corresponding experiments, using Pyramid Split Attention (PSA) [15] , Coordinate Attention (CA) [14] , SE (Squeeze-and-Excitation) [17] , and CBAM [47] for comparison. The experimental results in Table 8 show that different attention mechanisms perform variably across terrain classes, but overall, EMA exhibits the most significant advantage, particularly in the "bedrock" and "soil" classes. Specifically, while PSA excels in recognizing "big rock", and Coordinate Attention, SE, and CBAM perform well in some metrics, but their overall performance is surpassed by EMA. In summary, benefiting from its cross-spatial multi-scale attention mechanism, EMA can better capture local terrain features than other attention methods. 
conclusion: 
 In  
 
 Appendix:  


Decision: In the final ratings, two reviewers lean to accept the paper and one reviewer gives "borderline reject". Prior to the rebuttal, the major concerns are missing analysis on model efficiency, lack of network parameter analysis, and the weak performance on one dataset. In the rebuttal, the authors provide the requested analysis and clear most concerns. The AC agrees with the majority and recommends acceptance.

Reviews: review: paper_summary: paper_summary: This essay introduces a novel multi-view Martian terrain segmentation framework (MTSNet), designed to meet the needs of autonomous navigation and safe driving of Mars rovers, as well as global analysis of Martian geological landforms for Mars exploration missions. MTSNet combines an efficient Martian terrain text-guided Segment Anything Model (MTG-SAM) with a specially designed Local Terrain Feature Enhancement Network (LTEN) to capture intricate terrain details. The MTG-SAM is equipped with a Terrain Contextual Attention Adapter Module (TCAM) to enhance the model's adaptability and transferability to the specific distribution of Martian terrain. The LTEN is intended to compensate for the limitations of the MTG-SAM encoder in capturing fine-grained local terrain features on the Martian surface. Furthermore, MTSNet introduces a simple yet efficient Gated Fusion Module (GFM) to dynamically merge global contextual features from the MTG-SAM encoder with local refined features from the LTEN module for comprehensive terrain feature learning. MTSNet also supports the use of terrain-specific text as prompts, addressing the issue of costly annotation requirements for bounding boxes or foreground points in existing methods.

The essay also mentions the experimental results of MTSNet on the AI4Mars and ConeQuest datasets, demonstrating that the proposed MTSNet can effectively learn the unique distribution of Martian terrain features and achieve state-of-the-art performance in multi-view terrain segmentation. The introduction of MTSNet brings not only technical innovation but also provides a new efficient tool for terrain analysis in the field of Mars exploration.
strength_weakness: strength_weakness: 1.  This work is the first to tackle the multi-view Martian Terrain Segmentation task using only text prompts, achieving superior performance on two benchmarks with notable flexibility and efficiency.
2. The integration of MTG-SAM with LTEN and GFM represents a well-conceived theoretical approach, addressing the complexity of segmenting Martian terrain by considering both global and local features.
3. The paper outlines a technically sound architecture with components like TCAM for adaptability, LTEN for local feature enhancement, and GFM for dynamic feature fusion, along with a text prompt encoder that addresses annotation challenges.
4. The framework is evaluated on relevant datasets (AI4Mars and ConeQuest) using appropriate metrics (Dice scores and IoU scores), and the comparative analysis with existing methods is thorough, demonstrating clear performance improvements and insights from ablation studies.
limitations: limitations: 1.The essay has mentioned that the MTSNet might be applied to the self driving of Mars rovers. However, It is well known that autonomous driving requires making correct decisions about the environment in real time, which greatly tests the real-time performance of the algorithm. Unfortunately, the essay does not provide information such as the model's running time, and I believe that this information would be more helpful for an overall evaluation of the work presented in this paper.

2.During the evaluation on the IP region of the ConeQuest dataset, why does the method with TCAM and the LTEN not outperform the TCAM alone, and why does the method only show a modest improvement over the FPN? Does this indicate potential issues with the generalization capabilities of the method? It would be appreciated if you could offer a more in-depth explanation and propose possible plans for enhancement.

3.The essay frequently alludes to the incorporation of a lightweight TCAM component; however, it lacks any comparative data on the model's parameterization. To underscore the benefits of the lightweight design, it would be advantageous to present a comparative analysis of the parameter counts among various attention mechanisms.
paper_summary: paper_summary: This paper proposed a new framework MTSNet for the Martian terrain segmentation. 
It takes in a Mars image and text prompt of interested area to output a segmentation map. 
MTSNet combines several existing methods to achieve the goal of Martian terrain segmentation including SAM and Gated Fusion. It designs a new Local Terrain Feature Enhancement Network (LTEN in the paper) to extract and enhance the fine-grained details of the terrains. LTEN consists of convlutional layers and multi-scale attention layers to learn the local information of the terrains. Besides, based on SAM-adapter method, they propose a new module named Terrain Contextual Attention Adapter Module (TCAM) to fine-tune SAM on a specific domain, Martian Terrain domain in this paper. The local information and contexual information is fused by a gated fusion module. Then the fused features are fed into a mask decoder with corresponding text prompt.
strength_weakness: strength_weakness: Compared with the existing methods, MTSNet only uses the text prompt to guide the network, which does not need annotations of bounding boxes. It expolres the possibilites of using text prompt to guide the network and works well on the Martian terrain segmentation. Experiments on the Ai4Mars and ConeQuest dataset show that MTSNet outperforms the existing methods on most of the terrains. Sufficient ablation study shows the necessarity of the proposed LTEN and TCAM Module.
limitations: limitations: 1. There is no ablation study of the proposed Text Affine Layer. Also, the author does not explain how and why the Text Affine Layer works. Since it project the CLIP embeddings to the terrain terminology, there is no explanation or experiment to verify the effectiveness of this layer. 

The framework is too specific designed for the Martian terrain segmentation. It may not suitable to be used in other domains, which lacks generalization ability.

The source code is not available.

The layout of the four tables could be better.
typo: in line 359, 'LTE block is shown in Figure 2.' not 'Figure 3'
paper_summary: paper_summary: This paper presents a novel text-guided multi-view Martian Terrain Segmentation framework, called MSTNet. It consists of an efficient Martian Terrain text-Guided Segmentation Anything Model (MTS-SAM) and a tailored Local Terrain Feature Enhancement Network (LTEN) to capture intricate terrain details. Experimental results on public available datasets show state-of-the-art performance on multi-view terrain segmentation.
strength_weakness: strength_weakness: (1)The proposed MTG-SAM module enables to extract global features,  along with the lightweight TCAM to allow adapting to domain knowledge of Martian terrain segmentation.
(2)The LTEN module focuses on extracting local features with an efficient multi-scale attention module, which is complementary with the above MTG-SAM.
(3)The GFM module integrates the features of MTG-SAM and LTEN well, along with a text prompt encoder for efficient text embedding of Martian terrain segmentation.
(4)The experiments are well organized and state-of-the-art results are achieved compared with most existing methods.
limitations: limitations: (1)The description of the Text Prompt Encoder module is missing in the experiment. Besides facilitating regional retrieval, is it helpful for improving tasks?
(2)The description of the text prompt encoder in Figure 1 and Section 3.5 does not correspond well. The prompt encoder is not described in the text. Where are the learnable parameters?
(3)There is a citation error in the figure, which is to write Figure 2 as Figure 3 on page 4.
(4)There are some grammar errors in the text, including “a Efficient Multi-scale Attention ...” in page 4, “called MTSNet, it consists of ...” in page 8, and etc.