Title: EAGLE: Efficient Adaptive Geometry-based Learning in Cross-view Understanding
Venue: NeurIPS 2024
Accepted: yes
Keywords: Unsupervised Cross-view Adaptation

Cross-view Geometric Constraint

Semantic Segmentation

Geodesic Flow-based Correlation Metric
Abstract: Unsupervised Domain Adaptation has been an efficient approach to transferring the semantic segmentation model across data distributions. Meanwhile, the recent Open-vocabulary Semantic Scene understanding based on large-scale vision language models is effective in open-set settings because it can learn diverse concepts and categories. However, these prior methods fail to generalize across different camera views due to the lack of cross-view geometric modeling. At present, there are limited studies analyzing cross-view learning. To address this problem, we introduce a novel Unsupervised Cross-view Adaptation Learning approach to modeling the geometric structural change across views in Semantic Scene Understanding. First, we introduce a novel Cross-view Geometric Constraint on Unpaired Data to model structural changes in images and segmentation masks across cameras. Second, we present a new Geodesic Flow-based Correlation Metric to efficiently measure the geometric structural changes across camera views. Third, we introduce a novel view-condition prompting mechanism to enhance the view-information modeling of the open-vocabulary segmentation network in cross-view adaptation learning. The experiments on different cross-view adaptation benchmarks have shown the effectiveness of our approach in cross-view modeling, demonstrating that we achieve State-of-the-Art (SOTA) performance compared to prior unsupervised domain adaptation and open-vocabulary semantic segmentation methods.

EAGLE : Efficient Adaptive Geometry-based Learning in Cross-view Understanding

Abstract

Unsupervised Domain Adaptation has been an efficient approach to transferring the semantic segmentation model across data distributions. Meanwhile, the recent Open-vocabulary Semantic Scene understanding based on large-scale vision language models is effective in open-set settings because it can learn diverse concepts and categories. However, these prior methods fail to generalize across different camera views due to the lack of cross-view geometric modeling. At present, there are limited studies analyzing cross-view learning. To address this problem, we introduce a novel Unsupervised Cross-view Adaptation Learning approach to modeling the geometric structural change across views in Semantic Scene Understanding. First, we introduce a novel Cross-view Geometric Constraint on Unpaired Data to model structural changes in images and segmentation masks across cameras. Second, we present a new Geodesic Flow-based Correlation Metric to efficiently measure the geometric structural changes across camera views. Third, we introduce a novel view-condition prompting mechanism to enhance the view-information modeling of the open-vocabulary segmentation network in cross-view adaptation learning. The experiments on different cross-view adaptation benchmarks have shown the effectiveness of our approach in cross-view modeling, demonstrating that we achieve State-of-the-Art (SOTA) performance compared to prior unsupervised domain adaptation and open-vocabulary semantic segmentation methods.

1 Introduction

Modern segmentation models [3, 4, 63] have achieved remarkable results on the close-set training with a set of pre-defined categories and concepts. To work towards human-level perception where the scenes are interpreted with diverse categories and concepts, the openvocabulary (open-vocab) perception model [38, 40] based on the power of large vision-language models [30, 39] has been introduced to address the limitations of close-set training. By using the power of language as supervision, the largescale vision language model is able to learn the more powerful representations where languages offer better reasoning mechanisms and openword concept representations compared to traditional close-set training methods [3, 63, 9].

find cars, persons, trees

from the car view

Encoder

Encoder

Decoder

car

person

tree

DenseCLIP

Cross-View

Adaptation

FreeSeg

Encoder

Decoder

find cars, persons, trees

OurApproach

from the drone view

Encoder

Recent work is inspired by the success of large vision-language models [39, 27] that are able to learn informative feature representations of both visual and textual inputs from large-scale image-text pairs. These have been adopted to further develop open-vocab semantic segmentation models [38, 40, 31, 29] that can work well in open-world environments. However, the open-vocab perception models remain unable to generalize across camera viewpoints. As shown in Fig. 1, the open-vocab model trained on car views is not able to perform well on the images captured from unmanned aerial vehicles (UAVs) or drones. While this issue can be improved by training the segmentation model on drone-view

GroundTruth

Prediction

SourceView

Domain

Inputlmage

Prediction

Cross-ViewAdaptation

TargetView

Domain

data, the annotation process of high-resolution UA V data is costly and time-consuming. At present, there exist many large-scale datasets with dense labels captured from camera views on the ground, e.g., car views (SYNTHIA [44], GTA [43], Cityscapes [11], BDD100K [68]). They have been widely adopted to develop robust perception models. Since these car view and drone view datasets have many common objects of interest, incorporating knowledge from car views with drone views benefits the learning process by reusing large-scale annotations and saving efforts of manually labeling UA V images. Unsupervised domain adaptation (UDA) [58, 23, 1, 51, 53] is one of the potential approaches to transfer the knowledge from the car view (i.e., source domain) to the drone view (i.e., target domain). While UDA approaches have shown their effectiveness in transferring knowledge across domains, e.g., environment changes or geographical domain shifts, these methods remain limited in the cases of changing camera viewpoints. Indeed, the changes in camera positions, e.g., from the ground of cars to the high positions of drones, bring a significant difference in structures and topological layouts of scenes and objects (Fig. 2). Therefore, UDA is not a complete solution to this problem due to its lack of cross-view structural modeling. Additionally, although the open-vocab segmentation models have introduced several prompting mechanisms, e.g., context-aware prompting [40] or adaptive prompting [38] to improve context learning across various open-world concepts, they are unable to model the cross-view structure due to the lack of view-condition information in prompts and geometric modeling. To the best of our knowledge, there are limited studies that have exploited this cross-view learning. These limitations motivate us to develop a new adaptation learning paradigm, i.e., Unsupervised Cross-view Adaptation , that addresses prior methods to improve the performance of semantic segmentation models across views.

Contributions: This work introduces a novel E fficient A daptive G eometry-based L earning (EAGLE) to Unsupervised Cross-view Adaptation that can adaptively learn and improve the performance of semantic segmentation models across camera viewpoints. First, by analyzing the geometric correlations across views, we introduce a novel cross-view geometric constraint on unpaired data of structural changes in images and segmentation masks. Second, to efficiently model cross-view geometric structural changes , we introduce a new Geodesic Flow-based Metric to measure the structural changes across views via their manifold structures. In addition, to further improve the prompting mechanism of the open-vocab segmentation network in cross-view adaptation learning, we introduce a new view-condition prompting . Then, our cross-view geometric constraint is also imposed on its feature representations of view-condition prompts to leverage its geometric knowledge embedded in our prompting mechanism. Our proposed method holds a promise to be an effective approach to addressing the problem of cross-view learning and contributes to improving UDA and open-vocab segmentation in cross-view learning. Thus, it increases the generalizability of the segmentation models across camera views. Finally, our experiments on three presented cross-view adaptation benchmarks, i.e., SYNTHIA → UAVID, GTA → UAVID, BDD → UAVID, illustrate the effectiveness of our approach in cross-view modeling and our State-of-the-Art (SOTA) performance.

2 Related Work

Unsupervised Domain Adaptation Adversarial learning [6, 58, 59] and self-supervised learning [1, 69, 23, 14] are common approaches to UDA in semantic segmentation. The adversarial learning approaches are typically simultaneously trained on source and target data [58, 7, 6]. Chen et al. [7] first introduced an adversarial framework to domain adaptation. Later, several approaches improved

adversarial learning by utilizing generative models [74, 34, 21], using additional labels [28, 59], incorporating with entropy minimization [58, 65, 51, 52], or adopting the curriculum training [37]. Recently, the self-supervised approaches [1, 69, 23, 14] have achieved outstanding performance. Araslanov et al. [1] first proposed a self-supervised augmentation consistency framework for UDA. Hoyer et al. [23] utilized Transformers to improve the UDA performance. Later, this approach was further improved by utilizing multi-resolution cropped images [24] and masked image consistency strategy [25] to enhance contextual learning. Recent studies improved the self-supervised approach by aligning both output and attention levels via the cross-domain prediction consistency framework [60], using a prototypical representation [69], learning the cross-model consistency via depths [67], improving the class-relevant fairness [53, 55, 56], or exploring the relations of pseudo-labels [71]. Fashes et al. [15] introduced a prompt-based feature augmentation method to zero-shot UDA. Gong et al. [18] introduced a geodesic flow kernel to model the manifold structure between domains. Later, Simon et al. [47] designed distillation loss by the geodesic flow path.

Vision-Language and Open-Vocab Segmentation By pre-training on a large-scale vision-language dataset [39, 27], the vision-language models can learn various visual concepts and can further be transferred to other vision problems through ' prompting ' [17, 38, 31, 35], e.g., open-vocab segmentation [64, 13, 38]. Li et al. [29] first introduced the language-driving approach to semantic segmentation. Rao et al. [40] represented a context-aware prompting mechanism for dense prediction tasks. Ghiasi et al. [17] proposed an OpenSeg framework that learns the visual-semantic alignments. Qin et al. [38] presented a unified, universal, and open-vocab segmentation network based on Mask2Former [8] with an adaptive prompting mechanism. Xu et al. [64] proposed a two-stage open-vocab segmentation framework using the mask proposal generator and the pre-trained CLIP model. Ding et al. [13] decoupled the zero-shot semantic segmentation to class-agnostic segmentation and segment-level zero-shot classification. Liang et al. [31] improved the two-stage open-vocab segmentation model by further fine-tuning CLIP on masked image regions and corresponding descriptions.

Cross-view Learning The early studies exploited cross-view learning in geo-localization by using a polar transform across views [46, 45] or generative networks to cross-view images [41, 49]. Meanwhile, Zhu et al. [75] exploited the correlation between street- and aerial-view data via selfattention. In semantic segmentation, Coors et al. [10] first introduced a cross-view adaptation approach utilizing the depth labels and the cross-view transformation between car and truck views. However, this change of views in [10] is not as big a hurdle as the change of views in our problem, i.e., car view to drone view. Ren et al. [42] presented an adaptation approach across viewpoints using the 3D models of scenes to create pairs of cross-view images. Vidit et al. [57] modeled the geometric shift in cross FoV setting for object detection by learning position-invariant homography transform. Di Mauro et al. [12] introduced an adversarial method trained on a multi-view synthetic dataset where images are captured from different pitch and yaw angles at the same altitudes of the camera positions. Meanwhile, in our problem, the camera views could be placed at different altitudes (e.g., the car and the drone), which reveals large structural differences between the images. Truong et al. [50, 54] first introduced a simple approach to model the relation across views. CROVIA [50] measures the cross-view structural changes by measuring the distribution shift and only focuses on the cross-view adaptation setting in semantic segmentation. However, these methods [50, 54] lack a theory and a mechanism for cross-view geometric structural change modeling. To the best of our knowledge, there are limited studies exploiting cross-view adaptation in semantic segmentation. Therefore, our work presents a new approach to model the geometric correlation across views.

3 The Proposed EAGLE Approach

In this paper, we consider cross-view adaptation learning as UDA where the images of the source and target domains are captured from different camera positions (Fig. 2). Formally, let x s , x t be the input images in the source and target domains, p s , p t be the the corresponding prompts, and y s , y t be the segmentation masks of x s , x t . Then, the open-vocab segmentation model F maps the input x and the prompt p to the corresponding output y = F ( x , p ) . It should be noted that in the case of traditional semantic segmentation, the prompt p will be ignored, i.e., y = F ( x ) The cross-view adaptation learning can be formulated as Eqn. (1).

where θ is the parameters of F , ˆ y s is the ground truth, L Mask is the supervised (open-vocab) segmentation loss with ground truths, and L Adapt is unsupervised adaptation loss from the source to the target domain. In the open-vocab setting, we adopt the design of Open-Vocab Mask2Former [8, 38] to our network F . Prior UDA methods defined the adaptation loss L Adapt via the adversarial loss [28, 5], entropy loss [51, 58], or self-supervised loss [23, 25]. Although these prior results have illustrated their effectiveness in UDA, these losses remain limited in cross-view adaptation setup. Indeed, the adaptation setting in prior studies [58, 1, 23, 15] is typically deployed in the context of environmental changes (e.g., simulation to real [58, 59, 15], day to night [25, 15], etc) where the camera positions between domains remain similar. Meanwhile, in cross-view adaptation, the camera position of the source and target domain remains largely different (as shown in Fig. 2). This change in camera positions leads to significant differences in the geometric layout and topological structures between the source and target domains. As a result, direct adoption of prior UDA approaches to cross-view adaptation would be ineffective due to the lack of cross-view geometric correlation modeling. To effectively address cross-view adaptation, the adaptation loss L Adapt should be able to model (1) the geometric correlation between two views of source and target domains and (2) the structural changes across domains .

3.1 Cross-View Geometric Modeling

To efficiently address the cross-view adaptation learning task, it is essential to explicitly model cross-view geometric correlations by analyzing the relation between two camera views. Therefore, we first re-reconsider the cross-view geometric correlation. In particular, let ¯ x t be the corresponding image of x s captured from the target view, y s and ¯ y t be the semantic segmentation outputs of source image x s and target image ¯ x t , ¯ p t be the corresponding prompt of p s in target view, respectively. Formally, the images captured from the source and the target views can be modeled as Eqn. (2).

where R is the rendering function, K s and K t are the intrinsic matrices, [ R s , t s ] and [ R t , t t ] are the extrinsic matrices, and Θ represents the capturing scene. In addition, as the camera parameters of both source and target views are represented by matrices, there should exist linear transformations of camera parameters between two views as follow,

where T K and T Rt are the transformation matrices.

Remark 1: The Geometric Transformation Between Camera Views. From Eqn. (2) and Eqn. (3), we argue that there should exist a geometric transformation T of images between two camera views as: ¯ x t = T ( x s ; T K , T Rt ) .

Remark 2: The Equivalent Transformation Between Image and Segmentation Output. As RGB images and segmentation maps are pixel-wised corresponding, the same geometric transformation T in the image space can be adopted for segmentation space as: ¯ y t = T ( y s ; T K , T Rt )

Remarks 1-2 have depicted that the geometric transformation of both image and segmentation from the source to the target view can be represented by the shared transformation T with the camera transformation matrices T K , T Rt . Let D x ( x s , ¯ x t ) and D y ( y s , ¯ y t ) be the metrics the measure the cross-view structures changes of images and segmentation maps from the source to target domains.

We argue that the cross-view geometric correlation in the image space, i.e., D x ( x s , ¯ x t ) , is theoretically proportional to the one in the segmentation space, i.e., D y ( y s , ¯ y t ) . Since the camera transformations between the two views are linear (Eqn. (3)) and the images x and outputs y are pixel-wised corresponding, we hypothesize that the cross-view geometric correlation in the image space D x ( x s , ¯ x t ) and the segmentation space D y ( y s , ¯ y t ) can be modeled by a linear relation with linear scale α as follows:

3.2 Cross-view Geometric Learning on Unpaired Data

Eqn. (4) defines a necessary condition to explicitly model the cross-view geometric correlation. Therefore, cross-view adaptation learning in Eqn. (1) can be re-formed as follows:

where, L Adapt ( y s , ¯ y t ) = ||D x ( x s , ¯ x t ) -α D y ( y s , ¯ y t ) || is the cross-view geoemtric adaptation loss, ||·|| is the mean squared error loss. However, in practice, the pair data between source and target views are inaccessible as data from these two views are often collected independently. Thus, optimizing Eqn. (5) without cross-view pairs of data remains an ill-posed problem. To address this limitation, instead of learning Eqn. (5) on paired data, we proposed to model this correlation on unpaired data. Instead of solving the cross-view geometric constraint of Eqn. (5) on pair data, let us consider all cross-view unpaired samples ( x s , x t ) . Formally, learning the Cross-view Geometric Constraint between unpaired samples can be formulated as in Eqn. (6).

where x s and x t are unpaired data, and L Adapt ( y s , y t ) = ||D x ( x s , x t ) -α D y ( y s , y t ) || is the Crossview Geometric Adaptation loss on unpaired data. Intuitively, although the cross-view pair samples are not available, the cross-view geometric constraints on paired samples between two views can be indirectly imposed by modeling the cross-view geometric structural constraint among unpaired samples. Then, by modeling the cross-view structural changes in the image and segmentation spaces, the structural change on images of unpaired data could be considered as the reference for the cross-view structural change in the segmentation space during the optimization process. This action promotes the structures of segmentation that can be effectively adapted from the source view to the target view. Importantly, the cross-view geometric constraint imposed on unpaired data can be mathematically proved as an upper bound of the cross-view constraint on paired data as follows:

where O is the Big O notation. The upper bound in Eqn. (19) can be proved by using the properties of triangle inequality and our correlation metrics D x and D y (Sec. 3.3). The detailed proof is provided in the appendix. Eqn. (19) has illustrated that by minimizing the cross-view geometric constraint on unpaired samples in Eqn. (6), the cross-view constraint on paired samples in Eqn. (5) is also maintained due to the upper bound. Therefore, our proposed Cross-view Geometric Constraint loss does NOT require the pair data between source and target views during training. Fig. 3 illustrates our cross-view adaptation learning framework.

3.3 Cross-view Structural Change Modeling via Geodesic Flow Path

Modeling the correlation metrics D x and D y is an important task in our approach. Indeed, the metrics should be able to model the structure changes from the source to the target view. Intuitively, the changes from the source to the target view are essentially the geodesic flow between two subspaces on the Grassmann manifold. Then, the images (or segmentation) of two views can be projected along the geodesic flow path to capture the cross-view structural changes. Therefore, to model D x and D y , we adopt the Geodesic Flow path to measure the cross-view structural changes by modeling the geometry in the latent space.

TextualFeatures

Prompt:find car,

Pretrained CLIP

person,tree from the

drone view

TextEncoder

Encoder

Decoder

Image

Target

View

Language

Segmentation

Space

Space

Space

GeodesicFlow

GeodesicFlow

GeodesicFlow

Source

View

Decoder

Encoder

Prompt:find car,

Pretrained CLIP

person,tree from the

car view

TextEncoder

TextualFeatures

Remark 3: Grassmann Manifold is the set of N -dimensional linear subspaces of R D (0 < N < D ) , i.e, G ( N,D ) . A matrix with orthonormal columns P ∈ R D × N define a subspace of G ( N,D ) , i.e., P ∈ G ( N,D ) ⇒ P ⊤ P = I N where I N is the N × N identity matrix.

For simplicity, we present our approach to model the cross-view structural change D x in the image space. Formally, let P s and P t be the basis of the source and target domains. These bases can be obtained by the PCA algorithm. The geodesic flow between P s and P t in the manifold can be defined via the function Π : ν ∈ [0 .. 1] → Π ( ν ) , where Π ( ν ) ∈ G ( N,D ) is the subspace lying on

the geodesic flow path from the source to the target view:

where R ∈ R D × ( D -N ) is the orthogonal complement of P s , i.e., R ⊤ P s = 0 . Γ ( ν ) and Σ ( ν ) are the diagonal matrices whose diagonal element at row i can be defined as γ i = cos( νω i ) and σ i = sin( νω i ) . The list of ω i is the principal angles between source and target subspaces, i.e., 0 ≤ ω 1 ≤ ... ≤ ω N ≤ π 2 . U 1 and U 2 are the orthonormal matrices obtained by the following pair of SVDs:

Since P ⊤ s P t and R ⊤ P t share the same singular vectors V , we adopt the generalized Singular Value Decomposition (SVD) [18, 47] to decompose the matrices. In our approach, we model the cross-view structural changes D x by modeling the cosine similarity between projections along the geodesic flow Π ( ν ) . In particular, given a subspace Π ( ν ) on the geodesic flow path from the source to the target view, the cross-view geometric correlation of images between the source and target views can formulated by the inner product g Π ( ν ) ( x s , x t ) along the geodesic flow Π ( ν ) as follows:

where Q = ∫ 1 0 Π ( ν ) Π ( ν ) ⊤ dν . Intuitively, the matrix Q represents the manifold structure between the source to the target view. Then, Eqn. (10) measures the cross-view structural changes between the source and the target domain based on their manifold structures. The matrix Q can be obtained in a closed form [18, 47] as follows:

where Λ 1 , Λ 2 , and Λ 3 are the diagonal matrices, whose diagonal elements at row i can be defined as:

In practice, we model the cross-view structural changes D x via the cosine similarity along the geodesic flows. Finally, the cross-view structural changes D x can be formulate as:

Similarly, we can model the cross-view geometric correlation of segmentation D y via Geodesic Flow.

3.4 View-Condition Prompting to Cross-View Learning

View-Condition Prompting Previous efforts [40, 38, 16, 73] in open-vocab segmentation have shown that a better prompting mechanism can provide more meaningful textual and visual knowledge. Prior work in open-vocab segmentation designed the prompt via the class names [64, 13, 38], e.g., ' class 1 , class 1 , ..., class K '. Meanwhile, other methods improve the prompting mechanism by introducing the learnable variables into the prompt [40] or adding the task information [38]. This action helps to improve the context learning of the vision-language model. In our approach, we also exploit the effectiveness of designing prompting to cross-view learning. In particular, describing the view information can further improve the visual context learning, e.g., ' class 1 , class 1 , ..., class K captured from the [domain] view ', where [domain] could be car (source domain) or drone (target domain). Therefore, we introduce a view-condition prompting mechanism by introducing the view information, i.e., captured from the [domain] view ', into the prompt. Our view-condition prompt offers the context specific to visual learning, thus providing better transferability in cross-view segmentation.

Cross-view Correlation of View-Condition Prompts We hypothesize that the correlation of the input prompts across domains also provides the cross-view geometric correlation in their deep representations. In particular, let f p s and f p t be the deep textual embeddings of view-condition prompts p s and p t , and D p be metric measuring the correlation between f p s and f p t . In addition, since the textual encoder has been pre-trained on large-scale vision-language data [39, 27], the visual and the textual representations have been well aligned. Then, we argue that the correlation of

textual feature representations across views, i.e., D p ( f p s , f p t ) , also provides the cross-view geometric correlation due to the embedded view information in the deep representation of prompts aligned with visual representations. Therefore, similar to Eqn. (4), we hypothesize the cross-view correlation of segmentation masks and textual features can be modeled as a linear relation with a scale factor γ as:

Then, learning the cross-view adaptation with view-condition prompts can be formulated as follows:

where λ I and λ P are the balanced-weight of losses. Similar to metrics D x and D y , we also adopt the geodesic flow path to model the cross-view correlation metric D p .

4 Experiments

4.1 Datasets, Benchmarks, and Implementation

To efficiently evaluate cross-view adaptation, the cross-view benchmarks are set up from the car to the drone view. Following common practices in UDA [23, 58], we choose SYNTHIA [44], GTA [43], and BDD100K [68] as the source domains while UAVID [33] is chosen as the target domain. We chose to adopt these datasets because they share a class of interests and are commonly used in UDA and segmentation benchmarks [23, 61].

SYNTHIA → UAVID Benchmark SYNTHIA and UAVID share five classes of interest, i.e., Road , Building , Car , Tree , and Person . Since the UAVID dataset annotated cars, trucks, and buses as a class of Car , we collapse these classes in SYNTHIA into a single class of Car .

GTA → UAVID Benchmark consists of five classes in the SYNTHIA → UAVID benchmark and includes one more class of Terrain . Therefore, the GTA → UAVID benchmark has six classes of interest, i.e., Road , Building , Car , Tree , Terrain , and Person .

BDD → UAVID Benchmark is a real-to-real cross-view adaptation setting. Similar to GTA → UAVID benchmark, there are six classes of interest between BDD100K and UAVID. In our experiments, we adopt the mean Intersection over Union (mIoU) metric to measure the performance.

Implementation We adopt Mask2Former [8] (ResNet 101) with Semantic Context Interaction of FreeSeg [38] and pre-trained text encoder of CLIP [39] for our open-vocab segmentation networks. Our balanced weights of losses are set to λ I = 1 . 0 and λ P = 0 . 5 . Further details of our networks and hyper-parameters are provided in the appendix.

4.2 Ablation Study

Effectiveness of Cross-view Adaptation and Prompting Mechanisms Table 1 analyzes the effectiveness of prompting mechanisms, i.e., i.e., with and without Prompting , with and without Cross-view Adaptation (in Eqn. (6)), with and without View-Condition Prompting (in Eqn. (15)). For supervised results, we train two different models on UAVID with and without the

Terrain class on two benchmarks. As in Table 1, the cross-view adaptation loss in Eqn. (6) significantly improve the performance of segmentation models. With prompting and cross-view adaptation,

the mIoU performance is further boosted, i.e., the mIoU performance achieves 48 . 6% and 40 . 1% on two benchmarks. Additionally, by further using the view-condition prompting mechanism with our cross-view loss in Eqn. (15), the mIoU results are slightly improved by +1 . 1% and +1 . 7% on two benchmarks compared to the one without view-condition prompting. Our results have closed the gap with the upper-bound results where the models are trained on UAVID with labels.

Effectiveness of Cross-view Correlation Metrics and Network Backbones Table 2 studies the impact of choosing metrics and network backbones. We consider two options, i.e., Euclidean Metric and our Geodesic Flow-based Metric, for correlation metrics D x , D y , and D p . As shown in Table 2, our Geodesic Flow-based metrics significantly improve the performance of our cross-view adaptation. It has shown that our approach is able to measure the structural changes across views better than using the Euclidean metrics. In addition, by using the more powerful backbone (Swin), the performance of cross-view adaptation is further improved.

Effectiveness of Cross-view Learning Parameters Table 3 illustrates the impact of the linear scaling factors α and β . As in Table 3, the mIoU performance has been majorly affected by the relation between images and segmentation. The best performance is gained at the optimal value of α = 1 . 5 . Since the variation of RGB images is higher than the segmentation, the small value α could not correctly scale the rela-

tion between images and segmentation while the higher value of α exaggerates the structural change of segmentation masks. Additionally, the change of γ slightly affects the mIoU performance. Since the textual features are well-aligned with the image, the performance of segmentation models when changing γ also behaves similarly to the changes of α . However, the linear scale factor α is more sensitive to mIoU results since the images play a more important role in the segmentation results due to the pixel-wise corresponding of images and segmentation.

Effectiveness of Subspace Dimension in Geodesic Flow Table 3 reveals the importance of choosing the subspace dimension. The cross-view geometric structural change is better modeled by increasing the dimension of the subspaces. As in Table 3, the performance is improved when the dimension is increased from 96 to 256 . However, beyond that point, the mIoU performance tends to be dropped. We have observed that low dimensionality cannot model the structural changes across views since it captures small variations in structural changes. Conversely, higher dimensionality includes more noise in the cross-view structural changes and increases the computational cost. We also study the impact of batch size in our appendix.

Qualitative Results. To further illustrate the effectiveness of our proposed, we visualize the results produced by our model. In the model without prompting, Figure 4 illustrates the results of our cross-view adaptation compared to those without adaptation. As shown in the results, our approach can effectively segment the objects in the drone view. We also compare with the prior ProDA [69] and CROVIA [50] methods. Our qualitative results remain better than the prior adaptation method. For the model with prompting, Figure 5 illustrates the effectiveness of our approach in three cases:

Input

Adaptation

Without

Cross-View

Adaptation

CROVIA

ProDA

Input

Adaptation

Without

Cross-View

Adaptation

CROVIA

ProDA

Without

Cross-View

Cross-View

Without

Cross-View

Cross-View

Input

Adaptation

Input

Adaptation

Adaptation

Adaptation

Adaptation

Adaptation

+ViewCondition

+ViewCondition

without adaptation, with cross-view adaptation, and with view-condition prompting. As shown in the results, our cross-view adaptation can efficiently model the segmentation of the view. By using the view-condition prompting, our model can further improve the segmentation of persons and vehicles.

4.3 Comparisons with Prior UDA Methods

SYNTHIA → UAVID As shown in Table 4, our EAGLE has achieved SOTA results and outperforms prior view transformation (i.e., Polar Transform [45]) UDA methods by a large margin. For fair comparisons, we adopt the DeepLab [3] and DAFormer [23] for the segmentation network. In particular, our mIoU results using DeepLab and DAFormer are 45 . 2% and 50 . 8% . In the DAFormer backbone, the

Network

Method

SYNTHIA

→

UAVID

GTA

→

UAVID

Road Building Car Tree Person mIoU Road Building Car Tree Terrain Person mIoU

DeepLab

AdvEnt [58]

4.7

63.2

31.7 48.6

11.4

31.9

2.0

30.3

14.9 29.8

41.5

1.8

20.0

Polar Trans. [45]

20.5

10.9

38.2 22.6

4.3

19.3

19.4

9.1

37.8 20.7

15.6

2.5

17.5

DADA [59]

10.7

63.1

32.9 50.0

16.2

34.6

-

-

-

-

-

-

-

BiMaL [51]

5.4

62.1

34.8 50.7

12.7

33.1

1.3

44.6

10.1 49.2

20.0

10.9

22.7

SAC [1]

13.9

64.0

18.7 48.0

15.6

32.0

4.5

36.9

7.8

47.9

44.1

7.8

24.8

ProDA [69]

10.6

64.7

34.1 44.5

17.0

34.2

6.9

50.6

28.4 25.5

38.7

4.5

25.8

CROVIA [50]

10.6

65.7

51.7 55.6

17.0

40.1

18.2

49.8

10.4 48.1

44.0

8.0

29.7

EAGLE

29.9

65.7

55.5 56.8

18.3

45.2

20.5

53.0

37.6 50.7

45.3

13.0

36.7

Supervised

67.2

90.7

74.0 76.3

36.8

69.0

68.1

91.0

77.5 75.7

62.2

35.8

68.4

DAFormer

DAFormer [23]

7.3

75.1

51.7 48.0

15.1

39.4

15.3

51.6

33.6 27.8

38.5

4.0

28.5

MIC [25]

10.8

76.4

53.3 52.7

16.0

41.8

20.7

51.9

13.3 55.2

44.8

9.3

32.5

CROVIA [50]

16.3

75.1

59.6 60.0

19.1

46.0

20.5

56.1

37.6 50.7

45.3

10.9

36.8

EAGLE

30.6

75.3

59.7 63.1

25.3

50.8

23.9

65.0

38.5 53.5

49.3

14.1

40.7

Supervised

78.0

91.2

79.7 77.5

44.2

74.1

79.0

92.8

81.9 78.4

70.3

45.7

74.7

Mask2Former

EAGLE

31.4

75.1

57.5 59.2

19.5

48.6

22.9

64.6

37.8 52.8

48.5

13.8

40.1

Supervised

75.5

91.6

79.1 77.7

42.1

73.2

76.8

91.8

81.1 77.6

67.8

43.4

73.1

mIoU results of our approach are higher than CROVIA [50] and MIC [25] by +4 . 8% and +9 . 0% . The IoU result of each class also consistently outperformed the prior methods. Highlighted that although our approach does NOT use depth labels, our results still outperform the one using depths, i.e., DADA [59]. It has emphasized that our approach is able to better capture the cross-view structural changes compared to prior methods. Figure 4 illustrates our qualitative results compared to ProDA [69] and CROVIA [50].

GTA → UAVID As shown in Table 4, our effectiveness outperforms prior polar view transformation [45] and domain adaptation approaches when measured by both mIoU performance and the IoU accuracy of each class. In particular, our mIoU performance using DeepLab and DAFormer network achieves 36 . 7% and 40 . 7% , respectively. Our results have substantially closed the performance gap with the supervised results. By using the better segmentation-based network, i.e., Mask2Former with ResNet, the performance of our approach is further improved to 40 . 1% compared to DeepLab.

4.4 Comparisons with Open-vocab Segmentation

We compare EAGLE with the prior open-vocab segmentation methods, i.e., DenseCLIP [40] and an adaptive prompting FreeSeg [38] with four settings, i.e., Source Only, with AdvEnt [58], and with SAC [1], and our Cross-View Adaptation in Eqn. (6) (without view-condition).

Open-vocab Semantic Segmentation As in Table 5, the mIoU performance of our proposed approach with cross-view adaptation outperforms prior DenseCLIP by a large margin on SYNTHIA → UAVID. By using our cross-view geometric adaptation loss, the performance of DenseCLIP and FreeSeg is further enhanced, i.e., higher than DenseCLIP and FreeSeg with SAC by +3.7% and +5.0%. While FreeSeg [38] with our cross-view adaptation slightly outperforms EAGLE due to its adaptive prompting, our EAGLE approach with the better view-condition prompting achieves higher mIoU

performance. Similarly, our proposed cross-view loss consistently improves the performance of DenseCLIP and FreeSeg on GTA → UAVID. The mIoU results of DenseCLIP and FreeSeg using our cross-view loss achieve 37.3% and 44.4%. By further using the view-condition prompting mechanism, our mIoU result is considerably higher than FreeSeg with our cross-view adaptation by +1.3%. Figure 6 visualizes our qualitative results of our proposed approach.

Open-vocab Segmentation on Unseen Classes Table 6 illustrates the experimental results of our cross-view adaptation approach on unseen classes. In this experiment, we consider classes of Tree and Person as the unseen classes. As shown in the results, our cross-view adaptation approach with a view-condition prompting mechanism has achieved the best mIoU performance on unseen classes on both benchmarks, i.e., 39 . 3% and 39 . 6% on two benchmarks. Our experimental results have further confirmed the effectiveness and the generalizability of our cross-view geometric modeling and view-condition prompting approach to the open-vocab segmentation across views.

Real-to-Real Cross-view Adaptation Setting We evaluated our approach in the real-to-real setting, i.e., BDD → UAVID. Our approach is evaluated in two different settings, i.e., Unsupervised Domain Adaptation and Open-Vocab Semantic Segmentation. As shown in Table 7, our results have shown a significant improvement in our approach in real-to-real settings in both unsupervised domain adaptation and openvocab semantic segmentation. While the results of prior unsupervised domain adaptation, i.e.,

BiMaL [51] and DAFormer [23], gain limited performance due to their limits in cross-view learning, our method outperforms other methods these prior methods by a large margin.

5 Conclusions

This paper has presented a novel unsupervised cross-view adaptation approach that models the geometric correlation across views. We have introduced the Geodesic Flow-based metric to better model geometric structural changes across camera views. In addition, a new view-condition prompting mechanism has been presented to further improve the cross-view modeling. Through our theoretical analysis and SOTA performance on both unsupervised cross-view adaptation and open-vocab segmentation, our approach has shown its effectiveness in cross-view modeling and improved robustness of segmentation models across views.

Limitations Our study has selected a set of learning hyper-parameters to support our hypothesis and experiments. However, this work can potentially contain several limitations related to learning parameters and linear relation hypothesis in Eqn. (4). The details of the limitations are discussed in the appendix. We believe that these limitations will motivate future studies to improve our unsupervised cross-view adaptation learning approach.

Acknowledgment This work is partly supported by NSF Data Science, Data Analytics that are Robust and Trusted (DART), NSF SBIR Phase 2, and Arkansas Biosciences Institute (ABI) grants. We also acknowledge the Arkansas High-Performance Computing Center for providing GPUs.

References

N. Araslanov, , and S. Roth. Self-supervised augmentation consistency for adapting semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

M. Armando, S. Galaaoui, F. Baradel, T. Lucas, V. Leroy, R. Brégier, P. Weinzaepfel, and G. Rogez. Cross-view and cross-pose completion for 3d human understanding. arXiv preprint arXiv:2311.09104 , 2023.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. TPAMI , 2018.

L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

Y. Chen, W. Li, C. Sakaridis, D. Dai, and L. Van Gool. Domain adaptive faster r-cnn for object detection in the wild. In CVPR , 2018.

Y. Chen, W. Li, and L. Van Gool. Road: Reality oriented adaptation for semantic segmentation of urban scenes. In CVPR , 2018.

Y.-H. Chen, W.-Y. Chen, Y.-T. Chen, B.-C. Tsai, Y.-C. F. Wang, and M. Sun. No more discrimination: Cross city adaptation of road scene segmenters. In ICCV , 2017.

B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1290-1299, 2022.

B. Cheng, A. G. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS , 2021.

B. Coors, A. P. Condurache, and A. Geiger. Nova: Learning to see in novel viewpoints and domains. In 2019 International Conference on 3D Vision (3DV) , pages 116-125, 2019.

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR , 2016.

D. Di Mauro, A. Furnari, G. Patanè, S. Battiato, and G. M. Farinella. Sceneadapt: Scene-based domain adaptation for semantic segmentation using adversarial learning. Pattern Recognition Letters , 136:175-182, 2020.

J. Ding, N. Xue, G.-S. Xia, and D. Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11583-11592, 2022.

S. Ettedgui, S. Abu-Hussein, and R. Giryes. Procst: Boosting semantic segmentation using progressive cyclic style-transfer, 2022.

M. Fahes, T.-H. Vu, A. Bursuc, P. Pérez, and R. de Charette. Poda: Prompt-driven zero-shot domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 18623-18633, 2023.

P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision , pages 1-15, 2023.

G. Ghiasi, X. Gu, Y. Cui, and T.-Y. Lin. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision , pages 540-557. Springer, 2022.

B. Gong, Y. Shi, F. Sha, and K. Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 2066-2073, 2012.

G. Goyal, N. Noceti, and F. Odone. Cross-view action recognition with small-scale datasets. Image and Vision Computing , 120:104403, 2022.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , 2016.

J. Hoffman, E. Tzeng, T. Park, J.-Y . Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In ICML , 2018.

C.-A. Hou, Y.-R. Yeh, and Y.-C. F. Wang. An unsupervised domain adaptation approach for cross-domain visual classification. In 2015 12th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) , pages 1-6. IEEE, 2015.

L. Hoyer, D. Dai, and L. Van Gool. DAFormer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In CVPR , 2022.

L. Hoyer, D. Dai, and L. Van Gool. HRDA: Context-aware high-resolution domain-adaptive semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022.

L. Hoyer, D. Dai, H. Wang, and L. Van Gool. Mic: Masked image consistency for contextenhanced domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11721-11732, 2023.

J. Huang, D. Guan, A. Xiao, and S. Lu. Cross-view regularization for domain adaptive panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10133-10144, 2021.

C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning , pages 4904-4916. PMLR, 2021.

K.-H. Lee, G. Ros, J. Li, and A. Gaidon. SPIGAN: Privileged adversarial learning from simulation. In ICLR , 2019.

B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations , 2022.

L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang, K.-W. Chang, and J. Gao. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10965-10975, 2022.

F. Liang, B. Wu, X. Dai, K. Li, Y . Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu. Openvocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7061-7070, 2023.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Y. Lyu, G. Vosselman, G.-S. Xia, A. Yilmaz, and M. Y. Yang. Uavid: A semantic segmentation dataset for uav imagery. ISPRS Journal of Photogrammetry and Remote Sensing , 165:108 119, 2020.

Z. Murez, S. Kolouri, D. Kriegman, R. Ramamoorthi, and K. Kim. Image to image translation for domain adaptation. In CVPR , 2018.

H.-Q. Nguyen, T.-D. Truong, X. B. Nguyen, A. Dowling, X. Li, and K. Luu. Insect-foundation: A foundation model and large-scale 1m dataset for visual insect understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21945-21955, 2024.

B. Pan, J. Sun, H. Y. T. Leung, A. Andonian, and B. Zhou. Cross-view semantic segmentation for sensing surroundings. IEEE Robotics and Automation Letters , 5(3):4867-4873, 2020.

F. Pan, I. Shin, F. Rameau, S. Lee, and I. S. Kweon. Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In CVPR , 2020.

J. Qin, J. Wu, P. Yan, M. Li, R. Yuxi, X. Xiao, Y . Wang, R. Wang, S. Wen, X. Pan, et al. Freeseg: Unified, universal and open-vocabulary image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19446-19455, 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 87488763. PMLR, 2021.

Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang, J. Zhou, and J. Lu. Denseclip: Languageguided dense prediction with context-aware prompting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

K. Regmi and M. Shah. Bridging the domain gap for ground-to-aerial image matching. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 470-479, 2019.

H. Ren, Y. Yang, H. Wang, B. Shen, Q. Fan, Y. Zheng, C. K. Liu, and L. Guibas. Adela: Automatic dense labeling with attention for viewpoint shift in semantic segmentation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8069-8079, 2022.

S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games. In ECCV , 2016.

G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez. The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR , 2016.

Y. Shi, L. Liu, X. Yu, and H. Li. Spatial-aware feature aggregation for image based cross-view geo-localization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch'e-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 10090-10100. Curran Associates, Inc., 2019.

Y. Shi, X. Yu, D. Campbell, and H. Li. Where am i looking at? joint location and orientation estimation by cross-view matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2020.

C. Simon, P. Koniusz, and M. Harandi. On learning the geodesic path for incremental learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , pages 1591-1600, 2021.

N. Sun, Q. Lu, W. Zheng, J. Liu, and G. Han. Unsupervised cross-view facial expression image generation and recognition. IEEE Transactions on Affective Computing , 2020.

A. Toker, Q. Zhou, M. Maximov, and L. Leal-Taixe. Coming down to earth: Satellite-to-street view synthesis for geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6488-6497, June 2021.

T.-D. Truong, C. N. Duong, A. Dowling, S. L. Phung, J. Cothren, and K. Luu. Crovia: Seeing drone scenes from car perspective via cross-view adaptation, 2023.

T.-D. Truong, C. N. Duong, N. Le, S. L. Phung, C. Rainwater, and K. Luu. Bimal: Bijective maximum likelihood approach to domain adaptation in semantic scene segmentation. In IEEE International Conference on Computer Vision (ICCV) , pages 8548-8557, 2021.

T.-D. Truong, P. Helton, A. Moustafa, J. D. Cothren, and K. Luu. Conda: Continual unsupervised domain adaptation learning in visual perception for self-driving cars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5642-5650, 2024.

T.-D. Truong, N. Le, B. Raj, J. Cothren, and K. Luu. Fredom: Fairness domain adaptation approach to semantic scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

T.-D. Truong and K. Luu. Cross-view action recognition understanding from exocentric to egocentric perspective. arXiv preprint arXiv:2305.15699 , 2023.

T.-D. Truong, H.-Q. Nguyen, B. Raj, and K. Luu. Fairness continual learning approach to semantic scene understanding in open-world environments. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS) , 2023.

T.-D. Truong, U. Prabhu, B. Raj, J. Cothren, and K. Luu. Falcon: Fairness learning via contrastive attention approach to continual semantic scene understanding in open world. arXiv preprint arXiv:2311.15965 , 2023.

V. Vidit, M. Engilberge, and M. Salzmann. Learning transformations to reduce the geometric shift in object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17441-17450, 2023.

T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In CVPR , 2019.

T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez. Dada: Depth-aware domain adaptation in semantic segmentation. In ICCV , 2019.

K. Wang, D. Kim, R. Feris, and M. Betke. Cdac: Cross-domain attention consistency in transformer for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11519-11529, 2023.

L. Wang, R. Li, C. Zhang, S. Fang, C. Duan, X. Meng, and P. M. Atkinson. Unetformer: A unet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery. ISPRS Journal of Photogrammetry and Remote Sensing , 190:196-214, 2022.

Q. Wang, W. Min, Q. Han, Z. Yang, X. Xiong, M. Zhu, and H. Zhao. Viewpoint adaptation learning with cross-view distance metric for robust vehicle re-identification. Information Sciences , 564:71-84, 2021.

E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS , 2021.

M. Xu, Z. Zhang, F. Wei, Y. Lin, Y. Cao, H. Hu, and X. Bai. A simple baseline for openvocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision , pages 736-753. Springer, 2022.

Z. Yan, X. Yu, Y. Qin, Y. Wu, X. Han, and S. Cui. Pixel-level intra-domain adaptation for semantic segmentation. In Proceedings of the 29th ACM International Conference on Multimedia , pages 404-413, 2021.

Y. Yao and H. S. Park. Multiview cross-supervision for semantic segmentation. arXiv preprint arXiv:1812.01738 , 2018.

Y. Yin, W. Hu, Z. Liu, G. Wang, S. Xiang, and R. Zimmermann. Crossmatch: Source-free domain adaptive semantic segmentation via cross-modal consistency training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21786-21796, 2023.

F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2636-2645, 2020.

P. Zhang, B. Zhang, T. Zhang, D. Chen, Y. Wang, and F. Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. arXiv preprint arXiv:2101.10979 , 2021.

Q. Zhang, W. Lin, and A. B. Chan. Cross-view cross-scene multi-view crowd counting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 557-567, 2021.

D. Zhao, S. Wang, Q. Zang, D. Quan, X. Ye, R. Yang, and L. Jiao. Learning pseudo-relations for cross-domain semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19191-19203, 2023.

B. Zhou and P. Krähenbühl. Cross-view transformers for real-time map-view semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13760-13769, 2022.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV , 2017.

S. Zhu, M. Shah, and C. Chen. Transgeo: Transformer is all you need for cross-view image geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1162-1171, 2022.

1 Proof of Eqn. (9)

As shown in our Eqn. (16), our Geodesic Flow-based metrics have the upper bound as follows:

In addition, as D x is the distance metric, this metric should satisfy the following triangular inequality as follows:

Similarly, D y should satisfy the following triangular inequality as follows:

Then, from Eqn. (16) and Eqn. (17) above, we can further derive as follows:

Since α is the constant linear scale value, therefore, we can further derive as follows:

2 Implementation

We follow the implementation of Mask2Former [8] and FreeSeg [38] with ResNet [20] and Swin backbones [32] for our segmentation network. In particular, we adopt Mask2Former with Semantic Context Interaction of FreeSeg [38] for our open-vocab segmentation network. We use the pre-trained text encoder of CLIP [39]. The textual features f p s and f p t are obtained by the CLIP textual encoder. Following common practices [38, 31], we adopt the open-vocab segmentation loss of FreeSeg [38] to our supervised loss L Mask . For experiments without prompting, we use the Mask2Former network. Following the UAV protocol of [61], the image size is set to 512 × 512 . The linear scale factors α and γ are set to α = 1 . 5 and γ = 1 . 0 , respectively. For the Geodesic Flow modeling, we adopt the implementation of generalized SVD decomposition [18, 47] in the framework. The subspace dimension in our geodesic flow-based metrics is set to D = 256 . The batch size and the base learning rate in our experiments are set to 16 and 2 . 5 × 10 -4 . The balanced weights of losses in our experiments are set to λ I = 1 . 0 and λ P = 0 . 5 . During training, the classes in the prompts are generated similarly for both view images.

In our Geodesic Flow-based metrics, the subspaces of images and ground-truth segmentation of the source domain are pre-computed on the entire data. For the language space, we compute the subspaces of each view based on the textual feature representations of all possible prompts in each domain. Meanwhile, the subspaces of the segmentation on the target domain are computed based on the current batch of training. For the implementation of DenseCLIP [40] and FreeSeg [38] with AdvEnt [58], we perform the adaptation process on the mask predictions. Meanwhile, we adopt the pseudo labels and the self-supervised framework of SAC[1] for the implementation of DenseCLIP [40] and FreeSeg [38] with SAC [1].

Appendix

3 Ablation Study

Effectiveness of Batch Size Table 8 illustrates the impact of the batch size on the performance of cross-view domain adaptation. By increasing the batch size, the mIoU performance is also increased accordingly on both benchmarks. This result has illustrated that the small batch size could not have enough samples to approximate the subspace that represents geometric structures. Meanwhile, the subspace created from the large batch size will be ale to capture the geometric structure of drone-view scenes. However, due to the limitation of GPU resources, we could not evaluate the cross-view adaptation model with larger batch size.

Subspace Representation of Geodesic Flow-based Metrics To illustrate the ability of structural learning of our geodesic flow-based metrics, we use a subset of images of the car-view and the drone-view dataset to visualize the base structure of subspaces obtained from the PCA algorithm. Fig. 7 visualizes the mean structures of car-view and drone-view images. As shown in Fig. 7, The subspaces of car-view images represent the geometric structures of car-view data, i.e., the road in the middle, buildings, trees on two sides, etc. Meanwhile, the geometric structures of the drone view have also been illustrated in the figure with structures and topological distributions of objects (e.g., the road in the middle and trees and buildings on the sides) on the scenes. The results have illustrated the base geometric structures of the car-view and the drone-view data. Then, by modeling the geodesic flow path across two subspaces, our metric is able to measure the cross-view geometric structural changes (i.e., the change of structures and topological layouts of the scene) from the car view to the drone view. Our experimental results in other ablation studies have further confirmed our effectiveness in geometric structural modeling across views. Figure 8 illustrates the feature distributions with and without our proposed approach. As shown in Figure 8, our approach can help to improve the feature representations of classes, and the cluster of each class is more compact, especially in classes of car, tree, and person.

4 Discussion of Limitations and Broader Impact

Limitations. In our paper, we have specified a set of hyper-parameters and network designs to support our hypothesis and theoretical analysis. However, our proposed approach could potentially consist of several limitations. First, our work focuses on studying the impact of cross-view geometric adaptation loss and view-condition prompting mechanisms on the segmentation models across views. The balanced weights among weights, i.e., λ I and λ P , have not been fully exploited. We leave this investigation as our future experiments. Second, although the datasets and benchmarks used in our

StructureofCar-ViewImage

StructureofDrone-Viewlmage

WithoutAdaptation

WithCross-viewAdaptation

Road

Building

15

20

Car

Tree

Person

10

10

0

0

-10

-5

-20

-10

-30

-40

-20

0

20

40

7.5

-5.0

2.5

0.0

2.5

5.0

7.5

10.0

12.5

experiments have sufficiently illustrated the effectiveness of our proposed cross-view adaptation learning approach, the lack of diverse classes and categories in datasets is also a potential limitation. Third, the hypothesis of the linear relations across views of images and segmentation mask, i.e., α , and textual representations and segmentation masks, i.e., γ , could limit the performance of the relation. The non-trivial relations across views should be deeply exploited in future research. Also, while the implementation of Mask2Former and FreeSeg is adopted to develop our approach, the experiments with other open-vocab segmentation networks should be considered in subsequent research studies. These aforementioned limitations will motivate new studies to further improve the methodology, datasets, and benchmarks of the cross-view adaptation learning paradigm.

Broader Impact. Our paper could bring significant potential for various applications that require learning across camera viewpoints. Our approach enables generalizability across camera views, thus enhancing the robustness of the segmentation model across views. In addition, our approach helps to reuse off-the-shelf large-scale data while reducing the effort of manually labeling data of new camera views.

5 Other Related Work

While the important and closely related work to our approach has been presented in our main paper, we also would like to review some other research studies that are related to our method as follows. In particular, Brady et al. [72] presented a cross-view transformer that learns the camera-aware positional embeddings. Although the views are captured from left and right angles, the camera positions in the approach remain at the same altitude. Similarly, Pan et al. [36] present a View Parsing Network to accumulate features across first-view observations with multiple angles. Yao et al. [66] proposed a semi-supervised learning approach to learn the segmentation model from multiple views of an image. Huang et al. [26] a cross-style regularization for domain adaptation in panoptic segmentation by imposing the consistency of the segmentation between the target images and stylized target images. Wang et al. [62] proposed a viewpoint adaptation framework for the person re-identification problem by using the generative model to generate training data across various viewpoints. Hou et al. [22] presented a matching cross-domain data approach to domain adaptation in visual classification. Sun et al. [48] proposed a cross-view facial expression adaptation framework to parallel synthesize and recognize cross-view facial expressions. Goyal et al. [19] introduced a cross-view action recognition approach to transferring the feature representations to different views. Zhang et al. [70] proposed a multi-view crowd counting approach that adaptively chooses and aggregates multi-cameras and a noise view regularization. Armando et al. [2] proposed a self-supervised pre-training approach to human understanding learned on pairs of images captured from different viewpoints. Then, the pre-trained models are later used for various downstream human-centric tasks. In summary, these prior cross-view methods could require either a pair of cross-view images [2] or images captured at the same altitude with different angles [26, 72, 22]. In addition, the cross-view geometric correlation modeling has not been exploited in these prior studies [26, 72, 22, 2]

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims declared in the abstract match with the contributions, experimental results, and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the paper are discussed in the last section of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proof of formula is provided in the supplementary.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The details of datasets and implementations are presented in the experimental sections.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The code will be published may the paper be accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The details of training and testing are presented in the experimental section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Following the standard evaluation of semantic segmentation, we evaluate our model by the standard mIoU metrics instead of the statistical tests.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computational resources used in our experiments are presented in the experimental section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The content of the paper and datasets strictly follows the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper does not have a negative societal impact. The broader impact is discussed in the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not have a risk. The released models will be available may the paper be accepted.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper provides all the references to code, data, and models used in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce the new dataset. The code of the paper will be published may the paper be accepted.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The research in this paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The research in this paper does not involve crowdsourcing nor research with human subjects. Thus, there is no requirement for IRB.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a domain adaptive semantic segmentation method under the cross-view (front view to top view) setting. It addresses this using vision-language models for additional supervision. It proposes a cross-view geometric constraint to model the structural changes and similarities between two vastly different viewpoint. The paper presents detailed quantitative results, with comparisons and ablations\n\nSTRENGTHS:\n1. The paper addresses an important and practical problem.\n\n2. The method proposed in the paper is well-tailored for the problem at hand, and appears to be solid. It seems intuitive and logical.\n\n3. The quantitative results presented clearly show the efficacy of the method.\n\nWEAKNESSES:\nMy main problem with the paper is the lack of qualitative results. The main paper barely has any qualitative results. The supplementary video shows just one specific video. I am unable to judge the performance of the method without good qualitative analysis.\n\nAs in all semantic segmentation papers, it would be good to see where prior work fails, and how each of the components presented in the method help the case. Essentially, it is important to provide qualitative results for the comparisons, the ablations, and as well as the method.\n\nGiven that the performance improvement as depicted by the quantitative results is large, the qualitative improvements should be clearly visible too, and hence it is beneficial to add those results.\n\nAlso, one main motivation for using a VLM is the open set setting (as described in the abstract). While described as a key point in the paper, none of the experimental benchmarks deal with that.\n\nQUESTIONS:\nSince it is not possible to provide qualitative results in the rebuttal, and since it not possible to properly validate the effectiveness of the proposed semantic segmentation method without qualitative results, I am rejecting the paper at this stage.\n\nLIMITATIONS:\nYes, in page 9", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method for Unsupervised Domain Adaption to adapt an open-vocabulary segmentation model across different views. To achieve this, the authors introduce a cross-view geometric constraint that captures structural changes between different views. Further, a Geodesic Flow-based Metric is introduced to measure the structural changes across scenes. They also adapt the prompting scheme to take into account the change in viewpoints.\n\nSTRENGTHS:\n- The paper feature a good range of contributions, which are intuitive and interesting\n- The proposed task of unsupervised cross-view adaptation is novel and seems useful\n- The idea is well motivated \n- The paper is well written and clearly explains the proposed concepts\n- Table 2 nicely outlines the impact of the individual technical contributions\n\nWEAKNESSES:\n- The layout of the paper seems a bit cramped. It would have been nice to have larger visualizations. Also, many of the tables, especially those in the ablations, seem super cramped and the reader has to zoom in significantly. I understand the authors wanted to put as much information as possible into the paper, but this led to a presentation that is not top notch.\n- Table 3 has no visual separation between the results for different classes, and overall mIoU. This visual separation (e.g. vertical line) would be nice to have.\n\nQUESTIONS:\nPersonally, I would appreciate if the authors could further improve the presentation of the paper. While the technical contributions are solid and the writing is nice, the visualizations and tables would benefit from being larger and better integrated with the text. For example, by moving Table 6 to the supplementary, the authors could free up space for the very relevant Table 5. I would appreciate an improved presentation, but also understand that this can be challenging with the given space constraints.\n\nLIMITATIONS:\nThe authors have sufficiently addressed the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a novel unsupervised adaptation method for modeling structural change across different views. Additionally, the paper introduced a new metric for cross-view changes and a new prompting mechanism for cross-view open vocabulary segmentation. Through extensive experiments, the paper shows SoTA performance compared to previous unsupervised methods in cross-view semantic scene understanding.\n\nSTRENGTHS:\n1.The proposed method can be trained on unpaired data, enhancing its potential for practical applications.\n2.The paper proposed a new cross-view change modeling method via geodesic flow path, which effectively models the cross-view segmentation correlation.\n3.The newly introduced method demonstrated SoTA performance across various cross-view adaptation benchmarks.\n4.The paper provides thorough mathematical derivations and theoretical analysis, along with well-designed experiments.\n\nWEAKNESSES:\n1.The visualization results in this paper are limited. It is necessary to supplement more visualizations of segmentation results compared with the existing methods in the paper, which will help support the experimental conclusions.\n2.Generalizing the experiments to include more UAD datasets, such as SynDrone[1], UDD[2], and ICG Drone[3], would enhance the robustness and credibility of the results.\n3.The motivation behind the design of the adaptation loss should be explained more clearly.\n4.The performance improvement in open-vocab segmentation is not very significant.\n\n[1] SynDrone – Multi-modal UAV Dataset for Urban Scenarios. \n[2] Large-scale structure from motion with semantic constraints of aerial images.\n[3] ICG Drone Dataset.\n\nQUESTIONS:\n1.The performance of the building category in open-vocab segmentation on unseen classes is intriguing.\n2.The notation \"ν ∈ [0..1] → Π(ν)\" in line 241 is misleading.\n\nLIMITATIONS:\nThe authors discussed the limitation of this paper in the appendix, including the choice of hyperparameters, lack of more diverse class labels and flaws in their original mathematical hypothesis.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles a problem called cross-view semantic segmentation by using unsupervised domain adaptation methods. The cross-view means from the front-view to top-down view, ie, from car to drone. It recognizes the limitations of existing unsupervised domain adaptation and open-vocabulary semantic segmentation methods in handling geometric variations across different camera views. The paper demonstrates the effectiveness of the proposed approach through extensive experiments on cross-view adaptation benchmarks, achieving state-of-the-art performance compared to existing UDA methods.\n\nSTRENGTHS:\nThe authors propose a novel unsupervised cross-view adaptation approach. This approach includes a cross-view geometric constraint to model structural changes between images and segmentation masks, a geodesic flow-based correlation metric for efficient geometric comparison, and a view-condition prompting mechanism to enhance the capabilities.\n\n\nThe proposed method has significant gains as compared to the initial method CROVIA for car-to-drone scene segmentation UDA. Although this is not a new setting, the improvement compared to the baseline is good. \n\nThe experiments are conducted in three settings of the car-to-drone UDA, and it also includes ablation study to verify effect of different components.\n\nWEAKNESSES:\nThe comparison between the other view transformation methods and the proposed cross-view learning method is not included. \n\nMost of the UDA methods are tailored for the front-view street scenes, like AdvEnt, ProDA, SAC. More recent and advanced open-vocab semantic segmentation methods should be included in the experiment section. For example, x-decoder, cat-seg, clip-as-RNN, etc.\n\nQUESTIONS:\nHow is the difference of UDA settings between CROVIA and the proposed in this paper?\n\n\nIn the introduction, the authors claim that “However, the open-vocab perception models remain unable to generalize across camera viewpoints.” It would be better to show the limitation or performance along with this observation. For example, the model with and without using a cross-view setting. \n\n\nFrom eq 3 to 5, for the purpose of geometric adaptation on unpaired data, the problem is that the matrices between source and target domain data are not available. How about using estimated camera matrices for the geometric correlation in eq 3 and 4?\n\n\nThe cross-view learning framework is based on the geodesic flow. How about the comparison with other view transformation methods? For example, in the birds-eye-view semantic segmentation domain, there are different transformation methods, such as: \n[1] Pan, B., Sun, J., Leung, H. Y. T., Andonian, A., & Zhou, B. (2020). Cross-view semantic segmentation for sensing surroundings. IEEE Robotics and Automation Letters, 5(3), 4867-4873.\n\n\nIn Table 4, how about the comparison for BDD→ UAVID? Also, how about applying tye cross-view setting for other DA methods that are proposed for front-view? \n\n\nHow about using the state-of-the-art open-vocab methods? For example, x-decoder, cat-seg, clip-as-RNN, to name a few. \n\n[2] Zou, X., Dou, Z. Y., Yang, J., Gan, Z., Li, L., Li, C., ... & Gao, J. (2023). Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15116-15127).\n\n[3] Cho, S., Shin, H., Hong, S., Arnab, A., Seo, P. H., & Kim, S. (2024). Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4113-4123).\n\n[4] Sun, S., Li, R., Torr, P., Gu, X., & Li, S. (2024). Clip as rnn: Segment countless visual concepts without training endeavor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13171-13182).\n\n[5] Wysoczańska, M., Siméoni, O., Ramamonjisoa, M., Bursuc, A., Trzciński, T., & Pérez, P. (2023). Clip-dinoiser: Teaching clip a few dino tricks. arXiv preprint arXiv:2312.12359.\n\n\nIn Table 4, what is the reason that the method with DAFormer outperforms the one with Mask2Former. It is also very interesting why applying a larger or advanced model cannot obtain improvements. \n\n\nIn Remark 3, the grassmannn manifold is presented for the cross-view modeling method, how about the two subspaces of source and target domain? What is the difference of the feature distributions before and after domain adaptation, such as t-sne visualization? \n\n\nIt would be suggested to showcase the training process and the computational complexity of the proposed methods individually, such as in the ablation study. Apart from performance, training and runtime are also important for UAVs application.\n\nLIMITATIONS:\nThe target domain is limited in only one dataset. There are different UVA datasets public available. But this is not a very large concern, because the experiments are conducted with three different source domains.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}