# Domain Adaptation with a Single Vision-Language Embedding

Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, Raoul de Charette

► To cite this version:

Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, Raoul de Charette. Domain Adaptation with a Single Vision-Language Embedding. International Journal of Computer Vision, In press. hal-04986177

HAL Id: hal-04986177

https://inria.hal.science/hal-04986177v1

Submitted on 11 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
arXiv:2410.21361v1 [cs.CV] 28 Oct 2024

# Domain Adaptation with a Single Vision-Language Embedding

Mohammad Fahes¹*, Tuan-Hung Vu¹,², Andrei Bursuc¹,², Patrick Pérez³,
Raoul de Charette¹

¹Inria, Paris, France.
²Valeo.ai, Paris, France.
³Kyutai, Paris, France.

*Corresponding author(s). E-mail(s): mohammad.fahes@inria.fr;
Contributing authors: tuan-hung.vu@valeo.com; andrei.bursuc@valeo.com;
patrick@kyutai.org; raoul.de-charette@inria.fr;

## Abstract

Domain adaptation has been extensively investigated in computer vision but still requires access to target data at the training time, which might be difficult to obtain in some uncommon conditions. In this paper, we present a new framework for domain adaptation relying on a single Vision-Language (VL) latent embedding instead of full target data. First, leveraging a contrastive language-image pre-training model (CLIP), we propose prompt/photo-driven instance normalization (PIN). PIN is a feature augmentation method that mines multiple visual styles using a single target VL latent embedding, by optimizing affine transformations of low-level source features. The VL embedding can come from a language prompt describing the target domain, a partially optimized language prompt, or a single unlabeled target image. Second, we show that these mined styles (i.e., augmentations) can be used for zero-shot (i.e., target-free) and one-shot unsupervised domain adaptation. Experiments on semantic segmentation demonstrate the effectiveness of the proposed method, which outperforms relevant baselines in the zero-shot and one-shot settings.

**Keywords:** Zero-shot domain adaptation, vision-language models, feature augmentation, style mining, prompt/photo-driven instance normalization

## 1 Introduction

The success of deep learning in image classification (Krizhevsky et al, 2012) has led to a swift paradigm shift over the past decade. This shift has propelled the rapid evolution of algorithms, with deep neural networks (DNNs) forming the backbone of nearly every modern computer vision method. For instance, supervised semantic segmentation methods have achieved remarkable success in improving high-resolution predictions (Long et al, 2015; Chen et al, 2017, 2018;

Cheng et al, 2020; Wang et al, 2020a), incorporating multi-scale processing (Zhao et al, 2017; Lin et al, 2017) and enhancing computational efficiency (Zhao et al, 2018). Yet, DNN-based methods are still far from being reliably applied in many scenarios for critical applications like real-world autonomous driving. The reason is that learning-based systems suppose the training and testing data to be independent and identically distributed (i.i.d.), a hypothesis that is violated most of the times as the training data tend to under-represent the true generative distribution.
Consequently, in controlled settings where seg-
mentation models are trained using data from the
targeted operational design domains, the accu-
racy can meet the high industry-level expectations
on in-domain data; yet, when tested on out-
of-distribution data, these models often undergo
drastic performance drops (Ovadia et al, 2019).

To mitigate the so-called “domain-shift” problem (Ben-David et al, 2010), unsupervised domain adaptation (UDA) (Ganin et al, 2016; Sun and Saenko, 2016; Hoffman et al, 2018; Zou et al, 2018; Tsai et al, 2018; Vu et al, 2019) has been proposed; and aims at training on labeled data from a source domain and unlabeled data from the target domain. This alleviates the need for annotating data, which is often laborious, taking 2 to 3 hours per image on average (Cordts et al, 2016; Sakaridis et al, 2021). Moreover, modern architectures can require massive amounts of annotated images (Kirillov et al, 2023). UDA is thus seen as a label-efficient framework for dealing with domain shifts.

Despite its apparent simplicity, even gathering
unlabeled data can be challenging under cer-
tain conditions. For example, as driving through
fire or sandstorm rarely occurs in real life and
can be dangerous, capturing these conditions is
not trivial. One may argue on using internet
images for UDA. However, in the industrial con-
text, the practice of using public data is limited
or forbidden. Recent works aim to reduce the
burden of target data collection campaigns by
devising one-shot unsupervised domain adapta-
tion (OSUDA) (Luo et al, 2020; Wu et al, 2022)
methods, i.e., using one target image for train-
ing, which substitutes label-efficiency by a more
challenging data-efficiency setting.

In this paper, we frame the challenging new
task of domain adaptation using a single Vision-
Language (VL) latent embedding. At training
time, our method adapts the segmentation model
to the domain defined by either a single descrip-
tion in natural language (i.e., a *prompt*), a par-
tially optimized prompt, or a single unlabeled
target image. This makes the method suitable for
zero-shot domain adaptation when the embedding
comes from a prompt, and for one-shot unsu-
pervised domain adaptation (OSUDA) when it
comes from an unlabeled target image. Since our
approach leverages a single VL target embedding
which may originate from a prompt or an image,

<figure><img src="image_2.png" /><figcaption>**Fig. 1:** Domain adaptation with a single VL embedding. The proposed framework enables the adaptation of a segmenter model (here, DeepLabv3+ trained on the source dataset Cityscapes) to unseen conditions with only one embedding vector in shared VL space. (top) PØDA leverages a single text prompt. (bottom-left) PØDA-concept utilizes a prompt where the concept S* is optimized from the source images and conditions remain textually described. (bottom-right) PIDA adapts the model using a single unlabeled target image. Source-only predictions are shown as smaller segmentation masks to the left or right of the test images.</figcaption></figure>

we coin it *Prompt-driven Zero-shot Domain Adaptation* (PØDA) when using a fully verbalized target prompt, *PØDA-concept* when a part of the target prompt is optimized using source images, and *Photo-driven one-shot Domain Adaptation* (PIDA<fnref n="1" />) when using a target image.

Fig. 1 outlines the primary goal of our work with a few qualitative examples. Without seeing any fire or old movie images during training, and using a simple description of these conditions with a language prompt (i.e., PØDA), the adapted models succeed in segmenting out critical scene objects, exhibiting fewer errors than the original source-only model. Fig. 1 additionally shows the improvements brought by our framework when part of the prompt is optimized from source data (i.e., PØDA-concept) or when the VL embedding comes from an unlabeled target image (i.e., PIDA).

Our method, illustrated in Fig. 2, is made
possible by leveraging the VL connections from
the seminal CLIP model (Radford et al, 2021).
Trained on 400M web-crawled image-text pairs,

<footnote n="1">¹In PIDA the “I” stands for the roman number 1.</footnote>
CLIP has revolutionized multi-modal representation learning, bringing outstanding transfer capability to tasks such as image synthesis (Kwon and Ye, 2022; Gal et al, 2022; Patashnik et al, 2021), multi-modal fusion (Jatavallabhula et al, 2023), semantic segmentation (Li et al, 2022; Zhou et al, 2022a), few-shot learning (Zhou et al, 2022c; Gao et al, 2024; Fahes et al, 2024a), as well as open-vocabulary object detection (Minderer et al, 2022).

Our work exploits CLIP latent space and proposes a simple and effective method that converts source-domain embeddings into target-domain ones (Fig. 2, left), by optimizing style-specific components of low-level features. This procedure can be seen as a specific form of feature augmentation utilizing a single VL embedding guidance coming from either a prompt or an image. Fine-tuning the segmentation model using the optimized styles (Fig. 2, middle) helps mitigating the distribution gap between the two domains, thus improving the performance on target domains (Fig. 2, right).

This paper extends our previous work (Fahes et al, 2023), PØDA, and proposes a more general formulation. The extension is threefold. (i) We further analyze the effectiveness of relevant prompts in PØDA showing that it originates from the combined description of content (e.g., “driving”) and style (e.g., “at night”). (ii) Inspired by prompt learning (Zhou et al, 2022c,b), we show that rather than encompassing content and style in the prompt, the content concept $S^*$ can be optimized from source data and combined with a textual style (e.g., “$S^*$ at night”). (iii) Since our framework requires only a single VL embedding, we propose PIDA, which utilizes an unlabeled image instead of a prompt, thus allowing adaptation to conditions that are hard to describe with a textual prompt. These novel contributions not only improve the original PØDA but also extend its scope of application.

Overall, our contributions are the following:

* We define a novel task of domain adaptation, which aims at adapting a source-trained model to a target domain provided *only* a single VL embedding, and demonstrate its effectiveness with either a prompt or an image.

* Unlike other CLIP-based methods that navigate CLIP latent space using direct image representations, we alter only the features, without relying on the appearance in the pixel space. We argue that this is particularly useful for downstream tasks such as semantic segmentation, where good features are decisive for the performance of the segmentation head. Specifically, we introduce prompt/photo-driven instance normalization (PIN), which optimizes affine transformations of the low-level source features such that their embeddings match that of a single image/text condition describing the unseen target domain.
* We propose Prompt-driven Zero-shot Domain Adaptation (Fahes et al, 2023) (PØDA), which utilizes a *single prompt* to describe the content and style of the target condition, demonstrating its performance on a wide variety of scenarios: (i) from clear weather/daytime to adverse conditions (snow, rain, night), (ii) from synthetic to real, (iii) from real to synthetic.
* We introduce the novel PØDA-concept, building on a *concept optimization* strategy to relax the necessity of describing the content of the source images using language. Experiments show that when the optimized concept is coupled with a visual appearance (i.e., style) description, the downstream performance is significantly improved.
* We introduce PIDA which instead utilizes the VL embedding of a *single image* to adapt a trained model. For peculiar visual conditions which are difficult to describe with text prompts, we demonstrate that it further improves the performance of our method.

## 2 Related works

**Unsupervised Domain Adaptation (UDA).** The UDA literature is vast and encompasses different yet connected approaches: adversarial learning (Ganin et al, 2016; Tsai et al, 2018), self-training (Zou et al, 2019; Li et al, 2019), entropy minimization (Vu et al, 2019; Pan et al, 2020), generative-based adaptation (Hoffman et al, 2018), etc. The domain gap is commonly reduced at the level of the input (Hoffman et al, 2018; Yang and Soatto, 2020), of the features (Ganin et al, 2016; Sun and
Saenko, 2016; Wang et al, 2017; Long et al, 2018) or of the output (Tsai et al, 2018; Vu et al, 2019; Pan et al, 2020).

Recently, the more challenging setting of One-Shot Unsupervised Domain Adaptation (OSUDA) has been proposed. Luo et al (2020) show that traditional UDA methods fail when only a single unlabeled target image is available. To mitigate the risk of overfitting on the style of the single available image, the authors propose a style mining algorithm, based on both a stylized image generator and a task-specific module. Wu et al (2022) introduce an approach based on style mixing and patch-wise prototypical matching (SM-PPM). During training, channel-wise mean and standard deviation of a randomly sampled source image's features are linearly mixed with the target ones. Patch-wise prototypical matching helps addressing negative adaptation (Li et al, 2020).

In the more challenging zero-shot setting (where no target image is available), Lengyel et al (2021) tackle day-to-night domain adaptation using physics priors. They introduce a color invariant convolution layer (CICConv) that is added to make the network invariant to different lighting conditions. We note that this zero-shot adaptation method is orthogonal to ours and is restricted to a specific type of domain gap.

**Text-driven image synthesis.** Recently, contrastive vision-language pretraining has shown unprecedented success for multimodal learning in several downstream tasks such as zero-shot classification (Radford et al, 2021), multi-modal retrieval (Jia et al, 2021) and visual question answering (Li et al, 2021). This encouraged the community to edit images using text descriptions, a task that was previously challenging due to the gap between vision and language representations. For example, StyleCLIP (Patashnik et al, 2021) uses prompts to optimize StyleGAN (Karras et al, 2019) latent vectors and guide the generation process. However, the generation is limited to the training distribution of StyleGAN. To overcome this issue, StyleGAN-NADA (Gal et al, 2022) utilizes CLIP embeddings of text prompts to perform domain adaptation of the generator, which is in this case trainable.

For text-guided style transfer, CLIP-styler (Kwon and Ye, 2022) does not rely on a generative process. This setting is more realistic

for not being restricted to a specific distribution, and challenging at the same time for the use of the encapsulated information in CLIP latent space. Indeed, there is no one-to-one mapping between image and text representations and regularization is needed to extract the useful information from a text embedding. Thus, in the same work (Kwon and Ye, 2022), a U-net (Ronneberger et al, 2015) autoencoder that preserves the content is optimized while the output image embedding in CLIP latent space is varying during the optimization process.

We note that a common point in prior works is the mapping from pixel-space to CLIP latent space during the optimization process. In contrast with this, we directly manipulate shallow features of the pre-trained CLIP visual encoder.

**Prompt Learning.** Inspired by prompt embedding optimization and automatic prompt generation in natural language processing (Shin et al, 2020; Jiang et al, 2020; Zhong et al, 2021), Zhou et al (2022c) apply prompt learning for context optimization in VL models, and show that it outperforms prompt engineering in the few-shot setting for different image recognition downstream tasks. CoOp (for Context Optimization) (Zhou et al, 2022c) consists of replacing the prompt context by learnable vectors, which are optimized by minimizing a classification loss using few labeled support images. The weights of both the image and text encoders are frozen during this process. Subsequently, CoCoOp (Zhou et al, 2022b) adds a lightweight neural network that predicts input-conditional tokens, and improves the performance on unseen classes compared to CoOp. Instead of learning a single prompt, ProDA (Lu et al, 2022) estimates the distribution of prompts in the output embedding space, i.e., modeling the classifier weights distribution by a multivariate Gaussian. Such a strategy handles the variability of visual representations.

In this work, the embeddings of the source training images are used to optimize their common concept in the language space. This differs from strategies like (Zhou et al, 2022c) which fix the class name and learn the context in a supervised way using a cross-entropy loss. Our goal is to “search” in the text space for a representation that expresses the meta-domain common to the integrity of our source images, i.e., supposing that,
<figure><img src="image_3.png" /><figcaption>Fig. 2: Overview our framework of domain adaptation using a single VL embedding. (Left) Using only a single VL embedding representing a target domain (i.e., target prompt + or target image ▲), we leverage a frozen ResNet encoder with CLIP weights to optimize source→target low-level feature affine transformations saved in a style bank. (Middle) Zero-shot/One-shot unsupervised domain adaptation is achieved by fine-tuning a segmenter model (M) on features that are augmented using the learned transformations, here $f_{s→night}$. (Right) This enables inference on target domains.</figcaption></figure>

globally, all of them share the same semantic class (i.e., driving scenes).

**Normalization in Deep Learning of CNNs.**
To accelerate and stabilize the training of CNNs, Batch Normalization (BN) was proposed in (Ioffe and Szegedy, 2015). During mini-batch training, BN operation consists of normalizing the batch of features channel-wise and applying a learnable affine transformation. Specifically, BN reads as:

$$ \text{BN}(\mathbf{f}) = \gamma \left( \frac{\mathbf{f} - \mu_c(\mathbf{f})}{\sigma_c(\mathbf{f})} \right) + \beta, \quad (1) $$

where $\mathbf{f} \in \mathbb{R}^{B \times C \times H \times W}$ and $B, C, H, W$ are respectively the batch size, the number of channels, the height and the width of the feature activation; $\gamma, \beta \in \mathbb{R}^C$ are learnable parameters; $\mu_c(\cdot)$ and $\sigma_c(\cdot)$ are two functions that return the mean and standard deviation for each channel, computed across the remaining dimensions of $\mathbf{f}$. Specifically, for a channel $c$:

$$ \mu_c(\mathbf{f}) = \frac{1}{BHW} \sum_{b=1}^{B} \sum_{h=1}^{H} \sum_{w=1}^{W} \mathbf{f}_{bchw}, \quad (2) $$

$$ \sigma_c(\mathbf{f}) = \sqrt{\frac{1}{BHW} \sum_{b=1}^{B} \sum_{h=1}^{H} \sum_{w=1}^{W} (\mathbf{f}_{bchw} - \mu_c(\mathbf{f}))^2 + \epsilon}. \quad (3) $$

$\epsilon$ is a small positive constant used for numerical stability (e.g., $\epsilon = 10^{-5}$). The effectiveness of BN was initially associated to its effect on reducing internal covariate shift, while following research rather argued for its role in smoothing the

optimization landscape (Santurkar et al, 2018). Ulyanov et al (2016, 2017) show that replacing BN by Instance Normalization (IN) improves image stylization. IN reads as:

$$ \text{IN}(\mathbf{f}) = \gamma \left( \frac{\mathbf{f} - \mu(\mathbf{f})}{\sigma(\mathbf{f})} \right) + \beta, \quad (4) $$

with $\mu(\cdot)$ and $\sigma(\cdot)$ computed across the spatial dimension for each channel $c$ of an instance $b$ in a mini-batch:

$$ \mu_{bc}(\mathbf{f}) = \frac{1}{HW} \sum_{h=1}^{H} \sum_{w=1}^{W} \mathbf{f}_{bchw}, \quad (5) $$

$$ \sigma_{bc}(\mathbf{f}) = \sqrt{\frac{1}{HW} \sum_{h=1}^{H} \sum_{w=1}^{W} (\mathbf{f}_{bchw} - \mu_c(\mathbf{f}))^2 + \epsilon}. \quad (6) $$

Leveraging the relation between low-level feature statistics and image style, Adaptive Instance Normalization (AdaIN) (Huang and Belongie, 2017) transfers style-specific components across features. In AdaIN, the styles are represented by the channel-wise mean $\mu(\mathbf{f}) \in \mathbb{R}^C$ and standard deviation $\sigma(\mathbf{f}) \in \mathbb{R}^C$ of features. Stylizing a source feature $\mathbf{f}_s$ with an arbitrary target style ($\mu(\mathbf{f}_s), \sigma(\mathbf{f}_s)$) reads:

$$ \text{AdaIN}(\mathbf{f}_s, \mu(\mathbf{f}_t), \sigma(\mathbf{f}_t)) = \sigma(\mathbf{f}_t) \left( \frac{\mathbf{f}_s - \mu(\mathbf{f}_s)}{\sigma(\mathbf{f}_s)} \right) + \mu(\mathbf{f}_t), \quad (7) $$

where $\mu(\cdot)$ and $\sigma(\cdot)$ are defined as in Eq. (5) and Eq. (6). Note that there are no learnable parameters in Eq. (7).
These normalization methods, sharing similar mathematical formulations, have all different goals. In this paper, we propose prompt/photo-driven instance normalization (PIN), which is inspired by AdaIN, yet supposes having no access to $\mu(\mathbf{f}_t)$ and $\sigma(\mathbf{f}_t)$.

# 3 Domain Adaptation with a single VL Embedding

Our framework, illustrated in Fig. 2, builds upon CLIP (Radford et al, 2021), a Vision-Language model pre-trained on 400M image-text pairs crawled from the Internet. CLIP trains jointly an image encoder $E_{\text{img}}$ and a text encoder $E_{\text{txt}}$ to learn an expressive representation space that effectively bridges the two modalities. We use such bimodal space in this work to bring image embeddings from a source domain closer to a target domain defined by a single VL embedding, coming either from a textual description (i.e., zero-shot) or an unlabeled target image (i.e., one-shot). This is done by optimizing the style-specific components of the source low-level features, with the final values of these components corresponding to augmented data.

**For zero-shot**, we explore two types of guidance without access to target data: using either a full textual prompt or a textual prompt combined with an optimized concept. For the former, we explore the use of simple general prompts describing the target domain, e.g., “driving at night” for the night domain. Because simple prompts may not be sufficient to represent complex semantics of scenes, we also suggest optimizing a concept $S^*$ from the source images, inspired by context optimization (Zhou et al, 2022c). To form the target prompt, this concept is complemented with our target description, e.g., “$S^*$ at night”. Our zero-shot domain adaptation framework can interchangeably leverage any of these guidance signals.

**For one-shot**, we simply replace the single language embedding by the embedding of a single unlabeled target image. The optimization thus operates in the image embedding space, i.e., the space defined by the image encoder. We later show that this setting might be useful when the target domain conditions are nuanced and not trivial to verbalize.

Overall, our goal is to optimize affine transformations of low-level source features so that the embeddings of these features become closer to their imaginary counterparts in the targeted domain (Fig. 2, left), while crucially preserving their semantic content. The learned augmentations can then be applied to synthesize, in a zero-shot/one-shot fashion, features that correspond to the target domain and can subsequently be used to fine-tune the model (Fig. 2, middle). This ultimately allows inference on domains only described by either a single prompt or an image at training time (Fig. 2, right).

Our approach faces several challenges: (i) How to “mine” style information relative to a target domain using only one VL embedding? (ii) How to preserve pixel-wise semantics (i.e., content) during style mining? (iii) Based on such mined styles, how to adapt the source model to the target domain? We address these questions in the following.

**Problem formulation.** Our main task is semantic segmentation, that is, pixel-wise classification of input images into semantic segments. We start from a K-class segmentation model $M$, trained on a labeled source domain dataset $\mathcal{D}_s = \{(\mathbf{I}_s, \mathbf{y}_s) \mid \mathbf{I}_s \in \mathbb{R}^{H \times W \times 3}, \mathbf{y}_s \in \{0, 1\}^{H \times W \times K}\}$, where $\mathbf{I}_s$ represents an image and $\mathbf{y}_s$ its ground-truth annotation. The segmenter $M$ is composed of a CLIP image encoder $E_{\text{img}}$ (e.g., ResNet-50) as the frozen feature extractor backbone $M_{\text{feat}}$ and a randomly initialized pixel classification head $M_{\text{cls}}: M = (M_{\text{feat}}, M_{\text{cls}})$. Our goal is to adapt the model $M$ such that its performance on a test target dataset $\mathcal{D}_t = \{\mathbf{I}_t \mid \mathbf{I}_t \in \mathbb{R}^{H \times W \times 3}\}$ is improved, provided solely a single VL latent embedding as information about the target domain. This is enabled by access to a target embedding, $\mathbf{f}_t$, which is derived either from a prompt using the frozen text encoder, or from a single unlabeled target image using the frozen image encoder.

We train $M$ in a supervised manner for the semantic segmentation task on the source domain. Interestingly, we empirically show in Table 1 that keeping the feature extractor $M_{\text{feat}}$ frozen helps alleviate the overfitting risk to the source domain in favor of generalization; not a universal finding, this observation is limited to our experimental setting specified in Section 4.1. To minimize the
<table><caption>**Table 1: Segmentation with source-only trained models.** Performance (mIoU %) on “night”, “snow” and “rain” parts of ACDC (Sakaridis et al, 2021) validation set and on a subset of 1000 GTA5 images for models trained on Cityscapes (‘CS’). ‘M<sub>feat</sub>*’: frozen backbone.</caption><thead><tr><th>M<sub>feat</sub> *</th><th>CS</th><th>Night</th><th>Snow</th><th>Rain</th><th>GTA5</th></tr></thead><tbody><tr><td>Yes</td><td>66.82</td><td><b>18.31</b></td><td><b>39.28</b></td><td><b>38.20</b></td><td><b>39.59</b></td></tr><tr><td>No</td><td><b>69.17</b></td><td>14.40</td><td>22.27</td><td>26.33</td><td>32.91</td></tr></tbody></table>

interference of generalization effects caused by different training strategies in the adaptation results, we systematically freeze $M_{feat}$ when training on source and later fine-tuning for adaptation.

From the extractor, we remove the attention pooling head of $E_{img}$ to keep the spatial information for the pixel classifier. We denote $\mathbf{f}$ the intermediate feature activations extracted by $M_{feat}$ and $\bar{\mathbf{f}}$ their corresponding CLIP embeddings. In Fig. 4 we illustrate the difference between $\mathbf{f}$ and $\bar{\mathbf{f}}$.

**Overview of the proposed method.** Our solution is to mine styles (i.e. channel-wise statistics of the features) using source-domain low-level features set $\mathcal{F}_s=\{\mathbf{f}_s|\mathbf{f}_s=\text{feat-ext}(M_{feat}, \mathbf{I}_s)\}$ and $\bar{\mathbf{f}}_t$. For generality, $\text{feat-ext}(\cdot)$ can pull features from any desired layer. However, we later show that using the features from the earliest layers yields the best results.

The augment(·) operation, depicted in Fig. 4, augments the style-specific components of $\mathbf{f}_s$ with guidance from $\bar{\mathbf{f}}_t$, synthesizing $\mathbf{f}_{s\to t}$ with style information from the target domain. We emphasize that the features $\mathbf{f}_s$ and $\mathbf{f}_{s\to t}$ have the same size and identical semantic content, though they encapsulate different visual styles. For adaptation, the source features $\mathbf{f}_s$ are augmented with the mined styles then used to fine-tune the classifier $M_{cls}$, leading to the final adapted model.

## 3.1 Instance Normalization guided by single VL embedding

We draw inspiration from Adaptive Instance Normalization (AdaIN) (Huang and Belongie, 2017), an elegant formulation for transferring style-specific components across features. AdaIN is defined in Eq. (7).

We design our augmentation strategy around AdaIN as it can effectively manipulate the style information with a small set of parameters. In the

following, we present our augmentation strategy which mines target styles.

### 3.1.1 Prompt-driven Instance Normalization

In the zero-shot setting, we suppose having no access to any image from the target domain. Consequently, $\mu(\bar{\mathbf{f}}_t)$ and $\sigma(\bar{\mathbf{f}}_t)$ in Eq. (7) are unknown. Yet, we suppose having a single text condition in natural language (i.e., a prompt P) describing the target domain. Thus, we propose Prompt-driven Instance Normalization (PIN) as

$$ \text{PIN}_{\bar{\mathbf{f}}_t}(\mathbf{f}_s, \mu, \sigma) := \sigma \left( \frac{\mathbf{f}_s - \mu(\mathbf{f}_s)}{\sigma(\mathbf{f}_s)} \right) + \mu, \quad (8) $$

where $\mu \in \mathbb{R}^C$ and $\sigma \in \mathbb{R}^C$ are optimizable variables driven by $\bar{\mathbf{f}}_t = E_{txt}(\mathbf{P})$.

We aim to augment source image features $\mathcal{F}_s$ such that they capture the style of the unseen target domain. Here, the prompt describing a target domain could be fairly generic. For instance, one can use prompts like “driving at night” or “driving under rain” to bring source features closer to the nighttime or rainy domains.

### 3.1.2 Concept optimization

In domain adaptation, despite the discrepancies between source and target data, they still share common characteristics (e.g., “driving” data). Interestingly, our experiments show that prompt-based adaptation performs better when the prompt includes a word describing the common characteristics (e.g., the word “driving” in “driving at night”, “driving under rain”, etc.), though this improvement comes at the cost of prompt engineering. To prevent this, we draw inspiration from CoOp (Zhou et al, 2022c) and seek to optimize a word embedding describing the common characteristics. While similar in spirit, our approach differs from CoOp in methodology and implementation as the latter requires access to target data and their labels. Instead, we optimize a concept using only source images and a textual style description as guidance (e.g., “in clear weather”).

In practice, we apply concept optimization in two steps. In the first step, we construct a prompt P in the form of <concept> + “in clear weather”, where <concept> represents optimizable parameters in the word embedding space. The
<figure><img src="image_4.png" /><figcaption>**Fig. 3: Concept optimization.** A <concept> is optimized in the word embedding space by means of cosine distance $\mathcal{L}_{<concept>}$, such that the text embedding gets closer to source image embeddings. The final value of this optimizable word embedding is denoted $S^*$.</figcaption></figure>

optimization is done using SGD with the following objective:

$$ \mathcal{L}_{<\text{concept}>} = 1 - \frac{E_{\text{txt}}(\mathbf{P}).E_{\text{img}}(\mathbf{I}_s)}{\|E_{\text{txt}}(\mathbf{P})\|\|E_{\text{img}}(\mathbf{I}_s)\|} \cdot (9) $$

The word embedding is optimized by minimizing the cosine similarity between the representations of the source images and the prompt representation. The final value of <concept> is denoted $S^*$. In the second step, the target prompt $P^*$ is constructed by concatenating $S^*$ with target-specific texts, such as “$S^*$ at night” or “$S^*$ under rain”. Our proposed concept optimization is illustrated in Fig. 3. We then apply PIN (Eq. (8)), though using as $\tilde{\mathbf{f}}_t$ the VL embedding of $P^*$, i.e., $\tilde{\mathbf{f}}_t = E_{\text{txt}}(P^*)$.

### 3.1.3 Photo-driven Instance Normalization

Within our framework, $\tilde{\mathbf{f}}_t$ is simply an embedding guiding the optimization of $\mu$ and $\sigma$. While we have explored the use of text to obtain $\tilde{\mathbf{f}}_t$, certain peculiar conditions may be challenging to describe this way. Alternatively, our framework also allows getting the VL embedding from a single unlabeled target image $\mathbf{I}_t$. In such case, PIN (Eq. (8)) is applied using the embedding of the target image (i.e., $\tilde{\mathbf{f}}_t = E_{\text{img}}(\mathbf{I}_t)$) as guidance. In this setting, PIN refers to *Photo-driven Instance Normalization*.

<figure><img src="image_5.png" /><figcaption>**Fig. 4: Target style mining from a source image.** We illustrate here the optimization loop of Algorithm 1. The source image is forwarded through the CLIP image encoder $E_{\text{img}}$ to extract low-level features $\mathbf{f}_s$ and subsequent CLIP embedding $\tilde{\mathbf{f}}_s$. At each optimization step $i$, augment($\cdot$) takes the style of the previous iteration, $(\mu^{i-1}, \sigma^{i-1})$ and injects it within $\mathbf{f}_s$ via the PIN layer, to synthesize $\tilde{\mathbf{f}}_s^i$ and the corresponding embedding $\tilde{\mathbf{f}}_s^i$. The loss $\mathcal{L}_{\mu, \sigma}$ is the cosine distance between $\tilde{\mathbf{f}}_s^i$ and the target embedding $\tilde{\mathbf{f}}_t$, which can be derived from a prompt (e.g., “driving at night”), a partially optimized prompt (e.g., “$S^*$ at night”), or an unlabeled target image. Its optimization via gradient descent updates style to $(\mu^i, \sigma^i)$.</figcaption></figure>

## 3.2 Style mining with PIN

In the sections above, we described how PIN can leverage either a target prompt (Section 3.1.1), the combination of a source concept and a target prompt (Section 3.1.2), or a target image (Section 3.1.3). We now detail how to mine a collection of styles.

We describe in Algorithm 1 the first step of our feature augmentation procedure: mining the set $S_{s \to t}$ of styles in targeted domain. For each source feature activation $\mathbf{f}_s \in \mathcal{F}_s$, we want to mine style statistics corresponding to an imaginary target feature activation $\mathbf{f}_t$. To this end, we formulate style mining as an optimization problem over the original source feature $\mathbf{f}_s$, i.e., optimizing $(\mu, \sigma)$ in Eq. (8). The optimization objective is defined as the cosine distance in the CLIP latent space between the CLIP embedding $\tilde{\mathbf{f}}_{s \to t}$ of the stylized feature $\mathbf{f}_{s \to t} = \text{PIN}(\mathbf{f}_s, \mu, \sigma)$ and the VL embedding $\tilde{\mathbf{f}}_t$ of target domain:

$$ \mathcal{L}_{\mu, \sigma}(\tilde{\mathbf{f}}_{s \to t}, \tilde{\mathbf{f}}_t) = 1 - \frac{\tilde{\mathbf{f}}_{s \to t} \cdot \tilde{\mathbf{f}}_t}{\|\tilde{\mathbf{f}}_{s \to t}\| \|\tilde{\mathbf{f}}_t\|} \cdot (10) $$
<table><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Algorithm 1: Style Mining (see Fig. 4)</td><td></td></tr><tr><td>Input</td><td>Set F<sub>s</sub> of source image features.<br>VL embedding f&#773;<sub>t</sub>.</td></tr><tr><td>Param</td><td>Number N of optimization steps.<br>Learning rate lr and momentum m<br>of gradient descend (GD).</td></tr><tr><td>Output</td><td>Set S<sub>s→t</sub> of target styles.</td></tr><tr><td>1</td><td>S<sub>s→t</sub> ← ∅</td></tr><tr><td></td><td>foreach f<sub>s</sub> ∈ F<sub>s</sub> do</td></tr><tr><td>2</td><td>μ<sup>0</sup> ← mean(f<sub>s</sub>)</td></tr><tr><td>3</td><td>σ<sup>0</sup> ← std(f<sub>s</sub>)</td></tr><tr><td></td><td>// Optimization</td></tr><tr><td>4</td><td>for i = 1, 2, ..., N do</td></tr><tr><td>5</td><td>f&#773;<sup>i</sup><sub>s→t</sub> ← PIN<sub>f&#773;<sub>t</sub></sub>(f<sub>s</sub>, μ<sup>i-1</sup>, σ<sup>i-1</sup>)</td></tr><tr><td>6</td><td>f&#773;<sup>i</sup><sub>s→t</sub> ← get-embedding(f&#773;<sup>i</sup><sub>s→t</sub>)</td></tr><tr><td>7</td><td>μ<sup>i</sup> ← GD<sup>lr</sup><sub>m</sub>(μ<sup>i-1</sup>, ∇<sub>μ</sub>L<sub>μ,σ</sub>(f&#773;<sup>i</sup><sub>s→t</sub>, f&#773;<sub>t</sub>))</td></tr><tr><td>8</td><td>σ<sup>i</sup> ← GD<sup>lr</sup><sub>m</sub>(σ<sup>i-1</sup>, ∇<sub>σ</sub>L<sub>μ,σ</sub>(f&#773;<sup>i</sup><sub>s→t</sub>, f&#773;<sub>t</sub>))</td></tr><tr><td>9</td><td>end</td></tr><tr><td>10</td><td>(μ<sub>t</sub>, σ<sub>t</sub>) ← (μ<sup>N</sup>, σ<sup>N</sup>)</td></tr><tr><td>11</td><td>S<sub>s→t</sub> ← S<sub>s→t</sub> ∪ {(μ<sub>t</sub>, σ<sub>t</sub>)}</td></tr></tbody></table>

This CLIP-space cosine distance, already used in prior text-driven image editing works (Patashnik et al, 2021), aims to steer the stylized features in the direction of the target VL embedding. One step of the optimization is illustrated in Fig. 4. In practice, we run several such steps per image (i.e., instance) leading to the mined target style denoted $(\mu_t, \sigma_t)$.

As there might be a variety of styles in a target domain, our mining populates the $S_{s→t}$ set with as many variations of target style as there are source images, hence $|S_{s→t}| = |D_s|$.

Intuitively, our simple augmentation strategy can be seen as a cost-efficient way to cover the distribution of the target domain by starting from different anchor points in CLIP latent space coming from the source images and steering them in the direction of the target VL embedding. This mitigates the diversity problem discussed in one-shot feature augmentation (Luo et al, 2020; Wu et al, 2022).

## 3.3 Fine-tuning for Adaptation

For adaptation, at each training iteration, we stylize the source features using a mined target style $(\mu_t, \sigma_t)$ randomly selected from $S_{s→t}$. The augmented features are computed as $f_{s→t} = \text{AdaIN}(f_s, \mu_t, \sigma_t)$ and are used for fine-tuning the classifier $M_{cls}$ of the segmenter $M$

<table><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Algorithm 2: VL-guided UDA</td><td></td></tr><tr><td>Input</td><td>Source dataset D<sub>s</sub> = {(I<sub>s</sub>, y<sub>s</sub>)}.<br>CLIP encoders E<sub>img</sub> and E<sub>txt</sub>.<br>Target VL embedding f&#773;<sub>t</sub>.<br>Feature backbone M<sub>feat</sub> ← E<sub>img</sub>.<br>Source model: M = (M<sub>feat</sub>, M<sub>cls</sub>).</td></tr><tr><td>Result</td><td>Target-adapted model M' = (M<sub>feat</sub>, M'<sub>cls</sub>).</td></tr><tr><td>// Initialization</td><td></td></tr><tr><td>1</td><td>M ← train(M, D<sub>s</sub>)</td></tr><tr><td></td><td>// Feature Augmentation</td></tr><tr><td>2</td><td>F<sub>s</sub> ← feat-ext(M<sub>feat</sub>, {I<sub>s</sub>})</td></tr><tr><td>3</td><td>S<sub>s→t</sub> ← augment(F<sub>s</sub>, f&#773;<sub>t</sub>)</td></tr><tr><td></td><td>// Adaptation</td></tr><tr><td>4</td><td>M' ← fine-tune(M, F<sub>s</sub>, S<sub>s→t</sub>, {y<sub>s</sub>})</td></tr></tbody></table>

(Fig. 2, middle). As we only adjust the feature style, which keeps the semantic-content unchanged (Huang and Belongie, 2017), we can still use the labels $y_s$ to train the classifier in a supervised manner with a standard segmentation loss (i.e., cross-entropy). To this end, we simply forward augmented features through remaining layers in $M_{feat}$ followed by $M_{cls}$. In the backward pass, only weights of $M_{cls}$ are updated by the loss gradients. We denote the fine-tuned model as $M' = (M_{feat}, M'_{cls})$.

Algorithm 2 presents the high-level pseudo-code for VL-guided adaptation: from source-only training as model initialization, to VL embedding-driven feature augmentation, to zero-shot/one-shot model adaptation.

In PØDA and PØDA-concept, $M'$ is evaluated on images with conditions and styles that were never seen during any of the training stages yet described by a single prompt, while in PIDA it is evaluated on the validation set of the same dataset from which the target image was sampled.

# 4 Experiments

## 4.1 Implementation details

We use the DeepLabv3+ architecture (Chen et al, 2018) with the backbone $M_{feat}$ initialized from the image encoder $E_{img}$ of the pre-trained CLIP-ResNet-50 model.<fnref n="2" />

**Source-only training.** The network is trained for 200k iterations on random 768 × 768 crops

<footnote n="2">²https://github.com/openai/CLIP</footnote>
with a batch size of 2. We use a polynomial learning rate schedule with initial $lr=10^{-1}$ for the classifier and $lr=10^{-4}$ for backbone when it is not frozen (see Table 1). Optimization is done with Stochastic Gradient Descent (SGD) (Bottou, 2010), momentum 0.9 and weight decay $10^{-4}$. We apply standard color jittering and horizontal flip to crops.

**Concept optimization.** $S^*$ is optimized from the source images using SGD for 10 epochs, with a batch size of 16. The learning rate is set to $10^{-4}$.

**PIN optimization.** For the style mining step, we use the source feature activations after the first layer (Layer1): $f_s \in \mathbb{R}^{192 \times 192 \times 256}$. The style parameters $\mu$ and $\sigma$ are 256-dim real vectors. The CLIP embeddings are 1024-dim vectors for CLIP-ResNet-50 backbone, and 512-dim vectors for CLIP-ResNet-101. We adopt the Imagenet templates from (Radford et al, 2021) to encode the target descriptions P. For PØDA, PØDA-concept and PIDA, PIN is optimized for 100 iterations using Gradient Descent (GD) on batches of 16 feature instances with a learning rate $lr=1.0$.

**Classifier fine-tuning.** Starting from the source-only trained model, we fine-tune the classifier $M_{cls}$ on batches of 8 augmented features $f_{s \to t}$ for 2,000 iterations. Polynomial schedule is used with the initial $lr = 10^{-2}$. We always use the last checkpoint for evaluation.

**Datasets.** As source, we use Cityscapes (Cordts et al, 2016), composed of 2,975 training and 500 validation images featuring 19 semantic classes. Though we adapt towards a prompt or an image *not* a dataset, we need ad-hoc datasets to test on. We report main results using ACDC (Sakaridis et al, 2021) because it has urban images captured in adverse conditions. We also study the applicability of our method to the two settings of real→synthetic (Cityscapes as source, and evaluating on GTA5 (Richter et al, 2016)) and synthetic→real (GTA5 as source, and evaluating on Cityscapes). We evaluate on the validation set when provided, and for GTA5 evaluation we use a random subset of 1,000 images.

**Evaluation protocol.** Mean Intersection over Union (mIoU%) is used to measure adaptation performance. We test all models on target images at their original resolutions. We always report the mean and standard deviation over five models trained with different random seeds.

## 4.2 PØDA

We consider the following adaptation scenarios: day→night, clear→snow, clear→rain, real→synthetic and synthetic→real. We report zero-shot adaptation results of PØDA in the addressed set-ups, comparing against two main baselines: CLIPstyler (Kwon and Ye, 2022) for zero-shot style transfer and SM-PPM (Wu et al, 2022) for one-shot UDA. Both PØDA and CLIP-styler models don't see any target images during training. In this study, we arbitrarily choose a simple prompt to describe each domain. We show later in Section 4.2.1 more results using other relevant prompts with similar meanings – showing that our adaptation gain exhibits only a slight sensitivity to prompt selection. For SM-PPM, one random target image from the training set is used.

**Comparison to CLIPstyler.** CLIP-styler (Kwon and Ye, 2022) is a style transfer method that also makes use of the pre-trained CLIP model but for zero-shot stylizing of source images. We consider CLIPstyler<fnref n="3" /> as the most comparable zero-shot baseline for PØDA as both are built upon CLIP, though with different mechanisms and different objectives. Designed for style transfer, CLIPstyler produces images that exhibit characteristic styles of the input text prompt. However the stylized images can have multiple artifacts which hinder their usability in the downstream segmentation task. This is visible in Fig. 5 which shows stylized examples from CLIPstyler with PØDA target prompts. Zooming in, we note that stylization of snow or game added snowy roads or Atari game *on the buildings*, respectively.

Starting from source-only model, we fine-tune the classifier on stylized images, as similarly done in PØDA with the augmented features. Table 2 compares PØDA against the source-only model and CLIPstyler. PØDA consistently outperforms the two baselines. CLIP-styler brings some improvements over source-only in Cityscapes→Night and Cityscapes→Snow. In other scenarios, e.g., rain, CLIPstyler even performs worse than source-only.

Real→synthetic is an interesting though underexplored adaptation scenario. One potential application of real→synthetic is for model validation

<footnote n="3">³We use official code https://github.com/cyclomon/CLIPstyler and follow the recommended configs.</footnote>
<table><caption>**Table 2:** Zero-shot domain adaptation in semantic segmentation. Performance (mIoU%) of PØDA compared against CLIPstyler (Kwon and Ye, 2022) and source-only baseline. Results are grouped by source domain and prompts (P). Here, CS stands for Cityscapes (Cordts et al, 2016).</caption><thead><tr><th>Source</th><th>Target eval.</th><th>Method</th><th>mIoU[%]</th></tr></thead><tbody><tr><td rowspan="10">CS</td><td rowspan="3">ACDC Night</td><td>P = "driving at night"</td><td></td></tr><tr><td>source-only</td><td>18.31</td></tr><tr><td>CLIPstyler</td><td>21.38 ±0.36</td></tr><tr><td rowspan="3">ACDC Snow</td><td>PØDA</td><td><b>25.03</b> ±0.48</td></tr><tr><td>P = "driving in snow"</td><td></td></tr><tr><td>source-only</td><td>39.28</td></tr><tr><td rowspan="3">ACDC Rain</td><td>CLIPstyler</td><td>41.09 ±0.17</td></tr><tr><td>PØDA</td><td><b>43.90</b> ±0.53</td></tr><tr><td>P = "driving under rain"</td><td></td></tr><tr><td rowspan="4">GTA5</td><td rowspan="3">ACDC Rain</td><td>P = "driving in a game"</td><td></td></tr><tr><td>source-only</td><td>38.20</td></tr><tr><td>CLIPstyler</td><td>37.17 ±0.10</td></tr><tr><td>PØDA</td><td><b>42.31</b> ±0.55</td></tr><tr><td rowspan="3">GTA5</td><td rowspan="3">CS</td><td>P = "driving"</td><td></td></tr><tr><td>source-only</td><td>36.38</td></tr><tr><td>CLIPstyler</td><td>31.50 ±0.21</td></tr><tr><td></td><td></td><td>PØDA</td><td><b>40.08</b> ±0.52</td></tr></tbody></table>

Cityscapes night snow rain game

<figure><img src="image_6.png" /><figcaption>**Fig. 5:** CLIPstyler (Kwon and Ye, 2022) stylization. A sample Cityscapes image stylized using adhoc target prompts. Translated images exhibit visible artifacts, potentially harming adaptation, e.g., rain in Table 2.</figcaption></figure>

in the industry, where some hazardous validations like driving accidents must be done in the virtual space. Here we test if our zero-shot mechanism can be also applied to this particular setting. Similarly, PØDA outperforms both baselines. Also in the reverse synthetic→real setting, again our method performs the best. CLIPstyler undergoes almost 5% drops in mIoU compared to source-only.

We argue on the simplicity of our method that only introduces minimal changes to the feature statistics, yet such changes are crucial for target adaptation. CLIPstyler, designed for style transfer, involves training an additional StyleNet with ≈ 615k parameters for synthesizing the stylized images. We draw on the simplicity of PØDA to

<figure><img src="image_7.png" /><figcaption>**Fig. 6:** Qualitative results of zero-shot adaptation. (cols 1-2) Input images and their ground truths; (cols 3-4) Segmentation results of source-only and PØDA models.</figcaption></figure>

<table><caption>**Table 3:** Comparison with SM-PPM (Wu et al, 2022). Semantic segmentation performance (mIoU%) for source / adapted models, and gain provided by adaptation (Δ in mIoU). For adaptation, SM-PPM (ResNet-101 DeepLabv2) has access to one target image, while PØDA (ResNet-50 DeepLabv3+) leverages a target prompt and a text encoder.</caption><thead><tr><th>Source</th><th>Target eval.</th><th>1-shot SM-PPM</th><th>0-shot PØDA</th></tr></thead><tbody><tr><td rowspan="3">CS</td><td>ACDC Night</td><td>13.07 / 14.60 (Δ=1.53)</td><td>18.31 / <b>25.03</b> (Δ=6.72)</td></tr><tr><td>ACDC Snow</td><td>32.60 / 35.61 (Δ=3.01)</td><td>39.28 / <b>43.90</b> (Δ=4.62)</td></tr><tr><td>ACDC Rain</td><td>29.78 / 32.23 (Δ=2.45)</td><td>38.20 / <b>42.31</b> (Δ=4.11)</td></tr><tr><td>GTA5</td><td>CS</td><td>30.48 / 39.32 (Δ=8.84)</td><td>36.38 / <b>40.08</b> (Δ=3.70)</td></tr></tbody></table>

explain why it is more advantageous than CLIP-styler for downstream tasks like semantic segmentation: the minimal statistics changes help avoiding significant drifts on the feature manifold which may otherwise result in unwanted errors. For comparison, it takes us 0.3 seconds to augment one source feature, while stylizing an image with CLIPstyler takes 65 seconds (as measured on one RTX 2080TI GPU).

We show in Fig. 6 qualitative examples of predictions from source-only and PØDA models.

**Comparison to one-shot UDA (OSUDA).** We also compare PØDA against SM-PPM (Wu et al, 2022),<fnref n="4" /> a state-of-the-art OSUDA method, see Table 3. The OSUDA setting allows the access to a single unlabeled target domain image for domain adaptation. In SM-PPM, this image is considered as an anchor point for target style mining. Using 5 randomly selected target images, we trained, with each one, five models with different random seeds. The reported mIoUs are averaged over the 25 resulting models. We note

<footnote n="4">⁴We use official code https://github.com/W-zx-Y/SM-PPM</footnote>
that the absolute results of the two models are
not directly comparable due to the differences in
backbone (ResNet-101 in SM-PPM vs. ResNet-50
in PØDA) and in segmentation head (DeepLabv2
in SM-PPM vs. DeepLabv3+ in PØDA). We
thus analyze the improvement of each method
over the corresponding naive source-only baseline
while taking into account the source-only perfor-
mance. We first notice that our source-only (CLIP
ResNet) performs better than SM-PPM source-
only (ImageNet pretrained ResNet), demonstrat-
ing the overall robustness of the frozen CLIP-
based model. In Cityscapes→ACDC, both abso-
lute and relative improvements of PØDA over
source-only are greater than the ones of SM-
PPM. Overall, PØDA exhibits on par or greater
improvements over SM-PPM, despite the fact that
our method is purely zero-shot.

**Qualitative results on uncommon conditions.** Figure 7 shows some qualitative results, training on Cityscapes, and adapting to uncommon conditions never found in datasets because they are either rare (sandstorm), dangerous (fire), or not labeled (old movie). For all, PØDA improves over source-only, which demonstrates its true benefit.

4.2.1 Ablation studies

**Prompt selection.** Using any meaningful descriptions of the target domain as prompt P, one should obtain similar adaptation gain with PØDA. To verify this, we generate other relevant prompts by querying ChatGPT (GPT 3.5)<fnref n="5" /> with Give me 5 prompts that have the same exact meaning as [PROMPT] using same prompts as in Table 2. Results in Table 4 show that adaptation gains are rather independent of the textual expression. Inversely, we query irrelevant prompts with Give me 6 random prompts of length from 3 to 6 words describing a random photo, which could result in negative transfer (See Table 4). In some cases, small gains could occur; however we conjecture that such gains may originate from generalization by randomization rather than adaptation.

**Choice of features to augment.** DeepLabV3+ segmenter takes as inputs both low-level features from *Layer1* and high-level features from *Layer4*.

<figure><img src="image_8.png" /><figcaption>**Fig. 7: PØDA on uncommon conditions.** Qualitative results here all use Cityscapes as source and PØDA uses prompt P.</figcaption></figure>

In PØDA, we only augment the *Layer1* features (by optimizing PIN) and forward them through remaining layers 2-4 to obtain the *Layer4* features. The input to the segmentation head is the concatenation of both. We study in Table 5 if one should augment other features in addition to the ones in *Layer1*: we observe the best performance with only *Layer1* augmentation. We conjecture that it is important to preserve the consistency between the two inputs to the classifier, *i.e.*, *Layer4* features should be derived from the augmented ones of *Layer1*.

**Number of mined styles.** In our experiments, $|S_{s→t}| = |D_s|$. Here we study the effect of changing the number $|S_{s→t}|$ of styles on the target domain performance. By performing ablation on CS→Night with $|S_{s→t}| = 1, 10, 100, 1000, 2975$ (i.e., $|D_s|$), we obtain 16.00±5.01, 22.04±1.24, 23.90±0.96, 24.27±0.70, 25.03±0.48 (mIoU %) respectively. For $|S_{s→t}| < |D_s|$, the styles are sampled randomly from $D_s$ and results are reported in average on 5 different samplings. Interestingly,

<footnote n="5">⁵OpenAI's chatbot https://chatgpt.com/</footnote>
<table><caption>**Table 4: Effect of prompts on PØDA.** We show result for our prompt P (top) as well as ChatGPT-generated **relevant prompt** (middle) and **irrelevant prompt** (bottom). Please refer to Section 4.2.1 for details. Best results (bold) are always obtained with **relevant prompts** for which mean mIoU (italic) also proves to be better.</caption><thead><tr><th>Method</th><th>ACDC Night</th><th>ACDC Snow</th><th>ACDC Rain</th><th>GTA5</th></tr></thead><tbody><tr><td>Source only</td><td>18.31</td><td>39.28</td><td>38.20</td><td>39.59</td></tr><tr><td>Trg</td><td><b>"driving at night"</b><br>25.03 ±0.48</td><td><b>"driving in snow"</b><br>43.90 ±0.53</td><td><b>"driving under rain"</b><br>42.31 ±0.55</td><td><b>"driving in a game"</b><br>41.07 ±0.48</td></tr><tr><td rowspan="7">Relevant →</td><td><b>"operating a vehicle after sunset"</b><br>24.38 ±0.37</td><td><b>"operating a vehicle in snowy conditions"</b><br>44.33 ±0.36</td><td><b>"operating a vehicle in wet conditions"</b><br>42.21 ±0.47</td><td><b>"piloting a vehicle in a virtual world"</b><br>41.25 ±0.40</td></tr><tr><td><b>"driving during the nighttime hours"</b><br>25.22 ±0.64</td><td><b>"driving on snow-covered roads"</b><br>43.56 ±0.62</td><td><b>"driving on rain-soaked roads"</b><br>42.51 ±0.33</td><td><b>"controlling a car in a digital simulation"</b><br>41.19 ±0.14</td></tr><tr><td><b>"navigating the roads in darkness"</b><br>24.73 ±0.47</td><td><b>"piloting a vehicle in snowy terrain"</b><br>44.67 ±0.18</td><td><b>"navigating through rainfall while driving"</b><br>41.11 ±0.69</td><td><b>"maneuvering a vehicle in a computerized racing experience"</b><br>40.34 ±0.49</td></tr><tr><td><b>"driving in low-light conditions"</b><br>24.68 ±0.34</td><td><b>"driving in wintry precipitation"</b><br>43.11 ±0.56</td><td><b>"driving in inclement weather"</b><br>40.68 ±0.37</td><td><b>"operating a transport in a video game environment"</b><br>41.34 ±0.42</td></tr><tr><td><b>"travelling by car after dusk"</b><br>24.89 ±0.24</td><td><b>"travelling by car in a snowstorm"</b><br>43.83 ±0.17</td><td><b>"travelling by car during a downpour"</b><br>42.05 ±0.35</td><td><b>"navigating a machine through a digital driving simulation"</b><br>41.86 ±0.10</td></tr><tr><td><b>"mesmerizing northern lights display"</b><br>20.05 ±0.77</td><td><b>"playful dolphins in the ocean"</b><br>40.07 ±0.66</td><td><b>"breathtaking view from mountaintop"</b><br>39.87 ±0.26</td><td><b>"cheerful sunflower field in bloom"</b><br>42.08 ±0.28</td></tr><tr><td><b>"dramatic cliff overlooking the ocean"</b><br>20.09 ±0.98</td><td><b>"majestic eagle in flight over mountains"</b><br>38.20 ±0.54</td><td><b>"breathtaking view from mountaintop"</b><br>40.05 ±0.52</td><td><b>"cheerful sunflower field in bloom"</b><br>39.85 ±0.68</td></tr><tr><td>ChatGPT-generated</td><td><b>"mesmerizing northern lights display"</b><br>20.05 ±0.77</td><td><b>"playful dolphins in the ocean"</b><br>40.07 ±0.66</td><td><b>"breathtaking view from mountaintop"</b><br>39.87 ±0.26</td><td><b>"cheerful sunflower field in bloom"</b><br>42.08 ±0.28</td></tr><tr><td>Irrelevant →</td><td><b>"mesmerizing northern lights display"</b><br>20.05 ±0.77</td><td><b>"playful dolphins in the ocean"</b><br>40.07 ±0.66</td><td><b>"breathtaking view from mountaintop"</b><br>39.87 ±0.26</td><td><b>"cheerful sunflower field in bloom"</b><br>42.08 ±0.28</td></tr></tbody></table>

we observe that the variance decreases with the increase of $|S_{s→t}|$. Results also suggest that only few styles (e.g., $|S_{s→t}| = 10$) could be sufficient for feature translation, similarly to few-shot image-to-image translation (Pizzati et al, 2022), though at the cost of higher variance.

**Partial unfreezing of the backbone.** While our experiments use a frozen backbone due to the observed good out-of-distribution performance (Table 1), we highlight that during training only *Layer1* must be frozen to preserve its activation space where augmentations are done; the remaining three layers could be optionally fine-tuned. Results in Table 6 show that freezing the whole backbone (i.e., Layer1-4) achieves the best

<table><caption>**Table 5: Impact of selected layers for augmentation.** Performance (mIoU) of PØDA's day→night adaptation for different choices of ResNet layers for feature augmentation. In addition to augmenting features of *Layer1* (Row 1), one can augment *Layer2*, *Layer3*, or *Layer4* features (Rows 2-4).</caption><thead><tr><th>Layer1</th><th>Layer2</th><th>Layer3</th><th>Layer4</th><th>ACDC Night</th></tr></thead><tbody><tr><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td><b>25.03</b> ±0.48</td></tr><tr><td>✓</td><td>✓</td><td>✗</td><td>✗</td><td>23.43 ±0.51</td></tr><tr><td>✓</td><td>✗</td><td>✓</td><td>✗</td><td>22.93 ±0.53</td></tr><tr><td>✓</td><td>✗</td><td>✗</td><td>✓</td><td>21.05 ±0.55</td></tr></tbody></table>

<table><caption>**Table 6: PØDA when freezing (*) only Layer1.** Both models with *, reported in Table 2, freeze the whole backbone *Layer1-4*.</caption><thead><tr><th>Method</th><th>Night</th><th>Snow</th><th>Rain</th><th>GTA5</th></tr></thead><tbody><tr><td>src-only*</td><td>18.31</td><td>39.28</td><td>38.20</td><td>39.59</td></tr><tr><td>PØDA*</td><td><b>25.03</b> ±0.48</td><td><b>43.90</b> ±0.53</td><td><b>42.31</b> ±0.55</td><td><b>41.07</b> ±0.48</td></tr><tr><td>src-only (Layer1 *</td><td>9.60</td><td>30.99</td><td>30.89</td><td>29.38</td></tr><tr><td>PØDA (Layer1 *</td><td>19.43 ±0.69</td><td>37.80 ±2.65</td><td>40.71 ±1.06</td><td>39.09 ±1.23</td></tr></tbody></table>

results. In all cases, PØDA consistently improves the performance over source-only.

Note that for fine-tuning the backbone, different hyperparameters, optimizer, and/or fine-tuning strategies might bring performance improvement *w.r.t.* to the current results. This is beyond the scope of this work, which aims to show that PIN is a plug-and-play module that can be used for any CNN-based vision-language pretrained model. Other works have for example shown that PIN can be used for state-of-the-art language-driven domain generalization (Fahes et al, 2024b).

We show in Appendix A that PØDA can be applied on object detection and image classification. Additional ablations on PIN initialization and on training for adaptation are included in Appendix B.

## 4.3 PØDA-concept

Our initial prompts are intuitive descriptions verbalizing the target domains, e.g., “driving at night”, “driving under rain”. Each prompt is the natural combination of a global *content*, (i.e., “driving”) and a target style condition (e.g., “at night”, “under rain”). Equivalently, the **relevant prompts** queried using ChatGPT (See Table 4) follow the same construction. For instance, in
<table><caption>**Table 7: Effect of prompt construction.** Performance (mIoU %) on the target datasets. For PØDA, we use either the text concept “driving”, the target style conditions, or the combination of both elements as prompt P. For PØDA-concept, we show the use of the optimized concept (S*) alone or its combination with a target text style.</caption><thead><tr><th>Method</th><th>ACDC Night</th><th>ACDC Snow</th><th>ACDC Rain</th><th>GTA5</th></tr></thead><tbody><tr><td>Source only</td><td>18.31</td><td>39.28</td><td>38.20</td><td>39.59</td></tr><tr><td rowspan="4">PØDA</td><td>"driving"</td><td>"driving"</td><td>"driving"</td><td>"driving"</td></tr><tr><td>22.81 ± 0.34</td><td>43.34 ± 0.42</td><td>40.65 ± 0.99</td><td>40.33 ± 0.46</td></tr><tr><td>"night"</td><td>"snow"</td><td>"rain"</td><td>"game"</td></tr><tr><td>23.74 ± 0.55</td><td>40.45 ± 0.37</td><td>39.74 ± 0.35</td><td>40.69 ± 0.38</td></tr><tr><td rowspan="4">PØDA-concept</td><td>"driving at night"</td><td>"driving in snow"</td><td>"driving under rain"</td><td>"driving in a game"</td></tr><tr><td>25.03 ± 0.48</td><td>43.90 ± 0.53</td><td>42.31 ± 0.55</td><td>41.07 ± 0.48</td></tr><tr><td>"S""</td><td>"S""</td><td>"S""</td><td>"S""</td></tr><tr><td>21.90 ± 0.19</td><td>44.04 ± 0.34</td><td>43.25 ± 0.45</td><td>42.54 ± 0.29</td></tr><tr><td></td><td>"S" at night"</td><td>"S" in snow"</td><td>"S" under rain"</td><td>"S" in a game"</td></tr><tr><td></td><td>25.16 ± 0.31</td><td>44.72 ± 0.24</td><td>42.78 ± 0.35</td><td>42.90 ± 0.17</td></tr></tbody></table>

“navigating the roads in darkness”, the concept is expressed by “navigating the roads” and the style condition is described by “in darkness”. The presence of both elements in the target prompt is crucial for mining relevant styles. Indeed, in the rows of PØDA of Table 7, we show that while using either element alone improves over source-only, yet better results are obtained when combining both. Interestingly, optimizing towards “driving” improves all the target performances without any use of a textual style condition. Such prompt representation could describe driving scene images with all the styles included, bringing some generalization effects.

**Concept optimization from source data.** As introduced in Section 3.1.2, since the images of the source and unseen target domains share similar general content, we propose to optimize the latter using the source images instead of relying on text. This is done by optimizing a word embedding such that the text latent embedding is close to the source images’ embeddings in the bimodal space (See Eq. (9)). Results of PØDA-concept in Table 7 show superior performance compared to all variants of PØDA. Using a concept optimized from images better encodes the semantic content common to the data and results in better mined styles. Interestingly, using S* alone improves the performance w.r.t. source-only. This case is similar to the use of PØDA with “driving”, yet S* is more data-specialized. Note that for severe style

<figure><img src="image_9.png" /><figcaption>**Fig. 8: Effect of the number of optimization iterations.** Performance (mIoU %) of PØDA adaptation (including its concept version) from Cityscapes to ACDC subsets ((a) night, (b) snow, (c) rain) and to (d) GTA5 as a function of the number of statistics optimization iterations. The values are averages over 5 runs.</figcaption></figure>

shift, e.g., night, the mIoU still significantly lags behind S* + “at night”.

Fig. 8 shows that including an optimized concept in the prompt not only improves performance but also makes style mining less sensitive to the number of iterations. Indeed, optimizing towards a prompt including S* restricts the search space to a region encoding the semantics of the images, which reduces the risk of drifting from the content. For a high number of iterations, the performance is significantly harmed when using a concept expressed through language (i.e., “driving”). Interestingly, using an optimized S*, the same observations hold, however the decrease is significantly smaller. We conjecture that in both cases, this might be due to over-stylization (Kwon and Ye, 2022).

In short, using an optimized concept along with the style condition has two advantages: better performance and lower sensitivity to the optimization iterations.

## 4.4 PIDA

We now evaluate our one-shot adaptation, PIDA, which instead of a target prompt leverages a target
<table><caption>**Table 8: PIDA on discrete domains.** Performances (mIoU %) is reported for 0-shot using prompt (PØDA) and two 1-shot variants (AdaIN, PIDA) which utilize a target image.</caption><thead><tr><th></th><th>Method</th><th>ACDC Night</th><th>ACDC Snow</th><th>ACDC Rain</th><th>GTA5</th></tr></thead><tbody><tr><td rowspan="2">source-only</td><td></td><td>18.31</td><td>39.28</td><td>38.20</td><td>39.59</td></tr><tr><td>PØDA</td><td>"driving at night"<br>25.03 ± 0.48</td><td>"driving in snow"<br>43.90 ± 0.53</td><td>"driving under rain"<br>42.31 ± 0.55</td><td>"driving in a game"<br>41.07 ± 0.48</td></tr><tr><td rowspan="2">1-shot</td><td>AdaIN</td><td>18.15 ± 3.65</td><td>40.25 ± 1.72</td><td>42.61 ± 0.80</td><td>40.56 ± 0.97</td></tr><tr><td>PIDA</td><td>24.29 ± 1.07</td><td>43.66 ± 0.99</td><td>42.66 ± 1.21</td><td>41.02 ± 1.86</td></tr></tbody></table>

image. PIDA relies on the same mining strategy (Section 3.2) but using a target embedding derived from a target image $I_t$ (cf. Section 3.1.3). As in prior experiments, we always report the average performance over 5 runs though here, using a different $I_t$ randomly sampled from the target training set. Because one-shot adaptation enables nuanced settings that are hard to verbalize, we explore varying levels of granularity in the following experiments.

**Discrete domains.** Table 8 reports performance on the same discrete domains (i.e., rain, night, etc.) as done previously. Since PIDA has access to a target image, we also report a vanilla AdaIN-adaptation baseline which seeks direct alignment with the style of the target image. As expected, PIDA significantly outperforms AdaIN, a result we attribute to our mining strategy providing greater diversity. Conversely, PIDA is outperformed almost consistently by its 0-shot counterpart, PØDA. We conjecture that this is due to the prompts of PØDA better capturing the overall appearance of the target domain, whereas randomly selected target images may not adequately represent the entire distribution of appearances in the target domain. This interpretation is supported by the high variance of PIDA, suggesting that mined styles are highly dependent on the choice of the target image.

**Continuous domains.** To better show the interest of PIDA for nuanced target conditions, harder to verbalize, we report performance on timelapse data. As such data are lacking in the driving domain, we use CoMoGAN (Pizzati et al, 2021), a continuous image-to-image translation method trained on Waymo Open dataset (Sun et al, 2020), where the translation is parametrized by $\phi \in [0, \pi]$, mapping the sun elevation in $[+30^\circ, -40^\circ]$ where negative values indicates that the sun is below the horizon (i.e., nighttime). Using

<figure><img src="image_10.png" /><figcaption>**Fig. 9: PIDA on continuous time-of-day domains.** We report the 5 runs average performance on $\phi$-translated Cityscapes for PIDA, source-only, and PØDA with various prompts. Images shown as insets are examplar one-shot images used for PIDA.</figcaption></figure>

CoMoGAN, we translate Cityscapes training and validation sets for different $\phi$ values. For each $\phi$, we leverage PIDA using for adaptation a random image from the $\phi$-translated training set.

In Fig. 9, we plot the Cityscapes validation performance for each $\phi$ value, which shows that PIDA consistently outperforms the source-only model. In the latter plot, we also report performance of the 0-shot PØDA using various prompts constructed with “driving at <time>”. Interestingly, prompts referring to daytime (e.g., “driving at noon” or “at 2pm”) perform better than other prompts when the sun is higher (i.e., $\phi < \pi/2$) while night prompts (e.g., “driving at 10pm” or “at midnight”) are better after sunset (i.e., $\phi > \pi/2$). Still, for all $\phi$ values, PIDA consistently outperforms PØDA regardless of the prompt used. Such results hint that using an image as adaptation guidance can lead to better performance for peculiar conditions which are harder to describe with text.

# 5 Discussion

**Generalization with PØDA.** Inspired by the observation that some unrelated prompts improve performance on target domains (see Table 4), we study how PØDA can benefit from general style augmentation. First, we coin “Source-only-G” the generalized source-only model where we augment features by shifting the per-channel $(\mu, \sigma)$ with Gaussian noises sampled for each batch of features, such that the signal to noise ratio is 20 dB. This source-only variant takes inspiration from (Fan et al, 2023) where simple perturbations of feature channel statistics could help achieve SOTA generalization performance
<table><caption>**Table 9: Generalization with PØDA.** Source-only-G model is enhanced with a domain generalization technique. Training PØDA from Source-only-G ('PØDA-G') brings improvements. 'style-mix': style mixing as in (Wu et al, 2022).</caption><thead><tr><th>Method</th><th>Night</th><th>Snow</th><th>Rain</th><th>GTA5</th></tr></thead><tbody><tr><td>Source-only</td><td>18.31</td><td>39.28</td><td>38.20</td><td>39.59</td></tr><tr><td>Source-only-G</td><td>21.07</td><td>42.84</td><td>42.38</td><td>41.54</td></tr><tr><td>PØDA-G</td><td><b>24.86</b> ±0.70</td><td>44.34 ±0.36</td><td>43.17 ±0.63</td><td>41.73 ±0.39</td></tr><tr><td>PØDA-G+style-mix</td><td>24.18 ±0.23</td><td><b>44.46</b> ±0.34</td><td><b>43.56</b> ±0.46</td><td><b>42.98</b> ±0.12</td></tr></tbody></table>

in object detection. Table 9 shows that Source-only-G always improves over Source-only, demonstrating a generalization capability. When applying our zero-shot adaptation on Source-only-G (denoted “PØDA-G”), target performance again improves – always performing best on the desired target.

To further boost the performance, we perform style mixing. Wu et al (2022) showed that for OSUDA, mixing the source feature statistics and those of the single target image brings generalization effects. Later, Fahes et al (2024b) showed that there is no need for the target image: mixing original and PIN-augmented statistics helps generalization. We follow the same strategy here. The mixed statistics $\mu_{mix}, \sigma_{mix}$ are given by:

$$ \mu_{\text{mix}} = \alpha \odot \mu_{\text{t}} + (1 - \alpha) \odot \mu_{\text{s}}, \quad (11) $$

$$ \sigma_{\text{mix}} = \alpha \odot \sigma_{\text{t}} + (1 - \alpha) \odot \sigma_{\text{s}}, \quad (12) $$

where $\alpha \in \mathbb{R}^c$ are per-channel mixing weights uniformly sampled in $[0, 1]$; $\odot$ denotes element-wise multiplication. Addition and subtraction are also element-wise. Finally, the augmented features are computed as follows:

$$ f_{s \to t} = \text{AdaIN}(f_s, \mu_{\text{mix}}, \sigma_{\text{mix}}), \quad (13) $$

with AdaIN defined in Eq. (7).

**Other architectures.** We show in Table 10 consistent gains brought by PØDA using other backbone (RN101 (He et al, 2016)) and segmentation head (semantic FPN (Kirillov et al, 2019)).

# 6 Conclusion

In this work, we leverage the CLIP model to make possible a new challenging task of domain

<table><caption>**Table 10: PØDA with different architectures.** Backbones are RN50 for Semantic FPN ('Sem. FPN') and RN101 for DeepLabV3+ ('DLv3+').</caption><thead><tr><th>Backbone</th><th>Method</th><th>Night</th><th>Snow</th><th>Rain</th><th>GTA5</th></tr></thead><tbody><tr><td rowspan="2">Sem. FPN</td><td>src-only</td><td>18.10</td><td>35.75</td><td>36.07</td><td>40.67</td></tr><tr><td>PØDA</td><td><b>21.48</b> ±0.15</td><td><b>39.55</b> ±0.13</td><td><b>38.34</b> ±0.29</td><td><b>41.59</b> ±0.24</td></tr><tr><td rowspan="2">DLv3+</td><td>src-only</td><td>22.17</td><td>44.53</td><td>42.53</td><td>40.49</td></tr><tr><td>PØDA</td><td><b>26.54</b> ±0.12</td><td><b>46.71</b> ±0.43</td><td><b>46.36</b> ±0.20</td><td><b>43.17</b> ±0.13</td></tr></tbody></table>

adaptation using a single VL embedding. We propose a cost-effective feature augmentation mechanism that adjusts the style-specific statistics of source features to synthesize augmented features in the target domain, guided by a single VL embedding coming either from a prompt in natural language, a partially optimized prompt, or a single target image. Extensive experiments, particularly for semantic segmentation, have proven the effectiveness of our framework for semantic segmentation in particular. They also show its applicability to other tasks and various backbones. Our line of research aligns with the collective efforts of the community to leverage large-scale pre-trained models (so-called “foundation models” (Bommasani et al, 2021)) for data- and label-efficient training of perception models for real-world applications.

**Data availability statement.** All the datasets used in this paper are publicly available.

**Acknowledgment.** This work was partially funded by French project SIGHT (ANR-20-CE23-0016) and the European Union under grant agreement No. 101070617.
## A PØDA on other tasks

PØDA operates at the features level, which makes it task-agnostic. We show in the following the effectiveness of our method for object detection and image classification.

### PØDA for Object Detection.
We report results in Table 11 from straightforwardly applying PØDA to object detection. Our Faster-RCNN (Ren et al, 2015) models, initialized with two CLIP-pretrained backbones, are trained on two source datasets, either Cityscapes or the Day-Clear split in Diverse Weather Dataset (DWD) (Wu and Deng, 2022). We report adaptation results on Cityscapes-Foggy (Sakaridis et al, 2018) and four other conditions in DWD. For zero-shot feature augmentation in PØDA, we use simple prompts and take the default optimization parameters in previous experiments. PØDA obtains on par or better results than UDA methods (Chen et al, 2021b; Rezaeianaran et al, 2021) (which use target images) and domain generalization methods (Fan et al, 2023; Wu and Deng, 2022; Vidit et al, 2023). We also experimented with YOLOF (Chen et al, 2021a) for object detection in CS→Foggy; PØDA reaches 35.4%, improving 1.5% from the source-only model. These results open up potential combinations of PØDA with generalization techniques like (Fan et al, 2023) and (Vidit et al, 2023) for object detection.

In the Cityscapes-Foggy experiment, we freeze *Layer1* of the backbone, while in the DWD experiments we additionally freeze *Layer2*. The remaining layers are fine-tuned as commonly done. The code for PØDA in object detection was built upon the MMDetection library.<fnref n="6" />

### PØDA for Image Classification.
We show that PØDA can be also applied for image classification. We use the same augmentation strategy to adapt a linear probe on top of CLIP-RN50 features. In a first experiment we train a linear classifier on the features of CUB-200 dataset (Wah et al, 2011) of 200 real bird species. We then perform zero-shot adaptation to classify bird paintings of CUB-200-Paintings dataset (Wang et al, 2020b) using the single prompt “Painting of a bird”. PØDA improves the source-only performance from 28.90 to 30.91 ±0.69.

<table><caption>**Table 11: PØDA for object detection (mAP%).**
For Cityscapes→Cityscapes-Foggy adaptation, the backbone is ResNet-50, while it is ResNet-101 for adaption from DWD-Day-Clear to other conditions in DWD.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Target</th><th>CS→CS</th><th colspan="4">DWD-Day Clear →</th></tr><tr><th>Foggy</th><th>Night Clear</th><th>Dusk Rainy</th><th>Night Rainy</th><th>Day Foggy</th></tr></thead><tbody><tr><td>DA-Faster (Chen et al, 2021b)</td><td>✓</td><td>32.0</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ViSGA (Rezaeianaran et al, 2021)</td><td>✓</td><td>43.3</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>NP+ (Fan et al, 2023)</td><td>✗</td><td>46.3</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>S-DGOD (Wu and Deng, 2022)</td><td>✗</td><td>-</td><td>36.6</td><td>28.2</td><td>16.6</td><td>33.5</td></tr><tr><td>CLIP The Gap (Vidit et al, 2023)</td><td>✗</td><td>-</td><td>36.9</td><td>32.3</td><td>18.7</td><td>38.5</td></tr><tr><td>PØDA</td><td>✗</td><td>47.3</td><td>43.3</td><td>40.2</td><td>20.5</td><td>43.1</td></tr></tbody></table>

In our second experiment, we address the color bias in Colored MNIST (Arjovsky et al, 2019). While for training, even and odd digits are colored red and blue respectively, the test digits are randomly colored. We augment training digit features using the “Blue digit” and “Red digit” prompts for even and odd digits respectively, and create a separate set for each one to prevent styles from leaking, i.e., to avoid trivially using “red” styles coming from even digits to augment odd digits features and vice versa. Again, PØDA brings significant improvement over the source-only model (64.16 ±0.41 vs. 55.83).

## B Additional experiments

### Effect of style mining initialization.
In our feature optimization step, we initialize $(\mu, \sigma)$ with $(\mu(\mathbf{f}_s), \sigma(\mathbf{f}_s))$. In Table 12, we report results using different initialization strategies. Starting from pre-defined or random initialization, instead of from original statistics, degrades badly the performance. As we do not use any regularization term in the CLIP cosine distance loss, we argue that initializing the optimized statistics with those of the source images is a form of regularization, favoring augmented features in a neighborhood of $\tilde{\mathbf{f}}_s$ and better preserving the semantics.

### Diversity of optimized statistics.
To verify that the statistics — optimized for the same number of iterations with the same P but from different starting anchor points $\tilde{\mathbf{f}}_s$ — are diverse, we show in Fig. 10 the boxplots of optimized parameters on the first 20 channels of $\mathbf{f}_{s→t}$ (for prompt “driving at night”).

### Training from scratch on augmented features.
In PØDA, we start with a source-only

<footnote n="6">⁶https://github.com/open-mmlab/mmdetection</footnote>
<table><caption>**Table 12: Effect of style initialization.** Performance (in mIoU) of PØDA on ACDC-Night val set (Cityscapes as source), with different initializations of style statistics. Starting from source images' statistics works substantially better.</caption><thead><tr><th>μ<sup>0</sup></th><th>σ<sup>0</sup></th><th>mIoU</th></tr></thead><tbody><tr><td>μ(f<sub>s</sub>)</td><td>σ(f<sub>s</sub>)</td><td><b>25.03</b> ±0.48</td></tr><tr><td>0</td><td>1</td><td>8.59 ±0.82</td></tr><tr><td>~N(0, I)</td><td>~N(0, I)</td><td>6.80 ±0.92</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>**Fig. 10: Per-channel optimized statistics.** Distributions of the first 20 channels of the optimized statistics of **μ** (**Left**) and **σ** (**Right**). Each boxplot shows the interquartile range (IQR) that contains 50% of the data: Its bottom and top edges delimit the first and third quartiles respectively. The horizontal line inside the box denotes the data's median. The whiskers extend from the edges of the box to the furthest point within 1.5 times the IQR, in each direction. Outlier points beyond these limits are individually plotted (diamonds).</figcaption></figure>

<table><caption>**Table 13: Importance of source-only pre-training.** Semantic segmentation performance (mIoU %) of PØDA vs. its variant without source-only training, when adapting from Cityscapes to ACDC Nigt/Snow/Rain and to GTA5.</caption><thead><tr><th>Method</th><th>Night</th><th>Snow</th><th>Rain</th><th>GTA5</th></tr></thead><tbody><tr><td>PØDA no src. pretrain</td><td>22.46</td><td>36.73</td><td>39.70</td><td>39.57</td></tr><tr><td>PØDA</td><td><b>25.03</b></td><td><b>43.90</b></td><td><b>42.31</b></td><td><b>41.07</b></td></tr></tbody></table>

trained model (Algorithm 2, line 1) then we fine-tune it on augmented features (Algorithm 2, line 4). This is the general setting for domain adaptation. However, since our method performs domain adaptation under the assumption of label preservation, we also experimented training the model from scratch on augmented features. The results (Table 13) show the importance of the first, source-only training step.

## References

Arjovsky M, Bottou L, Gulrajani I, et al (2019) Invariant risk minimization. arXiv preprint arXiv:190702893

Ben-David S, Blitzer J, Crammer K, et al (2010) A theory of learning from different domains. ML

Bommasani R, Hudson DA, Adeli E, et al (2021) On the opportunities and risks of foundation models. arXiv preprint arXiv:210807258

Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: COMP-STAT

Chen LC, Papandreou G, Kokkinos I, et al (2017) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE T-PAMI

Chen LC, Zhu Y, Papandreou G, et al (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV

Chen Q, Wang Y, Yang T, et al (2021a) You only look one-level feature. In: CVPR

Chen Y, Wang H, Li W, et al (2021b) Scale-aware domain adaptive faster r-cnn. IJCV

Cheng B, Collins MD, Zhu Y, et al (2020) Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In: CVPR

Cordts M, Omran M, Ramos S, et al (2016) The cityscapes dataset for semantic urban scene understanding. In: CVPR

Fahes M, Vu TH, Bursuc A, et al (2023) Poda: Prompt-driven zero-shot domain adaptation. In: ICCV

Fahes M, Vu TH, Bursuc A, et al (2024a) Fine-tuning clip's last visual projector: A few-shot cornucopia. arXiv preprint arXiv:241005270

Fahes M, Vu TH, Bursuc A, et al (2024b) A simple recipe for language-guided domain generalized segmentation. In: CVPR
Fan Q, Segu M, Tai YW, et al (2023) Towards robust object detection invariant to real-world domain shifts. In: ICLR

Gal R, Patashnik O, Maron H, et al (2022) StyleGAN-NADA: CLIP-guided domain adaptation of image generators. In: SIGGRAPH

Ganin Y, Ustinova E, Ajakan H, et al (2016) Domain-adversarial training of neural networks. JMLR

Gao P, Geng S, Zhang R, et al (2024) Clip-adapter: Better vision-language models with feature adapters. IJCV

He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: CVPR

Hoffman J, Tzeng E, Park T, et al (2018) Cycada: Cycle-consistent adversarial domain adaptation. In: ICML

Huang X, Belongie S (2017) Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV

Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML

Jatavallabhula KM, Kuwajerwala A, Gu Q, et al (2023) Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:230207241

Jia C, Yang Y, Xia Y, et al (2021) Scaling up visual and vision-language representation learning with noisy text supervision. In: ICML

Jiang Z, Xu FF, Araki J, et al (2020) How can we know what language models know? ACL

Karras T, Laine S, Aila T (2019) A style-based generator architecture for generative adversarial networks. In: CVPR

Kirillov A, Girshick R, He K, et al (2019) Panoptic feature pyramid networks. In: CVPR

Kirillov A, Mintun E, Ravi N, et al (2023) Segment anything. In: ICCV

Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Neurips

Kwon G, Ye JC (2022) Clipstyler: Image style transfer with a single text condition. In: CVPR

Lengyel A, Garg S, Milford M, et al (2021) Zero-shot day-night domain adaptation with a physics prior. In: ICCV

Li B, Weinberger KQ, Belongie S, et al (2022) Language-driven semantic segmentation. In: ICLR

Li G, Kang G, Liu W, et al (2020) Content-consistent matching for domain adaptive semantic segmentation. In: ECCV

Li J, Selvaraju R, Gotmare A, et al (2021) Align before fuse: Vision and language representation learning with momentum distillation. In: NeurIPS

Li Y, Yuan L, Vasconcelos N (2019) Bidirectional learning for domain adaptation of semantic segmentation. In: CVPR

Lin TY, Dollár P, Girshick R, et al (2017) Feature pyramid networks for object detection. In: CVPR

Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: CVPR

Long M, Cao Z, Wang J, et al (2018) Conditional adversarial domain adaptation. In: NeurIPS

Lu Y, Liu J, Zhang Y, et al (2022) Prompt distribution learning. In: CVPR

Luo Y, Liu P, Guan T, et al (2020) Adversarial style mining for one-shot unsupervised domain adaptation. In: NeurIPS

Minderer M, Gritsenko A, Stone A, et al (2022) Simple open-vocabulary object detection with vision transformers. In: ECCV

Ovadia Y, Fertig E, Ren J, et al (2019) Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In:
Pan F, Shin I, Rameau F, et al (2020) Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In: CVPR

Patashnik O, Wu Z, Shechtman E, et al (2021) Styleclip: Text-driven manipulation of stylegan imagery. In: ICCV

Pizzati F, Cerri P, de Charette R (2021) Comogan: continuous model-guided image-to-image translation. In: CVPR

Pizzati F, Lalonde JF, de Charette R (2022) Manifest: Manifold deformation for few-shot image translation. In: ECCV

Radford A, Kim JW, Hallacy C, et al (2021) Learning transferable visual models from natural language supervision. In: ICML

Ren S, He K, Girshick R, et al (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: NeurIPS

Rezaianaran F, Shetty R, Aljundi R, et al (2021) Seeking similarities over differences: Similarity-based domain alignment for adaptive object detection. In: ICCV

Richter SR, Vineet V, Roth S, et al (2016) Playing for data: Ground truth from computer games. In: ECCV

Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: MICCAI

Sakaridis C, Dai D, Van Gool L (2018) Semantic foggy scene understanding with synthetic data. IJCV

Sakaridis C, Dai D, Van Gool L (2021) Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding. In: ICCV

Santurkar S, Tsipras D, Ilyas A, et al (2018) How does batch normalization help optimization? In: NeurIPS

Shin T, Razeghi Y, Logan IV RL, et al (2020) Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In: EMNLP

Sun B, Saenko K (2016) Deep coral: Correlation alignment for deep domain adaptation. In: ECCV

Sun P, Kretzschmar H, Dotiwalla X, et al (2020) Scalability in perception for autonomous driving: Waymo open dataset. In: CVPR

Tsai YH, Hung WC, Schulter S, et al (2018) Learning to adapt structured output space for semantic segmentation. In: CVPR

Ulyanov D, Vedaldi A, Lempitsky V (2016) Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:160708022

Ulyanov D, Vedaldi A, Lempitsky V (2017) Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In: CVPR

Vidit V, Engilberge M, Salzmann M (2023) Clip the gap: A single domain generalization approach for object detection. arXiv preprint arXiv:230105499

Vu TH, Jain H, Bucher M, et al (2019) Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In: CVPR

Wah C, Branson S, Welinder P, et al (2011) The caltech-ucsd birds-200-2011 dataset. California Institute of Technology

Wang J, Sun K, Cheng T, et al (2020a) Deep high-resolution representation learning for visual recognition. TPAMI

Wang S, Chen X, Wang Y, et al (2020b) Progressive adversarial networks for fine-grained domain adaptation. In: CVPR

Wang Y, Li W, Dai D, et al (2017) Deep domain adaptation by geodesic distance minimization. In: ICCV Workshops
Wu A, Deng C (2022) Single-domain generalized object detection in urban scene via cyclic-disentangled self-distillation. In: ECCV

Wu X, Wu Z, Lu Y, et al (2022) Style mixing and patchwise prototypical matching for one-shot unsupervised domain adaptive semantic segmentation. In: AAAI

Yang Y, Soatto S (2020) Fda: Fourier domain adaptation for semantic segmentation. In: CVPR

Zhao H, Shi J, Qi X, et al (2017) Pyramid scene parsing network. In: CVPR

Zhao H, Qi X, Shen X, et al (2018) Icnet for real-time semantic segmentation on high-resolution images. In: ECCV

Zhong Z, Friedman D, Chen D (2021) Factual probing is [mask]: Learning vs. learning to

recall. In: NAACL

Zhou C, Loy CC, Dai B (2022a) Extract free dense labels from clip. In: ECCV

Zhou K, Yang J, Loy CC, et al (2022b) Conditional prompt learning for vision-language models. In: CVPR

Zhou K, Yang J, Loy CC, et al (2022c) Learning to prompt for vision-language models. IJCV

Zou Y, Yu Z, Kumar B, et al (2018) Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: ECCV

Zou Y, Yu Z, Liu X, et al (2019) Confidence regularized self-training. In: ICCV