Using Language to Extend to Unseen Domains
Authors: Lisa Dunlap
Clara Mohri
Devin Guillory
Han Zhang
Trevor Darrell
Joseph E. Gonzalez
Aditi Raghunathan
Anna Rohrbach
Venue: ICLR 2023 notable top 25% (2023)
Keywords: vision and language
robust training
domain adaptation

Abstract: It is expensive to collect training data for every possible domain that a vision model may encounter when deployed. We instead consider how simply $\textit{verbalizing}$ the training domain (e.g.``photos of birds'') as well as domains we want to extend to but do not have data for (e.g.``paintings of birds'') can improve robustness. Using a multimodal model with a joint image and language embedding space, our method $\textit{LADS}$ learns a transformation of the image embeddings from the source domain to each target domain, while preserving task relevant information. Without using any images from the target domain, we show that over the $\textit{extended}$ domain containing both source and target, $\textit{LADS}$ outperforms standard fine-tuning and ensemble approaches over a suite of 4 benchmarks targeting domain adaptation and dataset bias.

Published as a conference paper at ICLR 2023
USING LANGUAGE TO EXTEND TO UNSEEN DOMAINS
Lisa Dunlap, Clara Mohri
Aditi Raghunathan
UC Berkeley
Camegie Mellon University radit t @cmu.ed
(lisabdunlap, cmohri (mmohri)bbereeeey.ee
Han Zhang, Devin Guillory, Trevor Darrell, Joseph E. Gonzalez, Anna Rohrbach
UC Berkeley
trevordarrell for. jegonzal. anna. row...hhhhacceeerreeee.....o
ABSTRACT
It is expensive to collect training data for every possible domain that vision model may encounter when deployed. We instead consider how simply verbal- izing the training domain (e.g. "photos of birds") well domains want to extend to but do not have data for (e.g. "paintings of birds") can improve ro- bustness. Using a multimodal model with joint image and language embedding space, our method LADS learns a transformation of the image embeddings from the training domain each unseen test domain, while preserving task relevant information. Without using any images from the unseen test domain, we show that over the extended domain containing both training and unseen test domains, LADS outperforms standard fine-tuning and ensemble approaches over suite of four benchmarks targeting domain adaptation and dataset bias. Code available at https: /tt::///giae /ggtttbtt...//// com/ lissldaaaue wwwaadnnaappADDD..
Training Domain
Augmentation using descriptionS
Extended Domain
"Sto Sipe" similartty
Training Domain
Unseen Domsin
STOP
STOP
LIKE
15
"SHOMy
"Sunny" similority
"Snowy"" similarity
A mix of training + domain samples
signs, want mnovy road signs.
Joint Vision & Language Embedding Space
Figure 1: Consider model trained recognize road signs in sunny weather. We aim extend to a new domain of snowy weather. Our method LADS (Latent Augmentation using Domain descrip- tionS) leverages multimodal model's knowledge the classes and the domain shift verbalized in natural language ("sunny" to "snowy") to train an augmentation network without any samples from unseen test domain. This network is used to translate multimodal image embeddings from the training domain to the unseen test domain, while retaining class-relevant information. Then, real and augmented embeddings are used jointly to train a classifier.
INTRODUCTION
The ability to extend model beyond the domain of the training data central building robust computer vision models. Methods for dealing with unseen test distributions often require leveraging additional image data, but linguistic knowledge of the anticipated domain shift is much cheaper and easier to obtain. For example, in many settings, the training images are collected in certain
"quull contribution

---

Published as a conference paper at ICLR 2023
conditions (e.g., daylight, clear weather, ...) but sensors may also experience less common but easy to anticipate conditions (e.g., night, snow, haze, illustrations, ...). Directly collecting or creating data in all possible anticipated settings is often prohibitively expensive. Thus it is of great interest how one can linguistically extend to unseen domains: that is, to utilize language to improve performance on an unseen test domain without sacrificing performance on the training domain.
The use of language in domain generalization has generated significant interest with the development of large vision-language models such as CLIP (Radford al., 2021), Flamingo (Alayrac et al., 2022), and ALIGN (Jia et al.. 2021), which allow users to create zero-shot classifiers using only class names. However, while these models have been shown achieve remarkable cross-domain generalization, their zero-shot classifiers often perform far worse than models trained for particular downstream task (Radford et al., 2021; Kumar et al., 2022). When training data available for the downstream task, a common practice is to fine-tune these models on the training data. While this significantly improves in-domain accuracy, it degrades performance on unseen domains.
We show that it is possible to leverage the domain-level knowledge (e.g. sunny environments vs. snowy environments in our example) contained in CLIP or similar models to deal with a variety of domain shifts in a way that requires no data from the new test domain, exploits the labeled training data, and is fast to train. Our method only requires users to imput text descriptions of the training and unseen test domains (e.g. "a sunny stop sign" and "a snowy stop sign") along with their training gaguuag--gudee domain generalization, we leverage the broad domain knowledge
data. To achieve encoded in CLIP coupled with its shared mmage-language embedding space perform latent feature augmentation of the training set.
More precisely, the embeddings of these textual descriptions are used to train an augmentation model which learns a transformation on the CLIP image embeddings of the training domain and "places" them in the new domain (see Figure 1). We train this augmentation model with two objectives: (1) translating the image embedding from the training domain to the unseen testing domain, while (2) retaining the class-specific information of the original image. Once this transformation is learned, we train a simple linear classifier on the combined augmented and unaugmented image embeddings, resulting in classifier that outperforms common fine-tuning methods on the extended domain while achieving similar performance on the training domain.
We introduce LADS, method to extend a model to new domains given only a language description of the distribution shift. Our main contributions are (1) the introduction of the Domain Extension with Language problem,
(2) a novel language-guii gugded latent feature augmentation training proce- dure, and (3) the extension of our method to address spurious correlation biases in the training data.
We evaluate LADS on two domain adaptation benchmarks, DomainNet (Peng et al., 2019) and CUB- Paintings (Wang et al., 2020), well as two benchmarks exhibiting color and contextual bias, Colored MNIST (Arjovsky et al., 2021) and Waterbirds (Sagawa et al., 2019). On the domain adap- tation benchmarks, we show that improve out-of-domain performance by 1-3% while matching in-domain performance of fine-tuned and ensembled models. On the biased benchmarks, we show an almost 2x improvement in out-of-domain performance over fine-tuned models. Across all bench- marks, LADS achieves the highest accuracy on the entire extended test domain containing both training and unseen test domain samples. Finally, we perform an in-depth analysis of the altered image embeddings, the effect of each loss function, and the effect of different vision and language models to understand our framework better.
2 RELATED WORK
Domain Adaptation/Gennerlizattii
The challenge of out-of-domain generalization is well stud- ied (Recht et al., 2019; Petryk et al., 2022; Kumar et al., 2022; Santurkar et al., 2021; Hendryeks & Dietterich, 2019) with large body of work in domain adaptation addressing the problem of adapt- ing a model to perform well on a new target domain. A typical domain adaptation approach involves collecting additional unlabeled data from the target domain (Ganin & Lempitsky, 2015; Saito al., 2017; Arjovsky et al., 2021: Kim et al., 2018; Tzeng et al., 2015), and aims to train a classifier such that it cannot tell the difference between source and target domain.
In the limited data setting, few-shot domain adaptation (Motian et al., 2017; Yue et al., 2021) aims to learn from little one example in the target domain. Work in domain generalization (Wang &
2

---

Published as a conference paper at ICLR 2023
Jiang. 2020; Gulrajani & Lopez-Paz. 2021; Koh et al., 2021) does not need target domain data but requires a set of several aligned and labeled source domains, and often shows only limited gains. While we evaluate on certain domain adaptation benchmarks, DA/DG methods primarily focus on maximizing target domain accuracy, while our work is interested in maximizing the accuracy of the extended domain. Furthermore, unlike previous works, assume have no access any target data (labeled or unlabeled), only a single source domain, and our domain shift can be verbalized.
Fine-tuning under Distribution Shift. The goal of fine-tuning under distribution shift is tailor pretrained models to a specific task without sacrificing their ability to deal with distribution shifts. Kumar al. (2022) found that it better to fit linear probe on the features and then fine-tune the model's 's backbone. For robust fine-tuning of CLIP specifically, Wortsman et al. (2021) proposed ensembling the weights of the fine-tuned image encoder with the zero-shot image encoder. We see our work as complementary to these ideas, targeting semantically defined domain shifts to increase OOD performance, while maintaining high ID performance.
Semantic Augmentation with CLIP. With the emergence of CLIP, several works (Ramesh et al., 2022; Patashnik et al., 2021: Gal et al., 2021) have used language to alter images using a combination of CLIP and generative model. Broadly, these works translate an image to a CLIP embedding, alter the image embedding with a text embedding of the desired augmentation, and use that embedding to generate altered image. These CLIP-based works do not attempt to use these data augmentations in the context of dataset bias or domain adaptation. Some prior work has explored augmentations using generative models (Sharmanska et al., 2020; Sankaranarayanan al., 2018; Yan et al., 2021), but since they generate images at the pixel level, they are often bottle-necked by the quality of the generative process. In contrast, we choose manipulate embeddings directly that allows us to effectively distill the knowledge in CLIP.
Removing Dataset Bias. In computer vision, several works debias data using extra information such as instance annotations (Hendricks et al., 2018; Li et al., 2018: Rieger et al., 2020), bounding boxes (Choi et al., 2019), or image-level bias annotations (Kim al., 2018). Some methods (Sharmanska et al., 2020; Bau et al., 2020; Santurkar et al., 2021) forego the need for expensive annotations by utilizing generative models, while Petryk et al. (2022) utilize CLIP to translate language descriptions of task into spatial guidance. In contrast, we do not limit ourselves to purely spatial bias or use per-image annotations of the bias, only a description of what biases may appear in the training data.
3 LATENT AUGMENTATION USING DOMAIN DESCRIPTIONS
We consider the supervised learning problem of generalizing to new unseen domains using only the verbal descriptions of the training domain and the anticipated but unseen new domains. More formally, we are given training dataset {x1, 3/ } drawn from the training domain D Dusining the class names by. a written description fraining of the training domain, and a set of written descriptions (finsen / of i unseen domains { (Dinseen 1 | that we expect to encounter at test time. Our goal is to train a model that performs well on both the original domain Dunining well as the unseen domains 1 We call this the Domain Extension with Language problem.
Large vision-language models have demonstrated the ability to generalize to new domains with language but only in the zero-shot setting. In order to utilize available training data, we explore the popular fine-tuning technique of linear probing: fitting linear classifier to the image embeddings of large vision-language models. We chose linear probing over full fine-tuning it faster train and has been shown to result in more robust classifiers (Kumar et al., 2022; Radford et al., 2021).
While standard linear probing only uses the image embeddings and the numerical labels, LADS also utilizes the text describing the classes and the descriptions of domain shift to augment the probe's training data to mimic samples from the unseen domain. Our two-stage approach first learns a network that transforms the image embeddings rather than the pixels themselves, with the goals of (I) augmenting the embedding to be aligned with the unseen domain while (2) retaining the features consistent with its class label. The second stage performs linear probing on the training set containing both the original image embeddings as well as the augmented image embeddings to produce classifier that is more robust to the specified domains. Note that we do not any data from D Dinseen in either stage-we we only use the class names and domain descriptions. An outline of the first stage of our method (training the augmentation network) is depicted in Figure 2.
3

---

Published as a conference paper at ICLR 2023
faug
10
LADS
Viston & Language Madel
LDA
domainx
LA
Frreniig ploto of'a bied!
Yallewere)
Tuscesn painting
To
Ccc
Loc
Inbels
y ("Puttir,, "Sparow"]
Tal Spuroo))
Anfficinsttd
User Supplied Imputs
image cmnbeldings
Figure 2: LADS. Let the task be classify Puffin vs. Sparnow. The training data Draining contains photos of the two classes but we would like to extend our classifier to paintings as well: that is, Dunscen- We aim do this using the text descriptions of the training and new domain, mmining and lunseen: respectively. The augmentation network sag is trained to transform image embeddings from D Drsining D Dusseen using a domain alignment loss LDA and a class consistency loss Lec. When LDA is low, the augmented embeddings are in the new domain but may have drifted from their class. may fail reflect the
When Lee is low, the augmented embeddings will retain class information desired change in domain. Saug aims to augment every image embedding to space with low domain alignment loss and low class consistency loss, resulting in ( having an image embedding similar to painting of Puffin. Note that the ballucinated image embeddings on the right are pictorial representation the effect of each loss function and not actually generated by LADS.
We choose CLIP (Radford et al., 2021) as our vision- Asioonlanguage model in our evaluations. Let Lg(x) = CLIP imge (x) I denote the image embedding of input image x and (t(t) CLIP denote the CLIP text embedding of some text Furthermore, let fraining a ly denote the composition of the domain description and the class name. For example, if trrainigg
photo of a", &unseen = "a painting of a" and ty could be "Puffin". The composition tuaining ty is "a photo of Puffin"
Stage 1: Training the augmentation network. The first stage of LADS is to learn augmen- tation network fing + I that transforms image embeddings from Duaining to D Dessenn using the corresponding language descriptions fuaining and t As mentioned previously, a valuable aug- mentation is one which places the transformed embedding in unseen domain D while retaining features relevant to the class label. To achieve this, we train f using a combination of two losses: Domain Alignment and Class Consistency. the setting of adapting multiple new domains at once, we train a unique flng network for each domain as described above.
Domain Alignment. The domain alignment loss encourages the augmented image embeddings f (I)(x)) to look like image embeddings from the new domain D D This loss is guided by the text embeddings of the domain descriptions 1 kussenn and liraining-
While CLIP is trained such that the space of image embeddings has some correspondence with the space of text embeddings T. it not obvious what mapping between I and T should look like. Thus, inspired by prior work (Patashnik et al., 2021; Gal et al., 2021), we assume the existence of shift from Dunining D & that is shared across both the
"global direction" that corresponds image embedding space and text embeddings space.
This "global direction" is defined as the normalized difference of the embeddings from the target domain and the embeddings from the source domain. Formally, the domain alignment loss of fl for training point (x,, Ui)
Sann (Ix()))) (xi) [ (lo(x)) (x)) [To "T (funseen: yi) Ta (farinnigg: y:)il
To Ttuunscea y:) To (faaining: Vi)
LDA(Ft sayg
(I)
Class Consistency. The domain alignment loss in Equation I encourages the augmented embeddings to only differ in the direction of change in the domain. If there were one global shared direction corresponding to the domain shift, optimizing Los would be sufficient. However, in practice,
4

---

Published as a conference paper at ICLR 2023
find that optimizing LDA alone removes some class relevant information and results in little diversity among the augmented embeddings different images (see Section 4.6 and Figure 11). Thus we add class consistency loss which preserves class information in the augmented embeddings. We measure class information in the image embeddings by our ability to classify the images accurately via CLIP zero-shot with the class names. Formally,
(fe(f
= -Eo 1=1
ceateentropyooffmaayy (To(x;) (o6)))).
Note that this is the same objective the standard CLIP loss. We CLIP zero-shot rather than trying to fine-tune CLIP because that could lead to overfitting where we classify the augmented training image embeddings correctly even when they do not contain class relevant information.
Our final objective LLADS( (fmg) to train the augmentation network as the first step in LADS is a linear combination of the the domain alignment loss and class consistency loss:
LADSS (f = aDL (f + (1 - ax) Lce (fmmg)
(3)
where & is hyperparameter dictating the trade-off between domain alignment and class consistent.
Stage 2: Fine-tuning. After the augmentation network aug is trained, we train a linear probe on the original image embeddings (x(x:) along with the augmented embeddings fig (Ix(.:). Infer- ence is straightforward: apply the linear probe on the CLIP image embeddings of the test images.
3.1 ADDRESSING DATASET BIAS
In addition to dealing with extended domains, LADS can also be used in the dataset bias setting where there are spurious correlations in the dataset. For example, in Waterbirds (Sagawa et al., 2019), we want to classify Landbirds vs. Waterbirds, where the spurious correlation is the back- ground (Landbirds appear on forest backgrounds and Waterbirds appear on water backgrounds in training). To prevent a classifier from using this correlation to make predictions, we can use LADS to generate augmentations that represent "Landbird on water" and "Waterbird on land".
We do this by using CLIP to label the backgrounds of each image and then decide what Graining and bunseen is per example. Given the domain information hand = "a O the forest" and bmatee = "a {} on the water", we can use zero-shot CLIP to determine if given image is land or water. If the image is predicted to on land, when training fagg. LDA for that particular example will use Eraining fland, tunseen twater and vice versa. The class consistency loss and the other parts of the pipeline remain unchanged. Because we are using the vision and language model to label the domains, we do not need per-image labels of the bias, only hypothesis of what the bias may be.
4 EXPERIMENTS
In this section we discuss our main experiments and results. We defer dataset details, the remainder of the experiments and their discussion to the Appendix (B, D. E). E)
4.1 IMPLEMENTATION DETAILS
In line with Radford et al. (2021), we normalize all text and image embeddings when performing zero-shot inference training with CLIP embeddings. The augmentation network faug used in LADS is a 2-layer MLP with input and output dimensions of 768 and a hidden dimension 384. Within LADS and all the relterelated baselines, backbone and resize all images to 224x224. We train on 10 GeForce RTX 2080 Ti GPUs.
the OpenAl CLIP model with VIT-L
For each baseline, do hyperparameter sweep across learning rate and weight decay and choose the parameters with the highest class-bal balanced validation accuracy. For LADS we also do a sweep across the parameters of the augmentation network, namely learning rate, weight decay, and and
5

---

Published as a conference paper at ICLR 2023
select checkpoint based on the validation loss. In general, we 0.5, Ir 0.001, wd = 0.05. Our hyperparameter search spaces and final choice of hyperparameters are listed in Table 4.
In our results report test accuracy on Darining Dunscen and the extended domain which averages the two. We run each method over 5 different random seeds and report the mean and standard deviation.
4.2 DATASETS
CUB-Paintings (one new domain) is composed of 2 datasets, CUB-200 (Wah al., 2011), tie-ggrined bird classification benchmark containing 200 different bird species and CUB-200- Paintings (Wang al., 2020), which contains the same classes as CUB-200 but instead of real images they are paintings collected from the web and filtered manually. We use the domain descrip- tions Vrrainigg = "a photo a (} bird", ( "a painting of a { bird".
DomainNet (multiple new domains) is a specific split (Tan et al., 2020) of the original Do- mainNet (Peng et al., 2019) dataset which contains the 40 most common classes from 4 do- mains: 'sketch', 'real', 'clipart', and 'painting'. Like prior work (Kumar et al., 2022; Tan et al., 2020), we train on sketches and evaluate on the three other domains. We use the domain de- scriptions furining = "a sketch of tu "clipart of a "a realistic photo of a.".
2 uunseen "a painting of +3 uuneen
Colored MNIST (color bias) (Arjovsky et al., 2021) was made by taking the original MNIST Digits (Deng, 2012), and coloring them red or blue. In the training and validation sets, numbers are red and odd numbers are blue, while in the randomly. The task is to classify the digits 0, ..,,, 9. We use the domain descriptions "a photo of red number "a photo of blue number
digits are colored
Waterbirds (contextual bias) (Sagawa et al., 2019) is a synthetically created dataset which creates contextual bias by taking species landbirds and waterbirds from the CUB-200 Wah et al. (2011) dataset and pasting them on forest and water backgrounds from the Places (Zhou et al., 2017) dataset. For the training and validation sets, all landbirds appear on forest backgrounds and waterbirds appear on water backgrounds while the test set has an even representation of backgrounds and bird types. We use the domain descriptions "a photo of a () in the forest", "a photo of a (} on the water".
4.3 BASELINES
Generic and Adaptive zero-shot CLIP are the zero-shot baselines proposed by Radford et al. (2021): (CLIP ZS (G)) uses the class name alone as the text prompt, while adaptive zero-shot CLIP (CLIP ZS (A)) caters the text prompts to the specific domains (e.g. "a painting of an airplane."). To do well on the extended domain, we average the text embeddings of each class across all possible domains.
CLIP I. fits a linear classifier on top of the CLIP image embeddings.
CLIP LP (ZS inăt) initializes the linear classifier with the text embeddings.
WISE -LP (Wortsman et al., 2021) is an ensembling technique which fine-tunes a CLIP model and does weighted average of the fine-tuned model's weights with the original. Due to the size of the vision and language models we are using, we did not fine-tune the entire backbone and instead ensembled the classifier with the linear classifier probe as explained by Wortsman al. (2021).
VQGAN + CLIP (Crowson et al., 2022) is a method that uses a VQGAN (Esser et al., 2021) trained with CLIP to augment images in pixel space. Using a text prompt and an image, we perform "style transfer" to the new domain in order to augment the training data. We then train a linear probe on the augmented and non-augmented CLIP embeddings. Due to the amount of time and compute required to generate images, we only ran this baseline for DomainNet and augmented approximately 15% of the training dataset. Examples of the augmented images are provided Table
4.4 RESULTS
Table I shows in-domain (ID) and oft-of-dommin (OOD) accuracy on CUB-Paintings and Domain- Net. The "Fxtended" column is the average accuracy of the two, corresponding the full extended

---

Published as a conference paper at ICLR 2023
domain. For CUB-P -UB-Paintings and DomainNet, LADS is able to match improve the ID accuracy of the fine-tuning baselines while improving over their OOD accuracy. Although CLIP zero-shot achieves higher OOD accuracy DomainNet, LADS achieves the highest result when evaluated on the full extended domain. We also improve over the VQGAN+CLIP baseline on DomainNet.
For Colored MNIST (Figure 3a) and Waterbirds (Figure 3b), LADS is able roughly match ID accuracy of the fine-tuned CLIP and OOD accuracy of CLIP zero-shot, resulting in approximately a 10% improvement the extended domain. We explore different weighted averages of ID and ODD accuarcy to compute the extended domain accuracy in Section of the Appendix.
Dataset
Method
ID
OOD
Extended
-UB-Paintings
CLIP 7S (G)
60.34%
52.84% 54.38%
56.59%
CUR-Paintings
CT.IP 7 (A)
61.93%
58.16%
CUB-Paiie CUB-Paintings
CLIP L.P
85.91=0.088 %
64.33++0.299
75.12:+0.18%
CUB-Paintings
CLIP LP (ZS imit)
8..08=0.111
65.05++0.05%
75.57=+0.064%
LUB-taint
WiSE-LP
81.7410..4%%
64.80-10.
73.27:10.22%
CUB-Pain PuB--ain
LADS
86.14.10.9%%
66.1811 0.25 %
76.16.10.23% %
DomainNet
CLIP ZS (G)
93.49%
95.94%
94.72%
DomainNet
CLIP (A)
93.24%
96.01% %
94.62%
DomainNet
CLIPLP
95.03=#0.07% %
93.75++0.02%
94.39
DomainNet
CLIP LP (ZS init)
95.21=0.211% %
93.95+00.03%
94.58 11%
DomainNet
WISE-LP
95.19 0.34%
93.68 0.12%
94.44
DomainNet
VQGAN+CLIP
95.54 0.09%
93.83-1 0.10%
94.67 1 0.09%
DomainNet
LADS
95.33 0.33%
95.21 +: 0.09%
95.27 + 0.14%
Table I: In-domain (ID), out-of-domain (OOD) and extended domain accuracy CUB-Paintings and Nem For NomainNet, we include the pixel augmentation baseline VQGAN+CLIP and OOD accuracy is the average of the 3 unseen domains. LADS is able to beat all methods on the extended domain for both datasets. Note that for tasks where CLIP zero-shot does not perform well, LADS is able significantly outperform zero-shot on the unseen domain.
100
100
1:0
60-
93-
02
50
Accurdyy (8)
71.30
(a) Colored MNIST.
103
21.79
$3.42
TUPEE
25.M0
TAPS
90
92
20
ACCuREy (8)
AcC/oXy
53
19
60
(b) Waterbirds.
Figure 3: Result on dataset bias benchmarks. Left and center plots show the training domain and unseen domain performance of the zeroshot and fine-tuned baselines respectively. For both Colored MNIST (a) and Waterbirds (b), LADS is able to roughly match the unseen domain accuracy of zero-shot methods and the seen domain accuracy of fine-tuned methods, resulting in improved performance on the extended domain (right).

---

Published as a conference paper at ICLR 2023
CLIP LP
VQGAN+CLIP
LADS
Domain Alignment
81.30.1
69.26.12.11%
83.44.10.61%
Class Consistency score
91.42:#0.4%%
77.14
87.26+t0.74%
Table 2: Augmentation Quality for DomainNet. The domain alignment and class consistency scores over 1000 randomly sampled training embeddings and their nearest neighbor in the test set. LADS achieves higher domain alignment and class consistency scores than VQGAN+CLII, and is able to obtain somewhat similar class consistency score to the unaugmented image embeddings.
4.5 ANALYSIS OF AUGMENTATION QUALITY
In this section, explore the quality of the embeddings generated by augmentation network. We perform our analysis for DomainNet below, defering the remaining results to Appendix D.I, D.2.
Since there is no publicly available model convert CLIP embeddings images, neighbors of the augmented embeddings from the extended test domain to confirm that our augmen- tations match our expectations. We take random subset of 1,000 samples from the image embed- dings used to train the linear probe: for CLIP LP, this is simply (x1(x), Mi 3 == for VQGAN+CLIP it is of mix of (x)) )i and GAN generated images, and for LADS is {1)(x;), 1 and
the nearest
the augmented embeddings - (Tox)())())),, yi Y= for each unseen domain j. We obtain the nearest neighbors in the extended test set (containing images from the training and unseen domain) with respect to cosine similarity of the image embeddings.
In line with our domain alignment and class consistency loss functions, we define metrics for (1) correctly altering the domain of the image embedding, while (2) retaining the class information. We define the percentage of the nearest neighbors that belong in the desired domain as the domain alignment score, and the percentage that belong to the original class as the class consistency score.
The CLIP LP scores can be viewed as an approximate upperbound for those of LADS since they reflect the nearest neighbors of only the original sketch embeddings in the extended domain. As shown in Table 2, LADS is able to beat the domain alignment score and closely fall behind the class consistency score of the linear probe, implying that the augmentations are of similar quality to the original image embeddings. Furthermore, LADS has better domain aligmment and class consis- tency than VQGAN+CLIB indicating that the long and laborious pixel-level augmentation may be producing lower quality training samples than our simple embedding augmentation.
For qualitative analysis of LADS, we visualize random sample of 10 nearest neighbors from Do- mainNet in Figure 4 (the sketch embeddings are non-augmented, all others augmented). The nearest neighbors of augmented embeddings closely resemble embeddings of similar images in the desired unseen domain. Even if the nearest neighbor is of a different class, it maintains some visual similarity to the original image.
sketch-> clipart
sketch- painting
sketch real
Training Image
NN from Extended Domain
Figure 4: Nearest Neighbors for LADS on DomainNet Sketch - Clipart, Painting, Real. The top row shows training images with the label on top being the intended domain augmentation for that embedding. The bottom row shows the nearest neighbor of the augmentation in the extended domain. Not only does LADS produce augmented embeddings within the correct domain, embed- dings often match the specific class and stylistic elements of each original image.
8

---

Published as a conference paper at ICLR 2023
4.6 ABLATIONS
In this section, ablate the class consisteney and domain alignment loss described in Section 2. We defer the remainder of the ablations, including ablations of the domain descriptions and the CLIP model, Appendix D.3, D.4, D.5.
In order to measure the impact of the domain alignment and class consistency loss functions, we ablate each one and report the accuracy, domain alignment score, and class consistency score. We also experiment with a domain-specifie class consistency loss, which replaces T(y:) with T(tuneen y:) in order to enforce the class and domain all in loss. We display our results the Waterbirds dataset below, with experiments on the other datasets in Appendix D.3.
As shown in Figure 5, the domain alignment loss alone results in high domain alignment score, but low accuracy due to losing some class specific information. Meanwhile, the class consistency loss alone achieves the highest class consistency score because it retains the relevant class information, but it fails to improve the OOD accuracy since the augmented embeddings are not within the new domain. Even in the case of domain specific Lee when the extended domain is incorporated into the class consistency loss, the scores only slightly improve. It is only when we combine both our losses that we are able to retain class information while transforming the image embeddings to the desired domain, leading to improved out-of-domain accuracy. Nearest neighbor visualizations of the different losses are given in Appendix D.3.
90-
17
CLIP LP
Dom Roc
Las lcc
100
Accuraty (%0
80
000
78
11
Score (50
80
70
70
60
50
50-
16
Extended
Domain Alignment Class Consistancy
Figure 5: Effect of the Loss Functions. We report the results of training with just the domain alignment loss, the class consistency loss, a domain-specifti class consistency loss, and the domain alignment class consistency loss, Waterbirds. The DA loss results in high DA score but low accuracy. The CC loss results in low DA score and does not improve the OOD accuracy; the domain- specific CC variant brings negligible gains. Our final design CCpx+Lec) works the best.
LIMITATIONS AND FUTURE WORK
Since one must input a natural language description of the distribution shift, LADS may not apply to "natural" distribution shifts where the change cannot be verbalized (Koh al., 2021). Furthermore, as our approach is reliant on the richness of concepts learned by pretrained vision-languan model, is also limited to domains that can accurately represented with textual descriptions, and are well covered in the data the pretrained models were trained on. As general rule of thumb, if CLIP zero- shot has very poor performance when it comes to classifying the domains and/or classes, LADS should not be used (see Section E of the Appendix).
We have presented LADS, a fine-tuning method for addressing the task of Domain Extension with Language. We view LADS as a jumping-off point for further exploration regarding how can the zero-shot capabilities of large multimodal models to improve accuracy on a desired domain given only language description input. We hope that future work is able perform reliable embedding augmentations independently hand. Furthermore, we hope future work is able to analyze more complicated domain shifts such as the ones seen in WILDS Koh et al. (2021) or Imagenet-A A (Hendrycks et al., 2021).
the ability of CLIP to correctly classify the domains and classes at
Acknowledgements. This work supported in part by DoD including DARPA's SemaFor, PTG and/or LwLL programs, as well as BAIR's industrial alliance programs.

---

Published as a conference paper at ICLR 2023
REFERENCES
HannBaptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Mcnick, Schastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zis- serman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
Martin Arjovsky. Léon Bottou, Ishaan Gulrajani. and David Lopez-Paz. Invariant risk minimization. In Inter- national Conference on Learning Representations (ICLR), 2021.
David Bau, Steven Liu, Tongzhou Wang, Jun- Yan Zhu, and Antonio Torralba. Rewriting a deep generative model. European Conference on Computer Vision (ECCV), 2020. URL https://aaat wwtp://iioooooooooooooooo / 2007. 15646.
Jinwoo Choi, Chen Gao. C. E. Joseph Messou, and Jia-Bin Huang. Why can't i dance in the mall? learning to mitigate scene bias in action recognition. In NeurIPS, 2019.
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. 0rXiv:2204.08583 2022.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Process- ing Magazine, 29(6)1141-14,, 2012.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873-12883, 2021.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators, arXiv:2108. 00946, 2021.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. International
Conference on Machine Learning, 2015.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations (ICLR), 2021.
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snow- board: Overcoming bias in captioning models. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 771-787, 2018.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference Learning Representations, 2019.
Dan Hendrycks. Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021.
Gabriel Ilaaroo, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https: / /tw.///oiioo////////ooo orr/io.rr11.... 5281/ zenodo. 5143773.
Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916 PMLR, 2021.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. Conference on Computer Vision and Pattern Recognition (CVPR), 2018. URL /tatt:://axxst httpaaawwww.ga.aa///////aa/oo tabb/1812 10352.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the- wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664 PMLR, 2021.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference Learning Representations, 2022. URL https: wwwpee/wwwweweeeeeeooooomomm
10

---

Published as a conference paper at ICLR 2023
Kumpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ersst, and Yun Fu. Tell me where to look: Guided attention inference network. Conference on Computer Vision and Pattern Recognition, abs/1802. 101711 2018.
Saeid Motiian, Quinn Jones, Seyed Mehdi Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation, 2017.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski, Styleclip: Text-driven manipulation of stylegan imagery. Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021.
Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi- source domain adaptation. In Proceedings of IEEE/CVF international conference on computer vision, pp. 1406-1441111 1415, 2019.
Suzanne Petryk. Lisa Dunlap, Keyan Nasseri, Joseph Gonzalez. Trevor Darrell, and Anna Rohrbach, On guid- ing visual attention with language specification. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. doi:
URL https: http/arx.vooooo////ee abs 22002. 08926.
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI,, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv:2204.06123 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers general- ize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Inter- national Conference Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 5389-5400. PMLR, 09-15 Jun 2019. URL http:tps: http///pcceengs..eeooppppo mlr. www.mrressss......mmmmm /v97 http//echh1111 9a. html.
Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. Interpretations are useful: Penalizing explana- tions to align neural networks with prior knowledge. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural net- works. In International Conference on Learning Representations, 2019.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. Conference on Computer Vision and Pattern Recognition, 2017. URL http:://rma org/ 111. /tt.b///...22..00.
Swami Sankaranarayanan, Yogesh Balaji, Carlos Domingo Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. Conference on Computer Vision and Pottern Recognition (CVPR). 2018.
Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Aleksander Madry. Editing classifier by rewriting its prediction rules. In Neural Information Processing Systems (NeurIPS), 2021.
Viktoriia Sharmanska, Lisa Aune Hendricks, Trevor Darrell, and Novi Quadrianto. Contrastive examples for addressing the tyranny of the majority. CoRR, abs/2004.06524, 2020. URL UtLhttps://arxlv.oor org/ abs/ 2004. 06524.
Shuhan Tan, Xingchao Peng, and Kate Sacnko. Class-imbalanced domain adaptation: empirical odyssey. In European Conference on Computer Vision, pp. 585-602. Springer, 2020.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE international conference on computer vision. pp. 10644008440 4076, 2015.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 5018 5027. 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongic. The caltech-uesd birds- 200-2011 I dataset, 2011.
Jinghua Wang and Jianmin Jiang. Adversarial learning for zero-shot domain adaptation. In European Confer- ence on Computer Vision, pp- 329-344. Springer, 2020.

---

Published as a conference paper at ICLR 2023
Sinan Wang, Xinyang Chen, Yunbo Wang. Mingsheng Long, and Jianmin Wang. Progressive adversarial networks fine-gran one-grained domain adaptation. The IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 9213 122222 2020.
Mitchell Wortsman, Gabriel Ilharco, Mike Jong Wook Kim, Hannaneh Hajishirzi, Farhadi, Hongseok Namkoong, and Ludwig Schmidt, Robust fine-tuning of zero-shot models. CoRR, abs/2109.01903, 2021. /tt:: /arxiv..////xi..ooo/ wttp/rr/g/////////// 019933.
Hao Yan, Yuhong Guo, and Chunsheng Yang. Source-free unsupervised domain adaptation with surrogate data generation. The British Machine Vision Conference (BMVC), 2021.
Xiangyu Yue, Zangwei Zheng, Hari Prasanna Das, Kurt Keutzer, and Alberto Sangiovanni Vincentelli, Multi- source few-shot domain adaptation. arXiv preprint arXiv:2/09 10:::21999//91,, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
12


Reviews:
[{"review_id": "uVNxGUL-TL", "reviewer": "Reviewer_hq6S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The problem of domain extension with language is addressed in this paper. The proposed method (LADS) uses a CLIP model's domain-level knowledge to learn a latent feature augmentation of the training set. It does not require any unseen domain samples and instead relies on written descriptions of the training and unseen domains. Domain alignment and class consistency losses are used to train the latent feature augmentation. Once trained, a simple linear classifier is trained on both the original and augmented image embeddings, resulting in improved in-domain and out-of-domain recognition performance. Experiments on DomainNet, CUB-Paintings, Colored MNIST, and Waterbirds demonstrate that LADS outperforms other methods.", "summary_of_the_review": "The paper's contributions are novel. The technique is simple, but it works well. However, it is suggested that more research into domain alignment loss is required. Given the novelty of the paper, I would recommend a 'marginally above the acceptance threshold' rating.", "strength_and_weaknesses": "(Strengths)\n\n- The proposed method does not require any unseen domain image samples. It significantly reduces the cost of data collection.\n\n- LADS augments data at the image space level rather than the pixel level. This approach is useful for avoiding bottlenecks caused by the generative process's quality.\n\n- The proposed method can be extended to address training data biases.\n\n- Extensive experiments are carried out to demonstrate the performance of LADS in comparison to the state-of-the-art. Furthermore, the authors discuss the proposed method's limitations on the 'natural' distribution shift. \n\n(Weaknesses)\n\n- The text embeddings of the domain descriptions guide the domain alignment loss. During training, the image embeddings space changes while the text embeddings space remains constant. It is suspected that the text embeddings function properly as guidance.\n- Furthermore, the method is heavily reliant on the domain descriptions used. Taking the first two rows of table 3, the results are affected by the prompts. Considering the variations of the 'direction' instead of fixing a 'global direction' may be helpful.\n- Though the ablation study investigated the role of each loss term, it is suggested that the role of $\\alpha$ be investigated as well.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "l7RyChztfpe", "reviewer": "Reviewer_AEAE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a strategy to augment the image feature into different domains. The process is guided by CLIP to not just transfer to another domain but also keep the semantic content. In the experiment ,we can see that the proposed method, LADS, indeed performs well in both in-distribution and out-of-distribution. We can also qualitatively (Fig4) and quantitatively (Tab2) see that the model indeed learns to transfer the input feature from the training domain to a new domain, and at the same time keep the semantic meaning.", "summary_of_the_review": "This paper did a good job in the proposed setting, where the unseen domain can be easily described in a few words and is given as prior knowledge during training. However, this seems like a rather contrived setting as described in the weakness section of the review.", "strength_and_weaknesses": "## Strength\n\n1. The proposed method achieves good performance in both the in-distribution and the out-of-distribution data, without explicitly training on OoD labeled data.\n\n1. The analyses demonstrate that the model indeed successfully learns to transfer the feature into another domain (contribution of $L_{DA}$) and keep the semantic meaning (contribution of $L_{CC}$.\n\n## Weakness\n\n1. The biggest concern the reviewer has is how to describe the unseen domain in the form of simple text. In this paper, the setting is rather simple: there is a clear distinction between training and testing domains, and the difference can be easily described in a few words. As also pointed out in Tab3, the model performance is quite sensitive to the prompt, or domain descriptions.\n\n1. Related to the previous point, in Sec 3.1 the waterbird example, how to discover the additional prior knowledge that the \"spurious correlation is the background\" may not be trivial.\n\n1. The author should also compare to other more general methods in domain generalization, where only the training domain is available and the domain descriptions of testing domains remain unknown.\n\n1. Since the gap between CLIP-LP and LADS is not that big, how did the authors train the CLIP-LP model? For example, the ELEVATOR paper [1] pointed out that it is beneficial to initialize the clf layer weight by CLIP's language branch.\n\n[1] https://openreview.net/forum?id=hGl8rsmNXzs", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "l3nnCblYhBS", "reviewer": "Reviewer_3yhi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the Latent Augmentation using Domain Descriptions (LADS) framework to enhance the generalization ability of image classifiers across various domains. Specifically, LADS trains an augmentation module to transform the source domain image embeddings to multiple pre-known target domains, where domain alignment and class consistency are respectively constrained. After that, a linear classifier is trained with both source and target domain embeddings to achieve better generalization. Under both the settings of domain shift and dataset shift, authors demonstrate that LADS can achieve state-of-the-art performance on the extended domain (the collection of source and target domains).", "summary_of_the_review": "In summary, I am convinced by the motivation of studying under the proposed setting and believe the technical soundness. I give an acceptance on the current version and suggest authors to do more studies under the domain shift setting during the response period.", "strength_and_weaknesses": "Strength:\n\n1. This paper studies a practical problem that has been rarely explored, domain extension with language, where the domain gap is filled by utilizing the text descriptions of domains and classes instead of unlabeled target domain samples as in UDA.\n2. The proposed method is technically sound. By utilizing pre-trained vision-language alignment models like CLIP, LADS can well represent different domains and classes and further transformer such knowledge to the image modality.\n3. Extensive empirical results (performance comparison and ablation study) demonstrate the effectiveness of LADS on tackling domain and dataset shifts. \n\n\nWeakness:\n\n1. The experimental results under domain shift are not strong enough. For image recognition under domain shift, people are always most interested in the performance on OOD domains. On the two datasets studied in this paper, LADS achieve gains on CUB-Paintings while underperforms zero-short CLIP on DomainNet. Therefore, it is interesting to study under what circumstances LADS is necessary against directly apply the zero-shot CLIP. The current experiments on two datasets are not sufficient to illustrate this. I suggest the authors to do performance comparison on a more diverse set of benchmark tasks under domain shift, e.g., the Office-Home dataset with moderate domain gaps and the VisDA dataset with a larger domain gap. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%