Feed-Forward Latent Domain Adaptation
Authors: Ondrej Bohdal
Da Li
Shell Xu Hu
Timothy Hospedales
Venue: Submitted to ICLR 2023 (2023)
Keywords: latent domain adaptation
source-free
cross-attention
meta-learning

Abstract: We study the highly practical but comparatively under-studied problem of latent-domain adaptation, where a source model should be adapted to a target dataset that contains a mixture of unlabelled domain-relevant and domain-irrelevant examples. Furthermore, motivated by the requirements for data privacy and the need for embedded and resource-constrained devices of all kinds to adapt to local data distributions, we focus on the setting of feed-forward source-free domain adaptation, where adaptation should not require access to the source dataset, and also be back propagation-free. Our solution is to meta-learn a network capable of embedding the mixed-relevance target dataset and dynamically adapting inference for target examples using cross-attention. The resulting framework leads to consistent  improvement on strong ERM baselines. We also show that our framework sometimes even improves on the upper bound of domain-supervised adaptation, where only domain-relevant instances are provided for adaptation. This suggests that human annotated domain labels may not always be optimal, and raises the possibility of doing better through automated instance selection.

examples. Under review as a conference paper at ICLR 2023
FEED-FORWARD LATENT DOMAIN ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
We study the highly practical but comparatively under-studied problem of latent- domain adaptation, where a source model should be adapted to a target dataset that contains mixture of unlabelled domain-relevant and Furthermore, motivated by the requirements for data privacy and the need for embedded and resource-constrained devices of all kinds to adapt to local data dis- tributions, we focus on the setting of feed-forward source-free domain adaptation, where adaptation should not require access to the source dataset. and also be back propagation-free free.. Our solution is to meta-learn a network capable of embedding the mixeedrrlevaanee larget dataset and dynamically adapting inference for target examples using cross- atteentttnnion. The resulting framework leads to consistent im- provement on strong ERM baselines. We also show that our framework sometimes even improves the upper bound of domain-supervised adaptation, where only domain-relevant instances are provided for adaptation. This suggests that human annotated domain labels may not always be optimal, and raises the possibility of doing better through automated instance selection.
INTRODUCTION
Domain shift presents a real-world challenge for the application of machine learning models because performance degrades when deployment data are not from the training data distribution. This issue is ubiquitious as it is often impossible or prohibitively costly to pre-collect and annotate training data that sufficiently representative of test data statisties. The field of domain adaptation (Kouw & Loog, 2021 Csurka et al., [022) has therefore attracted a lot of attention with the promise of adapting models during deployment to perform well using only unlabeled deployment data. The main body of work in deep domain adaptation assumes that there is a pre-specified source domain and a pre-specified target domain. An unlabeled adaptation set is provided from the target domain, and various methods define different learning objectives that update deep model on the unlabeled adaptation set, with the aim of improving performance on new test data drawn from the target domain.
In this paper make two main contributions: A conceptual contribution, of a highly practical variant of the domain adaptation problem; and an algorithm for effective domain adaptation in this condition.
A motivating scenario Let introduce potential illustrative application scenario that motivates the variant of the domain adaptation problem that we propose here. Suppose that a robot or other mobile embedded vision system needs recognise objects. Because it is mobile, it may encounter objects in different unconstrained contexts, e.g., indoor or outdoor backgrounds, sunny or rainy weather, rooms with lights or lights off, etc. The robot's object recognition model should adapt in order to maintain strong performance across all these conditions, for example by adapting based on buffer of recently experienced unlabelled images. However, unlike standard pre-defined domain adaptation benchmarks with neatly curated domains, there are two new challenges: 1) Using such a buffer as the adaptation set means that the adaptation data can be of mixed relevance to the test image to be processed at any given instant. For example the recent history used for adaptation may span multiple rooms, while any individual test image comes from a specific room. 2) The adaptation needs to happen on-board the robot and ideally happen in real-time as the adaptation set itself is updated over time. The first challenge is the latent domain challenge, wherein uncurated adaptation sets do not have consistent relevance to take place without back proppaattion (which is too costly and not supported on most embedded platforms). It means adaptation should be feed-forward.
given test image (Fig. [T) The second challenge requires adaptation

---

Under review as a conference paper at ICLR 2023
Latent domain adaptation While domain adaptation is very well studied (Kouw & Loog 2021 Csurka et al., 2022). the vast majority of work assumes that instances have been pre-grouped into one or more subsets (domains) that differ statistically across groups. while being similar within groups. We join a growing minority (Mancini et al. 2021: Deecke et al., 2022 Hoffman et al., 2014 Wang (et al., (022)) in arguing that this is an overly restrictive assumption that does not hold in most real applications of interest. Some collection processes may not provide meta-data suitable for defining domain groupings, Alternatively, for other data sources that occur with rich meta-data there may be no obviously correct grouping and existing domain definitions may be sub-optimal (Deecke et al. 2022). Consider the popular iWildCam (Beery et al. 2020) benchmark for animal detection within the WILDS Koh et al. (2021) suite. The default setup within WILDS defines domains by camera ID. But given that images span different weather conditions and day'night cycles as well as cameras, such domains may neither be internally homogenous, nor similarly distinct. There may be more transfcrability between images from nearby cameras at similar times of day than between images from the same camera taken sunny day vs snowy night, As remarked by [Hoffman et al. (2014); Wang et al. (2022), domains may more naturally define a continuum, rather than discrete groups Thaat continuum may even be nulti-dimmnnsinal such as timestamp of image and spatial proximity of cameras. Our latent domain formulation of the domain adaptation problem spans all these situations where domains are hard to define, while aligning with the requirements of real use cases.
Feed-forward domain adaptation Unsupervised domain adaptation aims to adapt models from source datasets (e.g. ImageNet) the peculiarities of target data distributions in the wild. The mainstream line of work here updates models by backpropagation on an adaptation set from the target data distribution (Kouw & Loog 202T Csurka et al. 2022) (and often simultancously uses the source data (Liang et al. 20200). We consider adaptation under the practical constraints of an edge device, namely that neither the hardware capability nor the software stack support back- propagation. Therefore we focus on the feed-forward condition where adaptation algorithms should use only feed-forward operations, and only the target dataset (source free condition (Liang et al., 2020)). For example, simply updating batch normalisation statistics, which can be done without back- propagation, provides a strong baseline for backprop- free adaptation (Schneider et al. 2020 Zhang et al. 2021).
Our solution To solve the challenge posed above, propose ford-forward adaptation framework based on cross-attention between test instances and the adaptation set. The cross-attention module is meta-learnet based a set of training domains, inspired by [Zhang et al. (2021). This is one-off cost paid up-front and performed on a server, after which the actual adaptation is fast. The deployed recognition model flexibly enables each inference operation to draw upon any part of the target adaptation set, exploiting each adaptation instance to a continuous degree. This can improve performance by eliminating adaptation instances that would be conventionally in-domain yet lead to negative transfer (c.g., same camera/opposite time of day), and include transfer from adaptation instances that would conventionally be -f-main but could benefit transfer (e.g.. similar images/different camera). Our experiments show that our cross-attention approach provides useful adaptation this highly practical setting across a variety of synthetic and real benchmarks.
BACKGROUND AND RELATED WORK
Test-time domain adaptation TTDA has emerged as a topical adaptation scenario that focuses on model adaptation without access to source data at adaptation time (i.e., source-free condition). and further adapts each mini-batch test time, aligning with an online adaptation scenario. meta- learning framework for TTDA has been recently proposed under the name adaptive risk minimization (ARM) (Zhang et al. 202T). ARM provides variety of options for how TTDA is done, including context network that embeds information from the whole minibatch, updates to batch normalization statistics and gradient- based fine-tuning on the minibatch. ARM learns to do TTDA by meta-learning across a large number of tasks. TENT (Wang al. 2021 is another TTDA method and is based on optimizing channel-wise affine transformation according to the current minibatch.
Latent domains In settings with latent domains, information about domains is not available Le. there are no domain labels. Further, some domains may be more similar to each other so the boundaries between domains are often blurred. Various approaches have been proposed to deal with latent domains, e.g. sparse latent adapters (Deecke et al. 2022), domain agnostic learning (Peng et al. 2019) that disentangles domain-specific features from class information using an autoencoder
2

---

Under review as a conference paper at ICLR 2023
and (Mancini et al. 2021) that discovers multiple latent domains using specialized architecture with multiple branches. However, these methods focus on standard domain adaptation or supervised learning that is done across many epochs of training.
Meta-learning Meta-learning can take various forms (Hospedales et al. 202II. but one of the most common is episodic meta-learning where the learning is done across tasks. As part of episodic meta-learning, the goal is to learn perameersmsters that will help achieve strong performance when presented with a new task. The most popular application of episodic meta-learning is few-shot learning, where we try to e.g. learn to distinguish between different classes after seeing only a few examples of each class. Example meta- pattaaaaaaeeersss include weight initialization (Finn al. 2017|.Antoniou et al., 22199] Li et al., (017) or metric space as is the case in Prototypical networks (Snell et al., 2017) and RelationNet (Sung et al., 2018). Prototypical networks and RelationNet are both examples of ford-forward methods. Episodic meta-learning has also been used in domain generalization (Li et al., 2019) and as part of the ARM framework (Zhang et al. 2021). We also use the episodic meta-learning paradigm.
Transformers As part of our method, we use cross-attention that takes inspiration from the attention mechanism found in the transformer architecture (Vaswani et al. 2017). After transformers became common in natural language processing, they have also led to strong results within computer vision, most prominently as part of the ViT model (Dosovitskiy et al., 2021). VIT model has served as foundation for more recent vision transformers, including Cross ViT (Chen et al.] 2021) that combines strong performance with efficiency. Our cross-attention mechanism is broadly inspired by the Cross ViT cross-attention module (Chen et al. 2022). Our approach has also been inspired by the idea of non- non-parametric transformers (Kossen et al. (021) that are able to reason about relationships between data points. Different to Cross we use image-image attention, rather patch-patch and show how to exploit this for forwafo source- latent domain adaptation. Cross-attention has also found use in few-shot classification (Hou et al. [2019] Doersch et al., 2020; Ye et al. 2020). but these approaches use to obtain better representations of class prototypes for few-shot supervised learning rather than to reason about which unlabelled examples come from relevant domains to drive many-shot unsupervised learning.
3 METHODS
3.1 SET-UP
During deployment the high-level assumption made by many source-free domain adaptation frameworks is that we are provided with a predictive model fy and an unlabeled target adaptation dataset x, whose label-space is the same as that of the pre-trained model (Liang et al. 2020). Given these, source-free free DA approaches define some algorithm AA that ultimately leads to classifying test instance F.y Vy = üg A(Igq. Lgg. v)). There are numerous existing algorithms for this. For example, pseudo-label strategies (Liang et al., 2020 et al. 2020 Yang et al. (002)) proceed by estimating labels ÿ. for the adaptation set x:,, treating these as ground-truth, backpropagating to update the model v/ such that it predicts ÿ. and then classifying the test point as for (z,). We address the feed-forward setting where algorithm A should not use pack-propagation. For example, BN-based approaches (Schneider et al. 2020, Zhang et al. 202T) use the adaptation set æ, to update the BN statistics in vi as v and then classify the test point as fe(()))
Preliminaries
While the conventional domain adaptation setting assumes that target domain test examples 2g and target domain training examples 2, are all drawn from common distribution, the latent domain assumption has no such requirement. For example, F, may be drawn from mixture distribution and Rig may be drawn from only one component of that mixture. In this case only a subset of elements in T, may be relevant to adapting the inference for zq..
Deployment phase Rather than explicitly updating model parameters, we aim to define a flexible inference routine fo that processes both Ti and I, to produce ún in feed-forward manner, i.e., 0g = A(r Lz. +) = fu (Zg. x., In this regard our inference procedure follows a similar flow to variants of ARM (Zhang al.] 2021). with the following key differences: (1) ARM is transductive: it processes a batch of instances at once without distinguishing test instances and target adaptation set, so elements Ty members of x,. (2) ARM makes the conventional dosain-observed assumption
3

---

Under review as a conference paper at ICLR 2023
Latent domain adaptation:
Adaptation
Test image
images
Unobserved
Cameea day
Camera day
Camera 2.
Camera 2.
Domain:
night
day
Which adaptation image is relevant?
Standard domain adaptation:
Adaptation images
Test image
Observed Domain:
Camera 2
Camera 2
Camera
Camera
Illustration the standard and latent domain adaptation problem settings. the LDA setting (support) images come from variety of domains of mixed and unknown relevance the test (query) image. In standard DA adaptation images are all assumed to be equally relevant.
Figure 1:
that domains have been defined external oracle that ensures all Vi and æs drawn from the same distribution. We do not make this assumption and assess robustness to irrelevant elements in æ,.
Training phase To train a model that can be used as described above, we follow an episodic meta-lcarning paradigm (Zhang et al. 2021 Hospedales et al., (021). This refers to training fo using of simulated domain adaptation tasks. each iteration, we generate task with unique pair of query and support instances (I,, (bg. Fig)) keeping label space the same across all tasks. We simulate training episodes where æ, contains instances with varying relevance to Tig. The goal for Lo to Vg.
learn how to select and exploit instances from æ, in order to adapt inference for Ty to better predict
In particular, task sampler defines each task as having support examples uniformly sampled across a random set ND domains, with the query example being from one these domains. More formally, each task can be defined as:
(I N..N. Fqy 3g
for N₂ unlabelled support examples vs,. and query example 2g with label yg-
Example We give illustration of task in Figure The chosen example comes from the real-world iWildCam dataset (Beery et al.) 2020), where cameras at different locations are meant to different domains. This challenging, annotation in the latent DA setting, and must learn to estimate relevance. On the other hand, we can see from this example that sometimes images from the same annotated domain (= camera) are not very similar, and conversely images from other domains may be quite similar. It may be possible for model in this setup do better than standard observed-domain methods that assume an external process provides exclusivel, same-domain data for adaptation. Our experiments will confirm that this can sometimes happen.
model
provided with this domain/camera
3.2 OBJECTIVE
Our goal is to train model that can adapt relevant examples from the support set and obtain superior performance on the query examples. We can formalize this using the following objective:
N
(folfonw( vo,k)) []
min E(O, Φb w) = Epul (di) 0,0..
E Eptepsall (a))
(1)
N
4

---

Under review as a conference paper at ICLR 2023
where 3 {0, G. ) are the parameters of the feature extractor, classifier and cross-attention module respectively (described in detail next), 2, are the support examples used for adaptation, while by are the query examples for which we make predictions and come from domain d. The support examples come from a set of domains {d;) with d E {d+)- There are Ng query examples and & represents the generalization error after adapting the model.
3.3 ARCHITECTURE
The key to solving Eq. I is defining an architecture fo that can identify and exploit relevant support instances within x..- Our solution to this relies on cross-attention between query and support images. We first embed the support and query examples using the same feature extractor, after which we pass the embeddings through the gross-attention module. The cross-attention module gives us transformed query examples that are then added to the embeddings of the query examples as a residual connection, after which the classifier makes predictions. Compared to CrossViT (Chen et al. 202T). we do cross-attention between support and query images from different domains, image-to-image rather than patch-to-patch and on extracted features right before the classifier layer.
Cross-attention module Given a set of support examples and query examples Bg, we use the
feature extractor fe to extract features fo (xx))) ). (2r,). Cross-attention module CA (x.); Se (2,,)) parameterized by w then transforms query embeddings (z,)). using support embeddings fo (xa) as keys. The output of the rross-attention module is added to the query example features as residual connection, which is then used by the classifier fo to predict labels of the query examples fo(xx))).
9g- = (f((f())))))))) + (fo fo(x))
The cross attenttoo module itself performs image-to-image cross- coossatteention, rather than patch-to aatch-to-patch to-patch. More specifically, after extracting the features we flatten all spatial dimensions and channels into one vector, which represents the whole image. Image-to-image attention is suitable for domain adaptation than patch-based option because the overall representation should better capture the nature of the domain rather than patch. Another benefit of image-to-image attention is also that it is significantly more efficient we attend to the whole image rather than patches, which makes the overall computations manageable even with more images.
Our cross- attention module is parameterized by a set of learnable projection matrices Wq. Wx. We (all of size CCx(C/R) R)) with additional projection matrix W RC/RR)xC to transform the queried outputs (we refer to all of these parameters collectively as w). The output of the feature extractor fe is flattened into one vector (any spatial information is flattened), giving C' channels, so fo(x,) E R-N,CC fo(ær,) E R.N,xC. We also specify ratio R that allows us to use rectangular projection matrices with fewer parameters, which improves efficiency and also provides regularization.
Formally we express CAw
= fo Wg. f=fo f= W.
= softmax
gkgt//// /VT////h). (fo((e)))) = Av.
Similarly as CiossViT (Chen et al. 2021) and self-attention more broadly, we use multiple heads ht so we refer to it as MCA. We also use layer normalization is the common practice. The output of MCA is added to the query example embeddings as a residual connection:
z fo x.,) MCA (IN(fo(x..) Lx(f)xx))))
which is then passed through the classifier fø to obtain predictions (z). Following CrossVit we do not apply a feed-forward network after cross-attention. We directly add the output via residual connection and pass it to the classifier.
3.4 META-LEARNING
We train the main model (composed of the feature extractor fe and classifier fo and the cross- attention module (parameterized by w) by meta-learning across many tasks. Each task has the structure described earlier in Section 3.1 Meta-learning is computationally efficient in this case because the inner loop does not include back-propagation proppgaaiion based optimization the adaptation to the support examples is done purely feed-forw forward. We provide a summary both how meta- vota-training and inference (meta-testing) is done in Algorithm
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Episodic meta-learning for source-free latent domain adaptation with CXDA
// Meta Meta-training
Require: training steps T. # latent domains in a task No. # support examples N., # query examples Ng-
learning rate
1: Initialize: 0, (0. w
2: for 1,..., do
3: Sample No support domains (D.), from training domains
4: Sample query domain D, from the set of support domains {D, H
5: Sample N, unlabelled support images æ, uniformly from the selected support domains (D.) No
6: Sample N₂ lahelled query images o. from domain 13g
7: Predict query labels D, + ff (a,,) + MCA (IN(fo(x.)) (N(f(
(0, Φb w) & (0, w.) 1f (.....)) Ex (00, ko
end for
// Inference
a new task Require: 0, Φ, w. support æ, and query 2, examples from new domains 10: Da + (x)) + MCA (LN(f (x:)); LN(fo (x,)) w)))
4 EXPERIMENTS
4.1 BENCHMARKS
We evaluate our approach variety of synthetic and real-world benchmarks, namely FEMNIST (Caldas et al. 2018). CIFAR-C (Hendrycks & Dietterich] 2019). TinyImageNet-C (Hendrycks & Dietterich 2019) and iWildCam (Beery et al. (2220). All of these benchmarks have large number of domains, e.g. around 100 for CIFAR-C and TinylmageNet-CC and around 300 for FEMNIST and iWildCam. Using large number of domains for pre-training is reasonable as for many practical problems it is possible to collect such pre-training data. We describe each benchmark next.
FEMNIST dataset includes images of handwritten letters and digits, and is derived from the EMNIST dataset (Cohen et al.] 2017) by treating each writer as a domain. CIFAR-C extends CIFAR-I0 (Krizheskyy 2009) by applying variety of corruptions such different brightness, snow various types of blurring. There are different levels of severity with which the corruptions are applied, giving rise to multiple domains for the different levels. TinyImagel InageeNettCCC is an extension of TinylmageNet analogous to CIFAR-C. iWildCam is a large-scale real-world dataset that includes images of different animal species taken by cameras in different locations. There is lot of variability in the style of images in different cameras, for example different illumination, camera angle or vegetation. The dataset has also substantial class imbalance, so macro Fl score is used for evaluation.
For FEMNIST, CIFAR-C and TinylmageNet-C we follow the splits into meta-training, meta- validation and meta-testing sets as selected (Zhang et al.] (202T). For iWildCam we follow the splits of domains selected in (Koh et al. 2021). Additionally for iWildCam we filter out all domains that have fewer than 40 examples.
4.2 BASELINES
Empirical risk minimization or ERM is baseline that simply trains on all training domains and performs no domain adaptation. It is known to work surprisingly well and is often difficult to beat when properly tuned (Gulrajani & Lopez- 2020). In our case is trained following the episodic pipeline for fair comparison i.c. it is directly trained using the query examples during meta-training.
ERM
A simple and often useful method for source-free domain adaptation is to update the batch normalization statistics using the unlabelled target domain data Schneider et al. (220)). It has achieved strong results in conventional SFDA (Ishii & Sugiyama (021). However, in the latent DA setting is unclear if statisties calculated across support set of varying relevance will be helpful for achieving better performance. During evaluation, the statistics are updated using all support examples, and directly used for the query examples.
BN

---

Under review as a conference paper at ICLR 2023
CML Contextual Meta-Learning is the main instantiation of ARM ZZang et al. 2021 way to extract information from the whole minibatch in test-time adaptation and use it to obtain better performance on test images. We apply the CML on the whole support set with images from different domains and then use it as additional information for making predictions on test images. CML is a feed-forward domain adaptation method, but it has not been designed for the latent domain adaptation problem.
Backprop-based Fine-tuning with standard domain adaptive losses such as entropy minimisation
(randvaleet& & Bengio 2004) (FT-EM) and infomax (Shi & Sha 2012) (FT-IM). Fine-tuning with these objectives is widely used in prior SFDA methods, and these baselines roughly correspond to applying methods such as TENT [Wang et al. (222]) and SHOT (Liang et al.) (2220) to our problem setting respectively. These results are presented for context rather than fair comparison, because they use backprop (which we do not) and they are not designed for latent domain adaptation (unlike us).
4.3 IMPLEMENTATION DETAILS
Our solution - CXDA Our cross-attention module first flattens all spatial information and channels into one vector for each image, so works taggee. In line with existing literature (Vaswani et al. 2017: Chen et al. 2021) we use 8 heads and layer normalization on the flattened features of support and query images. The use of layer normalization means that our approach does not rely on minibatch of query examples i.e. it natively supports streaming mode and does not need mutiple query examples to obtain strong results, unlike existing test-time domain adaptation approaches (Zhang et al. 2021 Wang et al. 2021).
Support images are projected into keys and values, while query images act as queries for cross- attention after transformation by projection matrix. After calculating the attention map and applying it to the values, we multiply the output by a further projection matrix. We use only one cross-attention layer and our projection matrices have rectangular shape of C' C/2 where C' is the dimensionality of the flattened features. No dropout is used.
Data augmentation We use weak data augmentation during meta-training. The exact augmenta- tions are cropping, horizontal flipping, small rotations (up to 30 degrees) and are different from the corruptions tested in some of the benchmarks. These are applied with probability 0.5 independently.
Task sampling Our tasks have 5 support domains, with 20 examples in each, overall 100 support examples. Query examples come from one randomly selected support set domain (out of 5 options) and there are 20 of them. Note that the method fully supports streaming mode, so no statistics are calculated across the batch and it works independently for each. The exact number of tasks for meta-validation and meta-testing is respectively (420, 420) for FEMNIST, (850, 11000) for CIFAR-C, (1700, 11000) for TinylmageNet-C. C, and (745, 2125) for iWildCam.
Training We follow the hyperparameters used by Zhang et al. (202T) for FEMNIST, CIFAR-C and TinylmageNet-C. C, and we also train the cross- tross-attention parameters with the same optimizer. For FEMNIST and CIFAR-C a small CNN model is used, while for TinyImageNet-Ca pre-trained ResNet-50 (He et al. 2015) is fine-tuned. For iWildCam we follow the hyperparameters selected in (Koh et al. 2021), but with images resized to 112 112, training for 50 epochs and with mini-batd size resulting from our task design (100 support and 20 query examples). All our experiments are repeated across three random seeds.
Evaluation metrics We follow Zhang et al. (201)) in reporting average and worst performance While/Zhang
over all testing tasks. While /Zhag et al. 2022) reports the worst single task, modify this metric to report the average performance of the worst decile of tasks. The reason is that for some benchmarks, among all 10,000+ test tasks with varying domain transfer difficulty there easily be at least one task with zero accuracy, making a less meaningful measure.
4.4 RESULTS
We report our results in Table [ for all bencharks: FEMNIST, CIFAR-C, TinylmageNet-C and large- scale real-world iWildCam benchmark. We include both average performance as well as reliability via the worst case performance (Zhang et al. 202T), with our bottom decile modification. From the results we can see our cross-attention approach results in consistent improvements over the strong
7

---

Under review as a conference paper at ICLR 2023
ERM baseline across all benchmarks, well as the CML and BN baselines. The encouraging result on iWildCam highlights our method works well also in practical real-world scenarios.
Overall CML and BN strategies that naively combine information from all support examples have limited success when the support set has both domain relevant and domain irrelevant examples. In fact, CML achieves lower performance than ERM in some of the benchmarks, despite having mechanism for domain adaptation. The results show the need to adaptively select the right examples from the support set when they come from domains of mixed relevance. The results confirm our cross- attention based framework can successfully select useful information from the set of examples with both relevant and irrelevant examples and ultimately achieve superior performance. Backprop-base alternatives usually perform worse, despite being slower, due to lack of support for latent domains.
Table 1: Main benchmark resuls: average and worst-case (worst 10% tasks) test performance, with standard error of the mean across 3 random seeds. Accuracy is reported for all except iWildCam, where F1 score is used (%). Feed-forward approaches the top and backprop ones the bottom.
FEMNIST
CIFAR-C
Tiny ImageNet-C
iWildCam
Approach
W10%
Avg
W10%
Avg
W10%
Avg
W10%
Avg
ERM
52.7 1 1.4
77.2 1.9
44.3 105
68.6 1.3
4.2
24
000000
38.7 1 0.8
CML Zhang et al. 2021
50.4 1.3
7..
448 03
0.5
48
-5
00=00
38.7 +
BN Echil Sugiyami 2021 Zhing
2011
1.22
7.0
45.4
0.4
0.2
203
19
0..
CXDXX
53.3
494
28.6
1.
EM Grandvalet Bengio 2001
51.7 4: 1.4
0.6
449 0.6
09
3.4
=3.
00000
38.6 0.8
FT-I
Sta & Sha 2012 Linng
2020
52.5 1151111 1.2
77.5 0.88
45.6
6.3
4.4
1.0
0.1000
38.7 1.7...0.
4.5 FURTHER ANALYSIS
As part of analysis we study several questions: 1) How does the performance of unsupervised cross- attention compare with a supervised version? 2) How does the inference and training time compare for our cross-attention method and the baselines? 3) What do the attention weights look like and what do they imply? 4) How does the performance vary with variable number domains in the support sets? 5) How does the size of the support set influence the performance?
Dumain-supervised vs domain-unsupervised adaptation Recall that our main CXDA algorithm and experiments above domain unsupervised. This may induce cost due to distraction by domain- irrelevant adaptation data (e.g.. as observed by CML underperforming ERM previously) or a potential benefit due enabling transfer. We therefore compare supervised alternative, with manually defined attention weights based on domain labels. Table 2 shows the results are dataset dependent. The fact that in at least some cases domain- -nsaaerrieseee adaptation outperforms the supervised case shows that the benefit can sometimes outweigh the cost, and that is possible for a suitable model to outperform manual domain annotations.
unsupervised method with domain-
Table 2: Comparison of unsupervised and supervised CXDA on our benchmarks. accuracy for all benchmarks apart from iWildCam where Fl score is reported (%).
Average test
Cross-attention
FEMNIST
CIFAR-C
TinylmageNet-C
iWildCam
Unsupervised
78.3 + 0.0
72.0 + 0.3
28.6 + 0.3
43.5 +: 1.5
Supervised
79.4 0.4
69.8 + 0.4
28.6 + 0.2
52.0 + 1.2
Comparison of inference and training times We show our approach is fast and can do adaptation quickly in Table (left), with inference time very similar feed-forward baselines. Table (right) shows that meta-training is longer for the smaller datasets, but the difference is small for large datasets approaches have significantly larges inference times than the
(left),
and models. Backpropagation-basee feedforward ones, showing the need for specialized feedforward adaptation methods. The total runtime is closer to the feedforward approaches because the training is done without any fine-tuning. All experiments within the same benchmark used the same GPU and number of CPUs.
Analysis of attention weights We have analysed the attention weights to understand the learned behaviour of the cross-attention mechanism. We have selected the large-scale iWildCam benchmark
8

---

Under review as a conference paper at ICLR 2023
Table 3: Comparative computational cost of different adaptation methods for adaptation and training.
Inference/Adaptationn (mshhask)
Train-Adaptution (minutes)
Appronch
PEMNIST
CIFAR-C
TinylmageNet-C
IWildCam
FEMNIST
CIFAR-C
'InyliageNNer-C
(WildCam
ERM
1..
+-.7
52.8 0.2
352.0 2.8
38.3 +
26.6 + 0.2
94,1 0.2
440.2 5.4
CML
1..1
1
0.1
385.2
55.2
35.2
131,8 0.2
461.9
9: 7.2
BN
16.5
1922
141=0.1
145.2
44.5 + 0.3
31.6 0.3
112.3 = 0.3
432.2 0.8
CXDA
1..8
0.2
=
392.5
110.0
12.0
63.2
167.9 1.9
491.6 1.0
FTEM
352.7::422
387.1
840.2
15.4
2064,7 = 238.0
97.5 +
156.6 3.6
483.4
907.1 = 28.0
FTIM
296.9 A 3.1
385.9 1: 10.9
830.7
16.6
1709.8 =: 16.5
78.2 a 0.4
150.2 3.9
470.8 = 7.3
688.0 :: 14.8
and used of the trained cross-attention models. Figure the density histogram of attention weights for same and different domain support examples, relative to the query examples in each task. From the plot we observe that: 1) There is significant amount of weight spent on attending to examples in different domains to the current query. This suggests that the model is exploiting knowledge transfer beyond the boundaries of the standard (camera- wisee) domain annotation in the benchmark, as illustrated in Figure I 2) Nevertheless overall the weight distribution tends to attend somewhat more strongly the in-domain instances than out-of-domain instances. This shows that our cross-attention module has successfully learned how to match query instances with corresponding domain instances in the support set, despite never experiencing domain- sunainnuupervision.
Variable number of domains Appendix sho that the best performance is obtained when there are fewer domains, confirm- ing our intuition. However, CXDA can handle well also cases when there is a large number of domains and consistently outperforms other ap- proaches irrespective of the number of domains.
Tables|[and] 5
domain Different domain
Variable support set size Tables 6 and in Appendix D analyse the impact of variable number of examples in the support set. The results confirm CXDA is scalable and outper- forms other approaches even if the support set size changes more fewer examples.
weight
Figure 2: Density histograms of attention weights for pairs of same and different domain examples in the test tasks of iWildCam benchmark.
4.6 DISCUSSION
Scalability and applicability Our cross- attention approach is fast the scenario when there are moderate number of examples for adaptation (in our experiments we use 100), and not too many inferences to be made per adaptation. In this regime it is much faster than the mainstream line of backpropagation-baet
adaptation solu- tions (Kouw & Loog 2021 Csurka et al. 2022, Liang et al. 2020 Wang et al.) (202I). Whereas, if there are very many inferences to be made then the overhead cost of mack-propagation would eventually be amortized. As for attention more broadly, its computational cost depends on the number of query and support examples, and the approach would be expensive there were many thousands or millions of examples for both. For a very large support set, one could simply take random or semi-ran random subset of images for adaptation. However, the most effective use case one where the adaptation set is smaller and/or changes rapidly compared to the frequency of inference.
CONCLUSION
We have introduced new highly practical setting where we adapt model using examples that come from a mixture of domains and are without domain or class labels. To answer this new highly challenging adaptation problem, we have developed novel solution based cross-attention that is able to automatically select relevant examples and use them for adaptation on the fly.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We include the code for our approach as part of the supplementary material. We also include details for our experiments within the main text the appendix.
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your MAML. In ICLR, 2019.
Sara Beery, Elijah Cole, and Arvi Gjoka, The iWildCam 2020 competition dataset. In arXiv, 2020.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H. Brendan
McMahan, Virginia Smith, and Ameet Talwalkar. LEAF: A benchmark for federated settings. In Workshop Federated Learning for Data Privacy and Confidentiality, 2018.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. CrossViT: Cross-attention multi-scale vision
transformer for image classification. In ICCV, 2021.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: an extension of MNIST to handwritten letters. In arXiv, 2017.
Gabriela Csurka, Timothy M Hospedales, Mathieu Salzmann, and Tatiana Tommasi. Visual domain adaptation in the deep learning cra. Synthesis Lectures on Computer Vision, 11(1):1-190, 2022.
Lucas Deecke, Timothy Hospedales, and Hakan Bilen. Visual representation learning over latent domains. In ICIRR, 2022.
Carl Doersch, Ankush Gupta, and Andrew Zisserman. CrossTransformers: spatially-aware few-shot transfer. In NeurIPS, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: transformers for image recognition scale. In ICLR, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NIPS, 2004.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In arXiv, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2015.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
Judy Hoffman, Trevor Darrell, and Kate Saenko. Continuous manifold based adaptation for evolving visual domains. In CVPR, 2014.
Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP: 1-1, 2021.
Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In NewrlPS, 2019.
Masato Ishii and Masashi Sugiyama. Source-free free domain adaptation via distributional alignment by matching batch normalization statisties. In arXiv, 2021.
10

---

Under review as a conference paper at ICLR 2023
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, lan Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, An- shul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In ICMI., 2021.
Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. In NeurIPS, 2021.
Wouter M. Kouw and Marco Loog. A review of domain adaptation without target labels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(3):766-785, 2021. doi: 10.1109/ TPAML.201..245594.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy Hospedales. Episodic training for domain generalization. In ICCV, 2019.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: unsupervised domain adaptation without source data. In CVPR, 2020.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD; learning to learn quickly for few-shot learning. In arXiv, 2017.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Inferring latent domains for unsupervised deep domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(2):485 498, 2021.
Xingchao Peng, Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with disentangled representations. In ICML, 2019.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 13:11539- 2020.
Yuan Shi and Fei Sha. information-thhoretiial learning of discriminative clusters for unsupervised domain adaptation. In ICML, 2012.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NIPS, 2017.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: relation network for few-shot learning. In CVPR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell, Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. CVPR, 2022.
Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Exploiting the intrinsic neighborhood structure for source-free domain adaptation. In NeurlPS, 2021.
Hin-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. CYPR, 2020.
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn.
Adaptive risk minimization: learning adapt domain shift. In NeurlPS, 2021.
11

---

Under review as a conference paper at ICLR 2023
A
QUALITATIVE ANALYSIS
Location #208
Query image
Largest attention weights
Location #301
Smallest attention weights
W: 0.009
W: 0.007
W: 0.006
W: 0.001
W: 0.001
W: 0.000
W: 0.000
W: 0.000
W: 0.000
W: 0.000
Location #288
W: 0.257
W: 0.126
W: O. 0.117
W: 0.099
W: 0.058
W: 0.007
W: 0.005
W: 0.002
W: 0.001
W: 0.000
Location #187
W: 0.001
W: 0.000
0.000
W: 0.000
W: 0.000
0.000
W: 0.000
W: 0.000
0.000
W: 0.000
Location #125
W: 0.077
0.022
0.009
W: 0.002
W: 0.002
0.000
W: 0,00000
W: 0,000
0.000
W: 0.000
Location #120
Figure 3: Analysis of attention weights for an example task in iWildCam, with a query image coming from location (camera trap) #288. We show the five support examples in each domain that have the largest and smallest attention weights. Similar images from the same location (#288) are given the largest weights, but also relevant images from other locations (e.g. #125) given larger weights. The examples with the smallest attention weights visually do not seem relevant.
B FORMAL DEFINITION OF LATENT DOMAINS
We provide both high-level and more detailed formal definition of how we use latent domains.
High-level explanation of latent domains: p(D) distribution over domains, and Pa y) is the distribution over data in domain d. I is a set of domain indexes drawn from p(D), D is dataset drawn from several domains within p(D) and the dataset does not contain the domain IDs corresponding to each instance (z, v). Equation 2 samples the domains to be used in a given dataset. Eq. samples the dataset from those domains, but doesn't 't record the domains.
(d p(D)
(2)
D = (=((x,9) (2.4) ~ pa(a, v), 1]
(3)
More detailed explanation latent domains assumption in each step of our algorithm: Eq.
says the adaptation (support) and test (query) set from disjoint domains to the pre-train set. Eq. says the query/test domain should observed in the support set, but the support set can contain irrelevant domains the query/test set.
12

---

Under review as a conference paper at ICLR 2023
paretrain
{d: d ~ p())
(4)
Doretrain {(x,)) (2.8) ~ pa(x, y), de Juretrasn]
(5)
Ideploy {d: (..:ddp(D,,..)) p
(6)
Drupport {(r,v): (z.y) ~ pa(x, v), d E pdeploy]
(7)
Iquery
C Ideploy
(8)
Diuery {(x, y) (z.0) ~ pa(a, u), de Jowery)
(9)
(10)
Note that d doesn't have to be either discrete scalar in our framework, unlike most others.
C ADDITIONAL DETAILS
We have followed the experimental choices from Zhang et al. 2021) for FEMNIST, CIFAR-10- JingeeegeNet-CC and Koh al. (202I) for iWildCam, unless we specify otherwise. This includes the splits of data into training, validation and test sets as well as the splits of domains into meta-training, meta-validation and meta-test sets of domains. For iWildCam we use the OOD splits from Koh et al. 1202T).
C.I MODELS
Feature extractor and classifier:
FEMNIST: CNN with three convolutional layers, hidden dimension of 128, batch normaliza- tion, ReLU activation, kernel of 5, padding of 2. Classifier consists of two fully-connected layers with 200 hidden units and ReLU activation in between. The input shape of images is 28 x 28.
CIFAR-C:
Same architecture as for FEMNIST, but with three input channels instead of 1 (colour images). The input shape of images is 32 x 32.
TinylmageNer-C: ImageNet pre-trained ResNet50 (He et al. 2015). of one fully connected layer. The input shape of images 64
The classifier consists
64.
iWildCam:
ImageNet pre-trained ResNet50 (He al., 2015). The classifier consists of one fully connected layer. The input shape of images is 112 x 112.
-daptation-specific components:
CML: Context network is used to transform the support examples three convolutional layers, 64 hidden units, kernel size of 5, padding of 2, with batch normalization and ReLU activation. The output of the network has the same shape as input. To create the context we average the context network outputs across the support examples and use the same context for all query examples in the task.
CXDA: All key details explained in the main text, and provide explanations here. The size of the fully-connected layers depends on the flattened shape of the features - only one cross-attention layer is used. After multiplying attention weights with projected values (Av), we transform the output further using projection matrix W, similarly as (Chen et 202T. However, do not use further MLP model that would contain multiple layers and non- non-linearity, so we also follow (Chen et al. 2021) in this aspect. The output of the cross cross-attention module has the same shape as input. As part of CXDA. batch normalization statistics of the feature extractor are updated too using the support set.
detailed
C.2 TRAINING
Dataset-specific training details:
13

---

Under review as a conference paper at ICLR 2023
FEMNIST: SGD with learning rate of 10"4. momentum of 0.9 and weight decay of 10 10- trained for 200 epochs, with validation set evaluated every 10 epochs, and early stopping based on accuracy.
CIFAR-C: SGD with learning rate of 10 momentum of 0.9 and weight decay of 10 10- trained for 100 epochs, with validation set evaluated every 10 epochs, and early stopping based accuracy.
TinylmageNett-C: SGD with learning rate of 10 momentum of 0.9 and weight decay of 10-4, trained for 50 epochs, with validation set evaluated every 5 epochs, and early stopping based on accuracy.
iWildCam: Adam with learning rate of 3 x 10-5, no weight decay, trained for 50 epochs, with validation set evaluated every 5 epochs, and early stopping based on macro F1 score.
In cases we use cross-entropy loss, and the cross-attention parameters are optimized in the same way as the main model. In each iteration we use task that has 5 domains with 20 support examples for each sampled domain, and there are 20 query examples from one selected domain from the set of current domains.
Details about fine-tuning (FT) of the pre-trained ERM model: we perform fine-tuning by taking 10x smaller learning rate compared to the ones used during training and then performing 10 steps the task's 's support data. We consider two losses: 1) entropy minimization (EM) and 2) information maximization (IM) of the support set example predictions during the fine-tuning, We reset the model to the pre-trained state for each test task.
D ADDITIONAL ANALYSES
For the additional analyses we use pre-trained models that were trained with tasks that have 100 support examples coming from 5 domains. We then deploy them to tasks that have 1) variable number of support domains 2) variable support sizes.
Table 4: Analysis of the impact of variable number of domains in the support set of examples worst 10% tasks test accuracy on CIFAR-C (%).
Approach
domain
2 domains
5 domains
10 domains
20 domains
ERM
58.3 + 0.4
54.2 + 0.3
44.3 + 0.5
37.9 + 0.5
29.4 + 0.4
CML
57.7 0.7
54.3 0.6
44.8 0.5
39.2 + 0.4
30.6 + 0.4
BN
55.4 0.5
45.4
0.7
30.2 + 0.6
CXDA
62.7
58.7
49.4
43.0
33.2 + 0.3
FT-EM
4.6
4.5
4.6
39.1 0.5
30.5 + 0.4
FT-IM
53.1 0.5
4..75.0.5
1.56
0.5
30.7 # 0.4
Table 5: Analysis of the impact of variable number of domains in the support set of examples average test task accuracy on CIFAR-C (%).
Approach
domain
2 domains
domains
10 domains
20 domains
ERM
68.7 + 0.3
68.7 + 0.3
68.6 + 0.3
68.5 + 0.2
68.6 + 0.2
CML
68.8 0.6
69.2 + 0.5
69,5
0.5
69.4 + 0.4
69.5 + 0.4
BN
69.7
69.4 0.4
69.3 : 0.4
69.1
0.4
69.2 + 0.4
CXDA
72.2 0.2
72.1 0.2
72.0 0.3
71.9 + 0.3
71.9 + 0.3
FT-EM
69.0 + 0.4
69.1 + 0.3
69,2 0.4
69.2 + 0.3
69.3 + 0.3
FT-IM
69.8 + 0.3
69.6 + 0.3
69.5 + 0.3
69.4 + 0.3
69.5 0.3
Finally, we further analyse the adaptation cost of CXDA vs fine-tuning based competitors in terms of wall-clock time. In Figure we compare our CXDA which uses iterative computation during
14

---

Under review as a conference paper at ICLR 2023
Table 6: Analysis of the impact of variable number of examples in the support worst 10% tasks test accuracy on CIFAR-C (%)
Approach
10 examples
20 examples
50 examples
100 examples
200 examples
500 examples
ERM
0.0
21.7 0.1
38.2 0.5
44.3 0.5
48.2
50.2 0.7
CML
0.0 0.0
21.7 0.2
38,9 + 0.5
44.8 0.5
48.6 + 0.5
50.6 0.4
BN
1,2 1.0
22.8 + 0.9
39.5 0.7
45.4
49.4 + 0.7
51.3 + 0.7
CXDA
1.8 1 0.9
26.4 1.4+0.7 0.7
43.0 1..0+00 0.5
49.4 0.6
53.8 + 0.6
56.2 + 0.8
FT-EM
0.8 - 0.7
22.4 1 0.6
0.6
39.0 1 0.5
44.9 0.6
48.9 1 0.5
50.8 0.5
FT-IM
1.3 1 0.9
22.9 1 0.8
39.7 1 0.5
45.6 1.5
49.6 1 0.5
51.7 1: 0.4
Table Analysis of the impact of variable number of examples in the support set average test task accuracy CIFAR-C (%).
Approach
10 examples
20 examples
50 examples
100 examples
200 examples
500 examples
ERM
68.7 + 0.3
68.7 + 0.3
68.6 0.3
68.6 + 0.3
68.6 0.3
68.6 0.3
CML
67.7 + 0.3
68.7 + 0.4
69.3 0.4
69.5 : 0.5
69.5 : 0.5
69,6 + 0.5
BN
69.3 + 0.4
69.3 + 0.4
69.2 + 0.4
69.3
69.2 + 0.4
69.3 + 0.4
CXDA
69.6 + 0.3
70.9 + 0.3
71.7 + 0.3
7
72.1 + 0.2
72.2 + 0.3
FT-EM
69.1 + 0.3
69.2 = 0.3
69.1 + 0.3
69.2 0.4
69.2 : 0.3
69.2 + 0.3
FT-IM
69.4 + 0.3
69.5 + 0.3
69.4 + 0.3
69.5 0.3
69.4 1 0.3
69.5 1 0.3
adaptation, with the alternatives using and 10 iterations of gradient-descent for adaptation. We can see that CXDA is faster than both versions of fine-tuning even if they use the minimum number of steps available. CXDA also significantly outperforms them in terms of the test score.
CIFAR-C
TinylmageNet-CC
X
Tost accuracy 1%)
Test (%)
78.2
43
CXDA
78.0
200 accuracy
71
27
42
XX FT-EM
70
77.4
69
25
time (ms)
Adaptation
(ms)
Adaptation time
Adaptation (ms)
Figure Analysis of test accuracy (%) vs adaptation time (ms) for CXDA and fine-tuning approaches that correspond to TENT (FT-EM) and SHOT (FT-IM) when the baselines use either 1 and 10 steps for fine-tuning.
15


Reviews:
[{"review_id": "xS53pRqTvZ", "reviewer": "Reviewer_tuYo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to meta-learn a network capable of embedding the mixed-relevance target dataset and dynamically adapting inference for target examples using cross-attention. It further analysis the method performance in the domain-supervised adaption. ", "summary_of_the_review": "----------------------------Updates---------------------------------\nAfter reading the response and other reviewers' comments, I would like to keep my score.\n", "strength_and_weaknesses": "Strength: \nthe writing of this papers seems easy to follow and it demonstrates its method from several perspectives. \n\nWeakness:\n1.\tThere are several implementation details associated with the proposed method such as the augmentation, cross attention, meta learning. It would be better to have  the ablation studies on decomposing the each components. \n2.\tI am concerned the performance boost with the relatively large computational time. Although in table 1, the authors shows the improvements but coming with larger computations. \n3.\tThe author claims privacy in the paper. However, there are not many privacy baselines are used as the baselines. I would recommend the authors to compare with some privacy related domain adaption such as black box related domain adaption. \n4.\tI also have the questions regarding the scalability of the proposed method. With the meta-training, how does the proposed method formulate on the larger dataset or settings? And what are there performance? It would be better to extend the proposed method on the larger dataset and with the detailed running time. I suspect that the running time will be much larger compared to the baselines. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QoCDGFZjlYu", "reviewer": "Reviewer_wyNm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors focus on source free latent domain adaptation. Here the content or label of  target data overlaps with the one of the source data. And authors doest not get access to the source data when using target data. Furthermore, authors also strict setting where the network is not updated which is backward-free. To achieve this goal, authors explore the probability of using meta-learning, and propose to use cross-attention to select relevant examples for adaptation. The experiment results demonstrate the effectiveness of the proposed method.   \n\n\n\n", "summary_of_the_review": "This paper use the meta-learning method to explore source-free latent domain adaptation, which sounds interesting. However, for forward-pass, it is less convincing. ", "strength_and_weaknesses": "1) This paper explore new setting about latent domain adaptation: source-free and forward-pass, which is a new highly practical setting.\n\n2) Authors try to give clear explanation about this setting, and show why it is interesting in practical application. \n\n3) The paper is well-written and easy to follow.\n\n4) The experiment is sufficient to support the proposed method. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K8o81v_tZS", "reviewer": "Reviewer_tHMU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new source-free domain adaptation method that can work without back-propagation and is capable of handling a target dataset containing both domain-relevant and domain-irrelevant samples (called LDA setting). The proposed method firstly embeds the target dataset into features and conduct adaptive inference for target samples using cross-attention mechanism with the embedded features. The modules for embedding, cross-attention, and classification are jointly trained via a scheme of meta-learning. Experimental results with several popular datasets show that the proposed method works well especially in the LDA setting.", "summary_of_the_review": "The proposed source-free domain adaptation is sufficiently novel and empirically works well. The manuscript is well-written. I vote for \"accept.\"\n", "strength_and_weaknesses": "<Strength>\n\n- Since the proposed method does not require any training process with back-propagation operations, it should be easy to use and can be applied to a wide range of applications. \n- The proposed method is also useful in terms of robustness against domain-irrelevant samples in the target dataset.\n- The proposed method consistently achieves good performance in several benchmark datasets.\n- This paper is well-written and easy to follow. I enjoyed reading it.\n\n<Weakness>\n\n- I do not have any major concern.\n- Shown below are minor things (almost just out of curiosity).\n\t- Do we need to fix the size of the support set? As a mechanism, the cross-attention module can process a variable size of the support set. Allowing this might be useful, because it makes it possible to handle a various size of a target dataset as is.\n\t- Can we use the proposed method in an online manner like test-time adaptation? Since LN(f(x_s)) can be stored for the subsequent inference process once we compute it, we can conduct online adaptation by using the proposed method, if we have a strategy to properly update the stored feature with LN(f(x_q)). It would be great if we can use the proposed method as an unified method to solve both source-free domain adaptation and test-time adaptation. (I understand that the above makes the proposed method much similar to ARM, though)\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FDkpJxgpHT", "reviewer": "Reviewer_zikv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed the Latent Domain Adaptation setting where the a source model should be adapted to a target dataset that contains a mixture of unlabelled domain-relevant and domain-irrelevant examples. The “latent'' means that we are not given the domain label in the training phase. To handle this problem, the authors proposed a feed-forward method which incorporates the cross-attention module with an episodic meta-learning framework. The authors have carried out extensive experiments to verify the performance of the proposed method. Furthermore, the authors analysed the training and inference time to show the strength of the proposed feed-forward method.", "summary_of_the_review": "Considering the limited technical novelty and insufficient comparison, e.g., with TENT and SHOT, the reviewer would not be able to recommend an acceptance.", "strength_and_weaknesses": "Strength: \n1. The authors proposed a new setting for source-free domain adaptation. This setting seems more practical in real-world scenarios compared with the conventional SFDA.\n\n2. The writing of this manuscript is clear and easy to follow. The overall presentation is nice, including smooth sentences, nice figures, and organised paragraphs.\n\n3. The pipeline of the method is technically sound and provides detailed implementation of the proposed method.\n\nWeakness:\n\n1. In Section 3.1, the authors mentioned that “While the conventional domain adaptation setting assumes that xq and xs are all drawn from a common distribution…”. In the conventional domain adaptation, however, they usually belong to different domains.\n\n2. Compared with ARM, the proposed method seems a bit incremental, it is just the simple combination of ARM and cross-attention module. The authors didn’t provide strong motivation for combining these techniques. In addition, neither ARM nor cross-attention module was proposed by the authors. \n\n3. Typo in Section 3.1: “To train a model than can be used as described above”. “than” should be “that”.\n\n4. In Sect. 3.3, the author mentioned that “image-to-image attention is more suitable for domain adaptation than patch-based option because the overall representation should better capture the nature of the domain rather than a patch.” Is there any theory that supports this conclusion?\n\n5. In my opinion, it is necessary to compare the proposed method with TENT and SHOT even though they are based on back-propagation. Although the proposed method does not require back-prop, this is a detail of the specific implementation and is not relevant to the task.\n\n6. Even with the feed-forward, the proposed method still requires several iterations for training. So, what are the advantages of using feed-forward?\n\n7. As the training time is heavily affected by the number of iterations of the method, the numbers in Table 3 are not convincing enough. Which method performs best with the same training time? Which method requires less training time to achieve the same performance? Furthermore, comparisons with TENT and SHOT are still missing from Table 3.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "EGNedLL9lrG", "reviewer": "Reviewer_J7SR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new problem setting: They assume a model is pretrained on source data, then the model is used to predict data in a target domain. They also assume that the resources are low and the method cannot update the parameters—i.e., cannot use backprop.\n\nThe proposed model is based on meta-learning. It is pretrained by taking batches as tasks and predicting target labels. The core computation part is to find the target image, in the given target set, that is most similar to the test image using a mechanism similar to self-attention in transformers. Then summing up the embedded vector of this image by the embedded vector of the test image to be used for prediction.\n\nThe model is evaluated in four datasets and shows some improvements.", "summary_of_the_review": "The paper proposes a new problem setting. But the presentation is horrible, and the problem setting needs further experiments to be justified.", "strength_and_weaknesses": "Reasons to accept:\n\nThe paper proposes a new research subject, and the results are good.\n\nReasons to reject:\n\nThe paper is extremely verbose, it is full of unnecessary information, it has some missing would-be useful information, and it lacks some visualizations to facilitate the convey of information. All together these have made the article extremely boring to read. As a person who has read many domain adaptation papers, in my opinion this paper is not worth reading, given the amount of information that it offers. Just a few examples (not a full list and not ordered): \n- You use too many mini-passages separated by a bold-faced phrase. These confuse the reader, particularly when each one focuses on a specific subject, and occasionally drift from the subject of the section.\n- You used these mini-passages in the incorrect sections. For example I don’t expect to see mini-passages called “Latent domain adaptation” or “Feed-forward domain adaptation” in the introduction section, when you already have a full section called “BACKGROUND AND RELATED WORK”. This continues all over your paper. For example, you have a mini-passage called “Preliminaries” in the section called methods, but weren’t you supposed to cover these in the section called background? Even worse, when you read this mini-passage, you would find that it is not actually preliminaries, it is a mix of background and notations.\n- You have a mini-passage called “Deployment phase”, aren’t you supposed to cover the model itself, then explain its deployment?\n- You have a section called “OBJECTIVE”, weren’t you supposed to explain the model first, and then explain its objective?\n\n   You see, this list goes on and on.\n\nI am not sure about the applicability of the proposed problem setting. This is important because failing to fully justify and evaluate this part, makes me think that the authors have just tried to come up with an unrealistic problem setting to publish a paper for. The authors assume the model cannot use backprop, but cannot we replace the encoder with a smaller one to use a smaller compute? Also based on the examples that you are using in you paper, I am not convinced that why we cannot use source data. You are already using target data, so memory usage shouldn’t be an issue.\n\nIn the mini-passage called “Training phase”, you have assumed that the domain of the query image appears in the target set. Why?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject