AnyDA: Anytime Domain Adaptation
Authors: Omprakash Chakraborty
Aadarsh Sahoo
Rameswar Panda
Abir Das
Venue: ICLR 2023 poster (2023)
Keywords: Efficient Domain Adaptation
Anytime Prediction
Knowledge Distillation
Resource-constrained Learning

Abstract: Unsupervised domain adaptation is an open and challenging problem in computer vision. While existing research shows encouraging results in addressing cross-domain distribution shift on common benchmarks, they are often constrained to testing under a specific target setting, limiting their impact for many real-world applications. In this paper, we introduce a simple yet effective framework for anytime domain adaptation that is executable with dynamic resource constraints to achieve accuracy-efficiency trade-offs under domain-shifts. We achieve this by training a single shared network using both labeled source and unlabeled data, with switchable depth, width and input resolutions on the fly to enable testing under a wide range of computation budgets. Starting with a teacher network trained from a label-rich source domain, we utilize bootstrapped recursive knowledge distillation as a nexus between source and target domains to jointly train the student network with switchable subnetworks. Experiments on multiple datasets well demonstrate the superiority of our approach over state-of-the-art methods.

Published as a conference paper at ICLR 2023
ANYDA: ANYTIME DOMAIN ADAPTATION
Omprakash Charkraborty Aadarsh Sahoo 'IT Kharagpur, 'MIT-IBM Watson AI Lab
Rameswar Panda Abir Das!
{opckgpê, abirêcse. abirrcs, Jatee..iittgpp jitkgp..d. ac. in, (aadarsh, rpanda] f.bm.ccmm
ABSTRACT
Unsupervised domain adaptation is an open and challenging problem in computer vision. While existing research shows encouraging results in addressing cross- domain distribution shift on common benchmarks, they are often constrained to testing under a specific target setting, limiting their impact for many real-world applications. In this paper, we introduce a simple yet effective framework for anytime domain adaptation that is executable with dynamic resource constraints to achieve aecuracy-efficiency trade-offs under domain-shifte shifts. We achieve this by training a single shared network using both labeled source and unlabeled data, with switchable depth, width and input resolutions on the fly to enable testing under wide range of computation budgets. Starting with teacher network trained from label-rich source domain, we utilize bootstrapped recursive knowl- edge distillation as a nexus between source and target domains to jointly train the student network with switchable subnetworks. Experiments multiple datasets well demonstrate the superiority of our approach over state-of-the-ar methods.
INTRODUCTION
Unsupervised Domain Adaptation (UDA) which aims to adapt models trained on a labeled source domain to unlabeled target domain has attracted intense attention in recent years. However, recent successful UDA approaches (Carlucci et al. 2019; Ganin et al. 22116] Li et al., [2000a] Prabhu et al. 2021 Sun al. 2019 Tan et 20200 zeng al. 2015 2017 often rely on complicated network architectures and are limited to testing under a specific target setting, which may not be particularly suitable for applications across a wide range of platforms that present different resource constraints (see Figure [p). While adapting the trained model independently for all testing scenarios in the target domain with drastically different resource requirements looks like a possible option at the first glance, it is not efficient and economical, because of tume-conumiigg training and benchmarking for each of these adaptation settings. Preferably, we want to able to adjust the model, without the need of re-training or re-adaptation in the target domain, to run in high accuracy mode when resources are sufficient and switch to low accuracy mode when resources are limited.
Motivated by this, in this paper, we investigate the problem of anytime domain adaptation where we have labeled training data from a source domain but no labeled data in the target domain and in addition testing at a resource setting with wide range of variation (e.g., see Figure Specifically, we aim to train a single network using both labeled source and unlabeled target data that can directly run at arbitrary resource budget while being invariant to distribution shifts across both domains. This is an extremely relevant problem to address as will provide a distinct opportunity for a more practical and efficient domain adaptation favor different scenarios with different resource budgets.
Recently, anytime prediction (Cai et al. 2019 Huang et al., 2018; Jie et al., 2019) that train a network carry out inference under varying budget constraints have witnessed great success in many vision tasks. However, all these methods assume that the models are trained and tested using data coming from some fixed distribution and lead to substantially poor generalization when the two data distributions are different. The twin goals of aligning two domains and operating at different constrained computation budgets bring in additional challenges for anytime domain adaptation.
To this end, we propose a simple yet effective method for anytime domain adaptation, called AnyDA, by considering domain alignment in addition to varying both network (width and depth) and input
Project page: https::///// //ei:////vrrr www.wri...g....o...oi.o..o.b http://cooooooo//ny......

---

Published as a conference paper at ICLR 2023
Training
Testing
Training
Anytime Testing
B
B
B
b2
B
b1
b3
Source
Targer
Target
Source (Labelled)
Target
Target
(Ladwled)
(Unlabeled)
(Unisbelled)
(a) Unsupervised Domain Adaptation
(b) Anytime Domain Adaptation
Figure I: Instead of conventional adaptation under specific computation budget, anytime domain adaptation focuses training model using both labeled source and unlabeled larget data that can directly run arbitrary resource budget the target domain while being invariant to distribution shifts across both domains.
(resolution) scales to enable testing under wide range of computation budgets. Such variation over width, depth and resolution enables tighter as well as finer coupling of efficiency-computatio trade-off than prior works that only focus one two out of the three dimensions (Li et al. 202 Yang et al., 2020; Yu et al., 2019b) for in-domain data. In particular, we adopt a switchable network where individual subnetworks executable at variable computation budget share parameters with the full-budget network, known as the supernet. However, inability to leverage on the higher capacity of complex networks may. in effect, cause such a model to severely underperform, leading sub- optimal performance across different resource budgets. To alleviate this, we propose to distill (Ba) & Caruana 2014, Hinton al., 2015) richer alignment information from higher capacity models to networks with limited computation. In particular, our proposed AnyDA, adopts two switchable networks teacher and student, that interact and learn from each other. The student subnetworks are trained recursively to fit the output logits of an ensemble of larger subnetworks of the teacher. Such recursive distillation within single network with only adaptive width shown improve generality and reduce performance gaps between high and low capacity networks (Li et al. 2021a).
Starting with the labeled source data, we build the teacher from past iterations of the student net- work as exponential moving average (ema) of the student. The bootstrapped teacher provides the targets to train the student for an enhanced representation. Once the target data is available, the bootstrapped recursive distillation not only brings the target features close to the source but also transfers the learned knowledge to a smaller network for efficient inference. Morcover, we harness the categorical information by leveraging self-supervision through pseudo-label loss the student supernet to ensure a discriminative latent space for the unlabelled target images. Interestingly, with- using any component for explicit domain alignment (e.g. a domain discriminator), we show that our approach trades the performance gracefully with decreasing budgets of the subnetworks. Our extensive experiments on benchmark datasets show very minimal drop in performance across wide range of computation budget (e.g., maximum drop of only 1.1% is observed when the range of computation budget gets x small during testing the Office-31 dataset (Saenko al. 2010).
Our work forges a connection between two literatures that have evolved mostly independently: any- time prediction and domain adaptation. This connection allows us leverage the progress made in unsupervised representation learning address the very practical problem of anytime domain adaptation. To summarize, our key contributions include:
We introduce novel approach for anytime domain adaptation, that is executable with dynamic resource constraints to achieve accuracy- accuracy-efficiency trade-offs under domain-shifte We achieve this by training two networks teacher and student with switchable depth, width and input resolutions to enable testing under a wide range of computation budgets.
We propose a bootstrapped recursive distillation approach to train the student subnets with the knowledge from the teacher network that not only brings the target features close to the source but also transfers the learned knowledge to a smaller network for efficient inference.
We perform extensive experiments on 4 benchmark datasets and demonstrate that AnyDA achieves superior performance over the oftt--of-theart methods, more significantly at lower computation budgets. We also include comprehensive ablation studies to depict the importance of each module of our proposed framework.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORKS
Efficient Domain Adaptation. UDA has been dominated by methods minimizing some measure
of domain discrepancy (Shen et al.
2018 Sun & Saenko 20166 Tzeng et al. 2013) or maximiz-
ing domain confusion (Ganin et al. 2016 Long et al. 2018 Pei et al. 2018 Tzeng al.. 2017)
to generate Comminninvvaiian features. Leveraging image translation (Hoffman et al. 2018 Murez
al.
(2018) or style transfer (Dundar al.
2020 Zhang et al.) 2018 is also another popular trend.
Sepe-supe approaches like solving pretext tasks (Carlucci et al. 2019|Mec et al.
2020: Sahoo
al.,
2023 Sun et al. 2019) and contrastive learning (Li et al. [0.00a] Prabhu et al.
2021 Sahoo
et al.
2021 Tan et al. 2020) have also recently enjoyed huge success in aligning domains. In spite
of the growing development of traditional UDA (Kouw & Loog 2019 Zhao et al. 2020), the chal-
lenging problem of efficient domain adaptation remains largely underexplored. Authors in (Jiang
ct al., 2020 Li al., (2021b) have proposed a multi-scale early-exit architecture (Huang et al., 2018) with DANN (Ganin et al. 2016) as the domain adaptation method. We, on the other hand, employ recursive knowledge distillation from an ensemble of teacher subnetworks that produces consistency supervision for the student subnetworks. The proposed approach eliminates the need for resource hungry spatial augmentation and further reduces the training burden by eliminating the domain dis- criminator which is unwieldy to train. The contemporary work SlimDA (Meng al. 2022), attaches a weight-sharing slimmable network to a domain symmetric adaptation network (SymNet) (Zhang al. 2019b). The stochastic distillation uses an empirical confidence score for the subnets requiring the bi-classifier of SymNet to be replaced by tri-classifier. Ours, on the other hand, is model ag- nostic showing strong transferability across backbones. In addition, unlike early-exit architectures with varying depth only or SlimDA with varying width only, we train network with switchable depth, width and input resolution. To the best of our knowledge, this work is the first attempt to address efficient domain adaptation under anytime prediction framework through switchable depth, width and input resolution with domain specific batch normalization.
Anytime Neural Networks. Anytime neural networks (Hu et al. 2019 Jie et al. 2019 Larsson jet al.
2017) are becoming increasingly attractive due to its computational efficiency. While MS-
DNet (Huang et al.] 2018) makes early exits to meet varying resource demands, MutualNet (Yang ett al. 2020) trains a single network to achieve accuracy-efficiency tradeoffs at runtime. Skipping unimportant channels (Chen et al., 2019) and layers (Wang al., 2018 Wu et al.; [018), dynamic
routing of multiple inference paths (Liu & Deng. 2018: Li et al. 2020b McGill & Perona, (017)
and input dependent adaptation (Chen et al. 2020 are investigated for efficient inference on many
applications. Slimmable networks ( Yu et al. 2019b) and it's variants (Yu & Huang] 2019 Li et al.)
2021a) train a model to support different width multipliers. Adjusting width, depth and kernel sizes simultaneously to achieve better accuracy- -effriiccfffc trade-off is also proposed in (Cai et al.) 2019 Han et al. 2020). Typically, anytime prediction works under the assumption of no distribution shift between train and test data. While our approach is inspired by these, in this paper, we propose framework that provides budget adaptive anytime predictions under change of domains.
Distilling knowledge by mimicking output (Hinton et al., 2015 Buciluã ct al. 2006) and intermediate features (Romero et al. 2014) from an ensemble of teachers or self- distilling (Furlanello et al., 2018 Ji et al.] [02 Lee al. 2020: Zhang al., 2019a) knowledge from previous iterations have obtained compact models without sacrificing the performance. Recent works have explored sele-super learning with knowledge distillation (Caron et al., 2021 Fang ee al. 2021 Tian et al. 2020 Xu et al. 2020) enabling model compression and performance gains without explicit supervision. However, these works rely on teacher trained on data from the same domain while our teacher is bootstrapped from the student that is trained with both source and target data stabilizing the feature alignment process between the two domains.
Knowledge Distillation.
METHODOLOGY
Anytime domain adaptation aims to achieve better generalization and low inference latency across different platforms in unlabeled target domain by transferring knowledge from a labelled source domain. Formally, we have a set of N, labelled source images D. ((x), } N. and a set of N₂ unlabelled target images D, = {af 1 N. 1= with a common label space L. The source and target samples have different data distributions. We also have set of computation budgets B (bi, by ..., by)) ) for inference with by < by when i < j. Our goal is to learn a single domain adaptive model executable wide range of computation budget on the target domain with minimal performance drop.
3

---

Published as a conference paper at ICLR 2023
g
Stea
münnet
subnefs
supernet
Teacher
Source Data Flow
Target (Unlabeled)
Target Data Flow
Stop Gradient
average prediction
ema
distillation
distillation
Source (Labeled)
Betu
Student
cross-entropy loss
stu
pseudo-label loss
Target (Unlabeled)
Figure 2: Illustration of our approach. We use teacher-student framework consisting of 2 networks. The student network g takes both source and target images as input, while the teacher network G takes only target images. Both networks are configured into n subnets with increasing budgets. We recursively distill knowledge from the teacher network to train the student subnets for anytime domain adaptation and bootstrap on previous representations with update for the teacher network. During inference, only student network used by selecting the appropriate subnet adhering to the budget constraints. See Section B for more details.
3.1 PRELIMINARIES
Budget Configurations. To make a single network executable at different budgets, we consider degrees of freedom both at the level of input (resolution) and network (width, depth), denoted by r, w. and d respectively.
takes positive integral values, while and d are numbers in the interval (0, 1]. Let R = {ri, fri,72, "....., .... T....a. W {wi, U2, .... Wh and D {dg d2 ......dde de} denote the sets of possible values of r, w, and b. Following (Yang al. 2020 Yu et al. 2019b), given a configuration tuple <Ta, Wy. d₂ > E R x W D we obtain subnetwork (or subnel) by executing only the first wy x 100% filters and dg x 100% blocks in each layer and take input images of rn X Ta resolution consequently utilizing specific budget B Let the set of all subnets for network G be denoted as g-=(51, $2 S2.., S.,,, ) where s; is a subnet with budget b; The network with the highest computation budget executable the highest width, depth and input resolution is denoted the supernet, while the one with the lowests is called the minnet. Rest of the networks are simply addressed as subnet. We compute the budget in terms of Aulliply-Acuumulae operations (MAC), following (Li et al. 2021b), and provide the detailed budget calculation for given tuple in Table the Appendix
Domain-Specific Switchable Batch Normalization. Independently reducing internal covariate shift through batch normalization (Ioffe & Szegedy] 2015) for each of the subnets is crucial for learning conditional information specific to the network configurations (Yu al., (019b)). Addi- tionally, batch normalization parameters have been used to encode domain-speciff information for unsupervised domain adaptation (Chang al. (019). For these reasons, in AnyDA, we learn batch normalization statistics individually for each domain as well as for each of the subnets.
3.2
APPROACH OVERVIEW
Figure 2 provides overview of AnyDA approach. We adopt teacher-student framework consisting of two switchable networks with the same architecture, namely gost and G parame- terized respectively by Osts and Oseas which following B. The student network is fed with both source and target images, while the teacher net- work takes only target images. Given an input image, each of the student subnets s:tu and teacher subnets sfna predicts probability distributions p, and p!na over the label space L by utilizing maximum budget b, per prediction. The student supernet, subnets, and minnet are trained using supervised learning on the source data. For target data, we propose a bootstrapped recursive dis- tillation approach to train the student subnets with knowledge obtained from the teacher network. Specifically, the student minnet is trained to match its output with the average prediction of the teacher subnets, while the student subnets are trained to match their outputs with the prediction of the teacher supernet. Teacher network parameters that of the student network. The teacher network only gets target data as input and provides the cross-domain knowledge essential for domain alignment to the lower computation budget student
configured into n subnets with increasing budgets
updated exponential moving average of
4

---

Published as a conference paper at ICLR 2023
networks. Additionally, to ensure discriminative latent space for the unlabelled target images, harness the categorical information by leveraging self-supervision through a pseudo-label loss on the student supernet. We now describe each of our proposed components individually in detail.
3.3 ANYTIME DOMAIN ADAPTATION
Bootstrapped Recursive Distillation. Self-distillation (Caron et al. (021) leverages multiple views of the same input to train a student network to match the output of a teacher network. To this end, we propose a recursive distillation learning paradigm where we emulate multiple views of an input through the outputs of the budgeted subnets. Each of the individual subnets encodes unique infor- mation specific to the configured width, depth and resolution providing us with diverse knowledge for a given input. For each of the subnets in G and Goe.* the predicted logits are normalized with softmax for any input to output probability distribution over the label space L as follows:
exp(5) stultt (st))
exp(s) tea(t) (x) (x))
slu(t) Py
tea(f) Pj
E (1)
E/S exp(s) stu(l) (x)/Tatu)
E exp(s) teall) (st)/Teea)
where, 5 and sf are any subnets in Goom** and G respectively, Tatas Thea > 0 are tempera- ture parameters controlling the sharpness of the output distributions. We allow the student network to observe data from both source and target domain while the teacher network sees data from target only. This is crucial to alleviate the distribution shift by encouraging "target-to-sone dences for distillation from the teacher to the student. Ensemble of teacher networks is shown to
correspon-
generate more accurate and general soft-labels for distillation (Shen et al. 2019; Shen & Savvides) 2020). So, consider the average prediction of the teacher subnets s to s as a representative of the diverse knowledge learned by them. Additionally, this reduces the effect of noisy subnets while distilling. The average prediction is obtained by averaging the logits and then softmaxing. Now, for a target image Ię E Dr, let the corresponding softmaxed logits from the student subnets be {Pit" p ,.., ptn J. the same from the teacher subnets be {r/ta plea ..... p..., } and plee be the P
average prediction of the subnets stea to tea
We formulate the recursive distillation loss as,
Cra(xe) = H(ptt" Praog) )
+ Σ H(P pto))
where, H(a, b) = - a log(b()) is cross-entropy loss and is minimized with respect to stu- dent parameters Ostu only. The first term in Eqn. Dannarrains student minnet to mimic average prediction of teacher subnets which helps to gain noise robustness using the ensemble especially when distill- ing to very low capacity networks. The second term encourages student subnets to mimic teacher supernet on target data, enabling recursive flow of knowledge. The teacher parameters Ozca are updated as: Orea < 20tea + (1 A)0stus where À represents the momentum hyperparameter. The teacher network is maintained exponential moving average of historical parameters over training iteration allowing to bootstrap on previous representations, as in (Grill et al.) 2020).
Target Pseudo-Labels. While the bootstrapped recursive distillation loss helps in learning low bud- get networks with simultaneous domain alignment, relying only on the source domain for categorical information be sub-optimal for the target domain. Therefore, we use pseudo-label loss on the student network to harness categorical information from the target domain through self-supervision. For target image H, D, the pseudo-label loss is formulated as:
(xx) = (Imax(pit") Z Tpt H(). Pit")) Tyt pseudo-label threshold
where, 0r = =arg max(pfta) is the pseudo-label obtained from the student supernet sffu for Fe.-
Optimization. While training, we follow the sandwich rule (Yu & Huang 2019) for better conver- gence behavior and overall performance. Specifically, in every iteration, we train the student minnet, supernet, and two randomly selected subnets. This inspired from the fact that performances at all budget are bounded by that of the model at smallest budget (minnet) and largest budget (supernet). Thus, optimizing performance lower bound and upper bound can implicitly optimize all subnetworks of different capacities. Additionally, this is computationally more efficient compared to training all the subnetworks in each iteration. Formally, at a given iteration with mini-batches 3x C D, and Be C D,, we optimize the following loss function:
Lrotat (8,, Bc) Br,e,, Lats + EZIES, (drahrd Apth
(4)
5

---

Published as a conference paper at ICLR 2023
where, Acta, Ayd, and Ayi are loss coefficients. Lets denotes the cross-entropy loss labeled source images on all the subnets (ref. Figure 2. Also, note that because of the sandwich rule, the second term of End (ref. Eqn. 2 now applies on the two selected subnets only. Following this, the teacher network parameters are updated using ema in every iteration.
Warm-up using Source Data. Absence of labels in the target domain may quickly make the net- works learn a degenerate solution if we do not take care with proper initialization (Tzeng et all] 2017). Additionally, random initialization is prone to provide high-confident noisy pseudo-labels leading to catastrophic outcomes. Therefore, before training AnyDA with Eqn. 4 we warmup the student network with the Late loss on the source data. Once the warmup is completed, we copy the learned weights 0. to dg to provide the same initialization to both the networks.
Inference. Once the networks are trained, we only use the student network for inference. Given any computational budget b; E B, we first obtain all possible configuration tuples in R. x W x 'D forming subnets of budget bg. E.g. in our experimental setup, given a budget constraint of 0.85 x 10 MACs, we have the tuples < 224, 0.9, 0.5 > and < 160, 0.9, 1.0 satisfying the requirement. We then test all of these subnets on a validation set, choose the best performing subnet and report its accuracy.
EXPERIMENTS
Datasets. We evaluate the performance of our proposed approach using 4 benchmark datasets, namely, Office-31 (Saenko el. 2010)). DomainNet (Peng et al. 2019). Office-Home Venkateswara al.] 2017) and ImageCLEF-DA (Long et al., 2017). Office-11 31 contains a total of 4,652 images belonging to 31 categories from 3 distinct domains: Amazon (A), Webcam (W) and Dsir (D). Do- mainNet is the largest available benchmark dataset containing images from 6 domains: Infograph (inf). Quickdraw (qdr), Real (rel), Sketch (skt), Clipart (clp), and Painting (pnt). 600K images distributed among 345 categories. Office-Home is a challenging dataset containing images from different domains: Artistic (Ar), Clipart (CI), Product (Pr), and Woal-World images (Rw) belonging to 65 categories. The ImageCLEF-DA dataset has around 1,800 images belonging to 12 categories in domains: Caltech 256 (C), ImageNet ILSVRC 2012 (I) and Pascal VOC 2012 (P).
Baselines and Implementation Details. We compare our approach with the following base- lines: ResNet (He al. 2016) models with varying depth (ResNet-18 to ResNet-152). Mo-
V3 (Howard et al. (019). MSDNet (Huang et al. 2018), REDA (Jiang et al. (0200) equipped with DANN (Ganin & Lempitsky 2015) as the domain adaptation method, TCP (Yu al. 2019a) and ResNet (18, 34, and 50) trained with knowledge distillation (Ba & Caruana 2014) from ResNet- 101 as the teacher. We also compare with DDA (Li et al., 2021b). For the baselines and DDA, we quote the numbers reported in the paper itself except for OfficeHome and ImageCLEF-DA for which we use the publicly available source code. We use ResNer-50 (He et al. 2016) architecture as the full network. For budget configuration, we use R {224, 192 160, 128). J. W = =1.0, 0.9), and D = (1.0, 0.5). providing us 16 subnets with computational budgets ranging from roughly 0.2 10 to 2.0 x 10 MACs. We further divide the range of budgets into 8 intervals of equal size. For each interval, the best performing subnet is used to report the accuracy. Warmup using source data was done top of networks initialized with Imagenet pretrained weights. In addition, following the general practice we use a validation set to obtain best hyperparameters. Additional implementation details including hyperparameter values are provided in the Appendix
4.1 RESULTS AND ANALYSIS
Office-31. Figure 3a shows the results on the Office-31 dataset. AnyDA achieves the best average accuracy of 85.1% at a budget of 0.7 x 10 MACs. Baseline methods specifically tailored for efficient domain adaptation (e.g., DDA and REDA) outperform other baselines like MSDNet, and ResNets. While comparing with DDA and REDA (Li et al. 2021b) over the same budget range, AnyDA performs significantly better, specifically for luw-budget networks. E.g., we outperform DDA by almost 8% at a budget of 0.2 x 10 MACs, while REDA by 6.9%. AnyDA's performance, unlike others, does not experience drastic drop when the available budget is decreased. This behavior clearly shows that our bootstrapped recursive distillation is not only able to train robust low-capacity models, but also ensures simultaneous domain alignment.
DomainNet. On the large-scale DomainNet dataset (ref Figure 3bb also, AnyDA outperforms DDA and other baselines. Similar to Office-31, AnyDA does especially well at very low computation

---

Published as a conference paper at ICLR 2023
Ascenacy folgetfor Ofice (lng)
Oudgat jugg
Acoescy
2
33
Sagget MUt A00)
Badget Io 2o000- A20)x1P
Bodge Er M.A-KGG
(a) Office311
(b) DomainNet
(c) Office-Home
(6) ImageCT.FF-DA
Figure 3: Performance Office-31, DomainNet, Office-Home and ImageCLEFF-A. The plots compare the average accuracy budget curves of our proposed approach, AnyDA with different baselines. AnyDA clearly outperforms the compared methods all the budget configurations on all the datasets. Specifically in the low budget settings, we observe the maximum improvement showing the effectiveness of AnyDA in learning domain adaptive networks for very low-resource applications. Best viewed in color.
Table 1: Performance on DomainNet. We show the task-wise accuracy the DomainNet dataset for 30 adaptation tasks, where each sub-table, the columns are the source domains and the rows are the target domains. Our proposed AnyDA outperforms DDA (w/ DANN) at both the compared budgets.
Ko 0.7
10" mass
155
10 macs
DDN/54)
inf
pnt
qdr
rel
ski
Avg
DBAS7)
clp
inf
qdr
rel
skt
Avg
clp
15.5
33.8
18.5
47.0
36.2
30.2
clp
16.8
26.3
20.7
51.3
39.0
32.8
inf
28.2
26.0
8,4
38.0
21.1
24.3
inf
29.4
28.1
10.0
43.8
23.0
26.9
pnt
37.6
15.9
8.9
48.1
31.8
28.5
pnt
40.4
17.2
10.9
52.1
33,9
30.9
qdr
21.5
288
7.4
15.1
13.0
11.9
qdr
21.2
3.0
8.4
18.6
14.1
13.1
rel
18.1
41.8
9.4
28.7
rel
46.0
18.7
33.9
31.1
skt
49.5
16.4
36.6
17.9
47.0
33.5
skt
51.1
17.3
40.0
20.5
50,9
36.0
Avg
36.0
13.7
29.1
12.6
39.0
26.6
26.2
Avg
37.6
14.6
31.5 14,8 43.3 28.8
28.4
ReaNosS0--0 0.7
10" moss
RosNos-90
1.6 x 10" moss
AnyDA
clp
inf
pnt
qdr
rel
Avg
AnyDA
clp
pnt
qdr
rel
skt
Avg
clp
33.2
38.4
21,2
48.8
49.3
38.2
clp
33.4
40.6
23.1
51.7
53.1
40.4
inf
13.1
14.2
2.5
17.1
13.3
12.0
inf
18.2
17.1
3.8
21.3
17.6
15.66
pnt
32.8
31.0
6.6
46.4
38.6
31.1
pnt
37.8
33.2
82
51.2
42.9
34.7
qdr
13.7
5.6
6.6
8.1
16.4
10.1
qdr
12.2
48
4.8
7.6
14.2
8.7
rel
47.9
44.7
$1.
13.7
48.5
41.3
rel
51.9
47.2
54.2
16.2
$2.9
44.5
skt
35,4
24.3
32.3
13.4
34.7
28.0
41.8
26.1
38.9
17.5
39.6
32.8
Avg
28.6
27.8
28.6
11.5
31.0
33.2
26.8
Avg
324
28.9
31.1
13.8
34.3
36.1
29.4
budgets. Our average accuracy at the lowest budget of 0.25 x 109 MACs is 24.4% which is about 6.6% more than DDA at a slightly higher lowest budget of 0.3 x 102 MACs. It worth noting that DDA used 7 layered ResNet-152 model in its S7 variant as the backbone while AnyDA uses a lower complexity backbone, ResNet-50 with comparatively weak representation learning ability. Simi- larly, when compared to the next best approach (MobileNet-v3) available at the same lowest budget as ours, the performance is better by 7.4%. At the highest budget (1.6 x 10 MACs) available for DDA, AnyDA achieves 1% improved performance of 29.4% over it. The performance improvement is continued as more computation in the anytime framework is available and reaches 29.7% at our highest budget of 2.0 10 MACs. To better illustrate that AnyDA can effectively enhance domain alignment at the highest budgets of DDA S4 and S7 architectures, we provide a finegrained compar- ative analysis on all 30 tasks of DomainNet in Table L It can be seen that even though AnyDA uses a low capacity backbone (ResNet-50 vis-a-vis ResNet-152)) our approach can exploit the goodness of the higher subnetworks for domain alignment in resource constrained scenario.
we again outperform DDA for all the budgets. Our average accuracy at the lowest budget of 0.25 x 10 MACs outperforms DDA by 30%, while by 7.9% at 0.7 x 109 MACs of budget, and achieves a best accuracy of 64% at 2.0 x 10th MACs.
Home. In Office-Home (ref Figure 3c
ImLgeCLEF-DA. Our performance in ImagoCLEF-DA (ref Figure 3d also in accordance with the trend in the rest of the datasets. We outperform DDA in all budgets significantly. At the lowest budget of 0.25 x 109 MACs AnyDA is better by an absolute margin of 23.1%, while by 4.7% at 0.7 x 10 MACs. As in other datasets, the performance of AnyDA is minimally hampered with decreasing computation budgets, showing the efficacy of our approach for low-capacity networks.
4.2 COMPARISON WITH SLIMDA
We also compared our performance with SlimDA (Meng et al. 2022), most close to ours. Due to unavailability of the SlimDA code the time of submission, we could compare the performance on
7

---

Published as a conference paper at ICLR 2023
Table 2: Comparision with SlimDA. We show average accuracy for Office-31, Office-Home and
ImageCLEF-DD AnyDA consistently shows better performance compared to SlimDA on 4 different budgets.
Office-31
Offece-Home
ImageCLEF-DA
Flops
1x
x
1
16
x
1x
x
x
1x
x
+
g
x
SlimDA MMeng et al. 2022
86.3
87.7
87.4
87.5
68.4
68.0
67.8
67.4
88.9
88.7
88.8
88.4
AnyDA
88.2
87.9
87.8
87.2
68.7
68.1
68.1
67.5
90.4
89.2
89.5
89.2
the three datasets Office-31, Office-Home and ImageCLEF-DA on which the reported results were obtained. SlimDA builds on top of more recent and stronger domain adaptation approach Sym- Net (Zhang et al., 2019b) by adding stochastic ensemble distillation with it. For a fair comparison, we also adopted AnyDA with the SymNet architecture by simply equipping the ResNet-50 back- bone of SymNet with switchable depth, width and input resolution and employing the bootstrapped recursive distillation therein without needing to change the original biclassifier design of SymNet. Table 2] shows AnyDA is better than SlimDA in all datasets across various computation budgets while being competitive at lowest compared budget of TO x on Office-31. On the other hand, the full budget comparison shows an absolute improvement of 1.9% on Office-31 dataset. This experi- ment shows the easy adaptability of AnyDA with other sophisticated domain adaptation frameworks towards efficient domain adaptation under budget constraints.
We also combine SymNet (Zhang et al., 2019b), with anytime prediction (US-Net (Yu & Huang. 2019) with slimmable width, depth and resolution ours) naively in three possible baseline settings on Office-31 and Office-Home to verify the performance of directly combining existing domain adaptation and anytime prediction techniques. Table 3 in Appendix A shows that direct combina- tion of these two existing methods fails miserably for the lower budget subnets even with additional pseudo-labeling added top of it for harnessing categorical information from the target domain. This clearly corroborates the importance of the proposed components in AnyDA towards learning a robust network executable at different computation budget as well alleviating the domain shift.
4.3 ABLATION STUDIES
We perform ablation studies on Office-31 dataset (unless otherwise specified) to test the effectiveness of different components of AnyDA and different variations of bootstrapped recursive distillation.
Training Supernet with Only Source Data. This ablation tests the naive scenario of employing a single high capacity network for anytime domain adaptation with a goal to set lower bound of the performance. We train only the supernet (a ResNet-50 model only) with source data and test it in all the computation budgets in the target domain. Figure 4a shows that the performance is poor as expected. Especially, the lower subnets suffer the most with fluctuations in the performance.
Effeet of Teacher Selection for Distillation. We verify the advantage of recursive distillation by comparing with two other variants. In the first variation, the recursive distillation loss encourages all student subnets including the minnet to mimic the highest capacity teacher supernet by modifying the first term in Eqn. 2 from H(pft" , placg)
H(pif" . From Figure 4b we consistent performance drop across all budgets by around 1% compared to the proposed approach where the minnet learns from the average of the teacher subnets. We conjecture that the progressively lower capacity and thus continually increasing gap of the student subnets with the teacher supernet can cause convergence hardship. In the second variation, we study the effect of mimicking the average of the soft logits of all the teacher subnets, supernet and minnet. Figure 4c shows that the performance drops by around 2% consistently across all computation budgets showing that our proposed recursive distillation strategy makes better use of the knowledge from multiple teacher subnets.
Training by Non-Recursive Distillation. Here, we experiment with non-recursive distillation where each student subnet learns from its companion subnet in the teacher. This is done by en- couraging each pft to mimic the corresponding soft logit pfse from the teacher resulting in Cra(zz) to be n H(p stu prts in Eqn. 2 As seen in Figure 4d this results in a drastic performance drop especially for the lower subnets showing the importance of the flow of information progressively from the top of the ladder to the bottom. Interestingly, the importance of recursive distillation can be appreciated from the fact that without it, the performance drops even below the naive approach of training a single supernet with only source data (compare with Figure 4a). Another variant of non-recursive learning is not to learn from the bootstrapped teacher, instead try mimic its nearest
8

---

Published as a conference paper at ICLR 2023
Budget (BLLAC3) x10
Ranger 581 w00)stm
Budgel MULAACC) <1d
Budgel MULL-M xtu
12
Budget MUL NOO
Source CE supernet
(b) Highest capacity
(c) Average
(d) Non-recursive
(e) ID w/o beotstrapping
Acarwoy
Budget HILLAGD)
Budgel Or MUL-ACC) STOP
Budget (1) KLL 1L1-441E/110
MULADO)
Busget AUL
(f) ID w/ bootstrapping
(g) w/ KL Divergence loss Domainn Specific RN (i) w/ Domain Discriminator
wlo PL loss.
Figure 4: Ablation Studies Office-31. The red line shows the accuracy curve for AnyDA while the black
line corresponds that of the baseline with specific ablation setting. AnyDA outperforms the corresponding baseline in all settings showing its effectiveness anytime domain adaptation. Best viewed color.
superior subnet from the student itself. Such implace distillation (ID) works well with no distribu- tion shift Yuu & Huang 2019). However such strategy (ID w/o bootstapping) naturally fails for the current task as does not use the information from supervised source data warmup. Thus, Figure 4c shows significant performance drop across all budgets. A fairer comparison would be to employ the teacher for inference which starting with source data warmup updates itself by exponential moving average of the student weights. However, Figure 4f shows that even such an inplace distillation with bootstapping (ID w/ bootstapping) does not also work well for anytime domain adaptation.
Ablation Distillation Loss. We investigate the effectiveness of cross-entropy loss (ref Eqn. by replacing it with KL Divergence loss traditionally used in knowledge distillation. Figure 4g shows that training with cross-entropy loss surpasses the performance training with KL Divergence loss by a high margin especially for the lower subnets (around 1.6 % average accuracy drop in minnet).
Ablation on Batch Normalization. Using batch normalization crucial for better convergence of both domain adaptation (Chang et al. (019) as well as slimmable networks (Yu et al., 2019b). We ran variant AnyDA where switchable batch normalization layers were used for the differ- ent subnets but these were not different for the source and target domain data. The performance degraded by roughly 1% across all computation budgets on average (ref Figure 44). We also tried using single domain-specific
noneaaaoomaiiitionnn for all the subnets but the testing accuracy was extremely poor. These results help us conclude the need for both domain as well as subnet specific batch normalization to achieve the twin goals of domain generalization and computation efficiency.
Need for Explicit Domain Discriminator. Using domain discriminators to confuse domains has been popular choice for domain adaptation. However, experiments in Figure 41 show that in anytime setting, adding an explicit domain discriminator (Ganin & Lempitsky 2015) may not offer any advantage, rather the presence of many subnetworks and result, multi-scale features can deteriorate the discriminability of the framework.
Role of Pseudo-Labeling. We remove the pseudo-label loss Cpt and observe that AnyDA without target pseudo-labels decreases the performance by more than 1% on average (ref Figure 4i indicat- ing the advantage of pseudo-labeling in enhancing discriminability by successfully leveraging label information from target domain. Additional results and analysis are included in the Appendix G
CONCLUSION
In this paper, we introduce novel approach for anytime domain adaptation by considering do- main alignment with switchable depth, width and input resolutions to achieve accuracy accuracy-effiiinncy trade-offs in the target domain for different resource constraints. In particular, we adopt teacher- student framework with bootstrapped recursive distillation to bring the target features close to the source and also to transfer the learned knowledge for efficient inference. We also leverage self- supervision via pseudo-labelin the unlabelled target images. We demonstrate the effectiveness our proposed approach on four benchmark datasets, outperforming several competing methods.
on the student supernets to ensure discriminative latent space for

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work was partially supported by Google India Research Award 2021. OC and AD sincerely acknowledge the travel reward partially supporting their travel for presenting this work. We also acknowledge support from the MIT-IBM Watson AI Lab, thich-throuee MIT Satori Supercom- puter contri-on the computational resources necessary to conduct the experiments in this paper.
REFERENCES
Jimmy Ba and Rich Caruana. Do Deep Nets Really Need to be Deep? Neural Information Process- ing Systems, 27, 2014.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizzi. Model Compression. In ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-All: Train One Network and Specialize it for Efficient Deployment. In International Conference on Learning Representations, 2019.
Fabio M Carlucci, Antonio D' D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main Gemeralization by Solving Jigsaw Puzzles. In IEEE Computer Vision and Pattern Recogni- tion, pp. 2229-2222222222222222222-------- 2238, 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Ar- mand Joulin. Emerging Properties in Sele-Supervised Vision Transformers. In IEEE International Conference on Computer Vision, pp. 9650-9660, 2021.
Woong-Gi Chang, Tackgeun You, Seonguk Sco, Suha Kwak, and Bohyung Han. Domain-Specine Batch Normalization for Unsupervised Domain Adaptation. In IEEE conference on Computer Vision and Pattern Recognition, pp. 7354-7362, 2019.
Yinpeng Chen, Xiyang Dai, Mengehen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic ReLU. In European Conference Computer Vision, pp. 351-367. Springer, 2020.
Zhourong Chen, Yang Li, Samy Bengio, and Si Si. You Look Twice: Gaternet for Dynamic Filter Selection in CNNs. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9172- 9180, 2019.
Aysegul Dundar, Ming- Yu Liu, Ting-Chun Wang, John Zedlewski, and Jan Kautz. Domain Styliza- tion: A Strong. Simple Baseline for Synthetic Real Image Domain Adaptation. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2020.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. SEED: Self-supervised Distillation for Visual Representation. In International Conference on Learning Representations, 2021.
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Ittil, and Anima Anandkumar. Born-Again Neural Networks. In International Conference on Machine Learning, pp. 1607-1616.
PMLR, 2018.
Yaroslav Ganin and Victor Lempitsky. Unsupervised Domain Adaptation by Backpropagation. In International conference on machine learning, pp. 1180-11888 PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Vietor Lempitsky. Domain-Adverssrria Training of Neural Net- works. The Journal of Machine Learning Research, (70))2096-2030, 2016.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot. Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap Your Own La- tent New Approach to Self-Supervised Learning. Neural Information Processing Systems, 33: 21271-21284, 2020.
10

---

Published as a conference paper at ICLR 2023
Hai Han, Yunhe Wang, Qiulin Zhang. Wei Zhang, Chunjing Xu, and Tong Zhang. Model Rubik's 's Cube: Twisting Resolution, Depth and Width for Tinynets. Neural Information Processing Sys- tems, 33:19353- 19364, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. arXiv preprint arXiv: 1503.02500 2015.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent Adversarial Domain Adaptation. In International Conference Machine Learning, pp- 1989-1998. PMLR, 2018.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for Mobilenetv3. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314-1324, 2019.
Hanzhang Hu, Debadeepta Dey, Martial Hebert, and Andrew Bagnell. Learning Anytime Pre- dictions in Neural Networks via Adaptive Loss Balancing. In AAAI Conference on Artificial Intelligence, volume 33, pp. 3812-3821, 2019.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger. Multi-Scale Dense Networks for Resource Efficient Image Classification. In International Con- ference on Learning Representations, 2018.
Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International conference machine learning, pp. 448-456. PMLR, 2015.
Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, and Il-Chul Moon. Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation. In IEEE Conference on Computer Vision and Puttern Recognition, pp. 10664-10673, 2021.
Junguang Jiang, Ximei Wang. Mingsheng Long, and Jianmin Wang. Resource Efficient Domain Adaptation. In ACM International Conference on Multimedia, pp. 2220-2228, 2020.
Zequn Jie, Peng Sun, Xin Li, Jiashi Feng, and Wei Liu, Anytime Recognition with Routing Con- volutional Networks. IEEE Transactions Puttern Analysis and Machine Intelligence, 43(6): 1875-1886, 2019.
Wouter M Kouw and Marco Loog. A review of domain adaptation without target labels. IEEE Iransactions on pattern analysis and machine intelligence, 43(3);766-785, 2019.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. FractalNet: Ultra-Deep Neural Net- works without Residuals. International Conference on Learning Representations, 2017.
Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Rethinking Data Augmentation: Self-Supervision and Seff-Distillation. In International Conference on Learning Representations, 2020.
Bo Li, Yezhen Wang, Tong Che, Shanghang Zhang, Sicheng Zhao, Pengfei Xu, Wei Zhou, Yoshua Bengio, and Kurt Keutzer. Rethinking Distributional Matching Based Domain daptation. arXiv preprint ar arXiv:2006.13:: 13352, 2020a.
Changlin Li, Guangrun Wang, Bing Wang. Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic Slimmable Network. In IEEE Conference Computer Vision and Pattern Recognition, pp. 8607- 8617, 2021a.
Shuang Li, Jinming Zhang, Wenxuan Ma, Chi Harold Liu, and Wei Li. Dynamic Domain Adaptation for Efficient Inference. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 7832-7841, 2021b.
11

---

Published as a conference paper at ICLR 2023
Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun. Learning Dynamic Routing for Semantic Segmentation. In IEEE Conference on Computer Vision and Pattern Recognition. pp.
2020b.
Lanlan Liu and Jia Deng. Dynamic Deep Neural Networks: Optimizing Accuracy-Efficienne Trade- Offs by Selective Execution. In AAAI Conference on Artificial Intelligence, 2018.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pp. 2208 2217. PMLR, 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional Adversarial Domain Adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
Mason McGill and Pietro Perona. Deciding How to Decide: Dynamic Routing in Artificial Neural Networks. International Conference on Machine Learning, pp. 2363-2372 2372. PMLR, 2017.
Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang. Instance Adaptive Self-Training for Unsu- pervised Domain Adaptation. European conference on computer vision, pp. 415-4344 430.. Springer, 2020.
Rang Meng, Weijie Chen, Shicai Yang, Jie Song, Luojun Lin, Di Xie. Shiliang Pu, Xinchao Wang, Mingli Song, and Yueting Zhuang. Slimmable Domain Adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7141-7150, 2022.
Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image to Image Translation for Domain Adaptation. In IEEE Conference Computer Vision and Pattern Recognition, pp. 4500-4509, 2018.
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-Adversarial Domain Adap- tation. In AAAI conference on artificial intelligence, 2018.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment Matching for Multi-Source Domain Adaptation. IEEE International Conference on Computer Vision, pp- 1406-14455 2019.
Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. SENTRY: Selective Entropy Optimization via Committee Consistency for Unsupervised Domain Adaptation. In International Conference on Computer Vision, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for Thin Deep Nets. arXiv preprint arXiv: 14Xiv:1412.65500 2014.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting Visual Category Models to New Domains. In European conference on computer vision, pp. 213-226. Springer, 2010.
Aadarsh Sahoo, Rutav Shah, Rameswar Panda, Kate Saenko, and Abir Das. Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing. In Neural Information Processing Systems, 2021.
Aadarsh Sahoo, Rameswar Panda, Rogerio Feris, Kate Saenko, and Abir Das. Seleet, Label, and Mix: Learning Discriminative Invariant Feature Representations for Partial Domain Adaptation. In IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4210-4219, 2023.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein Distance Guided Representation Learning for Domain Adaptation. In AAAI conference on artificial intelligence, 2018.
Zhiqiang Shen and Marios Savvides. Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks. arXiv preprint arXiv:2009.08553 2020.
Zhiqiang Shen, Zhankui He, and Xiangyang Xue. Meal: modtl ensemble via adversarial learning. In AAAI Conference Artificial Intelligence, volume 33, pp. 4886-4893, 2019.
12

---

Published as a conference paper at ICLR 2023
Baochen Sun and Kate Saenko. Deep Coral:
Correlation Alignment for Deep Domain Adaptation. In European conference on computer vision, pp. Springer, 2016.
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised Domain Adaptation through Self-Supervision. arXiv preprint
2019.
Shuhan Tan, Xingchao Peng, and Kate Saenko. Class-imbalanced Domain Adaptation: an Empirical Odyssey. In European Conference on Computer Vision Workshops (TASK-CV), pp. 585 302-602. Springer, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive Representation Distillation. In Inter- national Conference on Learning Representations, 2020.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous Deep Transfer across Domains and Tasks, In IEEE international conference on computer vision, pp. 4068-4076, 2015.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial Discriminative Domain Adaptation. In IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep Hashing Network for Unsupervised Domain Adaptation. In IEEE Conference Computer Vision and Puttern Recognition, pp. 5018-5188000 5018-5027, 2017.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning Dy- namic Routing in Convolutional Networks. In European Conference Computer Vision, pp- 409-424, 2018.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic Inference Paths in Residual Networks. In IEEE Confer- ence on Computer Vision and Puttern Recognition, pp. 8817-8826, 2018
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge Distillation Meets Self- Supervision. In European Conference on Computer Vision, pp. 588-604. Springer, 2020.
Taojiannan Yang, Sijie Zhu, Chen Chen, Shen Yan, Mi Zhang, and Andrew Willis. Mutualnet:
Adaptive Convnet via Mutual Learning from Network Width and Resolution. In European con- ference computer vision, pp. 299-315. Springer, 2020.
Chaohui Yu, Jindong Wang, Yiqiang Chen, and Zijing Wu. Accelerating deep unsupervised do- main adaptation with transfer channel pruning. In 2019 International Joint Conference on Neural Networks (IJCN), pp. 1-8. IEEE, 2019a.
Jiahui Yu and Thomas S Huang. Universally Slimmable Networks and Improved Training Tech- niques. In IEEE International Conference on Computer Vision, pp. 1803-1811 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable Neural Networks. In International Conference on Learning Representations, 2019b.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation. In IEEE International Conference on Computer Vision, pp. 3713-3722, 2019a.
Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Sommin-Symmetrrc Networks for Adversarial Domain Adaptation. In IEEE conference computer vision and pattern recognition, pp. 5031- 5040, 2019b.
Yiheng Zhang. Zhaofan Qiu, Ting Yao, Dong Liu, and Tao Mei. Fully Convolutional Adaptation Networks for Semantic Segmentation. In IEEE Conference Computer Vision and Pattern Recognition, pp. 6810-6818, 2018.
Sicheng Zhao, Xiangyu Yue, Shanghang Zhang, Bo Li, Han Zhao, Bichen Wu, Ravi Krishna, Joseph E Gonzalez, Alberto L Sangiovanni- Vincentelli. Sanjit A Seshia, et al. A review of single- source deep unsupervised visual domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, 2020.
13

---

Published as a conference paper at ICLR 2023
A COMPARISON WITH NAIVE COMBINATION OF ANYTIME PREDICTION AND DOMAIN ADAPTATION
In Table 3 we combine one existing state-of-the-art domain adaptation (DA) method, namely SymNet (Zhagg et al. 2019b). with anytime prediction (AP) (US-Net (Yu & Huang, 2019) with slimmable width, depth and resolution as ours) naively in three settings: (I) Domain adaptation Anytime prediction (inference only): in this experiment, we first train a network using the SymNet domain adaptation method. Once trained, we used the adapted model directly for anytime inference to obtain its performance at various computation budgets on the target domain. (2) Domain adap- tation + Anytime prediction (w/ Pseudo-labeling): in this experiment, similar to point (I) above, we obtain an adapted network using the SymNet domain adaptation technique on source and target data. After that, we further train the network in an anytime fashion using US-Net on the target data using supervupenvi through pseudo-labels (PL). Note that since the target data is unlabeled, need to use pseudo-labels to train on them using existing anytime networks. (3) Anytime prediction Domain Adaptation: in this experiment, labeled source data. Then, we train it using the SymNet domain adaptation approach. We compare the performance of AnyDA using the same SymNet backbone network for fair comparison.
first train network for anytime prediction the
Table 3: Comparison AnyDA with naive combinations of anytime prediction and domain adaptation on Office-31 and Office-Home. DA: Domain Adaptation, AP: Anytime Prediction, PL: Pscudo- Labeling. We use SymNet backbone for all the experiments.
Office-31
Office-Home
Flops
1x
2
x
x
10
Avg
1x
x
10
X
Avg
DA + AP (inference only)
86.4
35.7
31.8
30.0
46.0
62.8
24.5
22.9
20.8
32.8
DA + AP (w/ PL)
87.9
43.6
38.5
37.7
51.9
62.7
31.9
28.4
27.6
37.7
DA
73.6
46.3
43.2
40.0
50.8
56.5
47.0
35.6
32.2
42.8
AnyDA
88.2
87.9
87.8
87.2
87.8
68.7
68.1
68.1
67.5
68.1
We have the following observations from Table (1) DA AP (inference only): simply performing anytime inference on domain adapted model performs poorly as compared to AnyDA even when using the full network with the highest budget (1.8% lower in Office-31, 5.9% lower in Office- Home) and fails miserably for the lower budget subnets (e.g. 57.2% lower in Office-31, 46.7% lower in Office-Home in the lowest budget configuration); (2) DA AP (w/ PL): as can be seen, the performance improves as compared to point (1) because the additional anytime training on the target data. But, the the huge drop in performance from the highest budget network to the lower budget subnets still significant (e.g. 49.5% and 39.9% lower than AnyDA in the lowest budget network); (3) While this variation performs better at lower budgets, the performance the highest budget is lower than the other two variants, showing that anytime training without exploiting target data can give a poor initialization for anytime domain adaptation. Our approach outperforms AP+DA variant by 14.6% and 12.2% at the highest budget, while significantly outperforming it by more than 30% at the lower budgets for the Office-31 dataset. Similar observations can be made in the Office-Home dataset as well. To summarize, all the findings above corroborate the importance of the proposed components in AnyDA towards learning a robust network executable at different computation budget as well as alleviating the domain shift.
14

---

Published as a conference paper at ICLR 2023
B PERFORMANCE USING LARGER BACKBONE
Accuracy vs Budget for Office-31 (evg)
In Figure 5 we perform additional experiments using ResNet- 101 on Office-31 dataset in order show the effectiveness of our approach in larger backbones. We train DDA using the largest available supported backbone of S7. As can be seen,
Accuracy (%)
AnyDA achieves the best average accuracy of 86,4% at a bud-
get of 3.9 x 10 MACs. At the highest comparable budget with
DDA of 1.6 10 MACs, AnyDA significantly outperforms by 4.2%. AnyDA's performance, unlike others, does not ex- perience drastic drop when the available budget is decreased. The improvement obtained at the lower budgets is even signifi- cantly more, e.g., at the corresponding lowest comparable bud- gets, AnyDA outperforms DDA by 16.2% at 0.3 x 10 MACs and ResNet by 5.9% 0.9 x 10 MACs. This behavior clearly shows that our bootstrapped recursive distillation is not only able to train robust low-capacity models, but also ensures si- multaneous domain alignment with larger backbones.
-0000))7 DANN
Ros
Budget (nn x1C)
Figure 5: Performance on Office- 31. Plots compare the average accu- racy vs budget curves for AnyDA using ResNet-101 101 backbone, with two other baselines. Best viewed in color.
C FEATURE VISUALIZATION UNDER DIFFERENT BUDGETS
Figure O shows the t-SNE visualizations on four adaptation tasks (A- D->A, A-D,, and D D-+) from the Office-31 dataset. The figure shows the clustering of the target features at various computa- tion budget subnets from the supernet in the left to the minnet in the right for AnyDA. As can be seen from Figure the clustering is fairly consistent and discriminative across the subnets till the network with budget 0.6z 10 MACs while slightly slackening for the very low budgets (after 5th column), showing the effectiveness of the bootstrapped recursive distillation in learning discriminative feature space different budget configurations.
2 x 10
1.5 10
x 10
0.65 10
0.6 10"
0.5 10
0.3x 10
0.2 10
2 x
1.5 x 10
x 10
0.65 10
0.6 x 10
0.5 10
0.3x10
0.2 10
2 x 10
1.5 x 10"
x 108
0.65 x 109
0.6 x 10
0.5 109
U.3 0.3x10 10
0.2 x 10
10
1.5 10
x 10
0.65 102
0.6 102
0.5 10
0.3
0.2 10
Figure 6: Feature Visualization using t-SNE. Figure shows the t-SNE visualizations the target features on four adaptation tasks (A-W D D-+, ADD, and D-W, from top to bottom, respectively) from the Office-31 dataset. The computation budgets in MACs mentioned under the plots, with supernet at the extreme left, while minnet the extreme right. Best viewed in color.
D DISCUSSION ON UTILITY OF ANYTIME DOMAIN ADAPTATION
The focus of our work on learning a single network which is domain invariant and can be executed at different computational budgets (MACs) in multiple devices with different budget requirements. Training a single network with the ability to be executed at different budgets is more feasible and efficient than training separate networks of corresponding budget configurations. In many practical applications, once the model is trained, it is an extremely important to perform inference many times (without retraining) due highly dynamic deployment environments (train once but inference
15

---

Published as a conference paper at ICLR 2023
many). Additionally, computation cost of training network for AnyDA is more than that of conventional network due to joint training of all the subnets. However, we follow sandwich rule to reduce the training-tinm time computation overhead by almost 4 times as compared to forward-passing through all the subnets. Focussing on both training-time as well as inference-time efficiency is an interesting research topic, which would be exciting future work.
E DATASET DETAILS
We evaluate the performance of our approach on four benchmark datasets, namely, (1) Office- 31 (Saenko et al. 2010), (2) Office-Home /Venkateswara et al., (017), (3) DomainNet (Peng et al., 2019) and (4) ImageCLEF-DA (Long et al. 2017). Below we provide them details.
Office-31. This dataset contains 4, 110 images distributed among 31 different categories and col- lected from three different domains: Amazon (A), Webcam (W) and Dsir (D), resulting in 6 transfer tasks. Amazon images are collected from merchant websites and possess a clear white background. Webcam images are captured using web-cam and are of lower resolution. DsIr images are low- noise high resolution images. The dataset is imbalanced across domains with 2, 817 images belong- ing to Amazon, 795 images to Webcam, and 498 images to Dsir, making Amazon a larger domain as compared to Webcam and Dsir. The dataset is publicly available to download at: https://people: eecs. berkeleyye www.eyecccu..... jhof fman/ www.ddaaaaaaaaadooom http:///aatttttsseooooaoe _ooee
This dataset contains 15, 588 images distributed among 65 different classes and collected from four different domains: Art (Ar), Clipart (CI), Product (Pr). and RealWorld (Rw), resulting in 12 transfer tasks. The Art domain has images of paintings, sketches and artistic depic- tions. The Clipart domain consists of clipart images, while Product domain has clear background images. The RealWorld domain has regular images captured with camera. The dataset is split across domains with 2427 images belonging to Art, 4365 images to Clipart, 4439 images to Product, and 4347 images to RealWorld The dataset is publicly available download at:
Office-Home.
ott/Offfiieooooooooooooooooooooe Dataset/
DomainNet. This dataset one of the largest domain adaptation benchmark datasets available containing around 0.6 million images. The images are distributed among 345 different categories and are collected from six different domains: Infograph (inf), Quickdraw (qdr), Real (rel). Sketch (skt). Clipart (clp), and Painting (pnt), resulting in 30 transfer tasks. We use the cleaned version of the dataset which is split across the domains with 51, 605 images belonging to Infograph, 172, 500 images Quickdraw, 172, 947 images to Real, 69, 128 images to Sketch, 48, 129 images to Clipart, and 72, 266 images to Painting. The dataset is publicly available to download at:
http://a bu. www.e3/u//....
ImageCLEF-DA.
The ImageCLEF-DA dataset is a benchmark dataset used in the ImageCLEF do- main adaptation challenge of 2014. Following the standard practise, carried out our experiments considering the three domains, Caltech-256 (C), ImageNet ILSVRC 2012 (I) and Pascal VOC 2012 (P). There total of 12 categories in each domain with each category having around 50 images. The dataset is publicly available at:
http:///imaeeeee...rr///3 wwtp://cccee.....11oo///// http://211//dapattioo.
F IMPLEMENTATION DETAILS
The pscudo-code for the training of AnyDA is shown in Algorithm I
Training Details. For Eqn. we use Tata 0.1 and Stea = 0.04. We use a momentum hy- perparameter value of X 0.96. In Eqn. 3 threshold value of Tn 0.9 was used for Office- 31 and Office-Honne
dataset, while Tpt = 0.4 for the DomainNet dataset. While in Eqn. we
use Xels 1, 15, 64, Xra 1, 1, 0.5 for Office-31, Office-Home and DomainNet, respectively, Ayt = 0.1 for all the datasets. We perform warm-up using source data for 100, 100, 30 epochs for Office-31, Office-Home and Net,inNet, respectively. The proposed approach An is trained
16

---

Published as a conference paper at ICLR 2023
Algorithm The training pseudocode for AnyDA
Data: source data D, and target data Dr, hyperparameters
Networks: Student network Ghstu* {s9 stu .... s and Teacher network gheen {sten .82 ..20 ,,"5,.....'' s
1: warmup the networks goau** using source data D.
2: Ite = fsta
# load mini-batch and iterate over dataloader
3: for s, f in loader do
4: obtain predictions pri for all seen in g
5: compute fre
# Sandwich rule
6: stu-subnets + minnet, two randomly sampled subnets, supernet from Goan*
# Iterate over the student subnets
7:
for o in stu subnets do
8:
obtain predictions pst .P compute cross- cros--entrppy loss for p.i compute recursive distillation loss for Pi using H()
9:
10:
11:
compute pseudo-label loss for pi if s7 is supernet end for
12:
13:
compute Cels, Lrd. Cpt
14:
optimize Goata using gradient descent
# Update teacher network using ema
15:
update teacher network: Atea t- Arrea + (1 2)0
end for def H(ps, Pt):
17:
Stop gradient deeaach)
18:
return (p: * (og(a:))sum(ddn=1))mmnng)
for 30, 100, 20 epochs, respectively. Different input configurations are handled by our network as: depth: for a depth of d, we only execute first dx 100% blocks in each layer. E.g. layer-1 of ResNet- 50 has 3 blocks, so for d=0.5, we execute only the first |3x0.5 |=1 block of layer- width: Similarly, for a width of w, we only execute first wx 100% filters/weights for all layers. resolution: we con- sider R [224, 192, 160, 128] which are compatible with convolution kernel size, padding and strides of ResNet-50 network. Following Yuu & Huang 2019), we use single linear layer as our classifier. Specifically, for linear layer of shape d x (n = # classes), given width scale of 0.9, the subset weights of shape 0.9d X n is used. We use per-gpu batch size of 64 (32 source + 32 target) for all the experiments. We while 3e-5 for DomainNet. We follow cosine annealing to update the learning rate. We report the average classification accuracy. We additionally use information maximization on target data regularizer. All the experiments were performed using 4 NVIDIA Tesla V100 GPUs.
learning rate of 2e-4 for Office-31 and Office-
Budget Calculation. In Table 4. we show the budget values for all the 16 configuration tuples. Specific budgets at which our network is executed during inference are not pre-set, instead only range of budgets (minimum and maximum) for defining subnets during training is used in our current work. As can be seen, we obtain 8 intervals of equal size from the 16 configurations. After training, our trained model (the student network) executable at various budget configurations. The goal is to find the best configuration under a particular resource constraint. We achieve this by using a query table. For example, in ResNet-50, we sample network width from {1, 0.9), network depth from {1. 0.5} and sample input resolution from {224, 192, 160, 128]. We test all these width- configurations on a validation set and choose the best one under a given budget at inference. Since there is no re-training, the whole process is once for all. For handling budgets that fall outside of this range, one can train a supernet with a very large backbone or by considering extreme low values of width, depth and resolution: leave this an interesting future work.
17

---

Published as a conference paper at ICLR 2023
Table 4: Budget Calculation. show the details all configuration tuples their corresponding
budget values. The third column groups the subnetworks having computation budget within a defined range. We divide the whole computation budget of ResNet-50 into equally spaced 8 ranges of size 0.25 x 10 macs. If multiple subnetworks come under this range (e-g., two subnetworks denoted by < 1 x 0.9 192 > and the range of 10" 1. 1.25 10"), then report the result
0.5 224 > in the first column given by the best performing subnetwork.
ResNet-50
R x x
Budget (macs x10%)
Budget Range (macs x10")
224 1.0 x 1.0
2.00
(1.75,2.00)
224 0.9 1.0
1.65
(1.50.1.75)
192 1.0 1.0
1.50
(1.25-1.50)
192 1.0
1.20
(1.00-1.25)
224 1.0 0.5
1.05
160 1.0 1.0
1.00
224 0.9 0.5
0.85
(0.75-1.00)
160 0.9 1.0
0.85
192 1.0 0.5
0.75
128 1.0 1.0
0.65
(0.50-0.75)
192 0.5
0.60)
128 1.0
160 1.0 0.5
0,50
160 0.9 x 0.5
0.40
(0.25, 0.50]
128 1.0 0.5
0.30
128 0.9 0.5
0.20
(0.00, 0.25]
G ADDITIONAL EXPERIMENTS
Effeet of Hyperparameters. We study the effect of the coefficient hyperparameters Ara and Apt- For Office-31,, 31, we consider values of Ayd and Apt an order above and below the best value used in experiments and show the obtained results in Figure] 7 We observe that X,d 1.0 and Au 0.1 gives the best performance.
Accuracy Budgel Office-31 (avg)
Accuracy vs Budget for Office-31 (avg)
Accuracy (%)
Accurecy
1.
25
2.5
Budget (in MUL-ACC) x10
Budget MUL-ACC) x10
Azd
Ayp
Figure 7: Effect of hyperparameters. The plots show average accuracy vs budget by varying the hyperpa- rameter values Xyd and Ap1 (ref. Eqn. in main paper) for the Office-31 datasct.
Ablation Studies on OfficeHome. Similar to the main paper, we perform ablation studies on the OfficeHome dataset, and report all the findings in Figure 8 For all the cases, AnyDA outperforms the corresponding ablation setup, which affirms the importance the proposed bootstrapped recursive distillation method along with the other components for anytime domain adaptation.
18

---

Published as a conference paper at ICLR 2023
Acsint
x11
Budget MULLADD) xto
Budgst MULAACO) stor
Budget MUL-ADOY
Source CE
Highest capacity
Average
Kon-recursive
Busget AMO xto
Budact MUL XDD
ACC
ID w/o hootstrapping
ID w/ bootstrapping
Divergence
w/o Domain Specific BN
Buaget X11
w/O PL. loss
Figure 8: Ablation Studies. The plots show average accuracy budget curves for different ablation studies performed on Office-Home dataset. responds to that of the baseline with specific ablation setting. Here we observe similar trend of the individual ahlation studies with those for Office-31 dataset. For Office-Home also, AnyDA outperforms the corresponding baseline in all settings showing its effectiveness anytime domain adaptation. Best viewed in color.
line shows accuracy curve for AnyDA while black line cor-
Benefit of Recursive Distillation. It was shown in (Yang et al. 2020) that decreasing the lower bound of computation budget can result in decrease of the model performance at higher budgets. In this experiment, we tested by running AnyDA with increased lower bound of the computation from 0.2 10th macs 0.5 10 macs on the Office-31 dataset and studied the performance in Figure We observe that the increased budget at lower end of the subnetworks provides very similar performance with slight improvements. This shows the effectiveness of the recursive distillation method in learning robust representations even if we have very low budget subnetworks which can not drag down the performance of the higher end subnetworks, rather the lower end subnetworks perform almost good as the higher end ones.
H BROADER IMPACT
Our research anytime domain adaptation can help reduce burden of collecting large-scale super- vised data in many real-world applications by transferring knowledge from auxiliary datasets. At the same time, it can have positive impact on many applications that require customization of single deep neural network in the target domain to meet the dynamically changing demand. Our research anytime domain adaptation high impact on the environment as Al systems become more prevalent. Potential negative impacts share many of the pitfalls associated with standard deep learning models such vulnerability to adversarial attacks and dataset bias, and lack of interpretability.cm etc.
also reduce the memory and power consumption, leading
19

---

Published as a conference paper at ICLR 2023
Accuracy vs Budget for Office-31 (avg)
84
Accuracy( (%)
82
80
AnyDA (lowest budget 0.55 x 109 macs)
AnyDA (lowest budget 0.2 109 macs)
78
0.2
1.2 Budget (in MUL-ACC) x109
0.7
1.7
2.2
Figure 9: Effect of increasing budget lower bound. Plot shows difference in average accuracy budget curves by increasing the lower bound of budget from 0.2 10" macs to 0.55 10 macs on Office-31.
Arcurngy Rudgel-
Aczuracy Budgel (D-W))
Ridget
Actuvey 850
050
A W
D-
W-D
Accurnty Budgee- Ortcall (A-D))
Acrunay Burgall Offedl (D-A)
Aczureyy vs Bhidgel Drteatt (W-A)
Bucget
Bage
A
D-A
W-A
Figure 10: Performance Office-31. The plots 10a to 100 show the accuracy budget curves for all the six adaptation tasks of the Office-31 dataset. Best viewed in color.
20

---

Published as a conference paper at ICLR 2023
Accussay Sudget Offic-Home (4--00
Accurazy Budgel OTcc--oome (Ar-
Accuresy Budget OTfcc-Hmme (Ae-Rw)
Budgglt-Oicc-Heme
400.00
Accussdy Budget Orice-Home (CI-AA)
Accurayy Budget Olzcetomee (CI-PP)
Accusasy Budget Orico-Hmme (CI-RW)
400
CI-AAr
Cr-PP
Cl Rw
Accusscy Budget Orice-Hnns (Pr-A))
Acuuaay Budget Ofce-tome (Pr-Ch
Accusscy Budget Olcc-Home (Pe-RW)
Pr-AAr
Pr-CI
Pr- Rw
Accurady Budget Off (RW-AA)
Accrazy Budgel Ofce-Hone (Ra-iD)
Accurac vi Budget OffceHooe (Ry-PP)
Rugget j1 400
Rutirii MuL 4008
Aw-+Ar
Rw-SI
Rw
Figure 11: Performance on Office-Home. The plots Ila the twelve adaptation tasks of the Office-Home dataset.
111 show the accuracy vs budget curves for all
21

---

Published as a conference paper at ICLR 2023
Table 5: Performance DomainNet. We show the task-wise accuracy the DomainNet dataset for 30 adaptation tasks and for all the budget intervals, where in cach sub-table, the column-wist domains are the source domain and the row-wise domains are the target domain. The corresponding budget values are given above cach of the sub-table. We use ResNet-50 backbone for all our experiments.
0.2 x 10" mass
0.3
10
AnyDA
clp
inf
pnt
qdr
rel
skt
Avg
AnyDA
clp
inf
pnt
qdr
rel
skt
Avg
clp
28.9
34.0
19.8
45.5
46.3
34.9
clp
29.0
34.7
19.8
45.7
46.3
35.1
inf
11.9
12.8
2.2
15.3
11.8
10.8
inf
12.3
13.0
2.3
15.9
12.4
11.2
pnt
29.9 28.7
5.6
43.6
35.7
28.7
pnt
30.6 28.6
5.8
44.1
36.7
29.2
qdr
11.6 3.2
4.8
5.6
14.3
7.9
qdr
11.6
4.1
4.6
6.3
14.4
8.2
rel
45.2 41.6
48.6
12.7
45.1
38.6
rel
45.7 41.7
48.8
12.4
45.8
38.9
skt
31.9 21.7
28.4 12.1
32.0
25.2
skt
32.9
21.6
28.3
12.2
32.5
25.5
Avg
26.1 24.8 25.7 10.5 28.4 30.6
24.4
Avg
26.6
25.0 25,9 10.5 28.9
31.1
24.7
0.7 x 10 macs
1.0 109 macs
AnyDA
clp
inf
pnt
qdr
rel
skt
Avg
AnyDA
clp
inf
pnt
qdr
rel
skt
Avg
clp
33.2 38.4 21.2 48.8
49.3
38.2
clp
32.9
39.1
23.0
51.1
52.9
39.8
inf
13.1
14.2
2.5
17.1
13.3
12.0
inf
16.5
16.5
2.9
20.1
16.6
14.5
pnt
32.8 31.0
6.6
46.4
38.6
31.1
pnt
36.1
32.2
7.4
49.9
42.6
33.6
qdr
13.7 5.6
6.6
8.1
16.4
10.1
qdr
12.2 4.4
4.8
7.6
14.2
8.6
rel
47.9 44.7
51.5
13.7
48.5
41.3
rel
51,4 46.9
54.1
14.9
51.9
43.8
skt
35.4 24.3 32.3 13.4 34.7
28.0
skt
39.8 25.9
35.0
15.2
38.5
30.9
Avg
28.6 27.8 28.6 11.5 31.0 33.2
26.8
Avg
31.2 28.5 29.9 12.7 33.4 35.6
28.6
1.2 x 10" macs
1.5 10 macs
AnyDA
clp
inf
pnt
qdr
rel
skt
Avg
AnyDA
clp
inf
pnt
qdr
rel
skt
Avg
clp
32.9 39.3 23.0 51.0
52.9
39.8
clp
33.4 39.3 23.3
51.0
52.5
39.9
inf
16.5
16.4
2.9
20.1
16.6
14.5
inf
17.3
17.4
2.8
20.1
17.3
15.0
pnt
36.1 32.0
7.4
49.6
42.6
33.5
pnt
36.2
32.2
7.4
49.9
42.7
33.7
qdr
12.4 4.4
4.9
7.6
14.2
8.7
qdr
12.4 4.4
4.9
7.2
14.3
8.6
rel
50.3 46.6
54.1
14.9
51.8
43.5
rel
51.6 46.6
54.5
15.1
51.9
43.9
skt
40.3 25.5 34.7
15.2
38.5
30.8
skt
40.5 25.2
35.3
15.2
38.9
31.0
Avg
31.1 28.3 29.9 12.7 33.4 35.6
28.5
Avg
31.6
28.4 30.3 12.8 33.4 35.7
28.7
1.6 x macs
2.0 maes
AnyDA
clp
inf
pnt qdr
rel
skt
Avg
AnyDA clp
clp
inf
pnt
qdr
rel
skt
Avg
clp
33.4
40.6 23.1 51.7
53.1
40.4
33.7
39.9
23.5
51.3
53.8
40.4
inf
18.2
17.1
3.8
21.3
17.6
15.6
inf
18.3
17.5
3.9
21.7
17.6
15.8
pnt
37.8 33.2
8.2
51.2
42.9
34.7
pnt
37.6 33.6
8.9
51.8
44.0
35.2
qdr
12.2 4.8
4.8
7.6
14.2
8.7
qdr
12,0 4.1
4.8
7.4
13.6
8.4
rel
51.9 47.2
54.2
16.2
52.9
44.5
rel
52.6 48.1
54.8
16.9
53.0
45.1
skt
41.8 26.1
38.9
17.5
39.6
32.8
skt
41.9 27.1
39.9
17.9
39.5
33.3
Avg
32.4 28.9
31.1
13.8
34.3
36.1
29.4
Avg
32.5 29.3
31.4
14.2
34.3
36.4
29.7
22


Reviews:
[{"review_id": "jQFUMTrxfvD", "reviewer": "Reviewer_Z9Q2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The  paper propose a new concept, which is anytime domain adaptation.  \n\nUnlike the normal domain adaptation setting, the authors aim to make a generalised version of DA, which is tested under both at the level of input (resolution) and network (width, depth).\n\nTo achieve it,  the main idea of the paper is training two networks as teacher and student with switchable depth, width and input resolutions to enable testing under a wide range of computation budgets.\n\nExtensive experiments are conducted on 4  benchmark datasets, and the most related work \"slim DA\" is also included for testing.", "summary_of_the_review": "So my concerns are:\n\n- What is the main contribution of this paper? What is the main difference between anytime network + da\n\n- Why choose to match average performance? why not apply it to other subset networks?\n\n- What about different architecture pairs.", "strength_and_weaknesses": "Firstly, I like the new concept proposed by this paper,  which considers domain alignment in addition to varying both network (width and depth) and input (resolution) scales to enable testing under a wide range of computation budgets. Such concept is quite pratical, since we need to apply different scales network on different terminals. \n\nBut also, although I like this concept, we feel it maybe a little bit like the mixture of several concepts. Just as mentioned in the paper, anytime network plus domain adaptation. What about first do the normal domain adaptation and then do the anytime inference? \n\nAnd as for the method part, it is said that \"the student minnet is trained to match its output with the average prediction of the teacher subnets, while the student subnets are trained to match their outputs with the prediction of the teacher supernet.\" why make this setting, I guess that it mainly for minimize the performance gap, but I don't think it would make much difference. And if it works, why don't make the same setting for the other subnet like the second small sub network?\n\nAnd in this setting, I see that the authors set the teacher and the student network the same architecture. It is not flexible enough I think.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RQozMS2tkB0", "reviewer": "Reviewer_Qxxc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an domain adaptation method, adapted to different computational budgets when applying in the target domain. Specially, it considers the network depth, width and input resolutions in different budgets, and aims to learn a network that can be tailored to different network depth/width and input data sizes. The basic idea is based on the student-teacher framework, that distills the knowledge from larger capacity sub-networks to smaller capacity sub-networks. It also takes advantage of the switchable BNs and pseudo-label in the proposed method. The experiments show that the proposed method achieves higher performance than the DDA method, and marginally better than SlimDA. ", "summary_of_the_review": "The paper tackles a realistic problem in domain adaptation, and proposed an effective method for the anytime domain adaptation task. My major questions are on the more comparisons/evaluations, and clarifications on the extensible to out-of-range budget and larger backbone network. ", "strength_and_weaknesses": "1. Strength\n\n(1) The task to tackle is a realistic setting that needs to adapt to different computational budgets in domain adaptation. \n\n(2) The paper proposed a progressive student-teacher model to learn the sub-nets in different budgets. The proposed training loss is reasonable and demonstrates its effectiveness in DA benchmark datasets.\n\n2. Weakness\n\n(1) The basic idea of this work is simple and reasonable, however, combines the existing ideas of switchable sub-nets, BN, and the pseudo-label approach in domain adaptation.  \n\n(2) In Fig.3, the proposed approach is compared with the other methods, e.g., DDA, MSDNet. The curves show better performance when the budget is lower. Comparisons for larger computational budget should be also shown on these datasets. \n\n(3) Since the training is based on distilling knowledge from larger sub-net to smaller sub-net. How does this training approach affect the higher capacity subnet. Will it may hurt the performance of the larger capacity subnet?\n\n(4) The budget is pre-set, limiting its application if taking budget, might be out of range of pre-set budgets. How to handle this case? How does the competitors handle these out-of-range cases?\n\n(5) How about the performance of extension to using larger backbones as teacher/student networks? \n\n(6) How to understand the 'anytime' in the proposed setup.  'Anytime' may not be accurate to express the idea of DA adapted to different budgets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ip88J8_xCb", "reviewer": "Reviewer_fqZv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to study a novel domain adaptation setting, i.e., anytime domain adaptation, by combining anytime prediction and domain adaptation. Specifically, two networks are trained as teacher and student with switchable depth, width, and input resolutions to enable testing under a wide range of computation budgets. A bootstrapped recursive distillation approach is designed to train the student subnets with the knowledge from the teacher network. In this way, the target features are brought close to the source and the learned knowledge can be better transferred to a smaller network for efficient inference. Experiments are conducted on four benchmark datasets to show the effectiveness of the proposed method.", "summary_of_the_review": "Novel and interesting adaptation setting, effective method, convincing experiments and superior results but insufficient analysis, generally good presentation but inconsistent reference format.", "strength_and_weaknesses": "Strengths:\n1. The studied topic, i.e. anytime domain adaptation, is a novel domain adaptation setting with practical applications. The idea is interesting and the motivation is clearly explained.\n2. The proposed method, i.e. teacher-student framework, with bootstrapped recursive distillation makes good sense and seems to be effective.\n3. The experiments are convincing. The results are much better than the compared baselines. The ablation studies are promising.\n4. The presentation is generally clear and fluent.\n\nWeaknesses:\n1. It needs more discussion on the limitations of directly combining existing domain adaptation and anytime prediction techniques.\n2. The visualization of domain adaptation results is insufficient, such as t-SNE. I am interested to see how the visualization looks like under different computation budgets.\n3. Some closely related reviews and surveys on domain adaptation are missing, such as \"A review of domain adaptation without target labels\", \"A Review of Single-Source Deep Unsupervised Visual Domain Adaptation\". The format of references is consisitent.\n4. The analysis on the reasons that the proposed method can perform better than the baselines is not well summarized.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1tVF21HY2F", "reviewer": "Reviewer_yBmg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Combining the interests to of two separate fields of inquiry the authors of this work tackle the problem of Anytime inference for Domain Adaptation through the use of a recursive knowledge distillation. Being able to perform inference over multiple computational budgets is a beneficial strategy for deployed machine learning approach and prior work has shown an undesirable sensitivity to domain shifts, this work proposes an useful extension which would allow models to perform anytime inference over shifted target domains.", "summary_of_the_review": "The paper provides an interest approach to tackling the underexplored problem of anytime domain adaptation. The improvements over anytime baselines are consistent though not substantial and general utility of this problem area are not immediately apparent. \nHowever, this work thorough and explores and interesting new direction, as we collectively work towards more reliable machine learning systems this work provides an interesting perspective.", "strength_and_weaknesses": "The paper tackles and underexplored problem area of anytime domain adaptation and presents an effective solution which substantively improves the performance of anytime networks over unsupervised target domains. \nThe paper is well written and easy to follow, with the exception of the sandwich rule, all components of the approach are intuitively motivated and empirically verified. \n\nThe main weaknesses of the paper is that the improvements over SlimDA appear to be marginal at best and its relationship with that work contemporary work is not well explained. Additional, the utility of anytime domain adaptation is not immediately evident as any system which employs it would necessarily need to expend substantial MACs in order to learn the adapted network for the target domain. \nIf the authors could directly address why the process of learning the adaptation network should not be considered in the need for efficient inference over a target dataset it would improve the quality of the work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster