Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Combining the interests to of two separate fields of inquiry the authors of this work tackle the problem of Anytime inference for Domain Adaptation through the use of a recursive knowledge distillation. Being able to perform inference over multiple computational budgets is a beneficial strategy for deployed machine learning approach and prior work has shown an undesirable sensitivity to domain shifts, this work proposes an useful extension which would allow models to perform anytime inference over shifted target domains.
strength_and_weaknesses: The paper tackles and underexplored problem area of anytime domain adaptation and presents an effective solution which substantively improves the performance of anytime networks over unsupervised target domains. 
The paper is well written and easy to follow, with the exception of the sandwich rule, all components of the approach are intuitively motivated and empirically verified. 

The main weaknesses of the paper is that the improvements over SlimDA appear to be marginal at best and its relationship with that work contemporary work is not well explained. Additional, the utility of anytime domain adaptation is not immediately evident as any system which employs it would necessarily need to expend substantial MACs in order to learn the adapted network for the target domain. 
If the authors could directly address why the process of learning the adaptation network should not be considered in the need for efficient inference over a target dataset it would improve the quality of the work.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and provides a novel solution to infrequently studied problem area.
While there are questions of how sensitive the presented results are towards hyperparameters of the method, reproducing the algorithm itself should be straightforward from the presented work.
summary_of_the_review: The paper provides an interest approach to tackling the underexplored problem of anytime domain adaptation. The improvements over anytime baselines are consistent though not substantial and general utility of this problem area are not immediately apparent. 
However, this work thorough and explores and interesting new direction, as we collectively work towards more reliable machine learning systems this work provides an interesting perspective.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to study a novel domain adaptation setting, i.e., anytime domain adaptation, by combining anytime prediction and domain adaptation. Specifically, two networks are trained as teacher and student with switchable depth, width, and input resolutions to enable testing under a wide range of computation budgets. A bootstrapped recursive distillation approach is designed to train the student subnets with the knowledge from the teacher network. In this way, the target features are brought close to the source and the learned knowledge can be better transferred to a smaller network for efficient inference. Experiments are conducted on four benchmark datasets to show the effectiveness of the proposed method.
strength_and_weaknesses: Strengths:
1. The studied topic, i.e. anytime domain adaptation, is a novel domain adaptation setting with practical applications. The idea is interesting and the motivation is clearly explained.
2. The proposed method, i.e. teacher-student framework, with bootstrapped recursive distillation makes good sense and seems to be effective.
3. The experiments are convincing. The results are much better than the compared baselines. The ablation studies are promising.
4. The presentation is generally clear and fluent.

Weaknesses:
1. It needs more discussion on the limitations of directly combining existing domain adaptation and anytime prediction techniques.
2. The visualization of domain adaptation results is insufficient, such as t-SNE. I am interested to see how the visualization looks like under different computation budgets.
3. Some closely related reviews and surveys on domain adaptation are missing, such as "A review of domain adaptation without target labels", "A Review of Single-Source Deep Unsupervised Visual Domain Adaptation". The format of references is consisitent.
4. The analysis on the reasons that the proposed method can perform better than the baselines is not well summarized.

clarity,_quality,_novelty_and_reproducibility: This paper studies a novel domain adaption by combining with anytime prediction. The proposed method is novel and effective. The new problem and method indicate high quality and novelty of this paper. The proposed method is clearly explained and implementation details can ensure reproducibility.
summary_of_the_review: Novel and interesting adaptation setting, effective method, convincing experiments and superior results but insufficient analysis, generally good presentation but inconsistent reference format.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an domain adaptation method, adapted to different computational budgets when applying in the target domain. Specially, it considers the network depth, width and input resolutions in different budgets, and aims to learn a network that can be tailored to different network depth/width and input data sizes. The basic idea is based on the student-teacher framework, that distills the knowledge from larger capacity sub-networks to smaller capacity sub-networks. It also takes advantage of the switchable BNs and pseudo-label in the proposed method. The experiments show that the proposed method achieves higher performance than the DDA method, and marginally better than SlimDA. 
strength_and_weaknesses: 1. Strength

(1) The task to tackle is a realistic setting that needs to adapt to different computational budgets in domain adaptation. 

(2) The paper proposed a progressive student-teacher model to learn the sub-nets in different budgets. The proposed training loss is reasonable and demonstrates its effectiveness in DA benchmark datasets.

2. Weakness

(1) The basic idea of this work is simple and reasonable, however, combines the existing ideas of switchable sub-nets, BN, and the pseudo-label approach in domain adaptation.  

(2) In Fig.3, the proposed approach is compared with the other methods, e.g., DDA, MSDNet. The curves show better performance when the budget is lower. Comparisons for larger computational budget should be also shown on these datasets. 

(3) Since the training is based on distilling knowledge from larger sub-net to smaller sub-net. How does this training approach affect the higher capacity subnet. Will it may hurt the performance of the larger capacity subnet?

(4) The budget is pre-set, limiting its application if taking budget, might be out of range of pre-set budgets. How to handle this case? How does the competitors handle these out-of-range cases?

(5) How about the performance of extension to using larger backbones as teacher/student networks? 

(6) How to understand the 'anytime' in the proposed setup.  'Anytime' may not be accurate to express the idea of DA adapted to different budgets.
clarity,_quality,_novelty_and_reproducibility: The writing is mostly clear and the authors are suggested to public the codes for reproducibility.
summary_of_the_review: The paper tackles a realistic problem in domain adaptation, and proposed an effective method for the anytime domain adaptation task. My major questions are on the more comparisons/evaluations, and clarifications on the extensible to out-of-range budget and larger backbone network.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The  paper propose a new concept, which is anytime domain adaptation.  

Unlike the normal domain adaptation setting, the authors aim to make a generalised version of DA, which is tested under both at the level of input (resolution) and network (width, depth).

To achieve it,  the main idea of the paper is training two networks as teacher and student with switchable depth, width and input resolutions to enable testing under a wide range of computation budgets.

Extensive experiments are conducted on 4  benchmark datasets, and the most related work "slim DA" is also included for testing.
strength_and_weaknesses: Firstly, I like the new concept proposed by this paper,  which considers domain alignment in addition to varying both network (width and depth) and input (resolution) scales to enable testing under a wide range of computation budgets. Such concept is quite pratical, since we need to apply different scales network on different terminals. 

But also, although I like this concept, we feel it maybe a little bit like the mixture of several concepts. Just as mentioned in the paper, anytime network plus domain adaptation. What about first do the normal domain adaptation and then do the anytime inference? 

And as for the method part, it is said that "the student minnet is trained to match its output with the average prediction of the teacher subnets, while the student subnets are trained to match their outputs with the prediction of the teacher supernet." why make this setting, I guess that it mainly for minimize the performance gap, but I don't think it would make much difference. And if it works, why don't make the same setting for the other subnet like the second small sub network?

And in this setting, I see that the authors set the teacher and the student network the same architecture. It is not flexible enough I think.
clarity,_quality,_novelty_and_reproducibility:  quality of the paper is good.

 clarity of the paper is good.

and originality of the paper is good.

summary_of_the_review: So my concerns are:

- What is the main contribution of this paper? What is the main difference between anytime network + da

- Why choose to match average performance? why not apply it to other subset networks?

- What about different architecture pairs.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies domain adaptation in resource-efficient application scenarios, with switchable depth, width and input resolutions on the fly to enable testing under a wide range of computation budgets. The approach, coined by Anytime Domain Adaptation (AnyDA), is a framework of bootstrapped recursive knowledge distillation that bridges the teacher network to the student network with switchable subnetworks. Empirical studies have been conducted on standard benchmarks, showing that AnyDA outperforms previous approaches such as REDA and DDA but performs on par with the concurrent approach SlimDA.

Four referees and the AC reviewed this paper with mixed initial ratings. After rebuttal and discussion, reviewers acknowledged that their concerns were addressed reasonably well, and the overall scores converged to be unanimous positive. AC considered the paper, the revision, the reviews, the rebuttal, and the discussion, and concurred with the reviewers.

However, AC has a personal suggestion regarding this particular subarea of domain adaptation: It is highly discouraged to claim new problem settings, such as Dynamic Domain Adaptation, Anytime Domain Adaptation, etc, given that these settings are essentially the same under the umbrella of "Resource-Efficient Domain Adaptation" (Jiang et al. 2020). It would be better to only initiate new settings unless they are essentially and substantially dissimilar. A better way is to claim a new approach towards the resource-efficient domain adaptation setting.
justification_for_why_not_higher_score: I agree with the four reviewers that this paper is above the bar of ICLR. However, as pointed out the reviewers, the problem setting of this paper is within the scope of resource-efficient domain adaptation, which was studied by several pieces of previous work in 2020-2022. Thus the problem setting is not firstly initiated in this work as implied by the AnyDA title (kind of hyperbole). The technical part can be seen as a nontrivial integration of existing methods but with some crucial designs beneficial for empirical improvements. Note that, the current paper performs on par with a concurrent work SlimDA (CVPR 2022).
justification_for_why_not_lower_score: All reviewers stand unanimously on the positive side with reasonable reviews. The paper is above the ICLR bar, free of technical flaws. AC has no grounds to override the decision.
Decision: Accept: poster