Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a direct target adaptation from the source trained model (without the source data).  It applied self-supervised and distillation methods to learn from unlabeled target data.
main_review: Strength:  The paper discusses a direct target adaptation to improve the performance of target data ( which is the partial objective of the traditional domain adaptation method). The paper shows a different variation of each component ( initialization, self-supervised, etc.). The paper is very easy to follow.

Weakness: The paper needs to improve significantly in terms of proper evaluations ( fair evaluations). Some of the points are listed here: 
 The paper's novelty is limited; it uses most of the well-studied techniques such as Infomax-based entropy loss, self-supervised methods, and distillation. The paper seems it is only combinations of those methods without proper justification ( motivation) why should these particular techniques are suitable for the current problem. Although they showed it only based on empirical performance, it should not be the motivation or intuition to use particular techniques.
In stage 1, the parameters of the feature extractor are updated, not the classifier; I wonder what would be effective if we update the classifier too. This will help the feature to be class discriminative for the target domain rather than the source domain.
Regarding comparing state-of-the-art methods, the performance of domain adaption methods on the source domain does not degrade while adapting the target domain. In this case, we concern only with target performance, so I suspect the model will perform poorly on the source domain. So the comparison in terms of the target domain is not fair enough.
In Table 2, the performance reported on TENT ( baseline)  with network  R50S but the proposed model trained on R18S; what is the reason to choose different base networks. For the fair comparing the base network must be the same for all the experiments. The same thing for the SHOT baseline.
In Table 5, again, the base network is different from the baseline or other state-of-art methods. In fact, authors should include the source-only models' performance in each table, which is the super baseline for all the source-free or ( not source-free) domain adaption methods.
Overall due to the missing proper fair comparison, the results do not show the effectiveness of the proposed methods. 
Thus I would like to see the results with fair base network.


summary_of_the_review: The paper needs a proper evaluation on the fair baseline. The weaknesses are discussed above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a multi-stage approach for source-free domain adaptation, namely when source data is not available and one can only use the model pre-trained on the source and adapt it based on unlabelled target domain data. The paper advocates for learning on target not via fine-tuning the source model, but rather distill from it initializing on source using contrastive learning (the authors use Moco V2). The source model is also adapted on target via InfoMax loss before being used as the teacher for the distillation. The authors propose to use the FixMatch strategy (with strong and weak aug taken from AutoAugment) for the distillation. The authors show some gains when combining with SOTA source-free adaptation methods: TENT and SHOT. The authors also show that improved accuracy can be obtained with much smaller models trained on target using their method (e.g. replacing Res50 or Res101 with Res18 on Visda-C. Extensive experiments, ablations, and results are provided.
main_review: Pros:
* I like the source-free adaptation problem setting, I think it is very practical
* Combining model fine-tuning on target (or invoking a SOTA source free method) generating a teacher for distillation and using contrastive self-supervised student is a natural idea to try and I think the community should be aware of the good results
* SOTA improvements in some cases are welcomed
* extensive experiments and ablations
* code included

Cons:
* novelty seems a bit limited, the paper combines some standard ideas (target adaptation, distillation, contrastive pre-train on target, fix-match), the main novelty seems to be in the system - combining these for source free adaptation
* writing could be improved, sometimes it is hard to follow the mass of details and some conclusions become less clear
* why in Table 1 some numbers are missing and X is drawn? did not find an explanation in the text
* in Table 2 "Ours" is only shown on top of TENT or SHOT, how about without source free sota methods at play? is it in table 3? but there the numbers are a bit low and disappointing...
* Tables 4 & 5 only do TENT / TENT + ours, I looked briefly in SHOT and saw higher numbers for Table 5 as it seems, though there the proposed method did not hit sota anyway, how about adding SHOT also to table 4 (and 5)?
* Presentation (more minor): ablation tables are better re-designed, e.g. look at table 7 the message there was to show that there is improvement in every row reading from left to right, but the V and the X leads the reader to wonder if adding more things improves or not in columns and it is actually opposite in that table (but was so in earlier tables). In short - please make the style consistent, it is really hard to follow. This was just an example.
summary_of_the_review: I see a balance between pros and cons in this paper, yet I do have some positive tendency as the method itself is clear and intuitive to me and can boost performance in some interesting cases (not always, but still). Therefore, at least until I see some more clarifications from the authors and other reviews, I prefer to rank it slightly above borderline, with an intention to revisit and yet in a positive spirit of leaning towards acceptance :-)
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work addresses the problem of source-free domain adaptation. Instead of fine-tuning *the source model* on the target data, this work proposes to fine-tune *a representation learned on the target data alone* using e.g. self-supervised contrastive learning. This permits distinct target architectures that can improve performance and/or reduce memory and computational cost.
main_review: **Strengths:** Interesting and intuitive idea that (i) is well motivated; and (ii) seems to perform well on a variety of datasets.

**Weaknesses:**
  - **Missing critical details:**
    - *Loss function:* It is not clear what is actually being done in stage 3 (teacher-student) due to the lack of an equation for the final loss function, e.g. soft vs. hard losses, "consistency regularization" vs. pseudo-labelling, etc. This should be added to Section 3 to better describe the method.
    - *Algorithm:* It is not clear what is actually being done in stage 3 (teacher-student) due to the lack of an algorithm, e.g. the replacing student and teacher models at certain phases, the swapping between soft and hard losses and certain phases, the re-initializing of student features at certain phases (to correct for incorrect pseudo-labels), etc. This should be added to Section 3 (or Suppl. Material, and pointed to) to better describe the method.
  - **Lack of uniformity in the evaluation (cherry picking?)**:
    - *Table 1:* Why are the 4 entries in the bottom right corner not filled in?
    - *Table 3:* Why does the right column with SHOT not contain the same rows as the other two columns, namely R50S and R50P? Why has R50S been replaced with R101P? Do we not want to see the performance of SHOT with R50S?
    - *Table 4*: Why are SHOT and SHOT + ours omitted from this table?
    - *Table 5*: Why are SHOT and SHOT + ours omitted from this table?
    - *Table 7:* Why have different networks (“network”), pre-training strategies (“imagenet pretrain”) and adaptation strategies (“source only”) been used for SHOT (left) and TENT (right)? Are these the combinations where your method achieved the biggest improvement?
  - **Some prior/related work is not properly cited**:
    - *Not transferring source parameters:* Prior work has already *not* transferred source parameters in the setting of *black-box* source-free domain adaptation [2,3,4]. In fact, Fig. 1c exactly depicts this setting of black-box source-free domain adaptation. This should be discussed.
    - *Factorizing the representation and classifier:* SHOT[1] does exactly this, training the representation while holding fixed the classifier. I think you should cite them in Section 3 (last paragraph of page 3) and perhaps the introduction. Otherwise it (wrongly) comes across as a contribution of this paper.
    - *L_{ent} and L_{div}:* the use of both L_{ent} and L_{div} losses was introduced by SHOT[1] (i.e. their SHOT-IM loss), and should be cited accordingly in Section 3, Stage 1 (teacher).
    - *Using SHOT-IM loss with TENT (Section 4.2, Stage 1):* Since you replace the loss of TENT (L_{ent}) with that of SHOT-IM (L_{ent} and L_{div}), I don’t think you can refer to this method as TENT anymore because it is no longer TENT. Rather, it is a mix between TENT and SHOT-IM and should be labelled as such, e.g. TENT-IM.

**Further comments**:
 - **"Test-time":** I find the use of "test-time" a bit misleading for the *adaptation phase* of source-free domain adaptation. One is not really "testing" at that time, but rather adapting a model to the target data for many epochs (could be several days). Instead, it could simply be called adaptation, target adaptation, or on-target adaptation! The addition of “offline” and “online” test-time approaches further added to my confusion. 
 - **"Label smoothing *could* be leveraged":** Was label smoothing used or not? (Section 4.2, Stage 0).


[1] Liang, J., Hu, D., & Feng, J. (2020). Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In *International Conference on Machine Learning* (pp. 6028-6039).

[2] Liang, J., Hu, D., He, R., & Feng, J. (2021). Distill and Fine-tune: Effective Adaptation from a Black-box Source Model. *arXiv preprint arXiv:2104.01539*.

[3] Zhang, H., Zhang, Y., Jia, K., & Zhang, L. (2021). Unsupervised Domain Adaptation of Black-Box Source Models. *arXiv preprint arXiv:2101.02839*

[4] Wu, K., Shi, Y., Han, Y., Shao, Y., & Li, B. (2021). Black-box Probe for Unsupervised Domain Adaptation without Model Transferring. *arXiv preprint arXiv:2107.10174*.

summary_of_the_review: Overall, this seems to be a good paper. A well-motivated and intuitive idea is proposed, and it appears to work well across a variety of datasets. However, critical details are omitted (e.g. what is the complete loss function and algorithm?), there is a lack of uniformity in the evaluation (raising suspicions of cherry picking), and related works are not cited satisfactorily (e.g. black-box source-free domain adaptation). As a result, I believe this paper is only marginally above the acceptance threshold in its current form. If the aforementioned concerns can be addressed, I will improve my score.
review_title: null

Meta-review: The work proposed an interesting source free adaptation setting, where one is asked to adapt a pre-trained source model to a target domain without accessing data from the source domain. While reviewers find the setup interesting and the initial results encouraging, they expressed concerns on the limited novelty of the work as well as incomplete evaluation. Multiple reviewers (reviewer Lx65  and pNRq) raised concerns on the fairness of the evaluation, which was not fully addressed by the authors during rebuttal. Please consider addressing these comments in your draft.
Decision: Reject