Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Source-free domain adaptation (SFDA) methods usually use the pseudo-labels generated by source models which can be very noisy due to domain shifts. This paper studied and proposed an early learning regularization (ELR) based (Liu 2020) SFDA method to tackle the problem of learning with label noise (LLN). They show that pseudo-label noise generated by source models in SFDA is unbounded label noise. They theoretically and empirically proved that early-time training phenomenon exists in the unbounded label noise. Extensive expreiments were conducted to show that the proposed ELR based method can improve existing SFDA algorithms by around 1~2% on multiple DA benchmark datasets.      
strength_and_weaknesses: Strength:

This work theoretically and empirically justifies that the early-time training phenomenon exists in the unbounded label noise scenario, and proposed an extended ELR scheme for SFDA methods. The work has enough novelty for a practical problem. The paper provides reasonable experimental results and fair discussions. And the paper basically is well-written and easy to follow. 

Weakness:

The paper lacks explanations and details in some places for readers who are not familiar with the topic. The paper should be more self-contained. According to the experiment results, the improvement by the proposed technique is marginal 1~2%.  

In EQ1, should $d_1$ and $d_2$ be $d_1 = \frac{\mu_2 - \mu_1}{2} - C$ and $d_2 = \frac{\mu_2 - \mu_1}{2} + C$  ?
Please add a brief explanation of $l_{LLN} $ in Lemma 3.2 in the paper. 




  
clarity,_quality,_novelty_and_reproducibility: The work studied the LLN problem in SFDA, and showed that LLN is unbounded label noise, which had not been discussed before in previous literature. It theoretically and empirically proved that the early-time training phenomenon exists in the unbounded label noise scenario. The work actually is an extension of  ELR method (by Liu 2020)  on unbounded label noise. Their proposed early learning regularization (ELR) loss can be utilized on not only SFDA but also any LLN with unbounded noise. 
summary_of_the_review: In summary, the work theoretically and empirically studied the LLN in SFDA and the $\textit{early-time training phenomenon}$ in unbounded LLN which hadn't been studied before. It proposed a ELR based solution for the unbounded LLN.  The work has enough novelty for the practical problem of training with unbounded label noise. The paper is basically well-written with fair experimental results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors have proved that the “label noise” in source-free domain adaptation is unbounded. Therefore, existing LLN methods that rely on their distribution assumptions are unable to address the “label noise” in source-free domain adaptation (SFDA). The authors have also demonstrated theoretically that the early-time training phenomenon can also be observed in the SFDA problem.
strength_and_weaknesses: Strength:
+ The theoretical analysis that the label noise is unbound when there is a domain shift is novel to me.
+ The authors have demonstrated theoretically the early-time training phenomenon under certain conditions. Although the assumption can be strong, it is still interesting to see such an analysis.
+ This paper is well-motivated. It is important to understand the difference between two different but similar problem settings. It can be helpful for the designing of new algorithms for source-free domain adaptation.

Weakness:
+ The assumption for Theorem 4.1 that at most half of the samples are mislabelled could be a little bit strong. The proposed method based on the early-time training phenomenon may not be that practical.  

clarity,_quality,_novelty_and_reproducibility: This paper is novel to me. Although the designed method may not that practical, the theoretical analysis is important and can help design new algorithms for source-free domain adaptation.
summary_of_the_review: The major contribution of this paper is the theoretical analysis. The practical novelty may not be that significant. I would like to weakly accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of source-free domain adaptation (SFDA), where a pre-trained source model is adapted using unlabeled target domain data without accessing any source domain data. While previous works mainly focused on cluster assumption in the feature space, the authors propose a different perspective. They formulate the SFDA problem as learning with label noise (LLN). In addition, since label noise in SFDA is unbounded unlike the noise in conventional LLN scenarios, they leverage the early-time training phenomenon (ETP) with a simple early learning regularization (ELR) term. The authors demonstrate that it consistently improves existing SFDA algorithms by a large margin on four different SFDA benchmark datasets.
strength_and_weaknesses: Strengths
- I think the idea of formulating the SFDA problem as LLN is interesting. The authors also theoretically showed why the existing LLN methods cannot address the noise in SFDA due to the fundamental differences.
- The theoretical and empirical justification of ETP in the SFDA problem with the unbounded label noise seems to provide solid grounds for the proposed method.
- Leveraging ETP with ELR term seems simple but effective across a wide range of SFDA algorithms and benchmark datasets.

Weaknesses
- While the previous SFDA methods do not explicitly formulate the SFDA problem as LLN, I think they are still partially based on the LLN methods. As the authors explained in the related work section, the SFDA methods leverage the target “noisy” pseudo labels and also some type of pseudo-label purification processes as in the LLN methods. It would be great if the authors can provide more clear explanations of the similarities and differences between their motivations. 
- The proposed early learning regularization (ELR) term encourages the model prediction to stick to the early-time predictions for each data point. Meanwhile, NRC contains a loss term, called self-regularization loss, which has a similar purpose of reducing the potential impact of noisy pseudo-labels and not ignoring the current prediction. Please provide comparative explanations between the two and how the ELR still improves the NRC (Table1). 
- In Empirical Observations on Real-World Dataset, it seems the authors trained classifiers using the fixed noisy labels with cross-entropy loss. However, SFDA algorithms (e.g. NRC) gradually update pseudo-labels rather than using the fixed pseudo-labels. Please provide clarifications. It would be also nice if you can also show the training accuracy of SFDA algorithms in Figure 2. 
- How did the authors get the baseline results? There seem to be some discrepancies compared to the results reported in their original papers. For example, NRC results seem lower than those reported in the original paper. The reported average results are 72.2 for Office-Home and 85.9 for VisDA-C. Then, the performance difference between NRC+ELR in Table1 (72.6) and the NRC in the original paper (72.2) seems marginal.
- Please provide a more detailed protocol for the experiments. For example, assuming that no labeled target data is available, how do you decide (1) when to stop the training and (2) hyperparameters for each dataset? If the authors hand-picked the hyperparameters for each dataset, it might have caused over-optimistic results.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper was generally clear and easy to follow.
More detailed explanations of the experiment protocol would be helpful.

Quality: I think the paper holds good quality. It presents interesting motivations and demonstrates both theoretical and empirical justifications for the proposed method.

Novelty: The originality of the work seems a bit limited. While the idea of levering ETP for SFDA seems novel, they used the recently established ELR. In addition, it shares some similarities with the self-regularization term previously used in the NRC.

Reproducibility: The paper does not provide enough details to reproduce their experiment results. 
I assume their attached codes may resolve the reproducibility issue.

summary_of_the_review: While the paper has its merits, unfortunately, it also has several issues which need to be addressed: (1) clarification of the novelty of the proposed method, (2) comparative explanations between the proposed method and self-regularization loss in NRC, and (3) detailed experiment protocols.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers source-free domain adaptation, where a learner has access to only a source model and unlabeled target data due to privacy concerns regarding source data. Existing practices in source-free domain adaptation is to use the source model to pseudo-label target data, which may cause mislabeling occasionally, and moreover, the mislabeling probability is much higher than the noise probability that is usually assumed in learning from label noises. Indeed, the authors show a theoretical statement that the mislabeling probability can be arbitrarily large and approaching 1 under a simple setting. Then, the authors reveal that ETP (early-time training phenomenon), which observes that a learner with fewer training epochs can avoid overfitting and memorizing the mislabeled data perfectly, can be observed in the unbounded noise case as well. By leveraging this fact, the authors propose to use early learning regularization (ELR), which uses the moving average of the label prediction over training epochs to mitigate the aforementioned overfitting issue, with existing source-free domain adaptation methods. As a result, the existing methods can be improved with ELR.
strength_and_weaknesses: ### Strengths

- **Nice theory to demonstrate the issue of unbounded noises.** Although the setup is limited to a simple Gaussian case, the authors provide a nice theoretical illustration of how the labeling error of the pseudo-labeling in source-free domain adaptation can be arbitrarily large.
- **Simple remedy to existing source-free domain adaptation.** The proposed remedy to the unbounded noise is to use the moving average of the label prediction over training epochs, which is practically very simple to use. So far, this method has been used in the context of learning from label noises, but its applicability and effectiveness in source-free domain adaptation has not been witnessed. The authors contribute in this line to show ETP can be observed in the unbounded noise case as well.

### Weaknesses

Basically, I do not see any weaknesses of this paper except for small unclear points. I point out them in the subsequent "Clarity" section.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-written and easy to follow. Below, I have a few questions regarding unclear points that I encountered.

- Regarding Eq. (1), having a plot of the graph of the mislabeling rate as a function of $\\Delta$ may help readers to understand that the mislabeling rate increases as the magnitude of domain shift increases. It should be fine to have the plot in the supplementary material.
- In Theorem 3.1, it is not clear why we can assume $\\Delta$ is positively correlated with the vector $\\mu_2 - \\mu_1$ without loss of generality. (In any cases, I think we may "lose generality" because the aim of this theorem is to show an example where the mislabeling rate is unbounded.)
- Lemma 3.2 and the following paragraph do not explain $\\ell_\\mathrm{LLN}$ in its full detail so that the claim of the lemma is not clear enough to the readers not familiar with the existing work on LLN. It looks essential to introduce the background concept that a line of researches on LLN uses noise-robust loss functions instead of the usual classification loss to learn a noise robust classifier.
- In Figure 2, why do both green and blue curves suddenly change the tendency at epoch 90? It is explained "After 90 steps, we evaluate the prediction accuracy for every 0.33 epoch," but I do not see this affect that much.
- In Figure 2, I would like to see if well-known noise-robust loss functions such as GCE (Zhang & Sabuncu, 2018) and so on suffer from the memorizing issue as well. Adding a few more loss functions to Figure 2 would be comprehensive to argue the importance of considering the unbounded noise rate.
- In the experimental results, I do not see why the performance of UDA methods are uniformly worse than source-free domain adaptation (e.g. in Table 1). My intuition tells that the problem setting of UDA provides a learner more information about source data and hence she could leverage the richer information to improve the performance over source-free domain adaptation. Perhaps it gets off the topic here, but I appreciate if the authors can comment on this issue.
summary_of_the_review: The paper contributes to source-free domain adaptation from a new viewpoint, learning with label noises, which should be acknowledged. The overall quality of the manuscript is above the border. A few points I mentioned in "Clarity" section can be addressed, which I believe makes the paper much better.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission looks into source-free domain adaptation (SFDA) from the perspective of learning with label noise (LLN). It shows that the label noise distribution in source-free domain adaptation is different from that of the usual LLN setting, making LLN approaches unsuitable for SFDA. However, it demonstrates that the early-time training phenomenon (ETP) observed in LLN is also present in SFDA, and that using ETP to address label noise in SFDA yields significant improvements to existing SFDA approaches on four benchmarks (Office-31, Office-Home, VisDA, and DomainNet).

Reviewers find the submission clear (RMmn, 3S6g, nSg6), and the questions they raised in that regard were appropriately addressed by the authors in the rebuttal (Reviewer 3S6g on the connection between SFDA and LNN and the description of the baselines and experimental protocol; Reviewer nSg6 on theoretical and empirical results). Reviewers also note the submission's convincing theoretical and empirical contributions. Opinions were originally split on novelty: Reviewers RMmn found the paper novel enough and the theoretical analysis was new to Reviewer CRh4; Reviewer 3S6g was concerned that although leveraging ETP for SFDA is new, the mechanism for doing so is borrowed from existing work (ELR) and shares similarities with NRC. The authors' explanations addressed Reviewer 3S6g's concerns.

All reviewers now agree that the paper meets the bar for acceptance; I therefore recommend acceptance.
justification_for_why_not_higher_score: While of interest to the community, the submission does not have the kind of reviewer support to justify an oral presentation.
justification_for_why_not_lower_score: While reviewers unanimously assigned a score of 6 to the submission, I don't see any major unaddressed reviewer concern, and the paper makes enough new contributions to be worth highlighting in a spotlight.
Decision: Accept: notable-top-25%