Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a framework called DINO for domain adaptation regression, which is based on distribution-informed neural networks. This framework can be theoretically related to a) domain invariant learning, b) source data reweighting, and c) adpative gaussian processes.  Authors show that convergence of such networks can be guaranteed. Also, the framework uses the training dynamics of the network when measuring the distribution shift between domains. Experiments were conducted on dSprites, MPI3d, and Plant Phenotyping - two of which are image datasets. Overall, both DINO-INIT and DINO-TRAIN show strong performances in all three datasets.
strengths_and_weaknesses: **Strengths**
* DINO presents a unified framework that is well connected with a few previous works. In other words, the authors show that methods proposed in other works may be interpreted as a special instantiation of the DINO framework proposed in this paper.
* The authors theoretically verify that the proposed framework is able to converge.
* Overall, the paper is well written.

**Weaknesses**
* not clear what the difference between DINO-INIT and DINO-TRAIN is. First mention of DINO is in the intro, while the second mention is in Section 5. Explanation of DINO-INIT and DINO-TRAIN are in the Appendix (A.10), but should really be part of the main paper (or at least, it should be mentioned that DINO-INIT/TRAIN are explained in the Appendix). 
* Why is it that DINO-INIT outperforms DINO-TRAIN in both dSprites and MPI3D? More insights could be provided for this.
* I think the paper could be improved by reducing the amount of theoretical analysis in the main paper (could be moved to the Appendix), as I feel that it takes away from the main points of the paper. 
* It is not clear how DINO would be generalized to more practical datasets/network architectures. Overall, experiments that demonstrate practicality would make the submission much stronger.

**Other**
* L258 Typo: (8) can **not** only unify ...
* Some equations leak into the margins. For example, equation after Lines 229, 285, and 291. 

questions: * In this paper, the authors use an L-layer fully-connected network for all experiments. Does the theoretical analysis and method presented in this paper naturally extend to diverse architectures, such as CNNs and/or transformers?
* It seems like one underlying assumption is that the neural network should be over-parametrized. But can we say that a 6-layer fully-connected network is over-parametrized for dSprites or MPI3D, both of which are image datasets?
limitations: Limitations are mentioned in the Appendix. Authors note that the framework is based on single-source domain adaptation.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a domain adaptation (DA) regression framework based on the distribution-informed neural network, which connects to previous techniques, including domain invariant learning, reweighting, and adaptive Gaussian process. It also theoretically analyzes the convergence and generalization bound of the framework. The method is evaluated on three DA regression datasets and outperforms some state-of-the-art methods.
strengths_and_weaknesses: Strengths:
1. It is interesting and inspiring to explore the ideas of distribution-informed neural networks in the DA regression problem.
2. This paper provides theoretical analyses on the convergence and generalization of the proposed DA regression framework. Different from previous theories, distribution shifts are measured using the training dynamics of the neural network.
3. The proposed method outperforms some state-of-the-art DA regression methods on three benchmarks.

Weaknesses:
1. I think this paper is generally not clearly written and hard to follow, especially the most important sections 4.2 and 4.3. The titles of the subsections, such as ‘4.2 Initialization’, are confusing. I think it also needs more transitional sentences to lead or summarize the sections and to clarify the motivations.
2. The proposed framework is mainly evaluated on a shallow fully-connected neural network. It is unknown whether it can be extended to other architectures (such as CNN or Transformers), deeper models, and models already pre-trained from large datasets since these models may be more commonly used in practice.
3. The method is mainly evaluated on some toy datasets, and I think it needs some experiments on more realistic situations.
questions: Besides the points listed above, I have some questions:
1. Can you clarify why the method proposed in section 4.2 is called DINO-Init? How does it relate to section 4.3? In experiments, I find that DINO-Init and DINO-Train may outperform each other in different cases. Can you explain this? How could we choose between these two algorithms in practice?
2. Can you clarify what the ‘training dynamics’ mean in Line 251? Why is it called ‘training dynamics’? Why is it better than previous measurements?
3. In Line 238, why do we use all examples as the basis examples? Would this prevent the method from scaling to larger datasets? How do the different choices of basis example sets influence the method?
4. In practice, how should we choose hyper-parameters such as \alpha and \mu in Equation 7?
limitations: The authors addressed the limitation that the framework may not handle multi-source settings. I think another limitation is that it is unknown whether this paper can generalize to commonly-used deeper networks with diverse architectures.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a domain adaptation method through a learning algorithm that knows the distribution of source and target well. The proposed method performs regression task when the distribution of source and target are different. Their learning algorithm consider maximizing RKHS mean discrepancy. Various experiments show that the proposed method improve the performance of the previous methods.
strengths_and_weaknesses: Strengths 
+) Motivation and the description of the proposed algorithm is convincing.
+) Various experiments are performed, and the performance of the proposed method is better than previous regression DA methods.

Weaknesses
-) The paper representation needs to be improved. There are too many formulas in the main paper. Except for essential formulas, authors need to pass them all to the appendix and explain the concept and analysis in the main paper.
-) There are not enough SOTA methods in the experimental results. In particular, AT-GP was proposed in 2010 and the performance improvement of the proposed method is not remarkable.
questions: Q1) A lot of domain adaptation algorithms and models have been studied in the classification task. Why are these algorithms and models not applicable to regression task?
Why is the proposed method particularly suitable for regression problems?

Q2) In fact, domain adaptation for classification can be easily applied to regression problems. Can you do a performance comparison with them in regression task?

Q3) It is recommended to draw a workflow figure that shows everything from input to output at once. Can you draw a figure to understand the concept of the proposed method?
limitations: The authors explain the societal impact and limitation of the paper well.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose a distribution-informed neural network for domain adaptation regression, called DINO.  The proposed distribution-informed neural network can be explained by existing domain adaptative Gaussian process. The authors also analyze the convergence and generalization bound of DINO. Experiments are done to show the effectiveness of the proposed DINO framework.
strengths_and_weaknesses: Overall, the paper is well-written. The idea of using domain adaptative Gaussian process to explain the neural network feature extractor is interesting. The distribution shift measure using the training dynamics of the distribution-informed neural network is a novel and sound point. However, I also have the following comments as stated below.
questions: 1.The paper is specifically written for domain adaptation regression. However, to this reviewer, the technical contribution is not specific to regression. The authors claim that classification is naturally formulated as regression. Thus, the reviewer wonders how this method performs in those classification tasks (there are quite a lot SOTA UDA papers for classification, the authors may want to compare with some recent ones.).

2.To this reviewer, the related work section should be re-organized and further improved. Compared with discussing the big picture of domain adaptation, the authors should directly focus on the regression works on domain adaptation or transfer learning. The authors may want to add two more paragraphs, one discussing domain adaptation regression works and another one discussing the GP based adaptation setting, as these two research lines are closely related to the current paper.

More importantly, it is more convincing to discuss how those closely related works, e.g., [14] and [11], differ from the current paper, instead of just listing them out. 

3.To this reviewer, the setting studied in this paper is different from the settings of [14] and [11], where the former [14] only has unlabeled target data and the latter [11] only has limited labeled target data. This paper seems to be a mixed setting of the above two settings. In this sense, the reviewer has the following three concerns:

(1).  with both labeled and unlabeled target data, existing semi-supervised learning methods (here I give a reference example [ref1]. The authors may want to check the more recent SOTA methods). The authors should consider to show the superiority of the current paper over this kind of methods.
 
[ref1] Kostopoulos, Georgios, et al. "Semi-supervised regression: A recent review." Journal of Intelligent & Fuzzy Systems 35.2 (2018): 1483-1500.

(2). as the paper mixed the setting of [11] and [14], the reviewer expects the method should clearly outperforms [11] and [14] as it uses more target information. However, from Tabel 3, the methods is not as good as [11] at the task M -> Mu.

(3). How the scale of labelled target data affects the final performance? Furthermore, how the ratio of labelled and unlabeled target data matter?

4.The empirical evaluations are not very convincing. In Table 1 and 2, the DINO is only slightly better than those baselines. In table 3, DINO is not as good as [11]. Note that [11] is a 12-years ago paper, the authors should consider more recent adaptive GP methods for comparisons.



######################### After rebuttal ##################
I would like to thank the authors' detailed response, which well address my concerns. I am willing to increase my score accordingly. 


limitations: The limiatation is stated in Appendix, mainly for the possible future working direction. It is more interesting to discuss the limitations on the technical part of the method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to use a distribution-informed neural network for domain adaptation regression. The input-oriented feature representation is defined based on its distance to other examples in the same domain, thereby achieving distribution-conditioned representation of the input data. Theoretical analysis shows that the proposed distribution-informed neural network is equivalent to domain adaptive Gaussian Process. 
Empirical results on dSprites and MPI3D demonstrate that the proposed approach outperforms Gaussian Process, domain adaptive Gaussian Process and deep domain adaptation methods.
strengths_and_weaknesses: Strengths:

1. I think the proposal of distribution-informed neural network that is aware of the input-output training dynamic is novel. Although not mentioned in the paper, the proposed approach is somewhat related to self-attention where the model encodes each individual input based on its relations with other inputs. The illustration in Fig. 1 on how "local model evolution" relates to domain adaptation is insightful.
2. The proposed approach is theoretically grounded and the paper presents a unified framework for analyzing domain discrepancy that is connected to adaptive Gaussian process, sample re-weighting and domain-invariant representation learning.
3. The model outperforms other baselines in the empirical studies.


Weakness:
1. It is unclear what the "training dynamics" is referring to in the proposal. In the introduction, the dynamics are hinting at how the model changes during training, but the main methodological contribution seems to be the "distribution-informed neural network." If the domain divergence is defined on the training dynamics of a model, it would be susceptive to common problems when training a model, e.g., underfitting, overfitting, poor-calibration. 
2. Regarding the empirical results on MPI3D, it is counter-intuitive that the weakly supervised version "DINO-INIT" outperforms "DINO-TRAIN". The authors should clarify the difference between the two models and explain why additional training is detrimental.
questions: Q1-2: Please refer to my question in the weakness section.

Q3. In Eq. 5, it is not clear how $f_\theta(x)=\phi_{\theta<L}(x)^{T} w$ could represent $P(x)$. The probabilistic formation is not enforced in the training process.
limitations: No
review_title: null

Meta-review: metareview: This paper works towards domain adaptation in the regression setting through the distribution-informed neural networks. It provides a nice theoretical elaboration to the three mainstream methods in this field: domain invariant learning, source data re-weighting, and adaptive Gaussian processes. However, reviewers were on the borderline even after the author rebuttal and after the rating increase. AC carefully justified the merits and flaws of this paper and felt that the former slightly outweighs the latter, and this is in agreement with the SAC. The major concern is the use of fully-connected network for theoretical analysis and empirical evaluation --- the scalability of the theory to modern neural networks seems not easy while the paper does not provide sufficient evidence in this regard. However, considering this paper is more theory-oriented, such a concern is down-weighted a little bit. Thus the paper is recommended for acceptance. Authors are suggested to improve their paper by incorporating the rebuttal and further addressing the reviews, in particular improving the clarity and smoothness of the presentation to be more readable.
confidence: Less certain
Decision: Accept