Test-time Adaptation for Regression by Subspace Alignment

Authors: Kazuki Adachi, Shin'ya Yamaguchi, Atsutoshi Kumagai, Tomoki Hamagami
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Test-time adaptation, regression, distribution shift, deep learning

Abstract:
This paper investigates test-time adaptation (TTA) for regression, where a regression model pre-trained in a source domain is adapted to an unknown target distribution with unlabeled target data.
Although regression is one of the fundamental tasks in machine learning, most of the existing TTA methods have classification-specific designs, which assume that models output class-categorical predictions, whereas regression models typically output only single scalar values.
To enable TTA for regression, we adopt a feature alignment approach, which aligns the feature distributions between the source and target domains to mitigate the domain gap.
However, we found that naive feature alignment employed in existing TTA methods for classification is ineffective or even worse for regression because the features are distributed in a small subspace and many of the raw feature dimensions have little significance to the output.
For an effective feature alignment in TTA for regression, we propose Significant-subspace Alignment (SSA).
SSA consists of two components: subspace detection and dimension weighting.
Subspace detection finds the feature subspace that is representative and significant to the output.
Then, the feature alignment is performed in the subspace during TTA.
Meanwhile, dimension weighting raises the importance of the dimensions of the feature subspace that have greater significance to the output.
We experimentally show that SSA outperforms various baselines on real-world datasets.
The code is available at https://github.com/kzkadc/regression-tta.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730271588389

thread_type:
official_review

<thread_object>
creator: reviewer_gbnn

<content>
summary:
This paper investigates test-time adaptation (TTA) for regression, where a regression model pre-trained in a source domain is adapted to an unknown target distribution with unlabeled target data. To enable TTA for regression, the authors adopt a feature alignment approach, which aligns the feature distributions between the source and target domains to mitigate the domain gap. A propose Significant-subspace Alignment (SSA) is proposed for feature alignment in TTA for regression, which consists of two components: subspace detection and dimension weighting. Some experiments have been conducted to verify the performance of the proposed method.

strengths:
1.	A Significant-subspace Alignment (SSA) method is proposed to address TTA for regression on the basis of the feature alignment approach.
2.	SSA consists of two components: subspace detection and dimension weighting. 
3.	Subspace detection uses principal component analysis (PCA) to find a subspace and dimension weighting raises the importance of the subspace dimensions.

weaknesses:
1.	The description of Figure 1 is too simple to understand the procedure of the proposed method. Some key variables (i.e., gφ and hψ) should be introduced. For easy understand, I suggest the authors provide a step-by-step description of the workflow shown in the figure. 
2.	The used compared methods are all published before 2023, how the proposed method compares to or differs from some newer approaches, such as “Backpropagation-free Network for 3D Test-time Adaptation, CVPR 2024”, “Improved Self-Training for Test-Time Adaptation, CVPR, 2024”.
3.	Why no results for RSD are given in Table 3? Please provide the missing results for RSD in Table 3 or explain why these results were not included.
4.	What is the visualization effect of the proposed method? Provide T-SNE rendering.
5.	What is the limitation of the proposed method? Are there any situations where SSA might not perform well, or any assumptions it makes that might not hold in all regression tasks?

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730469714392

thread_type:
official_review

<thread_object>
creator: reviewer_xn4r

<content>
summary:
The paper proposes a test-time domain adaptation method for regression. They show that learned embeddings are more compressed in regression than classification. To address this, they introduce Significant-subspace Alignment (SSA). SSA uses PCA to detect a significant feature subspace where features are concentrated, then aligns the first two moments of source embeddings and target batch within this subspace. They also apply dimension weighting to prioritize subspace dimensions based on their significance to the output. The proposed method is evaluated on four regression datasets and demonstrates superior performance compared to baseline TTA methods designed for classification tasks, and domain adaptation for regression adapted for TTA .

strengths:
- The paper proposes a novel method for Test time domain adaptation for regression, and adapted common benchmarking procedures for classification to regression, especially on the UTK Face dataset.

- The paper is well-written and clear.

- The method and results seem convincing and easy to implement.

weaknesses:
- In the related work, feature alignment methods for TTA are reviewed with the claim, “Although some of these methods are directly applicable to regression, we have observed that they are not effective or even degrade regression performance.” However, these methods are not included in the experiments, leaving this claim unsupported.
- In Table 1, the subspace dimension for Biwi Kinect is shown as ‘34.5’, which is unclear, as dimensions are typically integers.
- MLPs and CNNs seem to produce embeddings with different ranks, with MLPs showing less low-rank behavior. A discussion or remark on this difference could provide useful insights.
- The test setup lacks clarity on whether SSA is applied in an online or episodic adaptation manner, and it’s unclear if batches are fed uniformly.
- The dimension weighting approach may lead to disproportionately high contributions from small-scale features if they are assigned high weights (when learned on source), potentially resulting in values similar to large-scale features with lower weights (when learned on source).   Did you consider any normalization or balancing mechanism to prevent this scaling effect?

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730637621046

thread_type:
official_review

<thread_object>
creator: reviewer_66u8

<content>
summary:
This paper investigates the application of Test-Time Adaptation for regression tasks, where a regression model pre-trained in a source domain is adapted to an unknown target distribution using unlabeled target data. The authors note that most existing TTA methods are designed for classification tasks and do not directly apply to regression models, which typically output single scalar values rather than class-categorical predictions. To address this, the paper proposes a feature alignment approach called Significant-subspace Alignment, which consists of two components: subspace detection and dimension weighting. SSA aims to align feature distributions between the source and target domains to mitigate the domain gap, focusing on a representative and significant subspace of the feature space. Experimental results on various real-world datasets demonstrate that SSA outperforms several baseline methods.

strengths:
1.SSA introduces a novel approach for TTA in regression tasks by combining subspace detection and dimension weighting, which is an innovative contribution to the field.
2.The paper conducts extensive experiments on multiple real-world datasets, validating the effectiveness of SSA.
3.Compared to the original model and other baseline methods, SSA achieves higher R2 scores across multiple datasets, demonstrating performance improvements in regression tasks.

weaknesses:
1.SSA assumes covariate shift, where p(y|x) remains unchanged. The paper does not address distribution shifts where p(y|x) changes, such as concept drift, limiting its applicability in broader scenarios.
2.While the paper mentions the selection of the parameter λ, it lacks a detailed discussion on the impact of other hyperparameters, which could affect the model's generalizability and adaptability.
3.The paper does not discuss the computational complexity of SSA, particularly its performance with large-scale datasets, which is crucial for practical applications.

questions:
How does SSA perform in multi-task and multi-class regression tasks? Can it be extended to these scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730730857936

thread_type:
official_review

<thread_object>
creator: reviewer_t9wm

<content>
summary:
This paper addresses a crucial yet under-explored area in machine learning: test-time adaptation (TTA) for regression models. The authors rightly highlight the limitations of existing TTA methods, which are primarily tailored for classification tasks, and propose a novel approach termed Significant-subspace Alignment (SSA). The paper presents a clear rationale for the need to adapt TTA techniques for regression and offers a compelling solution through feature alignment.

strengths:
The topic of adapting regression models to unknown target distributions is highly relevant, particularly given the increasing use of machine learning in diverse applications. The proposed SSA approach is innovative and addresses a gap in the literature.

weaknesses:
1. In Section 3.1 of the article, the use of "two diagonal Gaussian distributions" is not adequately justified, and the advantages and limitations of employing such distributions are not discussed.
2. The proposed method in this paper exhibits limited originality, as its various components are commonly found in existing models.
3. The dataset utilized in this study has a relatively small sample size, which weakens the persuasiveness of the findings. Additionally, there is a lack of comparative algorithms from the past three years, resulting in insufficient theoretical validation. Furthermore, the measurement standard is solely based on R²; incorporating additional metrics such as RMSE and RMAE would enhance the evaluation.

questions:
1. Why are "two diagonal Gaussian distributions" used in Chapter 3? What is the rationale behind this choice?
2. In Section 4.1, the first dataset utilizes a classification dataset as a regression task. Is this approach reasonable and justifiable? Regarding the second dataset, is it appropriate to consider a noisy version of the test set as the target domain?
3. Could you provide a proof or evidence demonstrating the effectiveness of Significant-subspace Alignment (SSA)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732179698465

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all the reviewers for their constructive feedbacks and suggestions, which have helped us improve the quality and clarity of our work.

Based on the reviewers' comments, we revised our paper as follows:

- Added discussion on employing diagonal Gaussian distribution in **Section 3.2** based on reviewer `t9WM`'s comment.
- Added evaluation in MAE in **Tables 2 and 3** of **Section 4.3.2** and **Tables 27 and 28** of **Appendix D.9.** based on reviewer `t9WM`'s comment.
- Added experiments with different learning rate and batch size in **Tables 21-26** in **Appendix D.8** based on reviewer `66U8`'s comment.
- Clarified the experimental setting in comparing the number of the feature dimensions in **Table 1** based on reviewer `xN4r`'s comment.
- Added explanation on the difference of the subspace dimensions between CNN and MLP in **Section 4.3.1** based on reviewer `xN4r`'s comment.
- Clarified the experimental setting in **Appendix C.3** and added results of batched online setting in **Tables 29-31** in **Appendix D.10** based on reviewer `xN4r`'s comment.
- Updated the overview of our SSA in **Figure 1** based on reviewer `gBNN`'s comment.
- Added citations of related papers in **Section 5.3** based on reviewer `gBNN`'s comment.
- Added feature visualization by UMAP in **Figure 5** based on reviewer `gBNN`'s comment.

We hope that the revised paper helps the reviewers to re-evaluate our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1742348417921

thread_type:
public_comment

<thread_object>
creator: ~kazuki_adachi1

<content>
comment:
The code is available at https://github.com/kzkadc/regression-tta
</content>
</thread_object>

</thread>


Metareview:
metareview:
After extensive author-reviewer discussions, all reviewers suggest that the paper should be accepted. In this case, I am happy to recommend accepting the paper.