Test-time Adaptation for Regression by Subspace Alignment
Authors: Kazuki Adachi
Shin'ya Yamaguchi
Atsutoshi Kumagai
Tomoki Hamagami
Venue: ICLR 2025 Poster (2025)
Keywords: Test-time adaptation
regression
distribution shift
deep learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a novel test-time adaptation (TTA) method for regression, which existing TTA methods are not designed for.

Abstract: This paper investigates test-time adaptation (TTA) for regression, where a regression model pre-trained in a source domain is adapted to an unknown target distribution with unlabeled target data.
Although regression is one of the fundamental tasks in machine learning, most of the existing TTA methods have classification-specific designs, which assume that models output class-categorical predictions, whereas regression models typically output only single scalar values.
To enable TTA for regression, we adopt a feature alignment approach, which aligns the feature distributions between the source and target domains to mitigate the domain gap.
However, we found that naive feature alignment employed in existing TTA methods for classification is ineffective or even worse for regression because the features are distributed in a small subspace and many of the raw feature dimensions have little significance to the output.
For an effective feature alignment in TTA for regression, we propose Significant-subspace Alignment (SSA).
SSA consists of two components: subspace detection and dimension weighting.
Subspace detection finds the feature subspace that is representative and significant to the output.
Then, the feature alignment is performed in the subspace during TTA.
Meanwhile, dimension weighting raises the importance of the dimensions of the feature subspace that have greater significance to the output.
We experimentally show that SSA outperforms various baselines on real-world datasets.
The code is available at https://github.com/kzkadc/regression-tta.

Published as a conference paper at ICLR 2025
TEST-TIME ADAPTATION FOR REGRESSION BY SUBSPACE ALIGNMENT
Kazuki A Shin'ya Yamaguchi*t Atsutoshi Kumagai" Tomoki Hamagami! "NTT Corporation
Kyoto University Yokohama National University
(kazuki adachi, wwwp:aaymmmucc.ttsssitaaa atsutoshi. . Jentt..cencccoommmmmm
hamagami eem.mmiiiiuuaammm jp
ABSTRACT
This paper investigates test-time adaptation (TTA) for regression, where a regres- sion model pre-trained in a source domain is adapted to an unknown target distri- bution with unlabeled target data. Although regression is one of the fundamental tasks in machine learning, most of the existing TTA methods have classification-- specific designs, which assume that models output class-categorical predictions, whereas regression models typically output only single scalar values. To enable TTA for regression, we adopt feature alignment approach, which aligns the fea- ture distributions between the source and target domains to mitigate the domain gap. However, we found that naive feature aligmment employed in existing TTA methods for classification is ineffective or even worse for regression because the features are distributed in a small subspace and many of the raw feature dimen- sions have little significance to the output. For an effective feature alignment TTA for regression, we propose Significant-subbl subspace Aligument (SSA). SSA con- sists of two components: subspace detection and dimension weighting. Subspace detection finds the feature subspace that is representative and significant the output. Then, the feature alignment is performed in the subspace during TTA. Meanwhile, dimension weighting raises the importance of the dimensions of the feature subspace that have greater significance to the output. We experimentally show that SSA outperforms various baselines on real-world datasets. The code is available at
INTRODUCTION
Deep neural networks have achieved remarkable success in various tasks (LeCun et al.] 1998a Krizhevsky et al. 2012: He et al.) 2016: Dargan et al.) 2020). In particular, regression, which is one of the fundamental tasks machine learning, is widely used practical tasks such as hu- man pose estimation or age prediction (Lathuilière et al., 2019 The successes of deep learning have usually relied on the assumption that the training and test datasets are sampled from an i..d. distribution. In the real world, however, such assumption is often invalid since the test data are sampled from distributions different from the training one due to distribution shifts caused by changes in environments. The performance of these models thus deteriorates when a distribution shift occurs (Hendrycks & Dietterich, 2019 Recht et al.) 2019). To address this problem, test-time adaptation (TTA) (Liang et al. [022) has been studied. T TTA aims adapting a model pre-trained in source domain (training environment) to the target domain (test environment) with only unlabeled target data. However, most of the existing TTA methods are designed for classification; that is, TTA for regression has not been explored much (Liang et al. 2023). Regarding TTA for classification, two main approaches have been explored: entropy minimization and feature alignment.
The entropy minimization approach was introduced by Wang et al. (2021). and the subsequent meth- ods follow this approach (Zhou & Levine 2021 Niu et al. 2022 Zhang et al. 2022 Zhao et al.) 2023 Enomoto al.) 2024). Although entropy is a promising proxy of the performance on the tar- get domain, entropy minimization is cussiffiatioonssecificc because assumes that a model directly outputs predictive distributions, i.e., a probability for each class. In contrast, typical regression mod- els output only single scalar values, not distributions. Thus, we cannot use the entropy minimization approach for regression models.

---

Published as a conference paper at ICLR 2025
Table 1: Number of valid (having non-zero variance) feature dimensions and feature subspace di- mensions (i.e., the rank of the feature covariance matrix), and R2 scores on the test datasets. Al- though the original feature space has 2048 dimensions in the experiment except for the California Housing dataset, the features of the regression models are distributed within the subspaces that have less than hundred dimensions. Our method improves R scores by the feature subspace in most cases, whereas the naive feature alignment sometimes diverged (displayed as "-) because of too few valid dimensions. See Section 4.3. for more details.
Rt (1)
Dataset
#Valid dims.
#Suhspace dims.
Source
Naive feature alignment SSA (ours)
SVHN
353
14
0.406
0.511
UTKFaac
204]
76
0.020
0,705
0. 0.731
Biwi Kinect"
713
34.5
0.706
0.753
0.778
California Housing (100 dims.)
45
40
0.605
0.639
The average model trained cach gender/target is repoeted.
Another approach, the feature alignment, preliminarily computes the statistics of intermediate fea- tures of the source dataset after pre-training in the source domain (Ishii & Sugiyama! 2021; Kojima et al. 2022; Eastwood et al., 20222 Adachi et al. [2033] Jung et al., 2023). Then, upon moving to the target domain, the feature distribution of the target data is aligned with the source distribution by matching the target feature statistics with the pre-computed source ones without accessing the source dataset. Although this approach seems applicable to regression because it allows arbitrary forms of the model output, it assumes to all feature dimensions to be aligned, and does not sufficiently consider the nature of regression tasks. For instance, regression models trained with standard mean squared error (MSE) loss tend to make features less diverse than classification mod- els do (Zhang et al.] 2023). In particular, we experimentally observed that the features of a trained regression model are distributed in only a small subspace of the entire feature space (Table [T. In this sense, naively aligning all feature dimensions makes the performance suboptimal or even be harmful in regression as shown in Table because it equally treats important feature dimensions and degenerated unused ones.
In this paper, we address TTA for regression on the basis of the feature alignment approach. To re- solve the aforementioned feature alignment problem in TTA for regression, we propose Significant- subspace Alignment (SSA). SSA consists of two components: subspace detection and dimension weighting. Subspace detection uses principal component analysis (PCA) to find subspace of the feature space in which the features are concentrated. This subspace is representative and significant to the model output. Then, we perform feature alignment within this subspace, which improves the effectiveness and stability of TTA by focusing only on valid feature dimensions in the subspace. Further, in regression, feature vector is finally projected onto one one-dimeniional line as to output a scalar value. Thus, the subspace dimensions that have an effect on the line need a precise feature alignment. To do so, dimension weighting raises the importance of the subspace dimensions with respect to their effect on the output.
We conducted experiments on various regression tasks, such as UTKFace (Zhang al. 2017), Biwi Kinect (Fanelli et al. 2013), and California Housing (Nugent) 2017). The results showed that our SSA retains the important feature subspace during TTA and outperforms existing TTA baselines that were originally designed for classification by aligning the feature subspace.
2 PROBLEM SETTING
We consider a setting with neural network regression model fo X' + R pre-trained on labeled source dataset S {(xi, vf) E X x RR), R where x² and yÿ are an input and its label, and X' is the input space. Our goal is to adapt fo to the target domain by using an unlabeled target dataset T = E x X without accessing S. Note that the target labels y E R are not available. In the source dataset S, the data {xx), n²)}} are sampled from the source distribution ps over X' x R. In the target dataset T, we assume covariate shift (Shimodaira) 2000), which is a distribution shift that often occurs in the real world. In other words, the target data x; are sampled from the target distribution Pt over X' that is different from p., but the predictive distribution the same, i.e., (x() A (xx) and (x(y) x) px()))).
2

---

Published as a conference paper at ICLR 2025
Forture
w
Source domain Pre-training
9 Fostuo PAENCING
hug f-
(10) 601)
Sorane droutt
Pre-tpuis modal
Procoeeppte egrittcrt reheuce , from
min CYTa
A.Algga fnztre Sora
0
Target domain adaptation
(x
g+
h
Figure 1: Overview of significant--sbspacc alignment (SSA). A more detailed procedure is listed in Algorithm
We split the regression model fo into a feature extractor 96 : R and linear regressor hy (z) = w z + b, where D is the number of feature dimensions, w E RD, E R. ø and & = (w, b) are the parameters of the models. The whole regression model using the feature extractor and linear regressor is denoted by fo hip Jo. where (Ø, vb).
TEST-TIME ADAPTATION FOR REGRESSION
In this section, we describe the basic idea behind Significamt-subspacee Alignment (SSA) in Sec- tion [.1 and describe it in detail in Section 3.2 Our method can be applied regardless of the form of input data since SSA does not rely on input-specific method, such as image data augmentations or self-supervisee tasks.
3.1 BASIC IDEA: FEATURE ALIGNMENT
The basic idea of our TTA method for regression is to align the feature distributions of the source and target domains instead of using entropy minimization, as usually done in TTA for classifica- tion. As assume covariate shift where the input distribution changes, we update the feature extractor Is to pull back the target feature distribution to the source one. Here, we describe a naive implementation of the idea and its problem.
First, in the source domain, we compute the source feature statistics (mean and variance of each dimension) on S after the source training:
No
0²
d²2=
Σ N
N. 1 о (2) 1²),,
(Z)
N.
where 2; = 96 (x:) E R a source feature and @ the element-wiss product.
Then, we move to the target domain, where we cannot access the source dataset S. Given target mini-batch = {xt ) sampled from T. we compute the mini-batch mean and variance ği' and 212 analogously Equation (Ip.
For feature alignment, we seek to make the target statistics similar to the source ones. For this purpose, we use the KI. divergence as Nguyen et al. (2022) proved that included in an upper bound of the target enoor in unsupervised domain adaptation. Concretely, we minimize the KI. divergence between two diagonal Gaussian distributions. A N(H², x²2) and (ft,
D (oIA(0) = (0) - DEL (Nirg o3 3 023) DEL (N(R), (MWLOWI
(2)
where the subscripts represent the d-th elements of the mean and variance vectors. Here, we used both directions of the KL divergence because it empirically had good results recommended by Nguyen et al. (2022). The KL divergence between two univariate Gaussians can be written closed form (Duchi 2007)
DxL. (N( o [log(0ž/0)) ((n 22)² + +oi)/0²
(3)

---

Published as a conference paper at ICLR 2025
However, in regression models, the features tend to be less diverse than in classification (Zhang et al., 2023). In addition to the insight from Zhang et al.] (2023), we observed that the features of a regression model trained on S are distributed in only small subspace of the feature space and many dimensions of the feature space had zero variances. Table [hoows the numbers of valid (having non-zero variance) feature dimensions and feature subspace dimensions (see Section 4.3.1 for more details). This property makes the naive feature alignment described above unstable since the KL divergence in Equation 3 includes the variance in the denominator. Also, this naive feature alignment is ineffective because many of the feature dimensions have small effect on the subspace.
3.2 SIGNIFICANT- ALIGNMENT
In this section, we describe our method, Significant-susspe Aligmment (SSA), to tackle the afore- mentioned problem of naive feature alignment. Figure shows an overview of SSA. As described in Section 3.1, the features of a regression model tend to be distributed in small subspace of the feature space. Thus, we introduce subspace detection and significant to the output and then perform feature alignment in the subspace. Subspace detec- tion similar to principal component analysis (PCA). Further, in our regression model, a feature z = (x) is projected onto a one- vreddmmensional line determined by w of ha in order to output scalar value w z + b. Thus, subspace basis va whose direction is not orthogonal to w needs precise feature alignment. We use dimension weighting to prioritize such dimensions.
detect subspace that is representative
(x)
Subspace detection. After the training on the source dataset S, we detect the subspace in which the source features are distributed. Instead of computing the variance of each dimension as Equation (I. we compute the covariance matrix:
(2)
=T²)T
(4)
N₂
where the source mean vector μ³ is the same as Equation (..
Then, we detect the source subspace. On the basis of PCA, the subspace is spanned by the eigen- vectors of the covariance matrix Σ,, denoted by v (llv) 1). The corresponding eigenvalues Xi represent the variance of the source features along the direction vi. We use the top- largest eigenvalues Xi. ....... Xxx (Aj > > Xx). the corresponding source bases vi vi...., Vx. and the source mean Jı as the source statistics.
Dimension weighting. Since assume that the output is computed with linear regressor (z) w z + b, the effect of a subspace dimension v² to the output is determined by w v1. To prioritize the subspace dimensions that have larger effect on the output, we determine the weight of each subspace dimension follows:
od =
(5)
ad assigns larger weight when the direction along a subspace basis vž affects the output, or keep the weight to one when not.
Feature alignment. This step is done in the target domain. Given target mini-batch B sampled from the target dataset T, we project the target features z = Io (x!) into the source subspace and then compute the feature alignment loss. The projection of the target feature is computed as follows:
1 (2) 1²).
where V iv..., E RKXD With こ E RK we compute the projected target mean and variance over the mini-batch analogously to Equation [ this is denoted by jı' and 2²2 On the other hand, the projected source mean and variance are 0 and the eigenvalues A [Ai.... A since V'v; = v Thus, the KL divergence in the detected subspace is computed between two K-dimensional diagonal Gaussians (O X²) and "i Using subspace detection and dimension weighting, the loss of SSA is:
(Ø) = Σx
11 ad (DLL (N(0, (OI 8121) +DKL (N 822) xD))
K
K
(12)) + X/
(12)22 + + -2)
2 d=1 ad
012²
+
X²
4

---

Published as a conference paper at ICLR 2025
During TTA, we optimize the feature extractor 96 to minimize LIIA. i.e., we seek o' ming (o)) We update only the affine parameters Y and 3 of the normalization layers such as batch normalization (loffe & Szegedy 2013) or layer normalization (Ba et al., 2016) inspired by Tent (Wang et a 2021). This strategy is effective not only to retain the source knowledge but also to enable flexible adaptation (Frankle al. 2021 Burkholz, 2024). The procedure of SSA is listed in Algorithm of the Appendix.
Is diagonal Gaussian distribution appropriate? For computing KL divergence of LTTA, we as- sume the source and target feature distributions as diagonal Gaussian. This is reasonable because features are likely to follow Gaussian distribution when projected onto the feature subspace de- tected by subspace detection as the number of original feature dimensions increases by the central limit theorem, as described in Figure S and Section 4.3.4 Moreover, since subspace detection uses the PCA, the features projected onto the subspace are decorrelated. Thus, assuming that each di- mension is independent, i.e., diagonal, is also reasonable.
4 EXPERIMENT
This section provides empirical analysis of feature subspaces and evaluations of SSA on various regression tasks. First, we checked whether the learned features are distributed in small subspace (Section 4.3.I) and then evaluated the regression performance (Sections 4.3.2 and 4.3.3). We also analyzed the effect of the TTA methods from the perspective of the feature subspace (Section
4.1 DATASET
We used regression datasets with two types of covariate shift, i.e., domain shift and image corruption. SVHN-MNIST. SVHN (Netzer et al.) 2011) and MNIST (LeCun et al., 1998b) are famous digit- recognition datasets. Although they are mainly used for classification, we used them for regression by training models to directly output a scalar value of the label. We used SVHN and MNIST as the source and target domains, respectively.
UTKFace (Zhang et al.) 2017). UTKFace a dataset consisting of face images. The task is to predict the age of the person in input image. For the source model, we trained models on the original UTKFace images. For the target domain, we added corruptions such as noise or blur to the images. The types of corruption were the same those of ImageNet-C (Hendrycks & Dietterich 2019). We applied 13 types of corruption at the highest severity level of the five levels.
Biwi Kinect (Fanelli et al., 2013). Biwi Kinect is a dataset consisting of person images. The task is to predict the head pose of the person in an input image in terms of pitch. yaw, and roll angles. We separately trained models to predict each angle. The source and target domains are the gender of the person in the image. We conducted experiments on six combinations of the source/target gender and task, i.e., {male - female, female - male) {pitch, yaw, roll). We trained regression models to directly output head pose angles in radian, which are roughly in (-0.4 0.4.T).
California Housing (Nugent, 2017). California Housing is a tabular dataset aiming at predicting housing prices from the information of areas. We split the dataset into non-coastal and coastal areas for the source and target domains in accordance with He et al. 20044).
More details of the datasets are provided in Appendix C.I
4.2 SETTING
Source model. We used ResNet-26 (He et al.) 2016) for SVHN, ResNet-50 for UTKFaae and Biwi Kinect, and an MLP for California Housing. We modified the last fully-connected layer to output single scalar values and trained the models with the standard MSE loss on each dataset and task. The details of the training are provided in Appendix C
Test-time adaptation with SSA (ours). We minimized LITA on the target datasets. We used the outputs of the penultimate layer of the model as features, which had 2048 dimensions. We set the number of dimensions of the feature subspace to K = 100 as the default throughout the experi- ments. More detailed settings are provided in Appendix C.3
Baseline. Since there no TTA baselines designed for regression, compared SSA with TTA methods designed for classification but can be naively modified to regression: Source (no adapta- tion), BN-adapt (Benz et al. 2021), Feature restoration (FR) (Fastwood et al. 2022). Prototype,
5

---

Published as a conference paper at ICLR 2025
Table 2: Test scores on SVHN-MNIST. The
best scores are bolded.
Table 3: Test R score and RMSE on Cali-
Method
RPit)
RMSE ())
MAE (L)
fornia Housing.
Source
0.106
2,232
1.008
Method
R²(
RMSE (4)
MAE (L)
DANN
0,307 0.307 c0
2.406
1.489
Source
0.605
0.684
0.516
TTT
0.288
2.44320.03
1.507
BN-adapt
0.318 0.00
0.69910 0.809
adapt
0.396
2.251 20.01
1.458+0.00
Protolype
0.726_0.01
1.431
1.196
Protocype
0.491
2.065
1.47920.01
FR
0,369 10
2,300
1.631
FR
0.510_o.01
0.762
0.534
VM
685.
ssa
7..83=1.52
76.78+1.52
RSD SSA (ours) 0.639
RSD
0 0,252 10.12
2.197
1..00
0.655
0.469_0.00
(ours)
0.511
2.024.0.06
1.209.0.04
Oracle
0.729_o.50
0.567
0.404
Oracle
0.87440.00
1.028
0.575+0.00
Variance minimization (VM), and RSD (Chen et al. (021). In addition, we used the following meth- ods other than TTA as baselines: test-time training (TTT) (Sun et al., (2020), DANN (Ganin et al.] 2016), and Oracle (fine-tuning using labels; performance upper bound). The details of the baseline methods are described in Appendix C.3
4.3 RESULT
4.3.1 NUMBER OF DIMENSIONS OF THE FEATURE SUBSPACE
After the pre-training on the source dataset, we counted the numbers of valid feature dimensions (i.e., having non-zero variances) and dimensions of the feature subspace in which the souree features distributed. The latter value corresponds to the rank of the covariance matrix of the source features in Equation (4). Table [ shows the result of each source dataset and regression test R² scores. Although the number of feature dimensions is 2048 in ResNet, many feature dimensions of the regression models have zero variance because of ReLU activation. This is the cause of the failure of the naive feature alignment, as described in Section 3.1 Moreover, the source features are distributed in only a small subspace with fewer than a hundred dimensions. In the California Housing dataset, we can also see the same tendency that the number of the subspace dimensions is only 40 whereas the number of the entire feature dimensions 100. This property limits the performance of the naive feature alignment in regression since aligning the entire feature space is ineffective to the subspace in which the features are actually distributed.
In the MLP used for the California Housing dataset, the subspace dimensions compared to the entire feature space higher than the ResNets used for the other datasets. One explanation for this difference is model capacity and task complexity. MLP's capacity is low relative to the task (California Housing)'s ' complexity. On the other hand, the ResNet-26, which has high capacity, resulted in lower subspace dimensions SVHN which has low complexity.
4.3.2 REGRESSION PERFORMANCE
We evaluated the regression performance in terms of the R² score (coefficient of determination), which is widely used in regression tasks (see Appendix B). Tables and show the scores for the SVHN-MNIST and California Housing. In the both cases, SSA outperformed the baselines; some of them even underperformed the Source. This is because the baselines were designed for classification tasks and they broke the feature subspaces learned by the source model (see Section 4.3.4p. On the other hand, RSD (Chen et al., (02T) is originally designed for regression in UDA but did not work in the California Housing dataset because of numerical instability of SVD performed on every target feature batch. In contrast, our method is stable since it avoids degenerated dimensions during TTA. Table 4 shows the R² scores on the UTKFace with image corruption. We can see that SSA had the highest R² scores for most of the corruption types. In particular, SSA outperformed the baselines by a large margin on noise-type corruption which significantly degraded the performance of Source. Table shows Shhows the R² scores on Biwi Kinect with genders different from the source domains. SSA constantly had higher R² scores than the basclines; the baselines' scores sometimes significantly dropped or even diverged (Prototype). In summary, SSA consistently improved the scores whereas

---

Published as a conference paper at ICLR 2025
Table 4: Test R² scores on UTKeaaee with image corruption (higher is better). The best scores are bolded.
Defivaus
Eloadi
1
comp.
Camolan
1000c
Engalee
Shot
Brighna
2055
Snow
Method
Mion
Sousee
0.430
00.120
006638
3,000
0.711
0.063
0.501
-2.05
-2.022
0.601
00.120
00.544
-0.078
DANN
0.512
0..06
0.637
-0.730
0.729
0.508
0.907
-4.341
-3.114
-3.741
0.600
-0.131
-0.425
-0.000
TTT
0.748
0.761
0.773
0.778
0.826
0.772
0.581
0.525
0.532
0.477
0.775
0.399
0.800
0.671
RN-Adlpt
0.727
0.730
0.763
0,702
0,526
0.778
0.850
0.510
0.510
0.730
Prouotype
-1.00
-1.020
-1.018
-0.710
-0.907
0.988
-0.974
-0.514
-0.512
-0.512
-1.004
-0.823
-0.822
-0.630
FR
0.794
0.839
0.540
0.756
0.899
0.825
0.046
0.509
0.522
0.458
0.861
0.808
0.428
0.700
VMd
-2.000
-1.991
2.007
1.889
1.918
1.918
-1.751
-2.181
-2.207
-2.778
-1.027
-2.250
-2.197
-2.035
RSD
0.789
0.833
0.831
0.749
0.897
0.825
0.941
0.582
0.583
0.445
0.802
0.419
0.500
0.201
SSA (OurR)
0.803
0.839
0.851
0.792
0.899
0.829
0.945
0.580
0.502
0.560
0.863
0.440
0.517
0.781
Oradle
0.856
0.800
0.889
0.822
0.917
0.633
0.960
0.635
0.662
0.635
0.895
0.019
0.071
0.780
Table 5: Test R scores on Biwi Kinect (higher is better). The best scores are bolded.
Female
Male
Male
Female
Method
Pitch
Roll
Yaw
Pitch
Roll
Yaw
Mean
0.759
0.956
0.481
0.763
0.791
0.485
0.706
DANN
0.098
0.820
-0.030
-0.03920.0
0.71
0.850
0.078
0.520=0.02
0.0622g..0
0.606g
0.031
20.00
0.750_0.00
0.725 0n
0.28820.03
BN BN-adapl
2.771
0.953220.0
0.
0.832_0.00
0.842 on
0.585_o
0.746g0.00
Prototype
-318 10.8
FR
-1.27
0.742.0.05
-2.69
0.622_..0o
0.855 01
-0,406
-0.357
VM
0.3002 1o.00
0,062 10.00
-0.0891...
00,101 0.0
0.0013
0.0011aco
-0.1000_..00
RSD
0.783.00.0
0.954..0.0
0.489920.0
0.832_...0
0.8466
SSA (ours)
0..60_nn00
1.962_oo.o
0.
0.869
0. 886
0.375.a0c0
0.778 10.00
Oracle
0.966gm.00
0,981ga.o
0.801go.00
0.959_m.0v
0.970
0.811gac0
0.915
the baselines sometimes even underperformed Source. Moreover, SSA worked well not only on image data but also tabular data.
4.3.3 ABLATION STUDY
We performed ablation study the subspace detection and dimension weighting. For the SSA variant without subspace detection (i.e., naively aligning the entire feature space), we simply selected the top-K feature dimensions that had the largest variances. In this case. we directly used the weight of the linear regressor hy to compute the dimension weight aa as aa = 1 + |uaal instead of Equation 55) Table shows the test R² scores with and without subspace detection and dimension weighting each dataset. Without subspace detection, the scores MNIST and Biwi Kinect, and of the same level as simple baselines like BN-adapt (Benz et al,) ) UTKFace (Table (4). In contrast, subspace detection significantly improved the scores all three datasets. Dimension weighting also improved the scores, although the gain was smaller than in the of subspace detection. This is because the variance of the feature subspace dimension correlates with the weight; i.e., the top-K selected dimensions with respect to variance tended to have high importance the output. Table lists the correlation coefficients between the top K 100 variances of the source features along the source bases X and the corresponding dimension weight ag, We can see that there are strong correlations the three datasets we used. But we can
worse than Source
Table 6: Test R² scores of SSA with and without subspace detection and dimension weighting. Scores averaged over corruption types and gender-task combinations are reported for UTKFace and Biwi Kinect, respectively. The best scores are bolded.
Subspace
Weight
SVHN
UTKFace
Biwi Kinect
California
0.333
0.642
0.626 0.67210.24
0.508
0.728+0.16
0.778
0.633=
0.51 1.51--000
0.731+0.18
0.778+0.17
0.639+0.00
Source
0.406
0.020
0.706
0.605

---

Published as a conference paper at ICLR 2025
Table 7: Correlation coefficients between the top K 100 variances of source features along the source bases X2 and weight an in Equation 5
Biwi Kineet (Female, Pitch)
Dataset
SVHN
UTKFaaee
Correlation
0.787
0.917
0.782
Jour adaat
emrrr
12.5
(curs)
7.5
Mathod
Soucce Pu00yyle
10.0
BN-adapt
Sourcc, Protoyyoe
3.0
Source, Protobspc
Bt--cast
1.0
0.5
Wean
5.0
Mean
7.5
supspace
(c) Biwi Kinect
(a) SVHN
(b) UTKEaae (Gaussian noise)
(Female + Male, Pitch)
Figure 2: Reconstruction error of features reconstructed with the source bases relative to the original target features. Note that Source and Prototype are the same, since Prototype does not update the feature extractor of the model. FR (Fastwood et al. 2022) and VM are not plotted in (a) because they had huge errors.
expect that dimension weighting can raise the importance of the feature dimensions that have low variance but affect the output, which further improves regression performance.
Next, we investigated the effect of the number of feature subspace dimensions K. We varied K within (10, 25, 50, 75, 100, 200, 400, 1000, 2048). Table g shows the test R² scores. Although the best K differs among the datasets, K 100 consistently produced competitive results. With increasing K, the best or competitive scores were when K was close to the number of the subspace dimensions in Table T This indicates the importance of the subspace feature alignment. When K 2 400 in MNIST and K 2 1000 in Biwi Kinect, the loss became unstable or diverged because SSA attempted to align too many degenerated feature dimensions. In contrast, although setting K > 1000 gave the good scores on UTKFaaee K = 100 produced a competitive score. Appendix [..] provides the results for California Housing, which three datasets. From these results, although K is hyperparameter, we can determine K before accessing the test dataset by calculating the number of the source feature subspace dimensions.
observed the same tendency with the other
4.3.4 FEATURE SUBSPACE ANALYSIS
Feature reconstruction. To verify that the reason why the bascline methods degrade the regression performance is that they affect the feature subspace learned by the source model as mentioned in (ection[4.3.2 we examined the reproducibility of the target features with the source bases V after TTA. That is, the target features can be represented by linear combination of the source bases if the model retains the source subspace throughout TTA and the target features fit within the subspace. To mea- sure this quantitatively, we computed the reconstruction error I. as the Euclidean distance between target fea- ture vector z' and z/. the one reconstructed with source bases:
Table 8: Test R² scores of SSA for different numbers of feature subspace dimensions K. The best scores are bolded.
MNIST
UTKKFace
Kinect
10
0.494
0.693
0.688
25
0.538
0.717
0.761
50
0.524
0.728
0.767
75
0.732
0.774
100
0.511
0.731
0.778
200
0.496
0.731
0.771
400
0.731
0.755
1000
0.732
L - z'11a
+ ='=
=
-" ((z)
"²) v2
(8)
d=
where z' is target feature vector extracted with the model after TTA, and is the number of dimensions of the source subspace listed in Table
the three datasets. The error decreased as increased for all methods, but SSA reduced the error with a smaller n than in those of the baselines, indicating that it could make the target features fit within the source subspace. Especially in the case
Figure plots the reconstruction error L versus
8

---

Published as a conference paper at ICLR 2025
Biwi Kinect
(a) SVHN-MNIST
(b) UTKFace (Gaussian noise)
(c) (Female > Male, Pitch)
Figure 3: Histograms of three randomly selected target feature dimensions. Left: Original features. Right: Projected features.
of Biwi Kinect (c), the baseline methods produced larger errors than Source; i.e., they broke the learned subspace.
In terms of feature alignment, Figure 4 in Appendix D.2 visualizes and evaluates the feature gap between the domains in the subspace.
Another effect of subspace detection. Subspace detection has another effect feature alignment. We used the KL divergence between two diagonal Gaussian distributions in Equation (3) to measure the distribution gap between the source and target features, under the assumption that the features follow Gaussian distribution. Here, while it is not clear that the assumption actually holds espe- cially when features output through activation functions like ReLU as in ResNet, argue that the subspace detection of SSA has, in addition to an effective feature alignment, the effect of making such features follow a distribution close Gaussian.
We visualized the histograms of the target features z! extracted with the source model and ones projected the source subspace with Equation (6). Figure show the histograms of three randomly selected dimensions of the original and projected features of the three datasets. In the left column of each figure, the histograms of the original features concentrate on zero because of the ReLU activation and do not follow Gaussian distribution, which makes the KL divergence computation with Equation (3) inaccurate. On the other hand, the histograms of the projected features the right columns are close to Gaussians. Thus, subspace detection makes it easier to align the features with Gaussian KL divergence.
The reason why the projected features follow Gaussian distribution can be interpreted as follows. From Equation (6), the k-th element of a projected feature vector ž
E = Σ Ed= (+i,a Pa)
Here, we regard each term a,d := (2/,a m2)ut as a random variable. Assuming that a,,d is independent of the feature dimension d, the central limit theorem guarantees that the distribution of the projected features, i.e., the sum of 0,d, becomes closer to Gaussian as the total number of dimensions increases.
RELATED WORK
5.1 UNSUPERVISED DOMAIN ADAPTATION
Unsupervised domain adaptation (UDA) has been actively studied as a way to transfer knowledge in the source domain the target domain (Csurka 2017). Theoretically, is known that the up- per bound of the етrоr on the target domain includes distribution gap term between the source and target domains David al. 2010 Ganin et al.) 2016, Nguyen et al. 2022). For regres- sion, Cortes & Mohri (2011) theoretically explored regression UDA. RSD (Chen et al. 2021 and DARE-GRAM (Nejjar et al. 2023) take into account that the feature scale matters in regression and explicitly align the feature scale during the feature alignment. However, UDA requires the source and target datasets to be accessed simultaneously during training, which can be restrictive when datasets cannot be accessed due privacy security concerns, storage limitations.
More recently. source-free domain adaptation (SFDA), which does not access the source dataset during adaptation, has been studied. The SFDA setting is similar to TTA in that SFDA adapts

---

Published as a conference paper at ICLR 2025
models with only unlabeled target data. However, SFDA requires to store the whole target dataset and access the dataset for multiple epochs to train additional models (Li et al., 2020 Xia et al.
2021 Chu et al. 2022; Sanyal et al. 2023; He et al. 2024) perform clustering (Ling et al, 2020). On the other hand, TTA does not train additional models nor access the target dataset for multiple epochs, which enables instant adaptation with low computational resource and storage.
5.2
TEST-TIME TRAINING
Test-time training (TTT) is also similar to TTA as it adapts models with unlabeled target data. The main difference is that TTT requires to modify the model architecture and training procedure in the source domain. The main approach of TTT is additionally training self-supervised branch simultaneously with the main supervised task in the source domain. Then, during adaptation, the model is updated via minimizing the self-supervised loss the target data. On the basis of this approach, TTT methods with various self-supervised tasks have been proposed such as rotation prediction (Sun al. 2020), contrastive learning (Liu et al.) [00TT, clustering (Hakim et al. (023), or distribution modeling with normalizing flow (Osowiechi et al. [202)]. However, adding additional losses the training prohibits the use of off-the-shelf pre-trained models may potentially affect the performance of the main task. In contrast, TTA accepts arbitrary training methods in the source domain and thus wwwe-shhlff-moeeeeee can be adapted.
5.3 TEST-TIME ADAPTATION
Test-time adaptation (TTA) aims to adapt model trained on the source domain to the target domain without accessing the source data (Liang et al.) (023). TTA can be applied in a wider range of situations than SFDA and TTT in that TTA does not train additional models modify the model architecture and source pre-training. TTA for classification has attracted attention for its practicality. Various types of TTA methods have been developed.
Entropy-based. [ang et al. (202)) found that the entropy of prediction strongly correlates with accuracy on the target domain and proposed test-time entropy minimization (Tent), which is the most representative of the TTA methods. BACS (Zhou & Levine 2021). MEMO (Zhang et al.) 2022]. EATA (Niu et al. 2022) and DELTA (Zhao et al..| 2023) follow the idea of Tent and improve adaptation performance. T3A (Iwasawa & Matsuo 2021) adjusts the prototype in the feature space during testing. IST (Mal 2024) employs graph- based pseudo label modification. However, these TTA methods are designed for classification and cannot be applied to regression. For instance, computing entropy, which is widely adopted in TTA, requires a predictive probability for each class, whereas ordinary regression models only output a single predicted value. Thus, we investigate an approach that does not rely on entropy.
Feature alignment. Feature alignment is based the insight of UDA and makes the target feature distribution close to the source one. Since accessing the source data is restricted in the TTA setting, methods based the feature alignment match the statistics of the target features
those of the pre-computed source. BN-adapt (Benz et al. (222]) updates the feature mean and variance stored in batch normalization (BN) layers (loffe& & Szegedy 2015). DELTA (Zhao et al. 2023) modifies BN and introduces class- wise loss re-weighting. CFA (Kojima et al. 2022). BUFR (Eastwood et al.) 2022). CAFe (Adachi al.) 2023), and CAFA (Jung al. (2023) incorporate pre-computed source statistics. Although some of these methods are directly applicable to regression, we have observed that they are not effective or even degrade regression performance.
Other tasks. TTA for depth estimation (Li et al. 2023) super resolution (Deng et al. 2024), point cloud WWang et al.) 2024). and person re-identification (Adachi et al., 2024) are proposed. But they have task-specific architectures methods and cannot be applied to ordinary regression.
6 CONCLUSION
We proposed crrnificant-subspacc alignment (SSA), a novel test-time adaptation method for regres- sion models. Since we have found that the naive feature alignment fails in regression TTA because of the learned features being distributed in small subspace, incorporated subspace detection and dimension weighting procedures into SSA. Experimental results show that SSA achieved higher R² scores on various regression tasks than did baselines that were originally designed for classification tasks. We will extend TTA to further broader tasks and settings such as concept drift in the future.
10

---

Published as a conference paper at ICLR 2025
Ethics statement. The potential ethical concern is that model may have fairesss or bias issues in certain sensitive applications if the model adapts to biased target data. The model's I's behavior should be carefully monitored in such situation.
Reproducibility statement. Details on the datasets and experimental settings are described in Sec- tions 4.1 and 4.2 and Appendix C
We also provide the code in the supplementary material.
REFERENCES
Kazuki Adachi, Shin'ya Yamaguchi, and Atsutoshi Kumagai. Covariance-aware feature alignment with pre-computed source statistics for test-time adaptation to multiple image corruptions. In IEEE International Conference on Image Processing (ICIP), 2023.
Kazuki Adachi, Shohei Enomoto, Taku Sasaki, and Shin' Ya Yamaguchi. Test-time similarity mod- ification for person re-identification toward temporal distribution shift. In 2024 International Joint Conference on Neural Networks (ICCNN, pp. 1-8, 2024. doi: 10. 10.1109//CN600999...0 2024.. 13.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06400 2016.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175, 2010.
Philipp Benz, Chaoning Zhang, Adil Karjauv, and In So Kweon. Revisiting Batch Normalization for Improving Corruption Robustness. In IEEF/CVE Winter Conference on Applications of Computer Vision (WACV), 2021.
Rebekka Burkholz. Batch normalization is sufficient for universal function approximation in CNNs. In International Conference on Learning Representations (ICLR), 2024.
Xinyang Chen, Sinan Wang, Jianmin Wang, and Mingsheng Long. Representation Subspace Dis- tance for Domain Adaptation Regression. In International Conference on Machine Learning (ICML), 2021.
Tong Chu, Yahao Liu, Jinhong Deng, Wen Li, and Lixin Duan. Denoised maximum classifier discrepancy for source-free unsupervised domain adaptation. AAAI Conference on Artificial Intelligence, 2022.
Corinna Cortes and Mehryar Mohri. Domain adaptation in regression. In International Conference on Algorithmic Learning Theory, pp. 308-323. Springer, 2011.
Gabriela Csurka. A Comprehensive Survey on Domain Adaptation for Visual Applications, pp. 1- 35. Springer International Publishing, Cham, 2017, ISBN 978-3-319-58347-1. doi: 10.1007/ 778-3-319-583477
URL http:////............00.01333313
Shaveta Dargan, Munish Kumar, Maruthi Rohit Ayyagari, and Gulshan Kumar. A survey of deep learning and its applications: new paradigm to machine learning. Archives of Computational Methods in Engineering, 27:1071-1092, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Zeshuai Deng, Zhuokun Chen, Shuaicheng Niu, Thomas Li, Bohan Zhuang, and Mingkui Tan. Effi cient test-time adaptation for super-resolution with second-order degradation and reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR), 2021.
11

---

Published as a conference paper at ICLR 2025
D.C. Dowson and B.V Landau. The Fréchet distance between multivariate normal distributions. Journal of Multivariatee Analysis, ((3))450-4555 1982.
John Duchi. Derivations for Linear Algebra and Optimization, 2007. httt::/aai.stanorrr
edd/ http://jdccch/po/ettss//// http://eetteeeeeereeeeeeeeeeee sattrraannttteeeem pdf
Cian Fastwood, lan Mason, Chris Williams, and Bernhard Schölkopf. Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration, International Conference on Learning Representations (ICLR), 2022.
Shohei Enomoto, Naoya Hasegawa, Kazuki Adachi, Taku Sasaki, Shin Ya Yamaguchi, Satoshi Suzuki, and Takeharu Eda. Test-time adaptation meets image enhancement: Improving accu- racy via uncertainty-aware logit switching. In IEEE International Joint Conference on Neural Networks (IJCNN), 2024.
Gabriele Fanelli, Matthias Dantone, Juergen Gall, Andrea Fossati, and Luc Van Gool. Random Forests for Real Time 3D Face Analysis. Int. J. Comput. Vision, (01()))337-455 February 2013.
V. Fomin, J. Anmol, S. Desroziers, J. Kriss, and A. Tejani, High-level library to help with training neural networks in pytorch. https: /ttt::/gghttbb cwn/ppytorcchiggnnteeee 2020.
Jonathan Frankle, David Schwab, and Ari S. Morcos. Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs. International Conference on Learning Representations (ICLR), 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, 770):209--2030, 2016.
Gustavo A. Vargas Hakim, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Ismail Ben Ayed, and Christian Desrosiers. ClusT3: Information Invariant Test- Time Training. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Tianlang He, Zhiqiu Xia, Jierun Chen, Haoliang Li, and S-H Gary Chan. Target-agnostic source-free domain adaptation for regression tasks. In IEEE International Conference on Data Engineering (ICDE), 2024.
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. International Conference on Learning Representations (ICLR), 2019.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning (ICML), 2015.
Masato Ishii and Masashi Sugiyama. Source-free Domain Adaptation via Distributional Alignment by Matching Batch Normalization Statistics. arXiv preprint arXiv:2101. 2021.
Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
Sanghun Jung, Jungsoo Lee, Nanhee Kim, Amirreza Shaban, Byron Boots, and Jacgul Choo. CAFA: Class-Aware Feature Alignment for Test-Time Adaptation. In IEEE/CVE International Confer- ence on Computer Vision (ICCV), 2023.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Takeshi Kojima, Yutaka Matsuo, and Yusuke Iwasawa. Robustifying vision transformer without retraining from scratch by test-time class-coniiiional feature alignment. International Joint Conference Artificial Intelligence (ICCAI), 2022.
12

---

Published as a conference paper at ICLR 2025
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's 's thesis, University of Tront, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2012.
Stéphane Lathuilière, Pablo Mesejo, Xavier Glameda-Pinedee and Radu Horaud. A comprehensive analysis of deep regression. IEEE transactions on pattern analysis and machine intelligence, 42 (9):2065-222 2081, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-25 222222244, 1998a.
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. MNIST handwritten digit database, 1998b. URL htLL/tttp///aa.eeccut wtt.euuu..oo//ex//miiiii////
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model Adaptation: Unsupervised Domain Adaptation Without Source Data. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Zhi Li, Shaoshuai Shi, Bernt Schiele, and Dengxin Dai. Test-time Domain Adaptation for Monoc- ular Depth Estimation. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICMI., 2020.
Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey test-time adaptation under distri- bution shifts. International Journal of Computer Vision (ICCV), 2023.
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexan- dre Alahi. TTT++: When Does Self-Supervised Test-Time Training Fail Thrive? Advances in Neural Information Processing Systems (NeurIPS), 2021.
Jing Ma. Improved self-training for test-time adaptation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Torch Vision maintainers and contributors. Torchvision: Pytorch's 's computer vision library. https: 2016.
//qithub.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. UMAP: Uniform Manifold Approximation and Projection. The Journal of Open Source Software, 3(29):861, 2018.
Ismail Nejjar, Qin Wang, and Olga Fink. DARE-GRAM: Unsupervised Domain Adaptation Regres- sion by Aligning Inverse Gram Matrices. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
A Tuan Nguyen, Toan Tran, Yarin Gal, Philip HS Torr, and Atilim Güneş Baydin. KL Guided Domain Adaptation. In International Conference Learning Representations (ICLR), 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui
Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning (ICML), 2022.
Cam Nugent. the California Housing Prices Dataset. Kaggle, 2017. https: http:://wwwwaggeg kaggle.
com/ dataaaeeee//aauuggeetteaaaaaaaea oogg
David Osowiechi, Gustavo A. Vargas Hakim, Mehrdad Noori, Milad Cheraghalikhani, Ismail Ben Ayed, and Christian Desrosiers. TTTFlow: Unsupervised Test-Time Training With Nor- malizing Flow. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
13

---

Published as a conference paper at ICLR 2025
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vitito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorhh: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML), 2019.
Sunandini Sanyal, Ashish Ramayee Asokan, Suvaansh Bhambri, Akshay Kulkarni, Jogendra Nath Kundu, and Venkatesh Babu. Domain-speciffiity inducing transformers for source-free domain adaptation. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time train- ing with supervsuev for generalization under distribution shifts. In International Conference on Machine Learning (ICML), 2020.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully Test-Time Adaptation by Entropy Minimization. International Conference on Learning Rep- resentations (ICLR), 2021.
Yanshuo Wang, Ali Cheraghian, Zeeshan Hayder, Jie Hong, Sameera Ramasinghe, Shafin Rahman, David Ahmedt-Aristizabal, Xuesong Li, Lars Petersson, and Mehrtash Harandi. Backpropagation-fies network for 3d test-time adaptation. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2024.
Ross Wightman. Pytorch image models.
https://githab.com rwi rwttom/arrtttta
Bencchiimage-mode
2019.
Haifeng Xia, Handong Zhao, and Zhengming Ding. Adaptive adversarial network for source-fine -feee
domain adaptation. In IEEE/CVF International Conference Computer Vision (ICCV), 2021.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.
Shihao Zhang, Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving Deep Re- gression with Ordinal Entropy. In International Conference on Learning Representations (ICLR), 2023.
Zhifei Zhang, Yang Song, and Hairong Qi. Age Progression/Regression by Conditional Adversarial Autoencoder. In IEEE/CVF International Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2017.
Bowen Zhao, Chen Chen, and Shu-Tao Xia. DELTA: DEGRADATION-FRED FULLY TEST-TIME ADAPTATION. In International Conference on Learning Representations (ICLR), 2023.
Aurick Zhou and Sergey Levine. Bayesian Adaptation for Covariate Shift. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A LIMITATION
One limitation of SSA is that it assumes a covariate shift, where p(y|x) does not change. Addressing distribution shifts where p(y]x) changes, e.g., concept drift, will specifically be addressed target in future work.
B EVALUATION METRIC
We used the R score (coefficient of determination) measure the performance of the regression models. Rº is computed as follows:
Σ i= (Úi - y:) Σ i= (1/ 8)
R =
(10)
where OJ is a predicted value of the regression model, yj is the ground-truth and y = (1/N) Σ y/ the mean of the ground truth values. R² is close to when the regression model is accurate.
C EXPERIMENTAL SETTINGS
We used Py PyTorch (Paszke et al. 2019) and PyTorch-Ignit (Fomin al. 2020) to make the imple- mentations of the source pre-traning, proposed method, and baselines. We conducted the experi- ments with single NVIDIA A100 GPU.
C.1 DATASETS
SVHN Netzer al. 2011): We downloaded SVHN via torchvision SVHN. can be used for non-commercial purposes only[]
MNIST (LeCun et
1998b): We downloaded MNIST via torchvison. dat.iiiissonddtssttss....s MNIST.
We could not find any license information for MNIST.
UTKFace Zhang et al.
2017): ; We downloaded UTKFace via the official site It is available for non-commercial research purposes only.
Biwi Kinect (Fanelli et 2013): We downloaded Biwi Kinect via Kaggle
released under
Crribution-onCCommeee CannCCommeeria-NDDeevvviiiieiiie -NDeeivvatvees 4.0 International (CC BY-NC-ND 4.0) license.
California Housing (Nugent 2017 We downloaded the dataset from Kaggle It is released in public domain.
C.2 PRE-TRAINING ON THE SOURCE DOMAIN
SVHN: We trained ResNet-26 (He et al., 2016) from scratch. For optimization, used Adam (Kingma & Ba 2015) and set the learning rate to 0.0001, weight decay to 0.0005, batch size 64, and number of epochs to 100. We minimized the MSE loss between the predicted values and digit labels. For the implementation of ResNet-26, we used the PyTorch Image Models (timm) library (Wightman) 2019).
UTKFaсe: We randomly split the dataset into 80% for training and 20% for validation. We fine- tuned ResNet-50 pre-trained on ImageNet (Deng et al.) 2009). We minimized the MSE loss to predict the ages of the persons in the images. For optimization, we used the same hyperparameters
15

---

Published as a conference paper at ICLR 2025
the above SVHN case. For the implementation of ResNet-50, used torchvision (maintainers & contributors 2016) with WHAGENETIK.V2 initial weights.
Biwi Kinect: We split the dataset into male and female images and further randomly split them into 80% for training and 20% for validation. We fine-tuned ResNet-50 pre-trained on ImageNet in the same way the UTKe case. We separately trained the models to predict each of the three head angles, ie., pitch, roll, and yaw, of the persons in the images. In total, we pre-trained six source models ((male, female} {pitch, yaw, roll)).
California Housing: We extracted the data of non-coastal areas for the source domain and split them into 90% for training and 10% for validation. We standardized the whole dataset using the source mean and standard deviation. We trained five-layer MLP, with 100-dimensional hidden layers, ReLU activation, and batch normalization.
C.3 TEST-TIME ADAPTATION
As for setting the hyperparameters of the baseline methods, basically followed their original papers. For adaptation, we adopted an offline manner for fair comparison, i.e., we ran TTA for one epoch and then ran evaluation, which is widely adopted in existing TTA works (Wang et al., 2021 Zhou & Levine, 2021 Eastwood et al. (022). For the evaluation, we ran each TTA three times with different random seeds and reported the means of the scores.
Source: We simply fixed the source-pretrained model (i.e., model. eval () in PyTorch) and per- formed inference.
BN-adapt (Benz et al. 2021): updates the feature mean and variance stored in the batch normaliza- tion layers during testing, L.e., ran inference with model. () mode in PyToorch.
Feature restoration (FR) (Eastwood et al. 2022): uses the source statistics of the features and outputs as a form of dimension- wise histogram and aligns the target feature histogram to the source one. The original FR uses the histograms of the features and logits pre-computed with the source dataset. Since our focus is on regression models, we used the outputs instead of logits. We set the number of bins of the histograms to eight and the temperature of soft-binning to 0.01 following Eastwood et al. (022). We set learning rate to 0.0001 (this value gave the best score).
Prototype: We tweaked T3A (Iwasawa & Matsuo, 202T), which regards the weights of the last fully-comnected layer as the prototype of each class and updates the prototypes with the mean of the arriving target features during testing. We regarded of the linear regressor he single prototype and updated it with the mean of the target features. Although T3A determines whether to feature for making an update by using entropy, we omitted this component since we cannot compute entropy with regression models.
Variance minimization (VM): makes augmented views of input images and minimizes output vari- ance. This is modification of MEMO (Zhang et al. (022). which minimizes the marginal entropy of the augmented views of inputs in the same manner. We set the number of augmented views to 32 input and set the learning rate to 0.001 with Adam optimizer.
Representation subspace distance (RSD) (Chen al. 2021): The original RSD is UDA method designed for regression, which aligns the SVD bases between source and target mini-batcher batches. To adapt TTA, we pre-computed the SVD of the source features wit the whole source dataset instead of mini-barr batches before TTA and align the target feature SVD with the RSD loss during testing.
SSA (ours): Algorithm D lists the procedure of SSA. For optimization, we used Adam (Kingma & Ba (015) with learning rate- 0.001, (3,. ,2) (0.9, 0.999), and weight decay= O, which is the default setting in Py PyTorch (Paszke al. 2019
We set the batch size to 64 following other TTA
baselines.
Test-time training (TTT) (Sun et al. 2020): incorporates a self-supervised rotation prediction task during pre-training in the source domain; then it updates the feature extractor by minimizing the self- supervised loss during testing. The rotation prediicionnniii branch is a linear layer that takes a feature z and outputs four logits corresponding the rotation angles {0",90 180 270%) of an input image. For optimization, we used SGD and set the learning rate to 0.001 in accordance with Sun et al. 2020)
16

---

Published as a conference paper at ICLR 2025
Algorithm 1 Significant-subssace alignment (SSA).
Input: Pre-trained source model fe. source bases V*, source mean μu¹, source variances X target
dataset '
Output: Adapted model før
Compute weights for each dimension of the source subspace aa according to Equation for all mini-batch {x:} in T do
Extract target features {z] go (x))² Project target features } into {2:] according to Equation 6
Compute projected target mean ji' and variances 2º2 analogously Equation
Update & of the feature extractor Js to minimize Cita according to Equation
end for
Table 9: Comparison between using KL divergence and 2- Wasserstein distance for feature alignment in SSA on SVHN-MNIST. The top row is our method.
Metric
Subspace detection
R² (t)
RMSE (L)
KI.
0.51 10.03
2.02440.08
KL
0.338 10.04
2.355
2WD
V
0.425=0.02
2. 2.19640.01
2WD
0.342=0.04
2.348
LI
V
0.472
2.104
LI
0.347700.00
2.337 #0.11
Source
0.406
2.232
Domain adversarial neural network (DANN) (Ganin et al., 2016): is an unsupervised domain adaptation method which adversarially trains feature extractor and domain discriminator to learn domain-invarian features. We trained the domain discriminator during training in addition to the main regression model. We used layer4 of ResNet for the discriminator, We scheduled the learning rate and the weight of the domain adaptation loss by following Ganin et al. (2016).
Oracle: fine-tunes the model using labels during testing, i.e., the upper bound of the performance.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 METRIC FOR FEATURE ALIGNMENT
We used the KL divergence between two Gaussian distributions in Equation (3) for the feature alignment in SSA. One may suppose that other metrics could also be used, since the variance term in the denominator makes the naive TTA loss in Equation (2) unstable, mentioned in Section 3.1 Here, we tried the 2- Wassersteei distance (2WD) between two Gaussian distributions (Dowson & Landau 1982) and the LI norm of the statisties:
W
of)) = -A1 - 22 + (a1 o22
(11)
L1 (Wi,,,
|u1 -12 + |o -022-
(12)
We replaced the KL divergence with the 2WD and LI in Equation as follows:
LITA-WWD -x K
(N(7) xD) - K
a- ad (1i)2+ (V08² 1x
(13)
LITA-LI Ex= d=1 aaLi af) -x ad
K
K
V0z3 - V
(14)
d=1
Tables 99 to compare the effects of using the KL divergence and 2WD on -NIST. UTK- Face, and Biwi Kinect. The KI. divergence with subspace detection (SSA) achieved highest R²
17

---

Published as a conference paper at ICLR 2025
Table 10: Test R² scores of cases using KL. divergence and 2- 2-Wasserstein distance for feature align- ment in SSA on UTKFace. The top row our method.
Elastic transform
Jpeg compression
Gaussian noise
Defocus blur
Metion blur
Zoom blur
Impulse noise
Brightness
Contrast
Pixelate
Subspace Metric detection
Shot
Fog
Snow
Mean
KI.
0.803
0.839
0.851
0.899
0.529
0.943
0.580
0.592
0.560
0.517
0.731
KL
0.826
0.853
0.825
0.752
0.904
0.843
0.944
0.377
0.421
0,294
0.842
0.246
0.205
0.641
2WD
0.816
0.843
0.826
0.729
0.903
0.830
0.946
0.353
0.424
0.302
0.824
0.177
0.192
0.628
2WD
0.827
0.854
0.832
0.755
0.906
0.845
0.946
0.389
0.439
0.298
0.846
0.223
0.238
0.646
LI
V
0.834
0.858
0.851
0.775
0.910
0.849
0.949
0.484
0.516
0.489
0.845
0.334
0.206
0.687
0.830
0.854
0.840
0.744
0.905
0.847
0.942
0.362
0.418
0.288
0.848
0.233
0.277
0.645
Source
0.410 0.159 0.658
3.906
0.711 0.069
0.595
2.536
2.539
2.522
0.661
0.029
0.544
00.78
Table 11: Test R2 scores of cases using KL divergence and 2- 2-Wasserstein distance for feature align- ment in SSA on Biwi Kinect. The top row is our method.
Female
Male
Male - Female
Suhspace
Metric detection
Pitch
Roll
Yaw
Pitch
Yaw
Roll
Mean
KL
0.860
0.962.1000
0.513
0.869
0,886
0.575
0.778
KL
0..05
0.945==0.00
0.240 0.03
0.835
0.874:0.01
0.61310.01
0.67220.01
2WD
0.70810.05
0.954_10.00
0.46550.01
0.76510.01
0.91610.01
0.61710.00
0.73810.01
2WD
0.5540...05
0.009
0.279
0.829
0.8621..02
0.59810.00
0.6761..01
LI
0.700
0.05811.00
0.48220.00
0.85810.00
0.92210.00
0.64110.00
0.76810.01
LI
0.562=20.0
0.04922.00
0.314
0.80220.00
0.861 20.02
0.613
0.6844a.01
Source
0.759
0.956
0.481
0.763
0.791
0.485
0.706
scores in almost all cases. In contrast, the 2WD variant of SSA produced only a slight improvement over Source SVHN-MNIST (Table and sometimes it had even worse scores than Source on Biwi Kinect (Table TT.. This degradation of 2WD is because the scale of the variance oî different among feature dimensions d. The KL divergence absorb the difference in scale since it includes the ratio of the variances, as in Equation
D.2 FEATURE VISUALIZATION
Figure illustrates PCA visualizations of the source and target features after TTA. In the visual- izations of SVHN-MNIST (a) and Biwi Kinect (c), can see that SSA makes the target feature distribution within the source distribution while the target features of the baselines protrude from the source distribution. In UTKFacee (b), the target features of Source significantly degenerate to single point, but the other methods alleviate this. In Figure 4 we also report the optimal transport distance (OTD) between the features of the both domains, which evaluates the feature alignment quantitatively. In addition to that our SSA alleviates the feature distribution gap in the subspace, SSA retains the source subspace better, as shown in Figure of Section 4.3.4
We also visualized the source and target features with UMAP (Melnnes et al. 2018 in Figure see the relation not limited within the top-2 principal components. We first trained the UMAP mapping with the source features projected onto the top K = 100 dimensional principal component space before TTA, which are shared among TTA methods. Then, we mapped the target features after TTA with the learned UMAP mapping. We can also see that the target feature distribution becomes closer to the source one by our SSA.
D.3 EFFECT OF ORIGINAL FEATURE DIMENSIONS ON THE SUBSPACE
As mentioned in Section 3.1 many of the feature dimensions have small effect the subspace, which makes the naive feature alignment ineffective. To verify the effect of changing the original features on the subspace, we computed the gradient of subspace feature vector & with respect to
18

---

Published as a conference paper at ICLR 2025
CTD:
QTD: 34.2
Source laget
Source
-10
Squrcce
-22
10
-00
40
10.0
-50
-25
6.
5.0
CTD: 0.266
OTD: 1.73
BN- adaat
Source Wagge
10
0+
-5
brrat
20
60
10.0
7.5
255
ca
2.5 5.0
QTT:: 6.91
Source Target
FR
-20
26
QTD: 0.126
OTD:
OTD: 1.79
SSA (Ours)
Soarce Target
10
Target
10
D+
Seu 10 Torget
-00
-10.0
-50
-25
co 25 5.0
(a) SVHN-MNIST
(b) UTKeeaee
(c) Biwi Kinect
Figure 4: PCA visualizations of source and target features of each dataset and method. The blue and orange dots represent the source and target features, respectively. We also report the optimal transport distance (OTD) between the principal components of the source and target features.
Source
Target
12
10-
Scurce Targer
Scurce Target
25
-25
0.5
25
3/0
7.5
10.0
125
-00
10
20
SSA (Ours)
Sourco Target
10
10
Source
Source
-10
-10
-25
00
23
5.0
7.5
30.0
12.5
-10
0
10
(a) SVHN-MNIST
(b) UTKFace
(c) Biwi Kinect
Figure 5: UMAP (Meinnes et al. 2018) visualizations of source and target features on each dataset. The blue and orange dots represent the source and target features, respectively.
the d-th dimension of the original feature za. With Equation (6). the norm of the gradient is:
Oz
8d
[I(v
Dall2 II[vi, .o......... ... 2222
(15)
Ozd
which is the norm of the d-th row of V
Figure 6 shows the histograms of 8a computed with the three datasets. As expected, most of the dimensions the original feature space had small effects; only a few dimensions had significant effect to the subspace. Specifically, SVHN-MNIST and Biwi Kinect strongly showed this tendency. In contrast, a larger number of raw feature dimensions affected the subspace in UTKFace compared with the other datasets. This is why the test R² score improved from Source without subspace detection UTKFace, while the scores on the other datasets were lower than those of Source, as shown in T
19

---

Published as a conference paper at ICLR 2025
1000
200
1000
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
(a) SVHN-MNIST
(b) UTKFace
(c) Biwi Kinect
Figure 6: Histograms of the gradient of a projected feature with respect to each original feature
dimension za computed with Equation 15
Table 12: Test R2 scores of SSA for different numbers of feature subspace dimensions on Cali- fornia Housing.
5
10
25
50
75
100
Ra
0.496
0.625
0.639
0.625
D.4 ADDITIONAL ABLATION
Table 12 shows the R scores on California Housing with changing K of SSA (the same experiment as Table 8) K 10 produced the best result, but K 5 and 25 also gave competitive scores. When 2 50, which exceeds the number of the subspace dimensions Table the loss became unstable or diverged because SSA attempted to align too many degenerated feature dimensions, as mentioned in Section 4.3.3
Tables 13 and 14 shows the test R2 scores without subspace detection (i.e., naively aligning raw features) with respect to K. Without subspace detection, the performance regression on the target domain is limited even when all dimensions of the original feature space aligned. This result supports the importance of aligning the representative feature subspace with subspace detection.
D.5 EXPERIMENTS ON VISION TRANSFORMER
We experimented TTA with vision transformer (ViT) Dosovitskiy et al. 2021) on SVHN-MNIST.
First, trained ViT-B/16 regressor SVHN and computed the number of feature dimensions as Section 4.3.4. We used the output of the penultimate layer for the features.
Table 15 shows the numbers of the valid (zero variance) and subspace dimensions. Since activa- tion function is applied to the feature vectors of ViT's unlike ResNets, all 768 feature dimensions are valid (non-zero variance). However, the number of the subspace dimensions is 503 the regression model, which is smaller that of the classification model. Although the subspace dimension is larger than the ResNet cases Table we can also see the same tendency.
Table 16 shows the TTA results on SVHN-MNIST. SSA outperforms the baselines.
D.6 MULTI-TASK REGRESSION MODEL
We applied TTA methods to multi-task regression models, which output multiple prediction values the same time. We trained ResNet-50 regression models on Biwi Kinect. Unlike the experiments single regression model outputs pitch, roll, and yaw angles.
on the same dataset in Section 4.3
Table 13: Test R scores of SSA without subspace detection. with subspace detection.
The right column represents the score
K
10
25
50
75
100
10 (ours)
R²
0.524
0.605
0.625
0.620
0.584
0.639
20

---

Published as a conference paper at ICLR 2025
Table 14: Test R² scores of SSA without subspace detection. The bottom row represents the scores
with subspace detection of K 100 from Table 8
K
SVHN
UTKFaae
Biwi Kinect
10
0.302
0.674
0.737
0.348
0.656
0.674
50
0.334
0.651
0.691
75
0.329
0.645
0.578
100
0.338
0.641
0.672
200
0.343
0.643
0.670
400
0.643
0.709
1000
0.656
0.755
2048
0.706
0.753
100 (ours) 0.511
0.731
0.778
Table 15: Number of valid (having non-zero variance) feature dimensions and feature subspace
dimensions of the ViT trained on SVHN.
Regression
Classification
#Valid dims #Subspace dims.
#Valid dims
#Subspace dims.
768
503
768
544
Table 17 shows the test R² score on each target angle. We compared TTA methods that can be easily extended to multi-task regression tasks. Our SSA outperformed the other baselines in most cases. The numbers of the source feature subspace dimensions were 134 and 132 for the male and female models, which larger than single-task models but much smaller than the entire feature dimensions (2048). Thus, SSA is also effective for multi-task regression models.
D.7 COMBINATION WITH CLASSIFICATION TTA
Although our SSA designed for regression, the subspace detection and feature aligmment loss can be used to classification since they operates in the feature space and works regardless of model output forms. Here, we applied the subspace detection and feature alignment loss classification models. We trained ResNet-26 on SVHN and CIFARIO (Krizhevsky et al., 2009). and tested on MNIST and CIFARIO-E (Hendrycks & Dietterich 2019).
Table 18 shows the number of the valid feature dimensions and subspace dimensions in classification and regression models on the source datasets. Although number of the feature dimensions the classification models are larger than those of the regression models, the subspace is smaller than the entire feature space. Thus, we expect that our SSA also effective on classification models.
Tables 19 and 20 show the classification accuracies. We can easily combine our method with entropy- based TTA methods in classification. Although SSA significantly improves the accuracy, combining SSA with Tent (Wang et al. 2021) further boosts the accuracy.
Table 16: Test I22 score and RMSE on SVHN-MNIST with VIT.
Method R2 (t)
RMSE (L)
Source
0.658
1.693
Prototype
0.468
2.111+0.00
FR
0.724+0..00
1.522+0.01
VM
1000
92.02+0.54
SSA (ours)
0. 0.741
1.471
Oracle
0.960+0.00
0.575+0.02
21

---

Published as a conference paper at ICLR 2025
Table 17: Test R2 scores on multi-task Biwi Kinect (higher better). The best scores are in bold- face.
Female -
Male
Male
Female
Method
Pitch
Yaw
Roll
Pitch
Yaw
Roll
Mcan
Source
0.904
0.628
0.960
0.794
0.419
0.854
0.760
BN-adapt
0.91220.00
0.60640.00
0.96740.00
0.791
0.40
0.921
0.778
VM
1.875
1.962 73
0.0993
-0.365
-0.1.. 0.1443=0.12
-0.00020.00
-0.740
RSD
0.912
0.606
0.967 +0.00
0.792
0.112200.00
0.921 10000
0.778
SSA (ours)
0.913=0.00
0.555
0. 0.970
00.00
0.540.20.om
0.942 o.
0.793
Oracle
0.976+0.00
0. 0.85940.00
0.988
0.958
0.804_o.00
0.979
0.928
Table 18: Comparison of the numbers of valid (having non-zero variance) feature dimensions and feature subspace dimensions (i.e., the rank of the feature covariance matrix) between classification and regression. When training classification models, we discretized the labels.
Classification
Regression
Dalasel
#Valid dimss
#Subspace dims..
#Valid dims
#Subspace diims.
SVHN
1946
CIFARIO
1521
UTKFacc
2048
1471
2041
Biwi Kineet (mean)
2048
277
713
34.5
California Housing (100 dims.)
100
100
45
40
D.8 HYPERPARAMETER SENSITIVITY
investigated SSA's 's sensitivity to other hyperparameters. Tables 21 and Tables to 26
show the results when varying the learning rate and batch size, respectively.
Typical ranges of the learning rate and batch size produce competitive performance. For the batch size, larger batch size results in better performance since the estimation of feature mean and vari- ance becomes more accurate. But batch sizes 2 16 produce competitive performance.
D.9 ADDITIONAL. RESULTS
Tables 27 and provide the performance measured by MAE on UTKFace and Biwi Kinect, which are corresponding to Tables 5
D010
ONLINE SETUP
We evaluated regression TTA in an batched online setting, where model update and evaluation are performed alternatively with batch in every iteration. Tables 3o 3o the results. Our SSA can outperform the baselines also the batched online setting.
Table 19: Test classification accuracy (%) on SVHN-MNIST.
Method
Accuracy
Source
53.82
Tent
75.68
SSA
79.97+0.67
Tent+SSA
80.69
Oracle
97.48+0.0p
22

---

Published as a conference paper at ICLR 2025
Table 20: Test classification accuracy (%) on CIFARIO-C. The best and second scores are in bold- face and underlined.
compression
Brightness
Contrast
Defocus
Elastic transform
Gaussian
noise
Impulse
Jpeg
Motion blur
Pixelate
blur
Fog
noise
Shot
noise
Snow
Zoom
blur
Method
Mcan
Source
77.80
19.97
57.74
72.33
47.62
69.06
46.97
79.82
58.92
76.88
70.44
72.18
60.75
62.34
Tent
79.06
56.88
77.00
73.70
67.41
76.98
70.02
79.22
74.28
77.80
77.70
75.27
77.08
74.03
SSA
81.34
64.98
79.11
74.48
71.89
78. 44
70.48
79.63
76.03
79.48
78.91
76.55
77.80
76.09
Tent+SSA
81.43
65.26
65.
79.30
74.61
71.98
78.41
70.59
79.57
76.13
79.51
78.96
76.50
77.84
76.16
Oracle
85.62
77.57
83.93
79.59
78.68
83.15
76.42
84.09
81.45
84.27
83.60
81.94
83.28
81.82
Table 21: Test R2 scores of SSA on SVHN-MNIST with different learning rates. Higher is better.
Learning rate
0.0001
0.0005
0.001
0.005
0.01
R2
0.47220.00
0.51610.01
0.509
0.510=0.07
0.212
Table 22: Test R² scores of SSA on UTKFace with different learning rates, Higher is better.
compression
Defiveus
blur
Motion
Zoom
blur
Contrast
Elastic
transform
Ipeg
Pixolate
Gaussion
nolse
Inpolke
noise
Shot
noise
Brighness
Fog
Snow
Learning rate
Mean
0.0001
0.781
0.828
0.836
0.755
0.895
0.820
0.941
0.526
0.534
0.469
0.860
0.420
0.474
0.703
0.0005
0.786
0.828
0.834
0.767
0.893
0.820
0.938
0.551
0.566
0.513
0.961
0.434
0.508
0.715
0.001
0.791
0.830
0.835
0.769
0.891
0.822
0.936
0.569
0.585
0.541
0.961
0.116
0.510
0.722
0.005
0.815
0.723
0.877
0.815
0.918
0.587
0.406
0.01
0.769
0.801
0.769
0.668
0.863
0.792
0.899
0.523
0.571
0. 0.161
0.811
0.327
0.293
0.658
Table 23: Test R² scores of SSA on Biwi Kinect with different learning rates. Higher is better.
Female >
Male
Male + Female
Learning rate
Pitch
Roll
Yaw
Pitch
Roll
Yaw
Mean
0.0001
0.787
0.955
0.505
0.842
0.849
0.581
0.753
0.0005
0.840
0.958
0.519
0.861
0.875
0.577
0.772
0.001
0.859
0.962
0.515
0.869
0.889
0.571
0.777
0.005
0.877
0.963
0.492
0.859
0.893
0.549
0.772
0.01
0.879
0.960
0.484
0.849
0.870
0.491
0.756
Table 24: Test R² scores of SSA on SVHN-MNIST with different batch sizes. Higher is better
Batch size 8
16
128
256
122
0.353+0.04
0.497+0.01
0.505+0.02
0.509
0.528
0.522+0.01
Table 25: Test R2 scores of SSA on UTKFace with different batch sizes. Higher is better.
compession
locus
blur
Motion
Zoom
Contrast
Elastic
transfom
Jpeg
Pixdate
Gaussian
Impulse
noise
Shot
moise
Brighmess
Snow
Batch size
Mean
8
0.804
0.796
0.651
0.867
0.499
0.516
0.416
0.834
0.668
16
0.782
0.809
0.808
0.699
0.874
0.507
0.915
0.545
0.562
0.476
0.848
0.421
0.421
0.690
0.792
0.821
0.530
0.749
0.885
0.816
0.929
0.566
0.583
0.333
03440
0.858
0.446
0.482
0.715
0.791
0.830
0.835
0.769
0.891
0.822
0.936
0.569
0.591
0.446
0.510
0.722
128
0.788
0.828
0.835
0.780
0.893
0.822
0.939
0.581
0.590
0,545
0.862
0.445
0.523
0.725
256
0.789
0.823
0.836
0.770
0.896
0.821
0.940
0.540
0.594
0.531
0.870
0.522
0.723
23

---

Published as a conference paper at ICLR 2025
Table 26: Test R scores of SSA on Biwi Kinect with different batch sizes. Higher is better.
Female +
Male
Male >> Female
Batch size
Pitch
Roll
Yaw
Pitch
Roll
Yaw
Mean
0.872
0.930
0.450
0.819
0.770
0.468
0.718
16
0.870
0.953
0.463
0.858
0.857
0.525
0.754
32
0.868
0.961
0.498
0.868
0.886
0.562
0.774
64
0.859
0.962
0.515
0.869
0.889
0.571
0.777
128
0.841
0.960
0.524
0.863
0.883
0.569
0.773
256
0.819
0.957
0.521
0.855
0.870
0.583
0.768
Table 27:
Test MAE scores on UTKFace with image corruption (lower better). The best scores are bolded.
compression
Brightness
Defocus
Motion
Zoom
Contrast
Elastic
trans
Jpeg
Pixelate
Gaussian
noi
Impulse
noise
Shot
noise
Snow
Fog
Method
Mcan
Source
10.12
12.64
7.68
39.99
7.92
13.91
9.65
32.00
32.02
31.92
8.81
15.52
19.76
18.61
DANN
9.44
8.88
8.73
19.59
7.52
8.01
6.09
41.26
35.56
38.21
9.11
16.07
18.11
17.43
TTT
6.78
6.55
6.61
6.51
5.77
6.50
5.13
9.56
9.46
10.00
6.51
10.96
7.71
BN-Adapt
7.23
6.91
6.69
7.59
5.97
6.81
5.40
10.37
10.32
10.99
6.44
11.51
10.55
8.21
Prototype
21.52
21.62
21.65
20.00
21. 21.28
21.02
21.27
18.46
18.45
18.42
21.71
20.71
20.70
20.52
FR
6.12
5.47
5.15
6.29
4.47
5.85
3.15
9.99
9.87
10.53
5.03
11.04
10.36
7.18
VM
28.28
28.14
28.42
27.45
27.75
27.74
26.97
29.07
29.21
29.05
27.80
29.50
29.19
28.35
RSD
6.35
5.68
5.25
6.60
4.61
5.93
3.34
10.26
10.35
10.94
5.12
11.13
9.98
7.35
SSA (ours)
6.05
5.52
5.09
6.05
4.46
5.72
3.27
9.05
8.95
9.29
5.01
10.60
9.50
6.81
Oracle
5.21
4.55
4.53
4.96
3.96
4.92
2.64
8.44
8,28
8,42
4.45
10.03
8.07
6.03
Table 28: Test MAE scores on Biwi Kinect (lower is better). The best scores are bolded.
Female +
Male
Male +
Female
Method
Pitch
Roll
Yaw
Pitch
Roll
Yaw
Mcan
Source DANN
0.150
0,081
0,087
0.160
0.171
0.093
0.124
1..00
0. 13610.01
0.181 18
0. 16.
0. 14.
0. 1.99..0.00
TTT
0.203
0.23620.00
0.1 0.14040.00
0,165
0.212_0.00
0.190
0.205
BN-adapt 0.146:20.00
0.085
0.090
0,153
0,134_0.00
0. 0.090=0.00
0.116+0.00
Prototype
6.935
6. 6.935
FR
0.37620.05
0.192220.01
0,248
0,212
0.150_o.01
0.182
0.226.0.01
VM
0.367 =0.00
0.407 20.00
0. 0.13640.00
0.418
0,463_0.00
0.135
0.321
RSD
1..122.0.01
0.08510.00
0.090
0.15410.00
0.132_a.00
0.1 10.00
SSA (ours)
0.112=20.00
0.07920.0m
0.086 oo
0.141 :0.00
0.126:00.00
0.090=a..0
0.106
Oracle
0.054.1..0
0.0544...0
0.05910.00
0.076
0.068
0.065
0.063
Table 29: Test scores on SVHN-MNIST in the batched online setting,
The best scores are bolded.
Method
R²(()
RMSE (L)
MAE (L)
Source
0.406
2.232
1.608
TTT
0.296+0.01
2.430+0.01
1.587+0.01
BN-adapt
0.38440.00
2.272=0.00
1.480+0.00
Prototype
0.48441.00
2.080=0.00
1.48940.00
FR
0.342.1..00
2.348 10.00
1.657
VM
43.729
37.021
RSD
0.312++..0
2.397=0.15
1.607
SSA (ours)
0.488+0.01
2.072
1.265+0.02
Oracle
0.745....00
1.463
0.88210.01
24

---

Published as a conference paper at ICLR 2025
Table 30: Test R² scores on UTKFaae with image corruption in the batched online setting. The best scores are bolded.
com compression
Brighttnss
Defocus
bler
Zoom
Mur
Comtroxt
Elottic
Jpeg
Pixelanc
Gaussian
10000
Impalee
D0000
Shot
Snow
Method
Mean
Source
0.410
0.187
0.658
-3.000
0..00
0.060
0.5415
-2.536
-2.530
-2.022
0.061
0.018
-0.443
-D-676
0.758
BN-Adapt
0.726
0.738
0.757
0.719
0.822
0.774
0.849
0.492
0.501
0.451
0.788
0.375
D.137
0.650
Prototype
-1001
-1.017
-1.014
-0.719
-0.965
-0.907
-0.972
-0.514
-0.513
-0.511
-1003
-0.523
-0.522
-0.829
0.409
VM
-1.457
-1.401
-1.503
-1 230
-1 327
-1 1.343
-1.251
-1.588
-1.538
-1.584
1.330
-1 1632
-1 -1.577
-1.136
RSD
0.783
0.529
0.543
0.761
0.895
0.820
0.940
0.503
0.500
0,147
0.858
0.416
0.455
0.699
SSA (ours)
0.794
0.834
0.841
0.789
0.896
0.823
0.953
0.550
0.304
0.624
0,861
0,427
0.520
0.721
Oracle
0.825
0.968
0.963
0.824
0.908
0.845
0.951
0.578
0.586
0.577
0.872
0.472
0.605
0.752
Table 31: Test R2 scores on Biwi Kinect the batched online setting. The best scores are bolded.
Female +
Male
Male +
Female
Method
Pitch
Roll
Yaw
Pitch
Roll
Yaw
Mean
Source
0.759
0.956
0.481
0.763
0.791
0.485
0.706
TTT
-0.207
0.610ga.00
0.010g0.02
0.743
0.722
0..29620.00
0.264.o.01
BN-adapt
0.750
0.951
0.487
+0.00
0.837 #0.00
0.567 20.00
0.738
Prototype
-317.740.n.c2
FR
-0.1244n.06
0.818
2.049 2.049
0.775
0.852
0.128
0.067 #0.06
-0.242.0..z
20.00
20.00
00.00 10.00
0.0051 10.00
0.00664..co
0.091 10.00
RSD
0.768
0.951
0.48020.00
0.826
0.838
SSA (ours)
0.82510.00
0.957
20.00_0.00
0.50210.00
0.85310.00
0.86510.00
0.56710.00
0.761 10.11.000
Oracle
0.923
0.971
0.67220.00
0.925
0.934.o.00
0.717
0.857
25


Reviews:
[{"review_id": "4PANmSjgqF", "reviewer": "Reviewer_t9WM", "summary": "This paper addresses a crucial yet under-explored area in machine learning: test-time adaptation (TTA) for regression models. The authors rightly highlight the limitations of existing TTA methods, which are primarily tailored for classification tasks, and propose a novel approach termed Significant-subspace Alignment (SSA). The paper presents a clear rationale for the need to adapt TTA techniques for regression and offers a compelling solution through feature alignment.", "questions": "1. Why are \"two diagonal Gaussian distributions\" used in Chapter 3? What is the rationale behind this choice?\n2. In Section 4.1, the first dataset utilizes a classification dataset as a regression task. Is this approach reasonable and justifiable? Regarding the second dataset, is it appropriate to consider a noisy version of the test set as the target domain?\n3. Could you provide a proof or evidence demonstrating the effectiveness of Significant-subspace Alignment (SSA)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The topic of adapting regression models to unknown target distributions is highly relevant, particularly given the increasing use of machine learning in diverse applications. The proposed SSA approach is innovative and addresses a gap in the literature.", "weaknesses": "1. In Section 3.1 of the article, the use of \"two diagonal Gaussian distributions\" is not adequately justified, and the advantages and limitations of employing such distributions are not discussed.\n2. The proposed method in this paper exhibits limited originality, as its various components are commonly found in existing models.\n3. The dataset utilized in this study has a relatively small sample size, which weakens the persuasiveness of the findings. Additionally, there is a lack of comparative algorithms from the past three years, resulting in insufficient theoretical validation. Furthermore, the measurement standard is solely based on R²; incorporating additional metrics such as RMSE and RMAE would enhance the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GMVvJjbRko", "reviewer": "Reviewer_66U8", "summary": "This paper investigates the application of Test-Time Adaptation for regression tasks, where a regression model pre-trained in a source domain is adapted to an unknown target distribution using unlabeled target data. The authors note that most existing TTA methods are designed for classification tasks and do not directly apply to regression models, which typically output single scalar values rather than class-categorical predictions. To address this, the paper proposes a feature alignment approach called Significant-subspace Alignment, which consists of two components: subspace detection and dimension weighting. SSA aims to align feature distributions between the source and target domains to mitigate the domain gap, focusing on a representative and significant subspace of the feature space. Experimental results on various real-world datasets demonstrate that SSA outperforms several baseline methods.", "questions": "How does SSA perform in multi-task and multi-class regression tasks? Can it be extended to these scenarios?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.SSA introduces a novel approach for TTA in regression tasks by combining subspace detection and dimension weighting, which is an innovative contribution to the field.\n2.The paper conducts extensive experiments on multiple real-world datasets, validating the effectiveness of SSA.\n3.Compared to the original model and other baseline methods, SSA achieves higher R2 scores across multiple datasets, demonstrating performance improvements in regression tasks.", "weaknesses": "1.SSA assumes covariate shift, where p(y|x) remains unchanged. The paper does not address distribution shifts where p(y|x) changes, such as concept drift, limiting its applicability in broader scenarios.\n2.While the paper mentions the selection of the parameter λ, it lacks a detailed discussion on the impact of other hyperparameters, which could affect the model's generalizability and adaptability.\n3.The paper does not discuss the computational complexity of SSA, particularly its performance with large-scale datasets, which is crucial for practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6rqHCgpwGb", "reviewer": "Reviewer_xN4r", "summary": "The paper proposes a test-time domain adaptation method for regression. They show that learned embeddings are more compressed in regression than classification. To address this, they introduce Significant-subspace Alignment (SSA). SSA uses PCA to detect a significant feature subspace where features are concentrated, then aligns the first two moments of source embeddings and target batch within this subspace. They also apply dimension weighting to prioritize subspace dimensions based on their significance to the output. The proposed method is evaluated on four regression datasets and demonstrates superior performance compared to baseline TTA methods designed for classification tasks, and domain adaptation for regression adapted for TTA .", "questions": "See Weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a novel method for Test time domain adaptation for regression, and adapted common benchmarking procedures for classification to regression, especially on the UTK Face dataset.\n\n- The paper is well-written and clear.\n\n- The method and results seem convincing and easy to implement.", "weaknesses": "- In the related work, feature alignment methods for TTA are reviewed with the claim, “Although some of these methods are directly applicable to regression, we have observed that they are not effective or even degrade regression performance.” However, these methods are not included in the experiments, leaving this claim unsupported.\n- In Table 1, the subspace dimension for Biwi Kinect is shown as ‘34.5’, which is unclear, as dimensions are typically integers.\n- MLPs and CNNs seem to produce embeddings with different ranks, with MLPs showing less low-rank behavior. A discussion or remark on this difference could provide useful insights.\n- The test setup lacks clarity on whether SSA is applied in an online or episodic adaptation manner, and it’s unclear if batches are fed uniformly.\n- The dimension weighting approach may lead to disproportionately high contributions from small-scale features if they are assigned high weights (when learned on source), potentially resulting in values similar to large-scale features with lower weights (when learned on source).   Did you consider any normalization or balancing mechanism to prevent this scaling effect?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cVdaWQm1E7", "reviewer": "Reviewer_gBNN", "summary": "This paper investigates test-time adaptation (TTA) for regression, where a regression model pre-trained in a source domain is adapted to an unknown target distribution with unlabeled target data. To enable TTA for regression, the authors adopt a feature alignment approach, which aligns the feature distributions between the source and target domains to mitigate the domain gap. A propose Significant-subspace Alignment (SSA) is proposed for feature alignment in TTA for regression, which consists of two components: subspace detection and dimension weighting. Some experiments have been conducted to verify the performance of the proposed method.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tA Significant-subspace Alignment (SSA) method is proposed to address TTA for regression on the basis of the feature alignment approach.\n2.\tSSA consists of two components: subspace detection and dimension weighting. \n3.\tSubspace detection uses principal component analysis (PCA) to find a subspace and dimension weighting raises the importance of the subspace dimensions.", "weaknesses": "1.\tThe description of Figure 1 is too simple to understand the procedure of the proposed method. Some key variables (i.e., gφ and hψ) should be introduced. For easy understand, I suggest the authors provide a step-by-step description of the workflow shown in the figure. \n2.\tThe used compared methods are all published before 2023, how the proposed method compares to or differs from some newer approaches, such as “Backpropagation-free Network for 3D Test-time Adaptation, CVPR 2024”, “Improved Self-Training for Test-Time Adaptation, CVPR, 2024”.\n3.\tWhy no results for RSD are given in Table 3? Please provide the missing results for RSD in Table 3 or explain why these results were not included.\n4.\tWhat is the visualization effect of the proposed method? Provide T-SNE rendering.\n5.\tWhat is the limitation of the proposed method? Are there any situations where SSA might not perform well, or any assumptions it makes that might not hold in all regression tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)