Year: 2021

Abstract:
A new influential task called imbalanced regression has received a great deal of attention. Different from previous works, this task particularly aims to solve the imbalanced problem in the regression world. The exploration of this task is still at a preliminary stage, so more attempts are needed. In this paper, we first discover that the benefit of Self-supervised Learning (SSL) can help reveal the bias caused by the severe skewness in the training data. Therefore, we work on a seamless marriage of imbalanced regression and SSL. But with this comes the first question of how to measure the similarity and dissimilarity between raw and augmented samples under the regression sense, for which the definition is clear in the classification. To tackle this issue, we provide a formal definition of similarity and dissimilarity in the regression task. In addition, a common practice in SSL to produce augmented samples is to put noise onto the raw samples; the second problem is, it is not guaranteed that the augmented samples are similar to original samples when scaling to a deep network by directly adding random noise to the input. To approach this problem, we specifically limit the volume of noise on the output to get a meaningful augmented sample by back propagation. Experimental results show that our approach achieves state-of-the-art performance.

heading: 1 INTRODUCTION
text: Regression is a fundamental task in machine learning and statistical analysis, which involves modeling the relationship, between one or more independent and dependent variables. As distinct from the discrete output values in classification, regression makes predictions for continuous values. The issue of imbalanced recognition, arising from the conjunction of imbalance and classification, has been drawing the attention of researchers recently (Shu et al., 2019; Cao et al., 2019; Ren et al., 2020; Jamal et al., 2020; Kang et al., 2020; 2021). By analogy, it is natural to question whether long-tailed regression exists (or one can also call it imbalanced regression, the equivalence of these two names in the paper). Exactly what we expected, a fresh task, namely long-tailed regression, has recently been proposed by Yang et al. (2021). Regardless of the newness of the task, there is no way to neglect its immense significance in terms of application and research. For example, age estimation from facial images; health indicators for various populations in healthcare (Soualhi et al., 2014), these quantities usually exhibit a long-tailed distribution. Going further, in terms of performance, progress in the long-tailed regression research is also far from satisfactory.
More recently, self-supervised learning (SSL) has emerged as a new rage in the research community, owing to its straightforward learning paradigm yet excellent performance (Noroozi & Favaro, 2016; Gidaris et al., 2018; Wu et al., 2018; He et al., 2020). The excellence of the approach has been admired not only on classification with well-balanced datasets, but also surprisingly on long-tailed classification by Kang et al. (2021); Wang et al. (2021); Cui et al. (2021); Samuel & Chechik (2021) who found its benefits. It prompts some thoughts on the potential and practical applications, in particular, applications to SSL for long-tailed regression (Yang et al., 2021). Mindful that the majority of available techniques on SSL are in the context of classification (Oord et al., 2018; Hjelm et al., 2018; Chen et al., 2020a; Henaff, 2020). Due to the difference between classification and regression (no hard boundary existing between outputs in regression task as shown in Figure 1a) (Razi & Athappilly, 2005; Breiman et al., 2017; Liaw et al., 2002; Yang et al., 2021), it is not guaranteed that SSL can help address the imbalanced regression problem well, and it may be difficult to make a straightforward migration from classification to regression task. Upon deeper pondering, there are
obstacles and problems that we face: Can SSL really relieve the long-tailed regression problem? If so, how might we solve the imbalanced regression problem by utilizing SSL?
Figure 1a illustrates the motivation, by adding noise to the image of the cat, it is very unlikely to be misclassified into a class that differs significantly from it, such as a bird, since they are discrete and the categories diverge greatly from each other. In the case of regression, however, it is not at all feasible when a similar migration was made. We imagine that if noise is added to a face image, then the network’s estimation of the person’s age may be significantly biased. In this regard, it becomes essential to work out and comprehend the degree of noise that induces “similarity and dissimilarity” amongst output values from a regression standpoint. On the foundation of this definition, questions are raised about exploiting noise sensibly and effectively.
With this paper, we try for the first time to propose answers to these questions. And we summarize our contributions as follows:
• To the best of our knowledge, we are the first to broaden the concept of similarity and dissimilarity between augmented samples and original samples from classification to regression. And we are also the first to reveal imbalanced regression problem with SSL.
• Having been convinced of the beneficial effect of applying SSL on a simple experiment, we extend it to the neural network-based long-tailed regression task. To ensure that the noise is manageable w.r.t. the similarity of the regression, we propose to generate noise on regression values within a predefined threshold.
• We obtain an optimization problem on noise with an efficient approximate solution algorithm based on a first-order Taylor expansion. The augmented input samples are obtained by back-propagation.
• As a practical trial for the proposed method, the results reveal that we attained not only the best results, but also the possibility to mix it freely with other training techniques.

heading: 2 PRELIMINARY
text: In this section, we cover the description of imbalanced regression, and then, with reference to the concept of similarity and dissimilarity in classification, we broaden this concept to regression, and provide a new definition of similarity and dissimilarity in regression.
Notations. In this paper, lowercase typeface letters (e.g., x) represent scalar, lowercase bold typeface letters (e.g., x) represent vectors, uppercase typeface letters (e.g., X) stand for random variable, uppercase bold typeface letters (e.g.,X) represent matrix. diag(·) means diagonalising a vector into a matrix.

heading: 2.1 PROBLEM STATEMENT
text: A collection of paired training data {(xsi , ysi )}Nsi=1, validation samples {(xvi , yvi )}Nvi=1 and testing samples {(xti, yti)}Nti=1 are presumed to be available, where xoi ∈ Rd, o ∈ {s, v, t}, and yoi are a continuous variables. We suppose that yoi is well-bounded and notate the maximum and minimum values of yoi as y o max and y o min accordingly. Further, we are able to partition y o i into B bins [yo0, y o 1), [y o 1, y o 2), . . . , [y o B−1, y o B) with equal interval ∆y = yomax−y o min
B . In so doing we could put every yoi in the dataset into the corresponding bin fulfilling the condition y o b < y o i ≤ yob+1, where b ∈ {0, 1, . . . , B − 1}. With three splits (training, validation, testing), if we count the number of data points that fall into each bin and form these into a vector mo = (mo1, . . . ,m o B). The imbalanced character of the normal regressions lies in the existence of msbi ,m s bj ∈ms, allowing msbi msbj to hold. It is the opposite of this that the values of the elements of the two vectorsmv andmt, are approximately equal. A desire for imbalanced regression enables model f(xoi ;θ) produced on an imbalanced training set to be potentially generalized to a balanced validation and testing set to the maximum extent possible. Typically, f(xoi ;θ) may be a neural network with parameter θ because of its impressive fitting ability. In what follows, for the sake of convenience, we will drop the superscript o.
In the introduction section, we elaborate on the predicament that the lack of a clear definition of the concepts of similarity and dissimilarity in the regression, as regression involves continuous targets, where hard boundaries between classes do not exist. As a solution to this problem, we formally propose the following definition:
Definition 1 There exists some threshold δ, regarding two data samples (xi, ŷi), (x̃i, ỹi), x̃i is obtained by adding some noise r to xi, and ỹi and ŷi is inferred by the model f(;θ) from x̃i and xi respectively. We remark that (xi, ŷi), (x̃i, ỹi) are similar samples of the model f(xi;θ) (no need for well-trained model) if the distance between the two samples is less than δ under some distance function D(·, ·), such as l1 or l2 distance. Mathematically, it can be phrased in this following:
I(xi, x̃i) = {
1, if D(ŷi, ỹi) < δ 0, if D(ŷi, ỹi) ≥ δ,
(1)
where I(xi, x̃i) indicates the trueness of the condition whether two samples xi and x̃i are similar samples. The parameter δ here is manually specified and task-dependent. We display a virtual example of this definition as illustrated in Figure 1b. We select images from AgeDB-DIR and attach noises to them, afterward we estimate the ages of the original images and their corresponding noisy images with the network. Finally, we identify a pair of original image and noisy image as not similar if their difference exceeds the threshold δ, and vice versa.

heading: 3 MOTIVATION
text: Herewith we illustrate, for simple shallow neural networks, that SSL facilitates long-tailed regression in a positive way. Aside from this, we would characterize the difficulties involved in extending our approach to deep neural networks, i.e. for controlling the level of noise to guarantee the creation of similar samples under the sense of regression. We first describe the data generation, together with a description of the specific experimental settings followed by an analysis. The network structure and implementation details are available in the appendix.
Data generation. Consider two components of data following two different Gaussian distribution, in particular, x1 ∼ N (µ1 · 1d, diag(σ1 · 1d)), x2 ∼ N (µ2 · 1d, diag(σ2 · 1d)). 1d denotes a vector with all 1’s. The values of µ1(σ1) and µ2(σ2) are quite different. With linear mapping to x1 and x2 with parameter w ∈ Rd, it yields the output value y = wTx. As a result, we get two collections of training data {xs1,i, ys1,i} Ns1 i=1, {xs2,j , ys2,j} Ns2 j=1. Suppose N s 1 > N s 2 , and we measure
imbalance here in terms of R = N s 1
Ns2 . We pick up the value of R in {1, 2, 4, 6, 8, 10, 100}. Despite
the training samples are imbalanced, the testing set {xt1,i, yt1,i} Nt1 i=1,{xt2,j , yt2,j} Nt2 j=1 we generate is balanced. Additionally, we produce similar samples by controlling the noise so that the condition min(y1,i, y2,j) δ, ∀i ∈ {1, . . . , N1}, j ∈ {1, . . . , N2} is satisfied. Experimental settings. We compare two training methods here: (i) Representation Learning: It is trained in the first T1 epochs using a self-supervised approach, and the resulting network is treated as an initialized network, and in the second T2 epochs it is trained in a standard supervised learning manner.(ii) Supervised Learning: it is entirely trained using supervised learning in the T1 +T2 epochs. As per the normal process, the model is trained on the training set and tested on the testing set. To evaluate we use Mean Square Error (MSE) as a criterion. Here we use the vanilla mean squared loss function.
Experimental findings. We plot the variation curve of MSE with the imbalance coefficient on the testing set for the two training methods. As illustrated in Figure 2a, what has been clearly observed is that the MSE for both methods grows with an increase in the imbalance coefficient. More importantly, it is found that SSL could alleviate this problem to some degree. The experimental findings, thus, lead us to mitigate the imbalance regression problem with SSL.
Remark It is true that the above experiments concede that SSL can be beneficial to long-tailed regression, albeit with some restrictions that remain when we adapt it to neural networks. Well, yes, we have imposed settings for noise, namely, the noise added is controllable in terms of its impact on the regression values. Unfortunately, scaling the technique beyond shallow to deep networks, there is a difficulty in keeping the difference between the output values of similar samples under regression within a threshold, a concern that arises in the case of a sample augmented with random noise, where the difference between the output values of the noisy sample and the original sample is more than the threshold, as shown in Figure 2b, hence the challenge stems from finding valid noise to ensure the generation of similar samples.

heading: 4 METHOD
text: Just as we showed before, it is not feasible to merely port the above method to regression tasks. Instead, it will backfire if at all we do so. Hence, it is essential to control the noise r to limit the gap between ŷi and ỹi in the length of a bin ∆y. We search for the most anisotropic noise causing the generated samples to be similar or dissimilar to the original samples, which means that we have to impose constraints on the output values. Rather than adding noise to the input, we blaze new a trail, that is, we apply noise to the output value. Mathematically, it can be expressed by the following
formula:
arg min θ arg max ri
D(ŷi, ỹi;θ), (2)
s.t. ||ri|| < , (3) where ŷi = f(xi;θ) and ỹi =ŷi + zi and ỹi = f(xi + ri;θ) (4)
where D denotes the distance loss function, and ỹi is the noisy output w.r.t. noises zi and ri. Here zi is randomly sampled from [−∆y,∆y]. This allows us to assure that ỹi and ŷi are similar, while simultaneously enhancing the local smoothness of the output. Normally it is not possible to obtain r to find a closed-form solution. Since neural networks are highly dimensional and nonlinear, the analytical solutions can not be derived for r . For the purpose of facilitating analysis and simultaneously providing efficient training of the network, we approximate D(ŷi, ỹi;θ) by the first-order Taylor expansion:
D(ŷi, ỹi;θ) ≈ ∇xiD(ŷi, ỹi;θ)T (x̃i − xi) + a = ∇xiD(ŷi, ỹi;θ)Tri + a (5) s.t. ||ri|| < (6)
Where a = D(ŷi, ỹi;θ)|x=xi is a constant term around xi. Based on this, the close-form solution of ri can be calculated as
ri : = arg max ri:||ri||≤
∇xiD(ŷi, ỹi;θ)Tri + a (7)
= ∇xiD(ŷi, ỹi;θ) ||∇xiD(ŷi, ỹi;θ)|| ||ri||max (8)
s.t. ||ri|| ≤ (9)
Therefore, it is obviously that ri = ∇xiD(ŷi,ỹi;θ) ||∇xiD(ŷi,ỹi;θ)||
. Furthermore, as Eq. 4 presented, ỹi = ŷi + zi, so ri can be further denoted as
ri = ∇xiD(ŷi, ŷi + zi;θ) ||∇xiD(ŷi, ŷi + zi;θ)||
(10)
Algorithm 1: Self-Supervised Imbalanced Regression (SSIR) Input: Mini-batchB with M samples, model f(·;θ) with its parameters θ, bin length ∆y,
limitation coefficient of noise , divergence distance function D, values range of input samples (vmin, vmax), function clip(·, vmax, vmin) to clip values that out of range of (vmin, vmax)
Result: Updated parameters of the model θ̂. 1 for xi inB do 2 Initial random noise zi, s.t. |zi| < ∆y; 3 ŷi = f(xi;θ), ỹi = ŷi + zi; 4 Calculate ri by taking the gradient of D(ŷi, ỹi;θ) w.r.t. xi : 5 gi = ∇xiD(ŷi, ỹi;θ) = ∇xiD(f(xi;θ), f(xi;θ) + zi) 6 ri ← × gi/||gi||2; 7 Update ỹi by taking xi + ri as the input of f : 8 x̃i = clip(xi + ri, vmax, vmin); 9 ỹi = f(x̃i;θ);
10 Update ri by by taking the gradient of D(ŷi, ỹi;θ)w.r.t. xi : 11 gi = ∇xiD(ŷi, ỹi;θ) 12 ri ← × gi/||gi||2; 13 x̃i = clip(xi + ri,vmax, vmin) 14 ỹi = f(x̃i;θ) 15 end
16 return θ̂ ← θ −∇θ( 1M M∑ i=1 D(ŷi, ỹi,θ))
x̃ = x + r obtained in this way, however, may exceed the boundary value of the feature (For example, the range of image pixels is [0, 255]). To cope with such an accident, we constrain the feature values to this range by clipping their values. On the other hand, considering that a single back-propagation optimization may not be an accurate approximation, we therefore performed a twostep optimization of r. In our experiments, one-step optimisation approach does not adequately yield optimal performance, and more iterations will introduce more computational cost to training without significant performance gain. Up to now, our proposed Self-Supervised Imbalanced Regression (SSIR) algorithm is completely introduced. In order to show the SSIR more clearly. We give the algorithm workflow in Algorithm 1.
Our training strategy includes normal regression training and self-supervised regression. For the regression training, we directly apply supervised training to minimize the loss function between the outputs of the model ŷ and the corresponding ground truth y. Besides, an additional coefficient λ is introduced to the final loss. Our overall training function is shown below:
Ljoint = LR + λLSSIR (11)
= 1
N N∑ i=1 [D(ŷi, yi;θ) + λD(ŷi, ỹi;θ)] (12)
= 1
N N∑ i=1 [D(f(xi;θ), yi) + λD(f(xi;θ), f(xi + ri;θ))] (13)

heading: 5 EXPERIMENTS
text: In this section, we report the results on three datasets to demonstrate the superiority of our proposed methods, when compared with state-of-the-art baselines. Specifically, we compare our method with VANILLA, SMOTER (Torgo et al., 2013), SMOGN (Branco et al., 2017), RRT (Yang et al., 2021), INV, SQINV, FOCAL-R (Yang et al., 2021), LDS (Yang et al., 2021), FDS (Yang et al., 2021) and some of their combinations on three imbalanced regression datasets including IMDN-WIKIDIR, AgeDB-DIR and NYUD2-DIR. Note that, in all the experiments, the LDS is combined with re-weighting methods i.e. SQINV or INV. In otherwise, our proposed SSIR can be combined with various methods in training process that demonstrates the flexibility of SSIR. The evaluation metrics used on IMDB-WIKI-DIR and AgeDB-DIR datasets are MAE and GM (Yang et al., 2021), besides, RMSE and δ1 are also employed on NYUD2-DIR dataset. For the details of datasets, baselines, evaluation metrics and implementation details, we put them in the appendix.

heading: 5.1 MAIN RESULTS
text: In the experiments on IMDB-WIKI-DIR dataset, we combine our proposed SSIR with SQINV and FOCAL-R respectively. As shown in Table 1, our method achieves the best performance with 7.63 and 4.26 on MAE and GM accordingly. Compared with the second-best method RRT + LDS + FDS, there has improvements of 0.02 and 0.05 in MAE and GM with our approach. Besides, in comparison with other FOCAL-R-based methods, greater performance can be achieved with the combination of FOCAL-R and SSIR with a better GM score of 4.39 and the MAE score of 7.88. For re-weighting-based methods, the combination of SQINV and SSIR outperforms the second-best re-weighting-based method SQINV + LDS + FDS and improves 0.15 and 0.11 on MAE and GM respectively.
In the experiments on AgeDB-DIR, with the same experimental settings on IMDB-WIKI-DIR, we combine our proposed SSIR with SQINV and FOCAL-R respectively. Table 2 illustrates the main results of our method and baselines on AgeDB-DIR dataset. Clearly our method achieves the best performance on both MAE and GM with 7.35 and 4.61 respectively. As the combination of the reweighting method and SSIR, SQINV + SSIR performs better than the other four re-weighting-based methods with 0.06 and 0.11 improvements on MAE and GM separately, compared to the second-best re-weighting-based method SQINV + LDS + FDS. For FOCAL-R-based methods, the association with SSIR can further decrease the testing error than FOCAL-R and FOCAL-R + LDS + FDS.
For NYUD2-DIR dataset, we combine SSIR with VANILLA and re-weighting method INV respectively. As presented in Table 3, our method INV + SSIR achieves the best performance among other methods. Specifically, INV + SSIR outperforms the second-best method INV + LDS + FDS on RMSE with 0.047, and our method achieves the best performance on δ1 with 0.075. Compared with VANILLA, the combination with SSIR can improve the performance significantly with 0.07 on RMSE and 0.007 on δ1.

heading: 5.2 PARAMETER AND CASE STUDY
text: Parameter Study. Figure 3a illustrates the results of parameter study of FOCAL-R + SSIR on AgeDB-DIR dataset. As it shows, each line denotes the results of different λ when fixing , and we conduct such experiments with the of 5e-3, 1e-3, 5e-4 and 1e-4. The overall results show that our method achieves the best performance with = 1e− 3 and λ = 1.0. Case Study. We also conduct case studies to visualize some results of selected examples. As Figure 3b shows, the gaps between the outputs and their associated noisy outputs of all the chosen 6 examples do not exceed a bin length, which means our proposed method can successfully control the noise to limit the noisy outputs within a reasonable range.
Visualization on Noisy Samples. Moreover, we also show three examples and their corresponding noisy examples with of 0.05, 0.1, 0.3, 0.5 and 1.0 respectively in Figure 4. Obviously, with the increasing, the noisy examples become unclearer that means the noise level is positively correlated with .

heading: 6 RELATED WORK
text: Presented here are the literature on imbalance problems and SSL. Concerning imbalance problems, the introduction focuses on classification and regression. For SSL, we review some of the advances and applications.

heading: 6.1 IMBALANCED PROBLEM
text: From the object of imbalanced problems, it can be divided into two categories: imbalanced classification and imbalanced regression.
Imbalanced classification. In the imbalanced classification task, existing methods mainly adopt data-based or model-based solutions (Yang et al., 2021). The core idea of data-based methods is reducing the gap of sampling numbers between many-shot classes and few-shot classes (Chawla et al., 2002; He et al., 2008). For the model-based methods, several works have been proposed to tackle such challenges based on lots of aspects. Jamal et al. (2020) proposed to solve long-tailed classification through domain adaptation perspective. Kang et al. (2020) used a two-stage learning strategy to learn representations for the classification layer using a long-tailed training set. Kang et al. (2021) explored the impact of balanced feature spaces when dealing with imbalanced data, and proposed a SSL-based method to better learn the representation for imbalanced classification. Besides, SSL on unbalanced classification was similarly explored in Wang et al. (2021); Cui et al. (2021); Samuel & Chechik (2021). Furthermore, there also has some methods addressing the problem by well-designed loss functions (Zhang et al., 2017; Cao et al., 2019; Cui et al., 2019; Ren et al., 2020) and other learning paradigms, i.e. meta learning (Shu et al., 2019; Wang et al., 2020).
Imbalanced regression. Imbalanced problem has also been explored in regression problems. Conventional works mainly applied data-based methods to solve the imbalanced problem in regression tasks. One of the first works to address this problem, is published by Torgo et al. (2013) using SMOTE (Chawla et al., 2002). Branco et al. (2017) proposed a Gaussian noise-based synthetic case generation method for imbalanced regression. After that, they further proposed a bagging-based method to aggregate various data pre-processing strategies (Branco et al., 2018) for the same task. Recently, Yang et al. (2021) further delved into such problem by analyzing the error distribution and the statistic distribution of features, both w.r.t. labels, and proposed two distribution smoothing methods for labels and features. And they also proposed the Focal-R, a new loss function for imbalanced regression, motivated by Focal loss (Lin et al., 2017).

heading: 6.2 SELF-SUPERVISED LEARNING
text: Recently, SSL has received extensive attention in the field of deep learning. Different from conventional unsupervised learning and supervised learning, SSL can use the semantic or contexture supervision information to learn robust and well-generalization representations without manual annotations, include contrastive learning-based methods (Chen et al., 2020a;b;c; He et al., 2020) and some other pretext tasks-based methods, such as image inpainting (Pathak et al., 2016; Jenni & Favaro, 2018), context prediction (Doersch et al., 2015), rotation prediction (Gidaris et al., 2018) and so on. Whereas we tackle the optimization problem in a very similar way, the difference concerns the fact that we apply noise to the output directly rather than to the input, as well as the fact that the problem we analyze varies from semi-supervised classification problems to imbalanced regression problems.

heading: 7 CONCLUSION
text: In this paper, we explore the Self-Supervised Learning (SSL) in imbalanced regression scenario. Specifically, we propose Self-Supervised Imbalanced Regression (SSIR) to better address the imbalanced problem in regression tasks. In SSIR, instead of adding noise to the inputs directly to generate the noisy samples, we add noise, which is limited by the bin size, to the output of the model to avoid the excessive dissimilarity between samples and noisy samples, followed by back propagation steps to obtain the meaningful noisy samples. This work provides a new paradigm to tackle this challenge. Further, extensive experiments have shown the superiority of our proposed SSIR over existing state-of-the-art baselines.

heading: 8 APPENDIX
text: In the appendix, we first offer two comparative experiments. Then we give the implementation details in the motivation, as well as the details of the datasets and baseline methods included in the experiments. Besides, we also present more detailed experimental results of parameter studies, and loss curves within different settings. Finally, We investigate the beneficial impact of our approach on feature imbalance.

heading: 8.1 COMPARISON WITH RANDOM NOISE METHOD
text: With the addition of random noise, the combined SSIR with FOCAL-R and SQINV approaches can achieve the performance on MAE with 7.57 and 7.71 respectively on AgeDB-DIR dataset. In contrast, the combined SSIR with FOCAL-R and SQINV approaches with noise generation proposed in our paper can achieve MAE performance with 7.35 and 7.49 respectively. The experimental results demonstrate the superiority of our noise generation approach.

heading: 8.2 COMPARISON WITH ONE-STEP OPTIMIZATION METHOD
text: To quantitatively compare the benefits of two-step approximation against one-step approximation, we conducted a comparison experiment on the AgeDB-DIR dataset. The setup of the comparison experiment is following: we perform a one-step optimization of r and provide results combined with SQINV; except for this, we keep the other hyperparameters consistent with our experiments. When performing one-step optimization, the combined SSIR with SQINV approach can achieve the performance on MAE with 7.62. In contrast, the combined SSIR with SQINV approach with a twostep approximation can achieve MAE performance with 7.49. The experimental results demonstrate the superiority of two-step approximation. However, we have found in our experiments that there is no significant increase in performance but an increase in computational overhead when the number of optimization steps is greater than or equal to 3. Therefore, in order to balance accuracy and efficiency, we have used two-step optimization in our paper.

heading: 8.3 EXPERIMENTS SETTINGS
text: Datasets We evaluate our proposed method on three large-scale deep imbalanced regression datasets, the tasks of which are age estimation and depth estimation. For all the datasets used in our experiments, we follow the settings in (Yang et al., 2021) and segment the target bins into three subsets: many-shot which contains more than 100 training images, medium-shot in which the number of training samples in each bin is in the range of 20 to 100, and few-shot that contains less than 20 training samples in each bin. The details of the datasets are as following:
• IMDB-WIKI-DIR: IMDB-WIKI-DIR is an imbalanced age estimation dataset. This dataset is created by Yang et al. (2021) based on the IMDB-WIKI dataset (Rothe et al., 2018). The original IMDB-WIKI contains 523.0K face images with associated ages. In IMDB-WIKIDIR, the training set contains 191.5K images, and both validation and testing sets include 11.0K bin-balanced images separately. The length of a bin is set to 1 year with the range from 0 to 186, and the density of each bin is from 1 to 7,149.
• AgeDB-DIR: AgeDB-DIR is also a imbalanced age estimation dataset proposed by Yang et al. (2021) using AgeDB dataset (Moschoglou et al., 2017). In the dataset, the age range is from 0 to 101 and the bin length is 1 year. The maximal and minimal numbers of images in a bin are 353 and 1 respectively. The training set contains 12,208 images with imbalanced densities among bins, while the validation and testing sets contain 2,140 bin-balanced images respectively.
• NYUD2-DIR: The NYUD2-DIR is a dataset constructed by Yang et al. (2021) for imbalanced indoor scenes depth estimation, which is an imbalanced version of NYU Depth Dataset V2 (Silberman et al., 2012). The dataset contains images of different indoor scenes and their corresponding depth maps. The bin length is set to 0.1 meters. The maximal and minimal depth is 10 meters and 0.7 meters separately in the dataset. Following Yang et al. (2021), there are 50K images in the training set and 654 images in the testing set. In each bin, the number of pixels is in the range from 1.13× 106 to 1.46× 108. Besides,
the number of testing pixels selected for each bin from testing images is set to 9,357 (the minimal number of pixels in each bin in the testing set), which aims to make the testing set balanced. The total number of pixels in the training set and testing set are 3.51× 109 and 8.70× 105 respectively.
Baselines We conduct experiments to compare our proposed method with the following baselines:
• Vanilla model: The pure backbone model that does not apply any techniques to improve the performance is presented as VANILLA. Following (Yang et al., 2021), we use ResNet50 (He et al., 2016) as VANILLA for both IMDB-WIKI-DIR and AgeDB-DIR datasets, and ResNet-50-based encoder-decoder method (Hu et al., 2019) for NYUD2-DIR dataset.
• Synthetic samples: We choose existing synthetic samples methods, which are used for imbalanced regression, as baselines, including SMOTER (Torgo et al., 2013) and SMOGN (Branco et al., 2017).
• Focal-R: FOCAL-R (Yang et al., 2021) is the regression version of Focal loss (Lin et al., 2017), which uses a continuous function, e.g. Sigmoid function, to map the absolute error into the range of 0 to 1 instead of the scaling factor in Focal loss.
• Regressor Re-Training (RRT): RRT (Yang et al., 2021) is a two-stage training strategy inspired by Kang et al. (2020), in which the feature and classifier are decoupled.
• Re-weighting: We also select Re-weighting methods as baselines, including inversefrequency weighting (INV) and its square-root weighting variant (SQINV) (Yang et al., 2021).
• Distribution Smoothing: Distribution Smoothing (Yang et al., 2021) includes Label Distribution Smoothing (LDS) and Feature Distribution Smoothing (FDS). Such methods are designed to utilize the information from nearby bins by smoothing the distribution of labels or features. Note that, LDS should be used with re-weighting methods, and we follow Yang et al. (2021) to combine LDS with INV and SQINV.
Evaluation metrics Here, we will introduce the details of all the evaluation metrics used in our experiments.
• Mean Absolute Error (MAE): MAE is a common evaluation metric in regression problems, which is used to measure the averaged absolute difference between ground truth and prediction results. MAE can be denoted as 1N ∑N i=1 |ŷi − yi|, where N denotes the number
of total samples, ŷi and yi are the prediction result and ground truth of the i-th sample respectively.
• Root Mean Squared Error (RMSE): Similar to MAE, RMSE is also a widely used evaluation metric in regression tasks. Different from MAE, RMSE is used to measure the rooted averaged squared difference between ground truth and prediction results, which can be presented as 1N ∑N i=1(ŷi − yi)2, where N denotes the number of total samples, ŷi and
yi are the prediction result and ground truth of the i-th sample respectively.
• error Geometric Mean (GM): GM is a new evaluation metric proposed by Yang et al. (2021). Instead of applying arithmetic mean, GM uses geometric mean over the prediction errors, which is defined as ( ∏N i=1 ei) 1 N , where ei , |ŷi − yi| denotes the L1 error of the
i-th sample.
• Threshold Accuracy (δi): For NYUD2-DIR dataset, we follow Yang et al. (2021) and use standard depth estimation evaluation metric δi which is defined as max(digi , gi di ) = δi <
1.25i, where gi and di denote the ground truth value and the the prediction value of a pixel respectively. Here, in our experiments on NYUD2-DIR, we set i to 1 as the settings in (Yang et al., 2021).

heading: 8.4 IMPLEMENTATION DETAILS IN MAIN EXPERIMENTS
text: IMDB-WIKI-DIR: For IMDB-WIKI-DIR dataset, we use ResNet-50 (He et al., 2016) as backbone and train it with our proposed SSIR method for 90 epochs using Adam optimizer (Kingma & Ba,
2014) with L1 loss. The initial learning rate is set to 1e-3 and decayed to 1e-4 and 1e-5 at the 60-th and 80-th epoch respectively. Besides, we set the batch size, λ and to 256, 0.1 and 1e-3 separately.
AgeDB-DIR: For AgeDB-DIR dataset, the settings of backbone, optimizer, learning rate, loss function and batch size are the same with that of IMDB-WIKI-DIR. For the training of FocalR+SSIR, we set the λ and to 1.0 and 1e-3 respectively. Besides, they are set to 0.1 and 1e-3 for SQINV+SSIR.
NYUD2-DIR: For the experiments on NYUD2-DIR, we apply a ResNet-50-based encoder-decoder architecture (Hu et al., 2019). We train our method by Adam optimizer with 20 epochs. The initial learning rate is set to 1e-4 and decayed by 0.1 every 5 epochs. We set the batch size and weight decay to 32 and 1e-4 respectively. For λ and , we set them to 1.0 and 5e-4 for VANILLA+SSIR, and 2.5, 5e-4 for INVERSE+SSIR.

heading: 8.5 IMPLEMENTATION DETAILS IN MOTIVATION EXPERIMENTS
text: For the settings in the data generation of our motivation, we set the mean and variance of the two Gaussian distribution µ1, σ1 and µ2, σ2 to 10, 25 and -10, 25 respectively, and the dimension of the generated data d is set to 100. We apply a two-layer Multi-Layer Perception (MLP) with 128 hidden neurons as the prediction model in the experiments. For the settings of the training process in the experiments, we set epochs T1 and T2 to 1000 and 500 respectively. The learning rates of the two training methods: Representation Learning and Supervised Learning, are set to 1e-3 and 1e-2 respectively. Besides, we set the weight decay of the training process in the experiments to 5e-5.

heading: 8.6 PARAMETER STUDIES
text: Here, we study the different parameter settings of and λ on all three datasets and report all the results of the experiments that we conducted.
Table 4 illustrates the experiments results of different settings of and λ. The top part of the table is the results of different parameters settings of SQINV + SSIR, and the bottom part shows the results of FOCAL-R + SSIR on different settings of and λ.
The results of the parameter study of our method on AgeDB-DIR dataset are presented in Table 5. As the table shows, we present the results of different and λ settings of our approach. The top part of the table is regarding with SQINV + SSIR, and the bottom part denotes the results of different parameter settings of FOCAL-R + SSIR.
Table 6 is the parameter study regards to and λ on NYUD2-DIR dataset. The top part of the table illustrates the results of different parameters settings of VANILLA + SSIR while the bottom part shows that of INVERSE + SSIR.

heading: 8.7 TRAINING AND VALIDATION LOSSES
text: We present the training and validation loss of our proposed approach on IMDB-WIKI-DIR and AgeDB-DIR datasets. Figure 5a and 5b present the training and validation losses w.r.t. training epochs of SQINV + SSIR and FOCAL-R + SSIR respectively on IMDB-WIKI-DIR dataset. Figure 5c
and 5d illustrate the training and validation losses of SQINV + SSIR and FOCAL-R + SSIR on AgeDB-DIR dataset.

heading: 8.8 FEATURE BALANCE UNDER REGRESSION SENSE
text: To qualitatively and quantitatively evaluate the balance of features under the regression sense, we first attempt to offer a definition of feature balance, and then assess the impact of our approach and the baseline approaches experimentally.

heading: 8.8.1 DEFINITION OF FEATURE BALANCE
text: As mentioned in (Kang et al., 2021), the feature space can be more balanced if the features from the same class or bin are more similar, and the balanced feature space can handle the imbalanced problem well. In this subsection, we give the definition of feature balance under regression sense. We assume that z with size c(z) is a feature vector extracted from the neural network corresponding to the input x. It is often the case that z is semantically rich. Each element in z signifies a different attribute. Without loss of generality, it is also reasonable to expect that the elements in z are mutually independent. And then, we assume that for data within a bin, its features all conform to a Gaussian distribution N (µi, σ2i ),∀i = 1, 2, ..., c(z) for each dimension. The reasons for the Gaussian assumption are: Gaussian distribution is the most commonly used, and it can fit deep learning features relatively well (Wang et al., 2017; Sun et al., 2020; Louizos & Welling, 2016). Therefore, the variances of the distributions of each dimension can be used as the evaluation metric, to evaluate the imbalanced ratio
of the feature space:
σ = ∑ σi (14)
s.t. i = 1, 2, ..., c(z) (15)
where smaller σ indicates more balanced feature space that can handle the imbalanced problem well.

heading: 8.8.2 EVALUATION OF FEATURE BALANCE ON IMDB-WIKI-DIR DATASET
text: We conduct experiments on IMDB-WIKI-DIR testing dataset, recall that previously we used f(x;θ) to denote a neural network. Here f(x;θ) is implemented via ResNet-50. z is the feature extracted from the penultimate layer in f(x;θ), which has a dimension of 2048. The methods we use for evaluation are VANILLA, SQINV and our SQINV+SSIR. In Figure 6, we show the variance values for the different methods on many-shot, middle-shot and few-shot classes, it is evident that our method yields a much smaller variance σ, which means the feature space generated by our method is more balanced. Figure 7 shows more variance values σ of the three methods within different bins.

Review summaries:
This paper researched on the recently proposed long-tailed regression problem with self-supervised learning method. Two questions are investigated by this paper: 1) how to measure the similarity and dissimilarity under the regression sense; 2) it is not guaranteed that the sampled with perturbations are similar to the original samples without any noise. The former problem is addressed by providing a formal definition of similarity and the later question is addressed by limiting the volume of noise on the output. Authors report the results on three datasets to demonstrate the superiority of our proposed methods, including IMDN-WIKI-DIR, AgeDB-DIR and NYUD2-DIR.

This paper proposes a novel algorithm SSIR to address the imbalanced problem in regression tasks, which seamlessly combines self-supervised learning and imbalanced regression by giving the formal definition of similarity in the regression task. Besides, the authors specifically propose to limit the volume of noise on the output, and in doing so to find meaningful noise on the input by back propagation. Experimental results show that our approach achieves the state-of-the-art performance.