Lifting Imbalanced Regression with Self-Supervised Learning

1 INTRODUCTION . Regression is a fundamental task in machine learning and statistical analysis , which involves modeling the relationship , between one or more independent and dependent variables . As distinct from the discrete output values in classification , regression makes predictions for continuous values . The issue of imbalanced recognition , arising from the conjunction of imbalance and classification , has been drawing the attention of researchers recently ( Shu et al. , 2019 ; Cao et al. , 2019 ; Ren et al. , 2020 ; Jamal et al. , 2020 ; Kang et al. , 2020 ; 2021 ) . By analogy , it is natural to question whether long-tailed regression exists ( or one can also call it imbalanced regression , the equivalence of these two names in the paper ) . Exactly what we expected , a fresh task , namely long-tailed regression , has recently been proposed by Yang et al . ( 2021 ) . Regardless of the newness of the task , there is no way to neglect its immense significance in terms of application and research . For example , age estimation from facial images ; health indicators for various populations in healthcare ( Soualhi et al. , 2014 ) , these quantities usually exhibit a long-tailed distribution . Going further , in terms of performance , progress in the long-tailed regression research is also far from satisfactory . More recently , self-supervised learning ( SSL ) has emerged as a new rage in the research community , owing to its straightforward learning paradigm yet excellent performance ( Noroozi & Favaro , 2016 ; Gidaris et al. , 2018 ; Wu et al. , 2018 ; He et al. , 2020 ) . The excellence of the approach has been admired not only on classification with well-balanced datasets , but also surprisingly on long-tailed classification by Kang et al . ( 2021 ) ; Wang et al . ( 2021 ) ; Cui et al . ( 2021 ) ; Samuel & Chechik ( 2021 ) who found its benefits . It prompts some thoughts on the potential and practical applications , in particular , applications to SSL for long-tailed regression ( Yang et al. , 2021 ) . Mindful that the majority of available techniques on SSL are in the context of classification ( Oord et al. , 2018 ; Hjelm et al. , 2018 ; Chen et al. , 2020a ; Henaff , 2020 ) . Due to the difference between classification and regression ( no hard boundary existing between outputs in regression task as shown in Figure 1a ) ( Razi & Athappilly , 2005 ; Breiman et al. , 2017 ; Liaw et al. , 2002 ; Yang et al. , 2021 ) , it is not guaranteed that SSL can help address the imbalanced regression problem well , and it may be difficult to make a straightforward migration from classification to regression task . Upon deeper pondering , there are obstacles and problems that we face : Can SSL really relieve the long-tailed regression problem ? If so , how might we solve the imbalanced regression problem by utilizing SSL ? Figure 1a illustrates the motivation , by adding noise to the image of the cat , it is very unlikely to be misclassified into a class that differs significantly from it , such as a bird , since they are discrete and the categories diverge greatly from each other . In the case of regression , however , it is not at all feasible when a similar migration was made . We imagine that if noise is added to a face image , then the network ’ s estimation of the person ’ s age may be significantly biased . In this regard , it becomes essential to work out and comprehend the degree of noise that induces “ similarity and dissimilarity ” amongst output values from a regression standpoint . On the foundation of this definition , questions are raised about exploiting noise sensibly and effectively . With this paper , we try for the first time to propose answers to these questions . And we summarize our contributions as follows : • To the best of our knowledge , we are the first to broaden the concept of similarity and dissimilarity between augmented samples and original samples from classification to regression . And we are also the first to reveal imbalanced regression problem with SSL . • Having been convinced of the beneficial effect of applying SSL on a simple experiment , we extend it to the neural network-based long-tailed regression task . To ensure that the noise is manageable w.r.t . the similarity of the regression , we propose to generate noise on regression values within a predefined threshold . • We obtain an optimization problem on noise with an efficient approximate solution algorithm based on a first-order Taylor expansion . The augmented input samples are obtained by back-propagation . • As a practical trial for the proposed method , the results reveal that we attained not only the best results , but also the possibility to mix it freely with other training techniques . 2 PRELIMINARY . In this section , we cover the description of imbalanced regression , and then , with reference to the concept of similarity and dissimilarity in classification , we broaden this concept to regression , and provide a new definition of similarity and dissimilarity in regression . Notations . In this paper , lowercase typeface letters ( e.g. , x ) represent scalar , lowercase bold typeface letters ( e.g. , x ) represent vectors , uppercase typeface letters ( e.g. , X ) stand for random variable , uppercase bold typeface letters ( e.g. , X ) represent matrix . diag ( · ) means diagonalising a vector into a matrix . 2.1 PROBLEM STATEMENT . A collection of paired training data { ( xsi , ysi ) } Nsi=1 , validation samples { ( xvi , yvi ) } Nvi=1 and testing samples { ( xti , yti ) } Nti=1 are presumed to be available , where xoi ∈ Rd , o ∈ { s , v , t } , and yoi are a continuous variables . We suppose that yoi is well-bounded and notate the maximum and minimum values of yoi as y o max and y o min accordingly . Further , we are able to partition y o i into B bins [ yo0 , y o 1 ) , [ y o 1 , y o 2 ) , . . . , [ y o B−1 , y o B ) with equal interval ∆y = yomax−y o min B . In so doing we could put every yoi in the dataset into the corresponding bin fulfilling the condition y o b < y o i ≤ yob+1 , where b ∈ { 0 , 1 , . . . , B − 1 } . With three splits ( training , validation , testing ) , if we count the number of data points that fall into each bin and form these into a vector mo = ( mo1 , . . . , m o B ) . The imbalanced character of the normal regressions lies in the existence of msbi , m s bj ∈ms , allowing msbi msbj to hold . It is the opposite of this that the values of the elements of the two vectorsmv andmt , are approximately equal . A desire for imbalanced regression enables model f ( xoi ; θ ) produced on an imbalanced training set to be potentially generalized to a balanced validation and testing set to the maximum extent possible . Typically , f ( xoi ; θ ) may be a neural network with parameter θ because of its impressive fitting ability . In what follows , for the sake of convenience , we will drop the superscript o . In the introduction section , we elaborate on the predicament that the lack of a clear definition of the concepts of similarity and dissimilarity in the regression , as regression involves continuous targets , where hard boundaries between classes do not exist . As a solution to this problem , we formally propose the following definition : Definition 1 There exists some threshold δ , regarding two data samples ( xi , ŷi ) , ( x̃i , ỹi ) , x̃i is obtained by adding some noise r to xi , and ỹi and ŷi is inferred by the model f ( ; θ ) from x̃i and xi respectively . We remark that ( xi , ŷi ) , ( x̃i , ỹi ) are similar samples of the model f ( xi ; θ ) ( no need for well-trained model ) if the distance between the two samples is less than δ under some distance function D ( · , · ) , such as l1 or l2 distance . Mathematically , it can be phrased in this following : I ( xi , x̃i ) = { 1 , if D ( ŷi , ỹi ) < δ 0 , if D ( ŷi , ỹi ) ≥ δ , ( 1 ) where I ( xi , x̃i ) indicates the trueness of the condition whether two samples xi and x̃i are similar samples . The parameter δ here is manually specified and task-dependent . We display a virtual example of this definition as illustrated in Figure 1b . We select images from AgeDB-DIR and attach noises to them , afterward we estimate the ages of the original images and their corresponding noisy images with the network . Finally , we identify a pair of original image and noisy image as not similar if their difference exceeds the threshold δ , and vice versa . 3 MOTIVATION . Herewith we illustrate , for simple shallow neural networks , that SSL facilitates long-tailed regression in a positive way . Aside from this , we would characterize the difficulties involved in extending our approach to deep neural networks , i.e . for controlling the level of noise to guarantee the creation of similar samples under the sense of regression . We first describe the data generation , together with a description of the specific experimental settings followed by an analysis . The network structure and implementation details are available in the appendix . Data generation . Consider two components of data following two different Gaussian distribution , in particular , x1 ∼ N ( µ1 · 1d , diag ( σ1 · 1d ) ) , x2 ∼ N ( µ2 · 1d , diag ( σ2 · 1d ) ) . 1d denotes a vector with all 1 ’ s . The values of µ1 ( σ1 ) and µ2 ( σ2 ) are quite different . With linear mapping to x1 and x2 with parameter w ∈ Rd , it yields the output value y = wTx . As a result , we get two collections of training data { xs1 , i , ys1 , i } Ns1 i=1 , { xs2 , j , ys2 , j } Ns2 j=1 . Suppose N s 1 > N s 2 , and we measure imbalance here in terms of R = N s 1 Ns2 . We pick up the value of R in { 1 , 2 , 4 , 6 , 8 , 10 , 100 } . Despite the training samples are imbalanced , the testing set { xt1 , i , yt1 , i } Nt1 i=1 , { xt2 , j , yt2 , j } Nt2 j=1 we generate is balanced . Additionally , we produce similar samples by controlling the noise so that the condition min ( y1 , i , y2 , j ) δ , ∀i ∈ { 1 , . . . , N1 } , j ∈ { 1 , . . . , N2 } is satisfied . Experimental settings . We compare two training methods here : ( i ) Representation Learning : It is trained in the first T1 epochs using a self-supervised approach , and the resulting network is treated as an initialized network , and in the second T2 epochs it is trained in a standard supervised learning manner . ( ii ) Supervised Learning : it is entirely trained using supervised learning in the T1 +T2 epochs . As per the normal process , the model is trained on the training set and tested on the testing set . To evaluate we use Mean Square Error ( MSE ) as a criterion . Here we use the vanilla mean squared loss function . Experimental findings . We plot the variation curve of MSE with the imbalance coefficient on the testing set for the two training methods . As illustrated in Figure 2a , what has been clearly observed is that the MSE for both methods grows with an increase in the imbalance coefficient . More importantly , it is found that SSL could alleviate this problem to some degree . The experimental findings , thus , lead us to mitigate the imbalance regression problem with SSL . Remark It is true that the above experiments concede that SSL can be beneficial to long-tailed regression , albeit with some restrictions that remain when we adapt it to neural networks . Well , yes , we have imposed settings for noise , namely , the noise added is controllable in terms of its impact on the regression values . Unfortunately , scaling the technique beyond shallow to deep networks , there is a difficulty in keeping the difference between the output values of similar samples under regression within a threshold , a concern that arises in the case of a sample augmented with random noise , where the difference between the output values of the noisy sample and the original sample is more than the threshold , as shown in Figure 2b , hence the challenge stems from finding valid noise to ensure the generation of similar samples .

Summary: This paper researched on the recently proposed long-tailed regression problem with self-supervised learning method. Two questions are investigated by this paper: 1) how to measure the similarity and dissimilarity under the regression sense; 2) it is not guaranteed that the sampled with perturbations are similar to the original samples without any noise. The former problem is addressed by providing a formal definition of similarity and the later question is addressed by limiting the volume of noise on the output. Authors report the results on three datasets to demonstrate the superiority of our proposed methods, including IMDN-WIKI-DIR, AgeDB-DIR and NYUD2-DIR.