A step towards understanding why classification helps regression
Silvia L. Pintea1,2

Yancong Lin3

Jouke Dijkstra2

Jan C. van Gemert112Computer Vision Lab, Delft University of Technology
Division of Image Processing (LKEB), Leiden University Medical Center
3Intelligent Vehicles Group, Delft University of Technology

Abstract
1.0

1. Introduction
Regression models predict continuous outputs. In contrast, classification models make discrete, binned, predictions. For a continuous task, regression targets are a superset of the classification labels: they are more precise, taking
values in-between the discrete classification bins. For regression, the error is only bounded by the precision of themeasurements, for classification this also depends on thebin sizes: e.g. an age estimation classifier that can predict only young/old classes, cannot discriminate between
middle-aged people. Additionally, when training a regression model, losses are proportional to the error magnitude,
while for classification all errors receive an equal penalty:
predicting bin 10 instead of 20, is just as incorrect as predicting bin 10 instead of bin 100. So classification cannot
add anything new to regression; or can it?
Surprisingly, adding a classification loss to the regression loss [30, 44, 46, 51], or even replacing the regression
loss with classification [10, 11, 38] is extensively used inpractice when training deep models for predicting continu-

0.5

f(x)

A number of computer vision deep regression approaches report improved results when adding a classification loss to the regression loss. Here, we explore why thisis useful in practice and when it is beneficial. To do so, we
start from precisely controlled dataset variations and data
samplings and find that the effect of adding a classification
loss is the most pronounced for regression with imbalanced
data. We explain these empirical findings by formalizing therelation between the balanced and imbalanced regression
losses. Finally, we show that our findings hold on two real
imbalanced image datasets for depth estimation (NYUD2DIR), and age estimation (IMDB-WIKI-DIR), and on the
problem of imbalanced video progress prediction (Breakfast). Our main takeaway is: for a regression task, if thedata sampling is imbalanced, then add a classification loss.

0.0
0.5
1.0

1.0

0.5

0.0

x

0.5

1.0

1.0

0.5

0.0

x

0.5

1.0

1.0

0.5

0.0

x

0.5

1.0

Figure 1. To probe “Why does classification help regression?” wedesign a fully controlled dataset including the following scenarios: Clean data – 1D non-linear functions defined by the sum of
two sine waves with different frequencies and amplitudes; Noisy
data – uniform noise added to the outputs; Out of distribution –sampling different regions of the input space during training andduring testing. (We show a single function here. The gray shadinggroups the function targets into 4 classes, as an example.)ous outputs. The classification is typically defined by binning the regression targets into a fixed number of classes.
This is shown to be beneficial for tasks such as: depth estimation [11], horizon line detection [44], object orientation
estimation [30, 51], age estimations [34]. And the reported
motivation for discretizing the regression loss is that: it improves performance [44, 46], or that it helps in dealing withnoisy data [44], or that it helps overcome the overly-smooth
regression predictions [30, 40], or that it helps better regularize the model [22, 44]. However, none of these assumptions has been thoroughly investigated.
In this work, we aim to explore in the context of deeplearning: Why does classification help regression? Intuitively, the regression targets contain more information thanthe classification labels. And adding a classification loss
does not contribute any novel information. What is it reallythat a classification loss can add to a standard MSE (meansquared error) regression loss? And why does it seem beneficial in practice?
To take a step towards understanding why classification
helps regression, we start the analysis in a fully controlled
setup, using a set of 1D synthetic functions. We consider
several prior hypothesis of when classification can help re-

1.0

f(x)

0.5
0.0
0.5
1.0

1.0

0.5

0.0

x

0.5

1.0

1.0

0.5

0.0

x

0.5

1.0

1.0

0.5

0.0

x

0.5

1.0 1.0

0.5

0.0

x

0.5

1.0

1.0
0.5

f(x)

gression: noisy data, out-of-distribution data, and the normal clean data case, as in Fig. 1. Additionally, we vary the
sampling of the data from uniform to highly imbalanced asin Fig. 2. We empirically find out in which of these cases
adding a classification loss improves the regression quality
on the test set. Moreover, we explain these empirical observations by formulating them into a probabilistic analysis.
We urge the reader to note that our goal is not proposing a novel regression loss, nor do we aim to improve results with “superior performance” over state-of-the-art regression models. But rather, we aim to investigate a common computer vision practice: i.e. adding a classification
loss to the regression loss, and we analyze for what kindof dataset properties and dataset samplings this practiceis useful. Finally, we show experimentally that our findings hold in practice on two imbalanced real-world computer vision datasets: NYUD2-DIR (depth estimation) and
IMDB-WIKI-DIR (age estimation) [47], and on the Breakfast dataset [23] when predicting video progression.

0.0
0.5
1.0

6 × 101

8 × 101

log-counts of f(x)

102

101

102

log-counts of f(x)

100

101

102

log-counts of f(x)

101

102

103

log-counts of f(x)Figure 2. Data sampling. On the columns we increase the data
imbalance from uniform (balanced) to severely imbalanced. On
the first row we show the function f (x) where darker datapointcolors visualize higher density. The gray shading on the first row
groups the targets into 4 classes. On the second row we show the
log-counts per function value, sampled for the training data. We
sample the test data uniformly.

2. Why does classification help regression?
For an input dataset, D, containing N samples of theform (x, y)∈D, we analyze what happens when we train
a deep network with parameters ω to predict a targety ∗ =f (x, ω) for a sample x, by minimizing NLL (negative
log-likelihood) or equivalently, minimizing the MSE (meansquared error) regression loss:
  \boldsymbol {\omega }^* &= \argmin _{\boldsymbol {\omega }} \sum _{(\mathbf {x}, {y}){\in }\mathcal {D}} L({y},\mathbf {x},\boldsymbol {\omega }) \\ &= \argmin _{\boldsymbol {\omega }} \left (\sum _{(\mathbf {x}, {y}){\in }\mathcal {D}} - \log p({y} |\mathbf {x},\boldsymbol {\omega }) \right ) \\ \label {eq:reg} &\equiv \argmin _{\boldsymbol {\omega }} \lambda \frac {1}{N}\sum _{(\mathbf {x}, {y}){\in }\mathcal {D}} \left ( {y} - {y}^* \right )^2(3)
where ω ∗ are the optimal parameters, and we can reinterpretthe imbalanced likelihood as the mean of a Gaussian distribution with σ noise: p(y|x, ω) = N (y; y ∗ , σ 2 I), in whichcase minimizing the NLL is equivalent to minimizing the
MSE loss [3], and λ is a function of the noise σ.
We contrast Eq. (3) to the case when we discretize thetargets y into a set of C classes and use a classification loss
next to the regression loss:  \label {eq:reg_cls} L({y},\mathbf {x},\boldsymbol {\omega }) =& \lambda \left ({y} - {y}^* \right )^2 - \log p( {y}^*_c | \mathbf {x}, \boldsymbol {\omega }),(4)

where yk∗ , k∈{1, .., C} denotes the model predictions
binned into classes, and specifically yc∗ is the prediction at
the true class indexed by c, for the sample x. For the classification term, we make the standard softmax distribution
assumption.

2.1. Controlled 1D analysis
To probe the question ”Why does classification help regression?” we first want to know in which cases does classification help regression. We measure test-time MSE scoresfor each case in Fig. 1, and compare training with a regression loss as in Eq. (3), with training using an extra
classification loss as in Eq. (4). We randomly sample 10
functions of the form: f (x)=a sin(cx) + b sin(dx), wheref (x)∈[−1.5, 1.5] and x∈[−1, 1]. For every function wevary the dataset scenario as in Fig. 1, and we also vary the
sampling of the data from uniform to severely imbalanced
sampling, as in Fig. 2. Each dataset sampling is repeatedly
performed with 5 different random seeds, where we always
sample ≈30, 000 samples in total, and then randomly pick
1/3 for training, for validation, and for testing, respectively.
In the out-of-distribution case, the training set misses certain function regions that are present in the validation/test
set, and vice-versa; and there is an overlap of 1/4 between
the function regions in the training set and the regions in thevalidation/test set. For the imbalanced sampling, we aim to
sample a range of the targets y more frequently than otherranges. For this, we randomly select a location along they-axis in each repetition: this defines the center of the peakin Fig. 2, second row. And depending on the sampling scenario, we use a fixed variance ratio around the peak (0.3, 0.1
and 0.03 for mild, moderate and severe sampling) to define
the region from which we draw the frequent samples. We
sample 75% of the samples from the peak region, and therest uniformly from the other function areas.
We train a simple MLP (multi layer perceptron) with
3 linear layers ([1×6], [6×16], [16×1]) and ReLU non-2.2. Anchoring 1D experimental observations
In Section 2.1 we observe that classification has a more
pronounced effect when the sampling of the data is imbalanced. Therefore, from here on we focus the analysis on imbalanced data sampling. We start from thederivations of Ren et al. [31] who define the relation between the NLL (negative log-likelihood) of imbalanced
samples − log pe(y|x, ω) and the NLL of the balanced samples − log p(y|x, ω):  \label {eq:imbalanced} -\log \widetilde {p}({y}|\mathbf {x},\boldsymbol {\omega }) &= - \log \frac {p({y}|\mathbf {x},\boldsymbol {\omega }) \widetilde {p}({y})}{\int _{{y}^\prime } p({y}^\prime | \mathbf {x},\boldsymbol {\omega }) \widetilde {p}({y}^\prime ) d{y}^\prime }(5)

where pe(y) denotes the prior over imbalanced targets.
Eq. (5) holds under the assumption that the data function
remains unchanged, pe(x|y)=p(x|y), which is the case forthe clean and noisy data scenarios above. We decompose
the log and rewrite the relation between the NLL of the balanced and the NLL of the imbalanced data:
  \label {eq:nll-extra} -\log \widetilde {p}({y}| \mathbf {x},\boldsymbol {\omega }) + L_\text {extra}({y},\mathbf {x},\boldsymbol {\omega }) = & - \log p({y}|\mathbf {x},\boldsymbol {\omega }), (6)Figure 3. MSE per class, where we vary the number of classes
from 4 to 1024. We evaluate on uniformly sampled test sets, across
5 repetitions. We plot means and standard deviations for eachdataset (rows) and sampling variation (columns), where the shading represents the standard deviation. The red line, reg, should beconstant across classes but it varies due to the sampling/training
randomness. We also print the gap between the reg and reg+cls
measured as absolute difference of MSE scores. The effect of theclassification loss is present when the sampling of the data is imbalanced for the clean and noisy data.

linearities. For setting the hyperparameter λ, we performa hyper-parameter search on the validation set. We find thebest λ to be 1e+2, 1e+3 and 1e+4 for the clean, noisy andout-of-distribution. We train for 80 epochs using an Adam
optimizer with a learning rate of 1e−3, 1e−2 and 1e−4 forclean, noisy and out-of-distribution respectively. We use aweight decay of 1e−3. For classification we add at trainingtime a linear layer ([16×C]) predicting C classes. Moredetails are in the supplementary material.
In Fig. 3 we show the MSE across all dataset and sampling variations, for {22 , 24 , 26 , 28 , 210 } classes. We define
the class ranges uniformly. The test sets are uniformly sampled and we perform 5 repetitions. We plot the means andstandard deviations. We print on every plot the gap between
the reg and reg+cls measured as the average absolute difference of MSE scores. From this 1D analysis, we observethat the effect of the classification loss is visible when the
training data is imbalanced for clean and noisy data.

Lextra contains all the information about the imbalanced regression targets:
  L_\text {extra} = & \log \widetilde {p}({y}){-}\log \int _{{y}^\prime } p({y}^\prime |\mathbf {x},\boldsymbol {\omega }) \widetilde {p}({y}^\prime ) d{y}^\prime ,\\ = & \log \widetilde {p}({y}){-}\log \int _{{y}^\prime } \mathcal {N}({y}^\prime ; {y}^*, \sigma ^2 I)\widetilde {p}({y}^\prime ) d{y}^\prime ,(8)
where again y ∗ are the predicted targets.
To derive the link between optimizing a model on imbalanced data and using both a regression MSE loss and a classification loss, we assume the imbalanced regression targets
y can be discretized into a set of classes, k∈{1, .., C} suchPC
that k=1 p(yk )=1. By going from continuous regression
targets to discrete classes, we change the form of the loglikelihood from Gaussian to softmax:
  \label {eq:discrete} L_\text {extra} &\approx \log \widetilde {p}({y}_c) -\log \sum _{k=1}^C p({y}_{k}^* | \mathbf {x}, \boldsymbol {\omega }) \widetilde {p}({y}_k),(9)where we denote the true class label by yc . Note that the
regression targets y are imbalanced, but the classes yk do
not necessarily need to be imbalanced. We analyze in theexperimental section the effect of defining balanced classes.
We make the observation that if we could optimize theclass assignment, the Lextra term would disappear. If theclasses are optimized, then the class likelihoods are close to
0 for all classes except the true class: p(yk∗ |x, ω)≈0, ∀k̸=c,where c indexes the true class. Using this in the expressionof Lextra , we obtain:
  L_\text {extra} \approx & \log \widetilde {p}({y}_c) - \log p({y}_{c}^* | \mathbf {x}, \boldsymbol {\omega }) \widetilde {p}({y}_c), \\ \approx & - \log p({y}_{c}^* | \mathbf {x}, \boldsymbol {\omega }),(11)where the pe(yc ) terms cancel out when decomposing thesecond log. Therefore, we can see that optimizing the classcross-entropy loss reduces the gap between between the
NLL of imbalanced data and NLL of balanced data. (Note:
we observe in practice that if the classifier fails to converge,adding a classification loss is detrimental to regression.)

2.3. Defining balanced classes in practice
Existing works show that optimizing imbalanced classes
is problematic [26, 49]. In practice, researchers opt for using balanced classes in combination with regression [30,44, 46, 51]. Here, the data is imbalanced, however we arefree to define the class ranges such that we obtain balanced
classes over imbalanced data sampling. To empirically test
the added value of using balanced classes, we need a way todefine balanced classes over imbalanced data sampling.
Given an imbalanced data sampling, we bin samples intoclasses, using uniform class ranges. This generates imbalanced classes, which we then re-balance by redefining theclass ranges such that the class histogram is approximately
uniform. To this end, we apply histogram equalization over
the original classes:
  \label {eq:balance_cls} q(k) = \left \lfloor \frac {C}{N} \sum _{j=1}^k \mathcal {H}_C(j)\right \rfloor ,(12)

Figure 4. Effect of the λ hyperparameter on the validation:
We evaluate a range of values λ∈{1e−3, 1e−2, 1e−1, 1, 1e+1,
1e+2, 1e+3, 1e+4} on the validation set. The shading represents
the standard deviation of the MSE error across 3 repetitions. Setting λ correctly is essential when using a classification loss next to
the regression loss.

3.1. Hypothesis analysis on 1D data

regression with classification — reg+cls from Eq. (4). We
start with the 1D data because it is easily interpretable andit offers a controlled environment to test the hypothesis thatclassification helps regression and to analyze the propertiesof the classes. 1
Effect of the λ hyperparameter. We perform hyperparameter search on the validation set for setting the λ in
Eq. (4). We vary λ∈{1e−3, 1e−2, 1e−1, 1, 1e+1, 1e+2,
1e+3, 1e+4}. Fig. 4 shows the MSE across 3 repetitions,when considering different number of classes, for clean andnoisy data scenarios, across sampling variations. Higher
values of λ typically perform better in this case. When thesampling of the data is imbalanced, there exists a value of
λ such that the reg baseline is outperformed by the reg+cls.
We use the best λ values found on the validation, when evaluating on the test set.
Effect of balancing the classes on imbalanced data. We
numerically evaluate in Tab. 1 if using balanced classes(as defined in Eq. (12)-Eq. (13)) is less sensitive to the
choice of λ. We perform 3 repetitions over all dataset
scenarios and sampling cases, and vary λ∈{1e−3, 1e−2,
1e−1, 1, 1e+1, 1e+2, 1e+3, 1e+4}. For this we consider
the percentage of runs (across different number of classes,
random seeds, and values of λ) where classification helps
regression — where reg+cls MSE is lower than reg MSE.
Ideally this number should be close to 100%. Balancing theclasses is more robust to the choice of λ, as on average there

We use the 10 randomly sampled 1D functions to further analyze the regression loss — reg from Eq. (3), and1 Our source code will be made available online, at the address:
https://github.com/SilviaLauraPintea/reg-cls.

where ⌊x⌋ rounds x down to the nearest integer, and HC (·)
computes the histogram of the samples per class, and q(·) isa mapping function that maps the old classes indexed byk∈{1, .., C} to a new set of classes {1, .., C}. Eq. (12)merges class ranges such that their counts are as close as
possible. Thus, the number of equalized classes is lower or
equal to the original number of classes, C ≤ C.
After class equalization, the new classes are not perfectly uniform. We further define a class-keeping probability ρ(k), as the ratio between the minimum class countand the current equalized class count HC (k):  \label {eq:bal_keep} \rho (k) = \frac {\min _{j=1}^{\overline {C}} \mathcal {H}_{\overline {C}}(j)}{\mathcal {H}_{\overline {C}}(k)},(13)

where HC (·) computes the histogram of equalized classes.
Selecting training samples using only Eq. (13), without firstequalizing the classes, will lead to never seeing samples
from the most frequent classes. During training, for the regression loss we use all samples, while for the classification
loss we pick samples (x, yk ) with a probability defined by
ρ(k). More details are in the supplementary material.

3. Empirical analysis

Imbalanced classes (↑)

Balanced classes (↑)

Dataset case

Clean

Noisy data

Clean

Noisy data

Uniform
Mild
Moderate
Severe55.42%
54.17%
47.50%
36.83%

59.83%
55.75%
56.58%
49.08%

59.00%
57.67%
47.50%
34.25%

65.00%
62.42%
56.00%
46.25%

Avg

48.48%

55.31%

49.60%

57.42%

Table 1. Effect of balancing the classes: On the validation sets
we test how sensitive reg+cls is to the choice of λ when balancing
classes versus when using imbalanced classes. For this we varyλ∈{1e−3, 1e−2, 1e−1, 1, 1e+1, 1e+2, 1e+3, 1e+4}. And wemeasure the percentage of runs where adding a classification loss
improves the regression predictions. Using balanced classes is lesssensitive to the choice of λ.

Figure 5. Effect of noisy targets: We vary the amount of noisein the targets, along the y-axis. We plot both using imbalanced
classes (green), and using balanced classes (lime) compared to thereg baseline (red). We report MSE on the test sets across 5 repetitions. Despite the the noise level increasing drastically, the benefitsof adding a classification loss to the regression loss remain visible.

are more runs where classification helps regression.
Effect of the noisy targets. In Fig. 3 we considered a single noise level for the Noisy data scenario. Here, we further analyze the effect on the MSE scores when varying thenoise level in the targets for the baseline reg (in red) andthe reg+cls with imbalanced classes (in green) as well as
reg+cls bal where the classes are balanced (in lime color).
We vary the level of noise σ of the targets on the y-axis
σ∈{0.05, 0.1, 0.5}, and plot mean MSE and standard deviation on uniform test sets across 5 repetitions. For the balanced case, we indicate the original number of classes on
the x-axis, while in practice the number of balanced classes

is typically 2× lower than the initial one. Fig. 5 shows that
despite severely increasing the noise level, adding a classification loss remains beneficial when the data is imbalanced.

3.2. Realistic image datasets
Imbalanced realistic image datasets. We run experimentson two realistic imbalanced datasets from Yang et al. [47]:depth estimation on the NYUD2-DIR dataset, and age estimation on the IMDB-WIKI-DIR. The supplementary material plots dataset statistics. For both datasets we use
the Adam optimizer and set the learning rate to 1e−4 for
NYUD2-DIR with 5 epochs1 and batch size 16 accumulated over 2 batches (to mimic batch size 32 on 2 GPUs),
while for IMDB-WIKI we use a learning rate of 1e−3 for
90 epochs and batch size 128. When comparing with Ren
et al. [31] we use their best results (their GAI method). We
also use the evaluation code provided in Yang et al. [47]and we report RMSE (root mean squared error) on NYUD2DIR and MAE (mean absolute error)/MSE (mean squared
error) for IMDB-WIKI-DIR as also done in [31, 47]. When
re-running the baseline reg results we observed a large variability across different runs by varying the random seed,especially on the NYUD2-DIR dataset, therefore we report results averaged over 3 random seeds. We use the architecture of Yang et al. [47]: ResNet-50 [17] for IMDBWIKI-DIR, and the ResNet-50-based model from [19] for
NYUD2-DIR.
For adding the classification loss on IMDB-WIKI-DIR
we append, only during training, a linear layer of size [F, C]
followed by a softmax activation and a cross-entropy loss,
where F is the number of channels in the one-to-last layer.
For the NYUD2-DIR the predictions are per-pixel, thus we
use the segmentation head from Mask R-CNN [16] composed of a transposed convolution of size 2×2, ReLU, anda 1×1 convolution predicting the number of classes, C. At
test time the classification branch is not used. We estimate
the λ hyperparameter using the validation set provided in[47] on IMDB-WIKI-DIR. For NYUD2-DIR we define avalidation set by randomly selecting 1/5 of the training directories with a seed of 0, and we use the same training/validation/test split for all our results. For NYUD2-DIR we
use λ=1.0 for 100 classes and λ=0.1 for 10 and 2 classes.
For IMDB-WIKI-DIR we set λ=0.1 for 100 classes and
λ=1.0 for 10 and 2 classes. We compare the reg results with
reg+cls adding the classification loss, and reg+cls bal. withbalanced classes, where we consider 2, 10 and 100 classes.
Tab. 2 shows the RMSE results on NYUD2-DIR whentraining with the standard MSE loss compared to adding aclassification loss. We report RMSE on the test, where thebest model is selected on the validation set across epochs— RMSE-val. To compare with previous work who selects
the best model on the test, we also report this as RMSE-test
1 Using more epochs on NYUD2-DIR seems to lead to overfitting.

NYUD2-DIR RMSE-val (↓)
Samples

NYUD2-DIR RMSE-test (↓)

All

Many

Med.

Few

All

Many

Med.

Few

Kernel [47]Balanced [31]reg (MSE)——
1.614 (±0.051)——
0.554 (±0.002)——
0.934 (±0.042)——
2.360 (±0.081)

1.338
1.251
1.499 (±0.083)

0.670
0.692
0.578 (±0.010)

0.851
0.959
0.896 (±0.043)

1.880
1.703
2.171 (±0.136)

reg+cls (2 cls)
reg+cls (10 cls)
reg+cls (100 cls)

1.587 (±0.026)
1.576 (±0.063)
1.536 (±0.090)

0.618 (±0.003)
0.585 (±0.008)
0.569 (±0.018)

1.062 (±0.025)
0.982 (±0.058)
0.966 (±0.041)

2.278 (±0.041)
2.282 (±0.095)
2.222 (±0.146)

1.532 (±0.082)
1.509 (±0.022)
1.488 (±0.028)

0.624 (±0.036)
0.582 (±0.013)
0.578 (±0.015)

0.946 (±0.032)
0.947 (±0.046)
0.971 (±0.049)

2.204 (±0.141)
2.178 (±0.047)
2.141 (±0.045)

reg+cls bal. (2 cls)
reg+cls bal. (10 cls)
reg+cls bal. (100 cls)

1.599 (±0.020)
1.454 (±0.044)
1.553 (±0.117)

0.616 (±0.026)
0.607 (±0.041)
0.563 (±0.033)

1.033 (±0.058)
0.965 (±0.023)
0.897 (±0.043)

2.304 (±0.044)
2.077 (±0.087)
2.263 (±0.193)

1.522 (±0.060)
1.454 (±0.044)
1.487 (±0.051)

0.665 (±0.033)
0.607 (±0.041)
0.574 (±0.014)

1.003 (±0.059)
0.965 (±0.023)
0.869 (±0.019)

2.166 (±0.118)
2.077 (±0.087)
2.156 (±0.084)

Table 2. Imbalanced realistic image data: NYUD2-DIR depth estimation. We evaluate the baseline reg trained with MSE, and thereg+cls variants. We report RMSE when the best model is selected on the validation (RMSE-val ), or as in [47] on the test set (RMSEtest). We average over 3 different random seeds. Adding a classification loss helps regression, which validates our hypothesis.
IMDB-WIKI-DIR MAE(↓)
Samples

IMDB-WIKI-DIR MSE(↓)

All

Many

Med.

Few

All

Many

Med.

Few

Focal [27]Kernel [47]Balanced [31]reg (MAE)

7.97
7.78
8.12
8.09 (±0.01)

7.12
7.20
7.58
7.23 (±0.02)

15.14
12.61
12.27
15.48 (±0.15)

26.96
22.19
23.05
26.81 (±0.48)

136.98129.35—138.53 (±1.17)

106.87106.52—107.82 (±0.96)

368.60311.49—375.27 (±6.97)

1002.90811.82—1017.59 (±27.51)

reg+cls (2 cls)
reg+cls (10 cls)
reg+cls (100 cls)

7.95 (±0.05)
7.93 (±0.06)
7.61 (±0.02)

7.11 (±0.03)
7.12 (±0.06)
6.90 (±0.03)

15.08 (±0.27)
14.93 (±0.17)
13.46 (±0.44)

26.15 (±0.06)
25.91 (±0.27)
25.04 (± 0.82)

135.15 (±0.40)
135.69 (±1.65)
129.73 (±1.33)

105.86 (±0.27)
106.58 (±1.42)
103.25 (±1.12)

361.02 (±7.00)
359.27 (±5.70)
328.17 (± 10.40)

973.53 (±11.52)
975.37 (±34.09)
933.33 (± 9.16)

reg+cls bal. (2 cls)
reg+cls bal. (10 cls)
reg+cls bal. (100 cls)

7.94 (±0.06)
7.92 (±0.06)
7.59 (±0.09)

7.14 (±0.09)
7.10 (±0.05)
6.86 (±0.08)

14.71 (±0.51)
14.99 (±0.21)
13.61 (±0.35)

26.20 (±0.73)
25.58 (±0.39)
25.30 (±0.19)

134.91 (±1.20)
134.53 (±0.75)
127.87 (±1.99)

106.36 (±1.68)
105.40 (±0.43)
101.34 (±1.74)

351.48 (±14.82)
362.48 (±4.41)
327.55 (±12.98)

980.61 (±39.44)
940.91 (±18.70)
925.65 (±21.96)

Table 3. Imbalanced realistic image data: IMDB-WIKI-DIR age estimation. We evaluate the baseline reg when trained with MAE,
and the reg+cls variants. We report test MSE and MAE on the test set, averaged over 3 repetitions with different random seeds. Adding
a classification loss next to the regression loss is specifically beneficial on this dataset, where this simple yet popular strategy performs on
par with state-of-the-art. (We use “—” where the authors’ results are missing).

.
Balanced NYUD2-DIR
reg (MSE)
reg+cls

All RMSE-val (↓)

All RMSE-test (↓)

1.442 (±0.077)
1.456 (±0.033)

1.492 (±0.042)
1.593 (±0.025)

Balanced IMDB-WIKI-DIR
reg (MSE)
reg+cls

All MAE (↓)

All MSE (↓)

7.74 (±0.04)
7.71 (±0.06)

131.03 (±1.44)
131.27 (±1.00)

Table 4. Balanced realistic image data: NYUD2-DIR depth estimation and IMDB-WIKI-DIR age estimation. We compare
the reg and reg+cls (using 100 classes) when the data is balanced.
We report RMSE and MAE/MSE, respectively, across 3 repetitions. There are no clear improvements when adding a classification loss to the regression on balanced data.

(despite this being a bad practice). Additionally, note that
our training set is slightly smaller because of using a validation set, so the reg results are worse than in [47] (i.e. 1.477
RMSE-test). We observe that there is an inconsistency between the training and test set, as the best model on the test
set does not correspond to the best model on the validation
set (which is a subset of the training). Despite all these,
adding a classification loss still improves across all classoptions, when selecting the best model on the validation set,

and for 100 classes and 10 balanced classes, when selecting
the best model on the test set. This may be due to the classifier overfitting on the training data for fewer classes.
Tab. 3 gives the MAE and MSE results on IMDB-WIKIDIR when using the standard MSE loss during training
compared to when adding the classification at training time.
The best results are obtained using 100 balanced classes.
Here, adding a classification loss not only improves over the
regression baseline, but it is also on-par with state-of-the-art
methods specifically designed for imbalanced regression,
such as [32, 47]. Adding a classification loss is similar to[32, 47], who define smooth classes over the data.
Balanced realistic image datasets. On the same twodatasets: NYUD2-DIR and IMDB-WIKI, we test the effectof adding a classification loss when we re-balance the databy binning the targets into 100 bins and selecting samples
per batch during training as defined as in Eq. (12)-Eq. (13)for both reg and reg+cls. Because we mask samples perbatch to balance the training data, for IMDB-WIKI here
we use a batch size of 128 and learning rate 1e−4 for 90
epochs. While for NYUD2-DIR we use a learning rate of
1e−4 and batch size of 8, accumulated over 4 batches (tomimic a batch of 32 on 1 GPU), for 5 epochs. Tab. 4 showsBreakfast RMSE % (↓)
Dataset split
Random baseline[24]
reg (RMSE)
reg+cls (100 cls)
reg+cls bal. (100 cls)

Breakfast RMSE frames (↓) – unnormalized

S1

S2

S3

S4

S1

S2

S3

S4

58.50 (±0.29)
31.24 (±0.80)
32.57 (±1.45)

58.37 (±0.07)
31.49 (±0.98)
33.30 (±1.69)

58.39 (±0.14)
30.85 (±0.66)
32.52 (±1.17)

58.38 (±0.12)
30.78 (±0.53)
33.08 (±1.49)

1394.48 (±992.50)
1079.38 (±775.17)
860.17 (±583.44)

1511.04 (±1132.60)
1235.02 (±932.89)
891.39 (±641.58)

1450.26 (±1085.56)
1172.67 (±880.69)
862.65 (±609.01)

1420.21 (±1063.16)
1170.07 (±886.92)
845.48 (±605.46)

28.71 (±0.38) 28.84 (±0.55) 28.46 (±0.48) 28.44 (±0.50)————(a) Percentage prediction (normalized).

837.59 (±573.15)
837.61 (±573.17)

870.55 (±630.89)
845.65 (±601.73)
870.54 (±630.88)
845.65 (±601.72)(b) Frame prediction (unnormalized).

809.76 (±595.37)
809.76 (±595.37)

Table 5. Imbalanced video progress prediction on Breakfast. We report mean RMSE and standard deviations averaged over all 10
cooking tasks of the Breakfast dataset. (a) Progress prediction in video percentages. (b) Progress prediction in frame numbers. The
overall progress prediction results leave space for improvements for all methods, because of the dataset challenges. However, also for thisregression problem adding a classification loss has benefits.

the results across 3 repetitions. When the data is alreadybalanced, adding a classification loss has limited effect.

3.3. Imbalanced video progress prediction
As an additional investigation on imbalanced data, we
explore videos which are naturally imbalanced in the number of frames. We perform video progress prediction on
the Breakfast video dataset [23] containing 52 participants
performing 10 cooking activities. We follow the standard
dataset split (S1, S2, S3, S4) [23] and use the corresponding train/test splits. We adopt the method in [24] and train asimple MLP on top of IDT (improved dense trajectory) features [41] of dimension 64 over trajectories of 15 frames.
We evaluate RMSE when predicting either video progresspercentages, or absolute frame numbers. Kukleva et al.
[24] use an MLP with 3 linear layers and sigmoid activations. We change the sigmoid activations into ReLU activations for the reg and reg+cls models, since it works better when predicting absolute frame numbers. For all methods we keep the training hyperparameters from [24]: learning rate 1e−3, Adam optimizer, and training for 40 epochswith the learning rate decreased by 0.1 at 30 epochs. We
also report the random baseline results when using the untrained model. The data is imbalanced, in the sense thatvideo lengths vary widely (see supplementary material fordata distribution). We test again if adding a classification
loss can benefit the regression predictions when using 100
classes in the reg+cls. We search for λ on a validation
set created by randomly selecting 1/3 of the training videos
with a seed of 0. We use λ=100 when predicting percentages, and λ=10 when predicting frame numbers. At training time, we add the classification loss via a linear layer on
top of the one-to-last layer, followed by softmax. We train
one model per task and report mean and standard deviations
over all 10 tasks.
Tab. 5 depicts the results across all 4 splits. We report RMSE scores when predicting video progress in termsof percentages in Tab. 5(a), and when predicting video
progress in terms of frames in Tab. 5(b). Even if we predictvideo progress percentages between [0,100]% in Tab. 5(a),because the video length varies widely, for some videos wewill have a lot more frame than for others, causing the data
sampling to still be imbalanced. In both cases there is gainfrom adding a classification loss to the regression loss.

4. Discussion and limitations
Relation to nonlinear ICA. Hyvärinen et al. [21] showthat there is a relation between learning to bin a continuous function and performing nonlinear ICA. Hyvärinen etal. [21] start from a continuous signal x generated by anon-linear combination of source-signals: x=f (s), wheref (·) is a non-linear function and s are the independentand non-stationary sources. They split the signal into C
temporal segments x={x1 , x2 , .., xC } and train an MLP
to predict for every sample xki the segment it belongs to:g(xki , θ)=k, k∈{1, .., C} and xki ⊂xk , and θ are the MLP
parameters. Hyvärinen et al. [21] prove that the last hidden layer of the MLP hg (·, θ) recovers the original sourcess within a linear transformation: hg (xki , θ) = wski + b,
where w, b define a linear transformation. Intuitively,
Hyvärinen et al. [21] discretize the signal into segments andclassify the segments, thus performing classification on acontinuous function. However, they do not focus on combining the binning with optimizing a regression problem.
Similar to Hyvärinen et al. [21], predicting discrete signal bins has been successfully used for unsupervised video
representation learning by discriminating between video
segments [9] or classifying video speed [42]. Up to thepoint which the underlying continuous regression function(e.g. speed, time, age, depth) can be assumed to be generated by a non-linear combination of non-stationary source(whose statistics change with the function), adding a classification loss decorrelates the independent sources. Additionally, we hypothesize that there may be a relation between nonlinear ICA and imbalanced sampling: the independent sources s are also continuous and shared acrosssamples. And having the hidden representation of the MLP
constrained to be independent across dimensions may leadto a better use of sparse samples in certain areas of thetarget-space. However, we leave this for future research.
Limitations of analysis. The analysis performed here is
still elementary and only aims to scratch the surface on theusefulness of adding a classification loss when performing
regression. A number of things have been disregarded here
such as: the effect of the model depth, while keeping themodel size fixed. Additionally, the choice of the optimizer
and the loss function during training may also play an important role. Finally, delving more into the relation between
nonlinear ICA and adding a classification loss to the regression, may be an interesting future research avenue.

regressing the precise value in each bin. Similarly, a jointclassification loss over discretized targets and a regression
loss is effective for horizon line detection [44]. Here, wewant to analyze why these prior works opt for discretizing
regression targets. More specifically, we analyze why andwhen a combination of a regression loss and a classification
loss over discretized continuous targets improves results.

5.3. Regression on imbalanced data

5. Related work
5.1. Improved deep regression
A thorough analysis of the effect of deep architecture
choices on regression scores is performed in [25]. Rather
than considering architecture choices, other works focus onregression robustness – defined as less influence from outliers [15, 20]. The benefits of having a weighted regression
loss has been extensively analyzed in classic work such as[6, 7]. With a similar goal, Barron [1] proposes a general regression loss that can be adapted to well-known functions.
Minimizing a Tukey’s biweight function also offers robustness to outliers [2]. Similarly, a smooth L1 loss is used in[12] for improved object bounding-box regression. Mapping regression targets to the hyper-sphere can also aid regression [29]. Instead of focusing on the loss function, asmooth adaptive activation function is used in [18]. From adifferent direction, ensemble networks have been successfully used for improved regression estimates [8, 14, 39].
Deep negative correlation learning is proposed in [48] tolearn diversified base networks for deep ensemble regression. Dissimilar to these works, our goal is not proposing a
new network architecture or a new regression loss function,
but rather analyzing why a combination between a classification and a regression loss can lead to improvements.

5.2. Discretized regression targets
Instead of optimizing continuous object pose angles,[38, 36] discretize them into classes and perform classification. Continuous prediction can be obtained back fromdiscretized targets, by using soft-min for disparity learning[22]. Similarly, ages are discretized in classes and the final
prediction is computed as an expectation over class probabilities in [34]. Rather than using predefined classes, clusters can be defined and, again, the final predication is aweighted sum over clusters [40, 43]. In a similar manner,a weighted combination of clustered regression targets isused for finding surface normals in [43]. Popular object detectors [28, 32, 33, 45] also rely on a set of predefined box
locations that can be seen as bin centers, and regress the final box locations with respect to these centers. An explicit
combination of regression and classification losses has beenshown to improve results for object orientation regression[30, 50, 51] by first splitting orientations into bins and thenPrior work has shown that imbalanced sampling can negatively affect the regression scores, and proposed ways tomitigate this by designing better data sampling techniques[4, 37]. How rare a data point is in the training set, can bemodeled with kernel density estimation [35]. Not only thedata sampling but also the target distribution can be fixed bysmoothing the distribution of both labels and features using
nearby targets [47]. Similarly, the learning model can be
regularized such that samples that are close in label space
are also close in the feature space [13]. While focusing
on the learning model, ensemble methods are a viable solution when working with imbalanced regression problems[5]. Instead of focusing on the data sampling or the model,
imbalanced regression estimates can be improved by adapting the MSE (mean squared error) loss [31]. Most similar
to us are [32, 47] whose methods can be seen as defining
smooth classes over the data. However, they aim to improve regression on imbalanced data, while we set off toanalyze in which cases adding a classification loss can help
regression, and what is the motivation behind this.

6. Conclusion
Here, we present a preliminary analysis on the effect ofadding a classification loss to the regression loss. We makethe observation that adding a classification loss to the regression loss has been used in computer vision for deep regression [30, 44, 46]. And we empirically test across data
variations and data samplings on a set of 1D functions, theeffect of adding a classification loss to a regression loss. We
find that for imbalanced regression, adding a classification
loss helps the most. Furthermore, we present an attempt atformalizing this observation starting from the derivations of
Ren et al. [31] for imbalanced regression.
Additionally, we validate that adding a classification loss
to the regression loss is beneficial on imbalanced real data,
where we evaluate on imbalanced image data on NUYD2DIR and IMDB-WIKI-DIR datasets [47], and imbalanced
video progress prediction on the Breakfast dataset [23].
Acknowledgements. This work was done with the support of the
Eureka cluster Program, IWISH project, grant number AI2021066. Jan van Gemert is financed by the Dutch Research Council(NWO) (project VI.Vidi.192.100).

References[1] Jonathan T Barron. A general and adaptive robust loss function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4331–4339,
2019. 8[2] Vasileios Belagiannis, Christian Rupprecht, Gustavo
Carneiro, and Nassir Navab. Robust optimization fordeep regression. In Proceedings of the IEEE international
conference on computer vision, pages 2830–2838, 2015. 8[3] Christopher M Bishop. Pattern recognition and machinelearning, volume 4. Springer, 2006. 2[4] Paula Branco, Luı́s Torgo, and Rita P Ribeiro. Smogn: a preprocessing approach for imbalanced regression. In First international workshop on learning with imbalanced domains:
Theory and applications, pages 36–50. PMLR, 2017. 8[5] Paula Branco, Luis Torgo, and Rita P Ribeiro. Rebagg: Resampled bagging for imbalanced regression. In Second International Workshop on Learning with Imbalanced Domains:
Theory and Applications, pages 67–81. PMLR, 2018. 8[6] William S Cleveland. Robust locally weighted regression
and smoothing scatterplots. Journal of the American statistical association, 74(368):829–836, 1979. 8[7] William S Cleveland and Susan J Devlin. Locally weighted
regression: an approach to regression analysis by local
fitting. Journal of the American statistical association,
83(403):596–610, 1988. 8[8] Corinna Cortes, Mehryar Mohri, and Umar Syed. Deepboosting. In International conference on machine learning,
pages 1179–1187. PMLR, 2014. 8[9] Ishan Dave, Rohit Gupta, Mamshad Nayeem Rizve, andMubarak Shah. Tclr: Temporal contrastive learning for video
representation. Computer Vision and Image Understanding,
219:103406, 2022. 7[10] Xin Ding, Qiong Zhang, and William J Welch. Classification beats regression: Counting of cells from greyscale microscopic images based on annotation-free training samples.
In CAAI International Conference on Artificial Intelligence,
pages 662–673, 2021. 1[11] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of theIEEE conference on computer vision and pattern recognition, pages 2002–2011, 2018. 1[12] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448,
2015. 8[13] Yu Gong, Greg Mori, and Frederick Tung. Ranksim: Ranking similarity regularization for deep imbalanced regression.
ICML, 2022. 8[14] Shizhong Han, Zibo Meng, Ahmed-Shehab Khan, and Yan
Tong. Incremental boosting convolutional neural network for
facial action unit recognition. Advances in neural information processing systems, 29, 2016. 8[15] Trevor Hastie, Robert Tibshirani, and Martin Wainwright.
Statistical learning with sparsity. Monographs on statistics
and applied probability, 143:143, 2015. 8[16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international
conference on computer vision, pages 2961–2969, 2017. 5[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and patternrecognition, pages 770–778, 2016. 5[18] Le Hou, Dimitris Samaras, Tahsin Kurc, Yi Gao, and Joel
Saltz. Convnets with smooth adaptive activation functions
for regression. In Artificial Intelligence and Statistics, pages
430–439. PMLR, 2017. 8[19] Junjie Hu, Mete Ozay, Yan Zhang, and Takayuki Okatani.
Revisiting single image depth estimation: Toward higher
resolution maps with accurate object boundaries. In 2019
IEEE winter conference on applications of computer vision(WACV), pages 1043–1051, 2019. 5[20] Peter J Huber. Robust statistics. In International encyclopedia of statistical science, pages 1248–1251. Springer, 2011.
8[21] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear
ica. Advances in neural information processing systems, 29,
2016. 7[22] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter
Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry.
End-to-end learning of geometry and context for deep stereo
regression. In Proceedings of the IEEE international conference on computer vision, pages 66–75, 2017. 1, 8
[23] H. Kuehne, A. B. Arslan, and T. Serre. The language of actions: Recovering the syntax and semantics of goal-directed
human activities. In Proceedings of Computer Vision and
Pattern Recognition Conference (CVPR), 2014. 2, 7, 8
[24] Anna Kukleva, Hilde Kuehne, Fadime Sener, and Jurgen
Gall. Unsupervised learning of action classes with continuous temporal embedding. In IEEE Conference on Computer
Vision and Pattern Recognition (CVPR’19), 2019. 7[25] Stéphane Lathuilière, Pablo Mesejo, Xavier AlamedaPineda, and Radu Horaud. A comprehensive analysis of deep
regression. IEEE transactions on pattern analysis and machine intelligence, 42(9):2065–2081, 2019. 8[26] Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng
Wang, Jintao Li, and Jiashi Feng. Overcoming classifier imbalance for long-tail object detection with balanced
group softmax. In Proceedings of the IEEE/CVF conferenceon computer vision and pattern recognition, pages 10991–
11000, 2020. 4[27] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, andPiotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer
vision, pages 2980–2988, 2017. 6[28] Wei Liu, Dragomir Anguelov, Dumitru Erhan, ChristianSzegedy, Scott Reed, Cheng-Yang Fu, and Alexander C
Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21–37, 2016. 8[29] Pascal Mettes, Elise van der Pol, and Cees Snoek. Hyperspherical prototype networks. Advances in neural information processing systems, 32, 2019. 8

[30] Arsalan Mousavian, Dragomir Anguelov, John Flynn, andJana Kosecka. 3d bounding box estimation using deep learning and geometry. In Proceedings of the IEEE conference
on Computer Vision and Pattern Recognition, pages 7074–
7082, 2017. 1, 4, 8
[31] Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu.
Balanced mse for imbalanced visual regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 7926–7935, 2022. 3, 5, 6, 8
[32] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster r-cnn: Towards real-time object detection with region
proposal networks. Advances in neural information processing systems, 28, 2015. 6, 8
[33] Gregory Rogez, Philippe Weinzaepfel, and Cordelia Schmid.
Lcr-net: Localization-classification-regression for humanpose. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pages 3433–3441, 2017. 8[34] Rasmus Rothe, Radu Timofte, and Luc Van Gool. Dex: Deep
expectation of apparent age from a single image. In Proceedings of the IEEE international conference on computer vision
workshops, pages 10–15, 2015. 1, 8[35] Michael Steininger, Konstantin Kobs, Padraig Davidson,
Anna Krause, and Andreas Hotho. Density-based weighting
for imbalanced regression. Machine Learning, 110(8):2187–
2211, 2021. 8[36] Hao Su, Charles R Qi, Yangyan Li, and Leonidas J Guibas.
Render for cnn: Viewpoint estimation in images using cnns
trained with rendered 3d model views. In Proceedings ofthe IEEE international conference on computer vision, pages
2686–2694, 2015. 8[37] Luı́s Torgo, Rita P Ribeiro, Bernhard Pfahringer, and Paula
Branco. Smote for regression. In Portuguese conference on
artificial intelligence, pages 378–389. Springer, 2013. 8[38] Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pages 1510–1519, 2015. 1,
8[39] Elad Walach and Lior Wolf. Learning to count with cnn
boosting. In European conference on computer vision, pages
660–676, 2016. 8[40] Jacob Walker, Abhinav Gupta, and Martial Hebert. Dense
optical flow prediction from a static image. In Proceedingsof the IEEE International Conference on Computer Vision,
pages 2443–2451, 2015. 1, 8
[41] Heng Wang and Cordelia Schmid. Action recognition withimproved trajectories. In Proceedings of the IEEE international conference on computer vision, pages 3551–3558,
2013. 7[42] Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu.
Selfsupervised video representation learning by pace prediction.
In European conference on computer vision, pages 504–521.
Springer, 2020. 7[43] Xiaolong Wang, David Fouhey, and Abhinav Gupta. Designing deep networks for surface normal estimation. In
Proceedings of the IEEE conference on computer vision andpattern recognition, pages 539–547, 2015. 8[44] Scott Workman, Menghua Zhai, and Nathan Jacobs. Horizonlines in the wild. BMVC, 2016. 1, 4, 8[45] Yu Xiang, Wongun Choi, Yuanqing Lin, and Silvio Savarese.
Subcategory-aware convolutional neural networks for object
proposals and detection. In 2017 IEEE winter conference
on applications of computer vision (WACV), pages 924–933.
IEEE, 2017. 8[46] Yang Xiao, Xuchong Qiu, Pierre-Alain Langlois, Mathieu
Aubry, and Renaud Marlet. Pose from shape: Deep pose
estimation for arbitrary 3d objects. CoRR, 2019. 1, 4, 8
[47] Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, andDina Katabi. Delving into deep imbalanced regression. In International Conference on Machine Learning, pages 11842–
11851. PMLR, 2021. 2, 5, 6, 8
[48] Le Zhang, Zenglin Shi, Ming-Ming Cheng, Yun Liu, JiaWang Bian, Joey Tianyi Zhou, Guoyan Zheng, and Zeng
Zeng. Nonlinear regression via deep negative correlation
learning. IEEE transactions on pattern analysis and machine
intelligence, 43(3):982–998, 2019. 8[49] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, andJiashi Feng. Deep long-tailed learning: A survey. IEEE
Transactions on Pattern Analysis and Machine Intelligence,
2023. 4[50] Chenchen Zhao, Yeqiang Qian, and Ming Yang. Monocularpedestrian orientation estimation based on deep 2d-3d feedforward. Pattern Recognition, 100:107182, 2020. 8[51] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. CoRR, 2019. 1, 4, 8

A step towards understanding why classification helps regression
Supplementary material

A. 1D experimental details
For the 1D synthetic experiments we use the function f (·) defined by a sum of two sines with different frequencies and amplitudes:  f (x)=a \sin (cx) + b \sin (dx), \label {eq:1d_func}(14)

where f (x) ∈ [−1.5, 1.5] and x ∈ [−1, 1]. We show here in Fig. 6 the 10 functions we have sampled from this function for the 1D
experiments presented in the paper.

Figure 6. The 10 sampled 1D functions used in the experiments in the main paper. Over these 10 functions we vary the sampling into:
4 sampling scenarios (uniform, mild, moderate and severe), 3 data scenarios (clean, noisy, and ood ), with 5 different seeds and 5 classes(22 , 24 , 26 , 28 , 210 ) resulting in 3000 experiments on the 1D data.
For the validation set we run the experiments over 3 repetitions with seeds: {0, 421, 8125}, while for testing we run the experimentsby setting the random seed everywhere in the code to {0, 421, 8125, 2481, 849}. For each dataset scenario (clean, noisy data, ood ) andsampling (uniform, moderate, mild, severe) we create 5 dataset versions. During training for every random seed we pick a dataset version,
giving rise to 5 repetitions per function, over which we report results in the Fig. 3 in the paper. We print here in Tab. 6 the model architecture
for the regression-only and regression and classification tasks.

Layer

Output Shape

Param

Layer

Output ShapeParam

Linear1[-1, 6]12
ReLU1[-1, 6]—
Linear2[-1, 16]112
ReLU2[-1, 16]–
Linear3a[-1, 1]17
Linear3b[-1, C]
16C(a) Regression only(b) Regression and classification
Table 6. MLP network architectures for the 1D experiments. When adding the classification loss during training, we use an additional
linear layer (emphasized here) which maps the features of the one-to-last layer to the desired number of classes, C. During testing thislayer is discarded.
Linear1
ReLU1
Linear2
ReLU2
Linear3[-1, 6][-1, 6][-1, 16][-1, 16][-1, 1]

12
—
112
–
17

Fig. 7 shows examples of predictions when using balanced and imbalanced classes on 3 severe sampling scenarios. The classification
methods can make different mistakes, but overall the predictions are more accurate than for the regression alone.

B. Re-balancing the 2D imbalanced training data
For re-balancing the classes, we first redefine the class ranges to have equalized class counts using Eq. (12). For the regression loss we
use all samples in each batch during training. For the classification loss we subsample the samples in the batch by considering the current

1.0

f(x)

0.5
0.0
0.5
1.0

101

102

0.50

0.00

0.50

0.50

x

log-counts of f(x)

0.00

0.50

0.50

0.00

0.50

x

x

1.0

f(x)

0.5

0.0

0.5

1.0
101

102

0.50

0.00

0.50

0.50

x

log-counts of f(x)

0.00

0.50

0.50

0.00

0.50

x

x

1.0

f(x)

0.5
0.0
0.5
1.0

101

102

103

log-counts of f(x)

0.50

0.00

x

0.50

0.50

0.00

0.50

x

0.50

0.00

0.50

x

Figure 7. Examples of 1D predictions: For moderate/mild/severely imbalanced data sampling, on clean data scenario, when using 64
classes. The classification variants: with balanced classes (lime) and with imbalanced classes (green) make more accurate predictions than
the regression baseline (red).

(a) Original NYUD2-DIR targets distribution.

(b) NYUD2-DIR probabilities of keeping samples per class.

(c) NYUD2-DIR re-balanced samples.

(c) Original IMDB-WIKI-DIR targets distribution.

(d) IMDB-WIKI-DIR probabilities of keeping samples per class.

(e) IMDB-WIKI-DIR re-balanced samples.

Figure 8. Re-balancing the training distribution in the classification loss. (a) The initial distribution per class for the NYUD2-DIR
dataset. (b) The probabilities of samples being retained per class, as described in Eq. (13). (c) The NYUD2-DIR samples re-balanced perclass as described in Section 2.3. (d) The initial distribution per class for the IMDB-WIKI-DIR dataset. (e) The probabilities of samples
being retained per class, as described in Eq. (13). (f) The IMDB-WIKI-DIR samples re-balanced per class as described in Section 2.3.
Here we consider 100 classes for binning the dataset targets. Equalizing the class ranges using Eq. (12) and sampling samples per classover the training epochs using Eq. (13) is effective at balancing the classes.

training sample (x, yk ) in the classification loss with probability ρ(k), where ρ(k) is defined in Eq. (13). Specifically, for every training
sample (x, yk ) we sample a random variable u from the uniform distribution and use this sample in the classification loss if the random
value is smaller than ρ(k):  & u \sim \mathcal {U}(0,1),\\ \text {we classify }(\mathbf {x}, y_k) \text { if } & u \le \rho (k).
(16)
This procedure is useful, because the complete dataset will be visited during training for classification (provided sufficient epochs), yet
the samples seen per class will be equal. To illustrate this, we show in Fig. 8(a) the original data distribution on NYUD2-DIR, and
IMDB-WIKI-DIR, the probabilities per class of keeping a sample for 100 equalized classes in Fig. 8(b), and the re-balanced training set
distribution in Fig. 8(c). Here we consider 100 classes to bin the training targets. We see that after class equalization and re-balancing theclass distribution is uniform. We could have only used the second option to balance the classes by sampling uniform samples per classusing Eq. (13) without first equalizing the classes with Eq. (12). However, if the classes are severely imbalanced (as in the case of our data),some class probabilities are extremely low which leads to never selecting samples from those classes, and thus having the classification fail
to converge.

C. Breakfast dataset task variations
Fig. 9 shows the large variation in video lengths across the 10 tasks on the Breakfast dataset on the training data. This large variability
makes predicting video progression extra challenging, even when predicting video progression in percentages. This is due to having amost-frequent task length, and the model learns to predict better for the videos belonging to the most-frequent task length, and it makes
mistakes when predicting progression on the videos that are a lot shorter or a lot longer than the average.

(a) Cereals(b) Coffee(c) Fried egg(d) Juice(e) Milk(a) Pancake(b) Salad(b) Sandwich(c) Scrambled egg(d) TeaFigure 9. Breakfast task length variations. Here we consider 100 classes for binning the video lengths. There is a considerable variationin the video lengths for the same cooking task. Because of this imbalance in the training data, the model makes more errors when predicting
video progression on videos that are either considerably longer or shorter than the average. Next to this, the large variety in appearance ofthe videos makes the video progress prediction challenging.