SwitchLoss: A Novel Optimization Scheme for Imbalanced Regression

Authors: Jovana Aleksic, Miguel García-Remesal
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: SwitchLoss, Cost-sensitive Methods, Imbalanced Regression, High-dimensional Regression

Abstract:
In the realm of machine learning, conventional techniques like neural networks often encounter challenges when dealing with imbalanced data. Unfortunately, imbalanced data is a common occurrence in real-world datasets, where collection methods may fail to capture sufficient data within specific target variable ranges. Additionally, certain tasks inherently involve imbalanced data, where the occurrences of normal events significantly outweigh those of edge cases. While the problem of imbalanced data has been extensively studied in the context of classification, only a limited number of methods have been proposed for regression tasks. Furthermore, the existing methods often yield suboptimal performance when applied to high-dimensional data, and the domain of imbalanced high-dimensional regression remains relatively unexplored. In response to the identified challenge, this paper presents SwitchLoss, a novel optimization scheme for neural networks, and SwitchLossR, a variant with a restricted search space. Diverging from conventional approaches, SwitchLoss and SwitchLossR integrate variable loss functions into the traditional training process. Our assessment of these methods spans 15 regression datasets across diverse imbalanced domains, 5 synthetic high-dimensional imbalanced datasets, and two imbalanced age estimation image datasets. Findings from our investigation demonstrate that the combined utilization of SwitchLoss and SwitchLossR not only leads to a notable reduction in validation error, but also surpasses prevailing state-of-the-art techniques dedicated to imbalanced regression.

Decision: reject
Reviewer scores: 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730467477188

thread_type:
official_review

<thread_object>
creator: reviewer_i2kq

<content>
summary:
The authors introduce SwitchLoss, an optimization framework in which loss function schemes are first selected in an exploration phase and then used in a training phase for neural network optimization.
The key idea is to alternate between several loss functions during optimization.
The proposed method and a restricted variation thereof are evaluated on imbalanced regression tasks ranging from standard tabular benchmark data to age estimation.

strengths:
- The paper is concerned with the interesting and practically important topic of imbalanced regression

weaknesses:
- The novelty of the proposed general SwitchLoss framework is limited. Consider e.g. (Sculley, 2010) who uses such an approach to optimize a combined loss of regression and ranking terms.

- I am highly doubtful regarding the experimental results. Consider the results given in Table 1. If I see it correctly, this table presents the results on the 15 standard datasets. First of all, the results do not coincide with the results in Appendix (see Table 6 and Table 7). Secondly, the indiidual rows do not sum up to 15. The first 4 itmes of each row do, while the "Combined" column is the sum of the SwitchLoss and SwitchLossR column. It is explained in the manuscript text, that this is the case because it harnesses the advantages of both and thus should be additive. To me, it did not become clear whether the experiments were actually run for a combined implementation or whether the other two columns where simply summed up. Also, I think this can not hold in general, as the exploration and training is done on train data. For the validation performance, I doubt that the number of winners for a combined approach is always the sum of the two others, because one approach may be better suited on the training data while another is advantageous on test data.
Additionally reporting only the number of best performing methods hides the margin between the methods performances. I understand that these numbers cannot be trivially aggregated for presentation, but I would like to see some RMSE values achieved by the methods at least in the appendix.

- I also question the experimental setup and choice of datasets. In many cases MSE outperforms SMOGN. Why is such a standard regression loss better suited than the SOTA for imbalanced regression?

- The presentation of the proposed method is highly redundant. The manuscript has 3 pseudocodes which are almost identical.

Literature:
Sculley, D. (2010, July). Combined regression and ranking. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 979-988).

Minor remarks:
- p. 7 line 377 "SwithLoss" -> "SwitchLoss"
- p. 8 Table 2 appears before Table 1

questions:
- Is the combined version of SwitchLoss and SwitchLossR implemented and evaluated or are the results for the runs of the individual versions summed up? 

- Why does MSE show such a competitive performance to SMOGN?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730509187528

thread_type:
official_review

<thread_object>
creator: reviewer_5hjb

<content>
summary:
The submission presents an approach to tackle deep regression problems that exhibit an imbalanced target variable. The basic idea is to switch between different loss functions during training; the submission considers MSE, KL-divergence, and a loss function that encourages the variance in the predicted target values to be similar to the variance in the ground truth. Given a grid on the number of epochs performed during training, the loss is switched randomly when an epoch in the grid is encountered.  This training process, with randomly switched loss functions, is repeated a certain number of times, as determined by a hyperparameter, and the model with the best performance, measured on validation data, is kept. The submission also has a modified version of this approach, where MSE is selected at every second grid point, and one of the other two losses is chosen randomly at the other grid points. In the submission's experiments with several configurations of fairly shallow multilayer perceptrons, a hybrid of the two approaches has a positive win/loss ratio when compared against a competing method for imbalanced regression called SMOGN, which is based on sampling. This hybrid also yields lower MSE than SMOGN on two age estimation datasets when ResNets are used.

strengths:
The idea of switching between loss functions during training appears novel, at least in the context of imbalanced regression. The proposed method is simple and appears to be a valid competitor when compared to sampling-based imbalanced regression.

weaknesses:
The submission does not present a strong theoretical justification for the proposed method.

There is no comparison to the DenseLoss method proposed by Steininger et al. that is cited in the paper.

The comparison to SMOGN does not seem entirely fair: the sequence of loss function can be viewed as a hyperparameter, which is optimized using random search on the validation data. Similarly, the hyperparameters of SMOGN should be optimized on validation data. The submission compares to SMOGN with default parameters instead.

The majority of the results are presented in terms of win/loss statistics. It is unclear how large the improvements in accuracy were.

Some important details seem to be missing:

a) KL-divergence requires probability estimates, and it is not stated how they were obtained.

b) It is not stated how the bins for the target variable were created.

c) It is not specified how the validation and the test data were balanced

There is a lot of redundancy in the pseudo code presented in the paper. Showing Procedure 2, a specialized version of Procedure 1, is not a good use of space. Similarly, Procedure 3 involves a minor modification of Procedure 2.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1731110296820

thread_type:
official_review

<thread_object>
creator: reviewer_qtc8

<content>
summary:
This paper investigate the possibility of addressing imbalanced data in regression problems using various loss function. In particular, this paper suggests to use thee different loss functions alternatively during the training process and shows empirically the performance can be improved (in terms of overall MSE and MSE for disjoint subsets of target space).

strengths:
The paper proposes a simple method to handle imbalanced dataset for regression tasks. The method is straight forward and easy to implement.

weaknesses:
1. The paper does not provide a rigorous  definition of imbalanced regression problem. There is not mathematical definition for imbalanced dataset. Does imbalanced data mean that the distribution of $Y$ has long tail and we do not see enough sample from the long tail? Is it possible that we have very few sample for a specific bin even if that bin has high probability? 

2. There is no clear definition of balanced test data. In the proposed algorithm, we need to have access to balanced test data set. How this balanced test data is generated? I am assuming that we need to split the target space to several bins and we should make sure that we have the same number of samples in each bin. However, it is not clear how we should split the target space. Depending how we define bins, the balanced dataset will be different. 

3. The experiment part is very limited. This paper is purely empirical paper. So in order to evaluate the algorithm, we need to see more experiments. For example, I did not find Table 2 useful. Because the results highly depend on how we define the bins for target space. I believe we can select the bins such that the proposed method looks better in terms of MSE in each bin. Should we discuss the impact of bins on the results? I believe more experiment is needed to understand the performance of the proposed under different target space split.

4. The authors compare their method only with one baseline that can handle imbalanced data. Is there any other baseline that we need to consider for imbalanced regression? Is there any method for imbalanced classification that can be extended to regression? 

5. The numbers in the tables are not reliable. In particular, there is no variance reported in the tables.

questions:
I want to ask authors to address the weaknesses that I pointed out above.
</content>
</thread_object>

</thread>


Metareview: