MT-GBM: A Multi-Task Gradient Boosting Machine  with Shared Decision Trees
Authors: Zhenzhe Ying
Zhuoer Xu
LANQING XUE
Changhua Meng
Weiqiang Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-Task learning
Classification

Abstract: Despite the success of deep learning in computer vision and natural language processing, Gradient Boosted Decision Tree (GBDT) is yet one of the most powerful tools for applications with tabular data such as e-commerce and FinTech. However, applying GBDT to multi-task learning is still a challenge. Unlike deep models that can jointly learn a shared latent representation across multiple tasks, GBDT can hardly learn a shared tree structure.

In this paper, we propose Multi-Task Gradient Boosting Machine (MT-GBM), a GBDT-based method for multi-task learning. The MT-GBM can find the shared tree structures and split branches according to multi-task losses. First, it assigns multiple outputs to each leaf node. Next, it computes the gradient corresponding to each output (task). Then, we also propose an algorithm to combine the gradients of all tasks and update the tree. Finally, we apply MT-GBM to LightGBM. Experiments show that our MT-GBM improves the performance of the main task significantly, which means the proposed MT-GBM is efficient and effective.

Under review as a conference paper at ICLR 2022
MT-GBM: A MULTI-TASK GRADIENT BOOSTING MACHINE WITH SHARED DECISION TREES
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the success of deep learning in computer vision and natural language pro- cessing, Gradient Boosted Decision Tree (GBDT) is yet of the most pow- erful tools for applications with tabular data such as e-commerce and FenTech. However, applying GBDT to multi-task learning is still challenge. Unlike deep models that can jointly learn a shared latent representation across multiple tasks, GBDT can hardly learn a shared tree structure.
In this paper, we propose Multi-task Gradient Boosting Machine (MT-GBM), a GBDT-based method for multi-task learning. The MT-GBM can find the shared tree structures and split branches according to multi-task losses. First, assigns multiple outputs each leaf node. Next, it computes the gradient corresponding to each output (task). Then, we also propose an algorithm to combine the gradi- ents of all tasks and update the tree. Finally, apply MT-GBM Lig.tGBM. Experiments show that our MT-GBM improves the performance of the main task significantly, which means the proposed MT-GBBM is efficient and effective.
INTRODUCTION
Gradient boosting is powerful machine-learning technique that achieves state-of-the-ar results in variety of applications. It is a process of constructing strong predictor by ensembling weak predic- tors and performing gradient descent in a functional space. Gradient boosting has achieved tremen- dous success in a wide range of machine learning tasks. It is popular in applications with heteroge- neous features, noisy tabular data, and complex dependencies, such as recommender systems (Wang et al. 2016; Wen al. 2019).. web search (Agrawal & Shanahan 2010), FinTech (Wang et al. 2018 Mei & Li-ren 2015; Yu-Zhuo & Xiao-Ni 2006). online advertising, etc (Roe et al.) 2004 Zhang & Haghani). Among all gradient boosting algorithms, XGBoost and LightGiMM are the two most famous algorithms. They have contributed to countless top solutions in machine learning and data mining competitions on tabular data.
Despite the considerable success of gradient boosting, applying GBDT models to multi-task learn- ing is still challenging. The GBDT model aims train gegh-peeformance discriminative model towards single loss function of single task. Therefore, it cannot learn a shared latent feature representation, which is essential for multi-task learning.
We list three things that limit the applicability of GBDT in multi-task learning.
First and foremost, the core of efficient multi-task learning is extract valuable hidden representa- tion that shares across all tasks, which is non-trivial for free-based ensemble learning methods like GBDT. For example, in news recommendation, user response prediction tasks include the prediction of click-through rate, dwell-time, thumbs, reviews, shares, and so on. These objects correspond to the same user; thus an informative representation of the user is the key to such multi-task learning problem. The recent development of deep learning achieve this naturally using hidden embed- dings and layers in neural networks (Ruder 2017 Huang et al. 20188] Gao et al. 2019). thus has shown its dominance for multi-tasi learning in computer vision and natural language processing (Su et al. 2013 Radford et al., 2019 Liu et al., 2011). However, is known that neural networks sometimes fail applications with tabular data.
Second, the splitting mechanism for decision trees aims to separate the input space as well as the training samples according single loss function, which could not be applied directly to multi-

---

Under review as a conference paper at ICLR 2022
task learning where there are multiple losses. When optimizing end-to-end neural networks, it is natural to sum up the gradients from all losses. However, for tree-based models, the splits that have maximal information gain for the losses can differ a lot, especially when training classification and regression tasks at the same time. Therefore, it is challenging to find tree splits that are suitable for all tasks.
Third, multi-task learning, the model should keep similar paces for each task for stable learning. For neural networks, current gradient descent methods with adaptive step size can adjust the learning rate for different tasks. However, the learning speed of GBDT is related to the residual errors. Therefore, when there are multiple tasks, a fixed learning rate is applied all tasks, the model tends to overfit certain tasks while underfit the others. Therefore, how to adjust learning rates for multiple tasks remains a challenge.
In this paper, we propose new algorithmic technique for multi-task learning based on GBDT, namely multi-task Gradient Boosting Machine (MT-GBM). The key novelty of MT-GBM is that it learns shared decision tree structures for multiple tasks, which can be understood as the hidden knowledge across all tasks. As the decision trees are shared, it requires almost the same com- putational overhead comparing to single task GBDT. Therefore, MT-GBM can be efficiently trained. We verified MT-GBM on two financial datasets. Experimental results show that MT- GBM can achieve significantly higher performance than existing state-of-the-an art gradient boosted decision trees methods, XGBoost (Roe et al., 2004), LightGBM (Prokhorenkova et al. 2018).
(Ke et al. 2017) and Cathoost
Our contribution can be summarized as follows:
We propose MT-GBM that learns an ensemble of shared decision trees learned from multi- ple losses simultaneously, which significantly improves the efficiency and effectiveness of GBDTs for multi-task learning.
We propose a new learning algorithm that adaptively adjusts the learning rate for each task by taking the scale of residual errors into consideration. It stabilizes training on arbitrarily complex problems.
Our method is easy implement upon existing GBDT models such as XGBoost and Light- GBM. We tested our method by modifying LightGBM and showed significant performance improvements.
PRELIMINARIES
2.1
MULTI-TASK LEARNING WITH DECISION TREES
widely known that multi-task learning could improve the performance of every single task by sharing knowledge from all tasks. When using max-margin machine learning (i.e., SVMs) or deep learning (i.c.. feed-forward neural networks), multi-task learning has achieved significant success (Caruana 1997: Evgeniou & Pontil 2004 Bingel & Søgaard 2017). However, with decision-tree- based models, it remains unclear how to learn shared knowledge from multiple tasks.
There are many potential reasons why adding extra outputs to model might improve generaliza- tion performance. For example, to the extent that tasks are uncorrelated, their contribution to the aggregated gradient, which guides the node splitting in GBDT, can appear as noise other tasks. Thus uncorrelated tasks might improve generalization by acting as a source of the noise. Figure (a) shows situation of data samples with a noisy label. One single sample contains two rectangles while the left represents label I and the right represents label 2. We can assume that task 1 clicks behavior and task shares behavior in the recommendation system. It makes clear that two tasks fix the splitting line from dotted line to solid line. Figure I (b) shows that Sub tasks also help the sample splitting when dataset has the class imbalance problem.
2.2 PROBLEM DEFINITION
GBDT is an ensemble model of decision trees, which are trained in sequence. In each iteration, GBDT leams decision tree by minimizing the negative gradients (also known as the residual
2

---

Under review as a conference paper at ICLR 2022
Big rough-group
Small strictly
tack
Takk1
Task
Task1
Task 2
Task2
Task 2-
Task
(+)
Figure I: (a) Samples with noisy tasks. Each sample contains two rectangles as tasks and the color means whether it is positive not. There some noises in each task in the middle of the figure. Take two tasks into consideration is better than any one of them;(b) Samples with sub tasks. Task subclass of task and it helps the model to split the small group from the big group which is not accurate enough. Finally model will predict a high probability for sample in small group and medium in big group.
errors) of previously trained trees. Specifically, to construct the decision trees, the algorithm finds the split points in the feature space according the residual emrors.
Assume we have n learning tasks E 21, n). each with a set of supervised data {(x1,11), y1), (mm: }mm). For simplicity, assume that the number of data points in each task is the same, and all the tasks share the same input space. For the 1-th task, there is a predefined learning objective I,, In regression, might consider the mean squared error:
L
y1 D2)²
(1)
1=1
And we use the negative log-likelihood as the classification loss:
1
m
L.
(y, logg; (1 yt) log(1 pp))
m
Finally, we assume the main task objective function Lı the metric to evaluate the model.
2.3 RELATED WORK
In this section, we ignore the related work of neural network because most of neural network work well in specific tabular dataset (Zhang Yang 2018
Ruder 2017)
To achieve multi-task tree, there were some works about multi-classifier tree on Adaboost (Faddoul al. 2012) This paper revises the information gain rule for learning decision trees and modified MT-Adaboost to combine multi-task Decision Trees as weak learners. Since its algorithm depends on their new equation of Information Gain, the model does not support doing the regression task yet. Obviously, it fails when it comes to residual enrors in GBDT cause the loss is float number even in classifier tasks.
Another approach to multi-task learning is the work sharing tree structure for different tasks (Chapelle et al. [011]) This paper presents a general solution boosting for different task types by minimizing the sum of losses "" Li It uses relative value to control the strength of the connec- tion between the tasks. We approve of the way about exploring the general solution. But the part of optimization in which splitting the node would be hard and slow when consider minimizing all loss functions. seems to be a bit complex and obsolete and is beaten by the existing implementations of gradient boosted decision trees such XGBoost and LightGBM.

---

Under review as a conference paper at ICLR 2022
However, the GBDTs has some tricks to deal with multi-class task (Fan al. 2017), i... training n trees in one iteration for n classes. When inferring, the trees belong to the specified class will be activated.
Algorithm 1 Traditional Multi-Class GBDT
Input: n the number of tasks, m the number of samples. Accumulative prediction of previous rounds:(P1.e..... Pm, 1t= Goundtruth label: (V1.s) Mm, -
Output: New decision trees T1, T2,... ..., Tn in this iteration.
1: for = 1..... m7 do
2: Pt,: - exp(p.
exp(ps.)
1, ,...,
3:
7 for ,, do
4:
// J is the number of nodes in the tree
5:
I/ N is the number of samples in the node j
6:
Uz.t = Vj.t Pj.t- j -,....,N N I/ Ryts means that the ith sample belongs to the j node at the fth task Il Titi means the prediction of sample at the ith task
7:
8:
9:
[Ryu 1/
J terminalNode (reee (yie, 21)(² }
11: end for Σ Mal(1 B.ll J 1, I,....J
10:
7jri K-1 Σ in
12:
end for
But the trees for different classes have no interaction when training except the last softmax transfor- mation. The trees for the label with few positive samples would have a terrible training, since it can not learn things from the other labels.
Moreover, it has big problem that it costs times of resources when training and inferring, There were papers that implied the multi-class by solving the sparse outputs problem (Si et al.) (017) and achieving order of magnitude improvements in model size and prediction time existing methods. But it still doesn't 't seem to be a multi-task solution, while realistic datasets above always have dense outputs and multi-class which can be positive the same time. Therefore call it multi-class solution, not multi-task.
To address the limitations of previous works, propose two new novel techniques called multi- output tree and multi-task splitting.
3 MULTI-OUTPUT TREE
The multi-task demands new structure called multi-output tree save the intermediate results. The new model should manage to specify weights for each label, and update leaf outputs with specified rates, which make necessary multiply gradients of the samples with the number of labels.
In this section, we propose a new structure tree by modifying the traditional decision tree, which almost has no disruptive innovation the GBDT training procedure. Multi-output Tree shares same structure and same leaf' instead of sharing other parameters. And we enrich the node and leaf information which contains Gradients, Hessians, mean value of the data the node and etc. There will be a discussion about them below:
Firstly, each sample in the dataset should have n predictions, n Gradients and n Hessians, where the n is the number of labels. They should be calculated at the beginning of each iteration like old GBDT. And it causes problem that the histogram object(Ke et al.) (017) which used to speed up splitting could only be calculated by one Gradient and one Hessian for each sample.
Then, we separate the concept of Gradients and Hessians. When training the tree, old Gradients and Hessians were designed both for splitting the node and updating the output. In MT-GBM, we denote the Gradients and Hessians used for updating as G, and H,, respectively. The Gradients and Hessians used for splitting the node are called ensemble G. and He, while it has one value for each sample, which determines how the tree splits. And more discussion about the ensemble G, & He and updating G., & H,, will be shown in the next sections.
4

---

Under review as a conference paper at ICLR 2022
Node
Node
Max gain to spltt
Max gain to spitt
Learl
Lear2
Lear1
Lear2
Tisk
G2
Task
(a)
leaf output
(b)
lesf output
Task
Figure 2: (a) Old structure of gradients in the GBDTs. Each rectangle is the G of one sample. As- sume the H equals constant in the square loss and model splits samples by the G; (b) New structure of gradients in the MT-GBM. The arrows in horizontal direction mean the transformation generating updating G and the arrows in vertical direction mean the transformation generating ensemble G.
At last, we only need save the mean of residual errors all the data at each leaf for each label, which was used to calculated residual errors in iterations below. It's easy to support the model to save part of trees for every single label. Therefore the dump model file will be the same size as the traditional GBDT and single label can be inferred the same speed.
MULTI-TASK SPLITTING
In this section, we define how to calculate the two types of G and H/ and their usages in MT-GBM.
XGBoost and LightGBBM have done lot of work make the GBDT efficient proposing new loss function and parallel computing when splitting, Their loss functions were based on Gradients and Hessians of the data:
G
L' 2- +TT
(3)
The loss serves as the most principle of the splitting selection, although XGBoost and LightGBM both have some techniques modify simplify them in computational procedure. At each leafs, the and means the sum of the Gradients and Hessians of all the data on it.
4.1 GRADIENTS FOR SPLITTING
The tree splits depending on the loss function, while the sum of two leafs' losses should be minimum the split position. would be large amount of computation model try minimize all the losses. Therefore we propose a new Ensemble Gradient G. and a new Ensemble Hessian He in the new equation:
T La =- Σ
G2 Hoj + +
(4)
To consider all the G and // of all type of labels, there would be two functions that calculate the G. and H.: G. = f.. GnJ. H.
Different tasks may bring different distributions of gradients to model. Set normalization factor w to normalize all the losses tasks would beneficial to the ensemble. We normalize them to distribution with 0.05 mean and 0.01 standard deviation.
We realizes the fc and fa have to be designed to outstand only one type of label or several rela- tive labels with positive correlation in iteration by trials. That is because different labels will be learned or fitted at different rates and the G and H will have a completely different order of magnitude, which is hard to ensemble. Thus define the fa end In by them with weights
5

---

Under review as a conference paper at ICLR 2022
Algorithm 2 The calculations of G. and H. for one sample before iteration
Input: predication trees above of each Pas. label of each task:/ 12.:1112,..... I2.3.....n Output: G..HH
n number of tasks
2: for = 1, ..... n. do d
3: G. - dp L(i,
4: H; = do L().pp))
5: end for
6: Randomly initialize W1, W2, War.-- Wn to make w, G, with 0.05 mean and standard deviation
0.01
7: Randomly choice one or more indexes of tasks from n kj to km
8: for k = ......, km do
9: Wk = 7w which 7 is parameter from 10 100
10: end for
G. = === Gj)
Randomly initialize V1, V2, V3,,, Va to make e H; with 1.0 mean and standard deviation 0.1 13: He = =- (v, Hj)
14: return Ge, He
which will randomly concentrate on one of few tasks. The probability of each task be chosen can be assigned which represents the importance of tasks.
The parameter y is used as factor to outstanding the chosen tasks. It could be verified from 10 to 100 by experiments from case by case. The smaller correlation, which means that there are strong conflicts between tasks, the better result with big y. From the algorithm we could find it complex to keep different G and I ensemble balance. We propose common one above, and believe there might be other better ensemble algorithms for particular problems and tasks.
When it comes to problem with main task and sub tasks, kj to km should not be selected randomly. We suggest that to always choice the main task label as k. And there is a helpful method to improve the learning process, which is to calculate or evaluate the correlation of the main task with others and give bigger w and smaller v for the task of positive correlation.
4.2 GRADIENTS FOR UPDATING LEAF VALUE
After defined the Ge and He, the GBDT tree learner could build trees for each iteration. The length of G. and He should be same as the length of the number of samples, while the length of Updating Gradient Gun. and Updating Hessian Hun same the length of the number of samples multiplying the number of labels.
is acceptable for the MT-GBM to set the same rate for each label with traditional GBDT. However, we still have some explorations about the Gun and Hun and propose the algorithm below with some tricks.
The function f we defined is used to calculate correlation of the gradients vector of each label in current iteration, which means the updating speed should be fast if all the gradients have the similar distribution. Actually whether the calculation of correlation works not depends many things such as type of tasks and meaning of tasks. Therefore set the corr as a constant equal 1.0 might be better on some complex datasets, which means the updating rate is same the traditional single task GBDT. And noticed that we multiply the same corr for each Gus in one iteration, is because that the GBDT learns things from residual errors. Trees get inconsistent paces if we set different rates for different tasks, which leads to bad results.
5 EXPERIMENTS
5.1 EXPERIMENT SETUP
Through experiments, seek to answer the following questions:

---

Under review as a conference paper at ICLR 2022
Algorithm 3 The calculations of Gun and Hun for one sample before an iteration
Input: predication trees above of each task PP.P22.P....Prtttiu label of each task: 122 13..... Output: Gu1. u22 Gu3...,, Hu,
n number of tasks do
2: for i 1,
3: G.i dp Pb),
4: Has = d:y L(4.p))
5: concatenate the Gui of all the samples as vector Gsui
6: end for
7: calculate the correlation corr with n vectors (Goul, Gauz, Geu3, my Goun): corr
f(Geul, Gzuz; Gau3 ...y Gsun)
8: for i = iori=, l....,.... do
9: Gu clip(corr, 0.5,
10: end for
return G,1, Gus...., Gun Hul Ho....... Han
1. Can MT-GBM achieve better performance state-of-the-art implementations of gradient boosted decision trees (GBDTs) XGBoost, LightGBM and Catboost?
2. Does neural networks of multi-task achieves same performance MT-GBM?
In this section, we report the experimental results regarding our proposed MT-GBM algorithm which was developed on the LightGBM(Thee code" http:///gggghub.oo////////////// GitHub). Particularly, the details of experimental setup, including data description, baseline models, and some specific experiments settings, will be presented below. All experiments are done with the same features in GBDTs, Neural Network, and MT-GBM, and the sub tasks in MT-GBM are derived from the main task. Our experimental environment is Linux server with two 79-0 CPUs (in total 36 cores) and 128GB memories.
is available at
5.2 DATASETS AND BASELINES
We tested the methods on two different real-world datasets. The first one is part China Foreign Currency Volume from 2017 to 2019. The dataset is about the currency exchange volume between one country and China by common people. The target is simple: use the historic data to predict next day volume and make the last 30 days value the testset. It is time-series problem, and it only contains one amount in every workday from 2017 to 2019 and has no features. Rmse and mape are used evaluate the models.
GBDT is proved as a great model in time-series cases and features are often be constructed by sliding window (Lee al., 2018 Liu et al. 2018 Xia & Chen 2017). We count the minimum, maximum, average, and variance of the last three days, one week, two weeks, and one month as the sliding window features. With these, bascline finished and we run XGBoost and LightGBM on it.
And the second dataset is Kaggle Competition:IEEE detection case from Vesta Corporation. The target of model fraud or not. There is about 3.5% fraud transactions in the dataset while ROC-AUC is good metric for the imbalance data. There are 600,000 samples the trainset and 500,000 in the testset. Since we can not get the true label of testset, we cut the last 20% samples in the trainset by datetime as the validation data, exactly same as the reality.
Fraud Detection which is a credit fraud predict whether this transaction is
About 400+ original features, contains user information, shop information and transactions details and anonymous features, are provided by the official. As a supplement, 100+ features are mined from raw features in baseline solution of the golden section. We ensure that the baseline is top single model by the leaderboard. We run XGBoost, LightGBM, CatBoost and MT-GBM with the setting copied from the open source code.
https://gthah.co//Mcooooffl//// httpss//www..aggg.ccooceeeeftrt/

---

Under review as a conference paper at ICLR 2022
5.3 RESULTS
Table Results of models for experiment 1
Models
best mape
mape of iteration 150
best rmse
rmse of iteration 150
XGBoost
0.0396
0.0475
316.43
362.50
LightGBM
0.0438
0.0452
400.32
403.47
NN 0.0652
453.87
MT-GBM
0.0387
0.0423
308.21
336.82
In the dataset of Currency Volume, the trend of amount may have some regular patterns. Only the label of amount itself is not enough. We propose that a sub label is the volume of next day divided by current day, while the main label is the volume of next day. Two labels have different orders of magnitude while the sub label is a number floating around 1.0 and the main label great than 10
All tree models in experiment perform not badly Table shown. XGBoost is better than Light- GBM at the same parameters on both metrics.
The result of first experiment shows the effects of ensemble numerical labels with different orders of magnitude. We compare the LightGBM and MT-GBM as the improvement multi-task model, we add the reduced from 4.3% to 3.9% and the best rmse is reduced from 400.32 to 308.21.
algorithm on the LightGBM repository. From Table 1, the best mape is
We conclude the results that multi-task teach model not only the volume of currency but also the rise and fall which make model robust at small dataset by splitting at a better position, meanwhile slowing down convergence speed of the model.
Table 2: Details of sub tasks for experiment 2
Task index
Definition
Correlation with main task
whether this card is fraud with amounts in two days
0.418
2
whether this ip is fraud in two days whether this card is fraud in same shop what percentage of the transactions of same amounts is fraud in two days
0.309
3
0.415
4
0.461
Table 3: Results of models for experiment 2
Models
sub task list
3-folds' ROC-AUC
fold
fold 2
fold 3
XGBoost
0.9443
0,9393
0.9417
0.9364
CatBoost
0,9371
0.9302
0.9309
0.9311
LightGBM
0,9442
0,9383
0.9408
0.9369
NN-4 Tasks
[1,2,3]
0.8410
0.8235
0.8346
0.8273
MT-GBM-2 Tasks
[1]
0.9447
0.9402
0.9410
0.9368
MT-GBM-2 Tasks
[2]
0.9441
0.9384
0.9402
0.9374
MT-GBM-2 Tasks
[3]
0,9446
0.9397
0.9405
0,9380
MT-GBM-3 Tasks
[1,2]
0.9441
0,9391
0.9388
0.9370
MT-GBM-3 Tasks
[2,3]
0.9448
0,9397
0.9405
0.9374
MT-GBM-3 Tasks
[1.3]
0.9441
0,9395
0.9396
0.9359
MT-GBM-4 Tasks
[1,2,]]
0.9454
0.9402
0.9407
0.9383
MT-GBM-5 Tasks
[1,2,3,4]
0.9443
0.9393
0.9400
0.9372
8

---

Under review as a conference paper at ICLR 2022
However, the second dataset has not provided multi labels too. In the risk management, fraud transaction often divided into many classes by the type of fraud. Table 2 shows the details of sub tasks.
The dataset is divided into folds in order to reduce the influence of randomness. The final result contains the average of folds and single fold versions. Since have sub tasks in the case, some combinations have been tested. From Table 3, most of the multi-task experiments behave better than single task on the ROC-AUC of the main task. We can find that 2-tasks models with sub task or 3 have presentable results and the 4-tasks model has the best performance through comparative analyses between LightGBM and MT-GBM. Low correlation task means irrelevant task which is helpless to the main task and high correlation seem to have less supplement for the main task, so the task and are the best for this case. And the Cathoost and XGBoost with single task can't reach the metric of MT-GBB. It is worthwhile mentioning that this experiment is training as the outstanding parameter 7 equals 50, cause the correlation between sub task and main task seems not so strong. And the improvement will be descended when y equals 10. We regard this case as an example that our algorithm makes a balance on the conflicting tasks.
Somehow the more tasks do not bring the better result. We concluded it the ensemble method of Gradients and Hessians not perfect enough, failing to express too many tasks.
5.4 COMPARISON WITH NEURAL NETWORKS
We also try neural networks because it a natural multi-task model. There are lots of big numerical numbers in the features disturbing the parameters in the neural cells. Therefore we have to transform them through F(x):
F(x) logio + 1)
(5)
On the other hand, we fill the NaN value in the features with 0. And after this data preprocessing, create simple Multi Layer Perceptron network structure with 2 hidden layers and GELUs activation function (Hendrycks & Gimpel 2016) to fit the dataset. The result is shown in Tables 1 and 3.
The neural network performs badly in both experiments as we explain before. It is not unexpected result while most of these problems are solved with GBDT in the industry.
CONCLUSION
In this paper, we have proposed a new general algorithm, multi-task gradient boosted tree, basing on gradient boosted decision tree, which provides powerful features to the tree model. We have per- formed both theoretical analysis and experimental studies on the implementation. The experimental results consistent with the theory and show that with the help of multi-task, MT-GBM sig- nificantly outperform traditional GDBTs. And the MT-GBM has great extensibility as it didn't place restrictions on the datasets and the type of task.
REFERENCES
Ritesh J Agrawal and James Shanahan. Location disambiguation in local searches using gradient boosted decision trees. In Proceedings of the 18th SIGSPATIAL International Conference Advances in Geographic Information Systems, pp. 129-136, 2010.
Joachim Bingel and Anders Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks. arXiv preprint arXiv: 170.00837 2017.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Olivier Chapelle, Pannagadatta Shivaswamy, Srinivas Vadrevu, Kilian Weinberger, Ya Zhang, and Belle Tseng. Boosted multi-rask learning. Machine learning, 85(1-2):149-17 2011.
Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp- 109-117, 2004.

---

Under review as a conference paper at ICLR 2022
Jean Baptiste Faddoul, Boris Chidlovski,, Rémi Gilleron, and Fabien Torre Learning multiple tasks with boosted decision trees. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 681-696. Springer, 2012.
Cun-qun Fan, Xian-gang Zhao, Man-yun Lin, Li-zi Xie, You Ma, and Xiao-hu Feng. A multitasking run time prediction method based on gbdt satellite ground application system. In Current Trends in Computer Science and Mechanical Automation Vol. 2, pp. 145-150 Sciendo Migration, 2017.
Chen Gao, Xiangnan He, Dahua Gan, Xiangning Chen, Fuli Feng, Yong Li, Tat-Seng Chua, and Depeng Jin. Neural multi-task recommendation from multi-behavior data. In 2019 IEEE 35th International Conference on Data Engineering (ICDE), pp. 1554-1557. IEEE, 2019.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv: 084466 2016.
arXiv preprint
Jizhou Huang, Wei Zhang, Yaming Sun, Haifeng Wang, and Ting Liu. Improving entity recommen- dation with search log and multi-task learning. In IJCAI, pp. 4107-4114, 2018.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in neural information processing systems, pp. 3146-3154 2017.
Ritchie Lee, Mykel Kochenderfer, Ole J Mengshoel, and Joshua Silbermann. Interpretable cat- egorization of heterogeneous time series data. In Proceedings of the 2018 SIAM International Conference on Data Mining, pp. 216-224. SIAM, 2018.
Song Liu, Yaming Cui, Yaze Ma, and Peng Liu. Short-term load forecasting based gbdt com- binatorial optimization. In 2018 2nd IEEE Conference on Energy Internet and Energy System Integration (E12), pp. 1-5. IEEE, 2018
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint 1x1.v:1901.11500 2019.
SHAO Mei and JIANG Li-ren. The application of decision tree in logistics finance credit risk. Logistics Sec-, 2015.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In Advances in neural information 2018.
processing systems, pp.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
Byron Roe, Hai Jun Yang. Ji Zhu, Yong Liu, and Gordon Mcgregor. Boosted decision trees as an alternative to artificial neural networks for particle identification. Nuclear Instruments & Methods in Physics Research, 543(2-3). 2004.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint (50.::770600099 2017.
Si Si, Huan Zhang, S Sathiya Keerthi, Dhruv Mahajan, Inderjit S Dhillon, and Cho-Jui Hsieh. Gradient boosted decision trees for high dimensional sparse output. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3182-3190. JMLR. org, 2017.
Chi Su, Fan Yang, Shiliang Zhang, Qi Tian, Larry S Davis, and Wen Gao. Multi-task learning with low rank attribute embedding for person re-identification. In Proceedings of the IEEE interna- tional conference on computer vision, pp. 3739-3747, 2015.
Maoguang Wang, Jiayu Yu, and Zijian Ji. Credit fraud risk detection based on xghoost-Ir hybrid model. 2018.
Yaozheng Wang. Dawei Feng, Dongsheng Li, Xinyuan Chen, Yunxiang Zhao, and Xin Niu. A mo- bile recommendation system based on logistic regression and gradient boosting decision trees. In 2016 International Joint Conference on Neural Networks (IJCNN) pp. 1896-1902. IEEE, 2016.
10

---

Under review as a conference paper at ICLR 2022
Hong Wen, Jing Zhang, Quan Lin, Keping Yang, and Pipei Huang. Multi-level deep cascade trees for conversion rate prediction in recommendation system. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 338-345, 2019.
Ying Xia and Jungang Chen. Traffic flow forecasting method based gradient boosting decision tree. In 2017 5th International Conference on Frontiers of Manufacturing Science and Measuring Technology (FMSMT 2017). Atlantis Press, 2017.
A. N. Yu-Zhuo and L. U. Xiao-Ni. Enterprise credit risk evaluation based on the decision tree. Journal of Hebei University of Technology. 35(6):64-68, 2006.
Yanru Zhang and Ali Haghani. A gradient boosting method improve travel time prediction. Transportation Research Part C Emerging Technologies, pp. S0968090X15000741.
Yu Zhang and Qiang Yang, An overview of multi-task learning. National Science Review, 5(1): 30-43, 2018.
A IMPLEMENTATION DETAILS
Codebase. For the leee- Fraud dataset, we use the codebase from The repository has done feature engineering work for the dataset from realistic scene. We use directly. We use the cross- validation evaluation method and divide the dataset into four folds as 3-folds train and hold-out validation. For each fold, we use a hold-out set as the validation set and train a machine to learn the model from scratch using the remaining data. We set the same random seed so that the result of each fold comparable.
Custom Object Function. The Ge. Gu. He, and H2 be implied in the fobj function in the Train API. We build the Multi-Task framework for GBDTs and leave the custom function because believe the high-level user would find better ensemble methods for Gradients and Hessians in different scenes.
Hy peperparrmeeers. For the Currency dataset, tuning the best hyperparameters for all models. We run XGBoost, LightGBM, and MT-GBM with the setting: a small tree depth as 6, learning rate 0.03, and lambda 7.1 as 0.1 since time-series cases is easy overfit.
For the leee-Fraud dataset, we copy the hyperparameters from Github: max depth as 16, learning rate as 0.03, lambda L1 0.1, and max leaves as 500. Settings of XGBoost and Catboost have small differences to get better results.
Comparison with neural networks. Note that the neural network in paper is the MLP, since other networks did not fit for tabular data. The network hyperparameters are the best in the two datasets while we get them by grip search.
B ADDITIONAL EXPERIMENT RESULTS
For clearness, we only show the final metric in the main text. There are some visualizations of intermediate process here.
Figures and show the metric during training the Currency dataset. The lines of single task(blue and yellow) are declined fast and overfit at about iteration 150. The lines of Multi-Tasse Task(green and red) behaves better at both metrics. However, the valid metric of Multi-Task overfits finally too, as result of a normal phenomenon in the tree model.
In leee- Fraud experiment we have defined sub tasks for the main task. The first three of them are the different types of fraud extracted from the main task. Take the label of main task and first three tasks as 1ot1 to 4914, the predictions of single task LGB model as Ysingles and the predictions of MT-GBM ymtt to Yma4. Figure 5 shows the Roc-aue curves of different outputs.
11

---

Under review as a conference paper at ICLR 2022
Metric during training
0.10
training valid
0.08
valid multi
train multi
0.06
mape
0.04
0.02
0.00
25
50
75
100
125
150
175
200
Iterations
Figure 3: Train and Valid MAPE.
Metric during training
1000
training
valid 1
800
valid multi
train multi
mse
400
200
0
25
50
75
100 Iterations
125
150
175
200
Figure Train and Valid MSE.
12

---

Under review as a conference paper at ICLR 2022
110
baseline(Igb)
mt-gbm(maiin task)
105
mt-gbm(sub tasks)
1.00
Roc-auc
0.95
0.90
0.85
0.80
0.0
10
20
3.0
Task index
Figure 5: Roc-auc curves.
13


Reviews:
[{"review_id": "jTo4cy3_Lpd", "reviewer": "Reviewer_CBKT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Authors claim “However, it is known that neural networks sometimes fail in applications with tabular data.” in the introductions. Can you cite the relevant work here?\nRephrase “It seems to be a bit complex and obsolete and is beaten by the existing implementations of gradient boosted decision trees such as XGBoost and LightGBM.”\nThe related works section is very short. \nConsidering that boosted decision trees are not the most popular approach in ML now, the authors could ease the readers by providing more background on the topic.\nThe authors evaluate their approach on just 2 datasets. More evaluation is needed to make a convincing argument for their approach.\n", "summary_of_the_paper": "Authors propose a way to perform multi-task learning with gradient boosted decision trees. Multitask learning has shown promise in deep learning and other machine learning systems. It is hard to perform multitask learning in decision trees. The main contributions from the paper are as follows\nAuthors propose a multitasking gradient boosted decision tree\nAuthors further propose a way to dynamically adjust the learning rate for each task so the learner does not overfit any specific task.\nThey evaluate their approach on two public datasets.\n", "summary_of_the_review": "The authors address a real problem with gradient boosted decision trees. But the results are limited and do not appear very convincing. They need to evaluate their approach on more datasets to make a convincing argument. The improvements from the proposed approach do not look very significant either.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bnlgmyScrb-", "reviewer": "Reviewer_8oMi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "One of the main strengths of this paper is its motivation and focus on the problem of multi-task learning of GBDTs. This is definitely an important problem, extending the ability of GBDTs similar to linear models and neural networks.\n\n\nHowever, one of the critical weaknesses of this paper is its lack of clarity. It is very hard to follow the precise problem being addressed in the paper and the precise choices made in the proposed scheme. This leads to various questions such as the following:\n\n- Regarding \"for tree-based models\", the splits that have ... are suitable for all tasks.\"\n  - Why is simple sum an issue when it finds a split that is \"good enough\" for all tasks? It works for neural networks, why should it be an issue with GBDTs especially if we consider a scaled sum of per-task losses? \n  - The proposed \"shared tree\" is probably essentially finding a tree split \"suitable\" for all tasks.\n- Regarding Figure 1:\n  - The motivating example is hard to follow. The intuition seems reasonable but the description of the specifics is hard to understand for me. I think we need to start from describing what each task is and why same sample has two labels.\n  - Is this a MT case where all tasks have the same covariates but different task-specific labels? That is, as training data, we have the same $x$ values but different $y$ values for each $x$ corresponding to each task?\n  - What is a \"noisy task\"? \n- Regarding \"main task objective function $L_1$ as the metric to evaluate the model\"\n  - Is this a MT setting where we evaluate only on one of the tasks? This does not appear to be standard practice. Can you please provide motivation for such a MT setting?\n- Regarding the positioning against Chapelle et al. (2011):\n  - The issue raised by the authors appear to be somewhat non-technical -- seems like the authors are claiming that minimizing a sum of relative losses over all tasks is \"complex and obsolete\" compared to XGBoost & LightGBM. It would be helpful to have this elaborated in more technical terms.\n- Regarding comment on GBDTs for multi-class classification while discussing Fan et al., 2017:\n  - It is an unclear comment and citation -- training one tree per class is standard in multiclass classification when minimizing the multinomial loss (or the cross-entropy loss) with GBDT. It is odd to cite a 2017 paper when this has been around since early 2000s.\n  - It is not clear what is meant by \"activation\" during \"inference\" based on specified class -- if we knew the \"specified class\" we won't need inference. This comment needs further clarification.\n- Regarding \"each sample in the dataset should have $n$ predictions, .... could only be calculated by one Gradient and one Hessian for each sample\"\n  - Is this implying the situation where all tasks have the same covariates but different labels? This is an unusual MT setting. Can you provide practical scenarios where this is true?\n  - The \"problem\" in the histogram object needs a bit more detailed description.\n\nIt would very helpful to start with a clear definition of the problem being considered, followed by the motivating example, connecting it to the aforementioned problem definition. Given the proper problem definition and motivation, it would be easier to distinguish between existing literature on boosting in a multi-task setup. In the current form, it is not clear why the existing schemes for multi-task boosted trees are not considered as baselines.\n\n\nMoreover, the proposed scheme appears to lack useful motivation and justification, making algorithms such as Algorithm 2 & 3 quite unintuitive. Specifically, I have the following concerns/questions/comments:\n\n- In Algorithm 2, it appears that the gradients are first scaled randomly across tasks (Algorithm 2, Line 6) and then the tasks are effectively chosen randomly and their (weighted) sum is used as the gradient to use for splitting (lines 7-11). This form of scaling and task-sampling seems very unintuitive to me without proper justification. Why can we not just consider simple normalization (across tasks to make the losses comparable across tasks) followed by a simple sum for the splitting gradient? This is important to discuss in my opinion.\n- There are magic numbers (0.05 for mean, 0.01 for standard deviation across tasks, [10, 100] as range for $\\gamma$) without any discussion or motivation. These are essentially hyperparameters of the proposed scheme and should be dealt with as such.\n- Gradient for leaf updates are also somewhat confusing in the Algorithm 3 -- it is not clear how an $n$-way correlation coefficient is computed and also how that allows for different \"learning rates\" for different tasks. If all the gradients for all tasks are just scaled with a single scalar, I am not sure it is doing anything. Maybe I misunderstand the procedure. Additionally, it would be good to understand how much effect this single-scalar-scaling has on the final performance (if any).\n\n\nFinally, the empirical evaluation is quite limited, with just two data sets and an artificially created multi-task setup. It is not clear how well the hyperparameters of the XGBoost, LightGBM and neural network baselines were optimized (if at all). Given the randomness present in MT-GBM, and the small margin of improvement over baselines, it is also important to have confidence intervals around the performance metrics utilizing different seeds and/or different train/test splits. It would be good to focus on a more natural multi-task setup with a clear multi-task metric to compare methods. Moreover, it would be good to study the effect of the different choices made in MT-GBM -- for example, compare a simple scaling + sum of multi-task losses to the proposed random scaling + random task selected losses in Algorithm 2 (something similar can also be done with Algorithm 3). This would help us highlight the gains obtained from the choices made by the authors.\n\n\n\nAdditional comments:\n\nWrong citation for XGBoost (Roe et al., 2004)! The correction citation is _\"Chen, Tianqi, and Carlos Guestrin. \"Xgboost: A scalable tree boosting system.\" Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 2016.\"_\n", "summary_of_the_paper": "This paper focuses on the multi-task learning for the popular gradient boosted decision trees (GBDTs) for tabular data. This paper proposes two novel techniques to the learning of GBDTs -- the multi-output tree and the mutli-task splitting -- that can be easily incorporated to the existing implementations of GBDTs without significant computational overhead. The multi-output tree shares the tree structure (split dimensions and thresholds) across all tasks but maintains different leaf statistics for each task. The multi-task splitting utilizes the per-task statistics in an internal node to generate a single statistic (gradient and hessian) that incorporates information from a small number of tasks to find the optimal split. The leaf value updating procedure is also modified to handle the multi-task setup. The proposed MT-GBM scheme is evaluated on two data sets where there is a single \"main\" tasks and auxilliary tasks are generated and used in the multi-task learning of MT-GBM. The empirical results indicate that the proposed MT-GBM is able to marginally outperform XGBoost and LightGBM trained on a single task, and a neural network trained with multiple tasks.\n\n", "summary_of_the_review": "I recommend a rejection based on the weaknesses I list above. Specifically, in the current state, it is very hard for me to follow the proposed scheme and why existing schemes do not apply. It is important that the relevant baselines are properly compared and contrasted against. Moreover, we only see a marginal improvement of the proposed scheme over the considered baselines.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "V7ier4LGAgW", "reviewer": "Reviewer_UURx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points**\n- Authors develop the algorithm for multi-task learning for gradient boosting which supports multiple tasks of different nature (classification, regression)\n- Algorithm is using special gradients and hessians aggregation to define the splitting criteria which is fast and does not need large search space and recomputing every task loss in every node\n- Accounting correlation between tasks to adjust learning rate during training\n- Fair experiment setting: taking top algorithms on particular datasets (with necessary data processing and hyperparameters) and simply applying proposed modifications on top with demonstrating improved results\n\n**Weak points**\n \n*Technical correctness*\n  - There is a strict motivation how loss Eq. (3) is obtained as it is optimal value when optimal weights are assigned in the leaves for a fixed tree structure. Then greedy procedure is used to construct the tree (as tree construction is NP-hard problem) based on this loss Eq. (3) and weights are assigned in the optimal way. Do authors can explain and give more details how Eq. (4) was obtained and is there any connection between weights in each leaf and loss Eq. (4)? Otherwise the provided Algorithm 2 looks artificially.\n  - Algorithm 2 looks ambiguous and it is not clear why it is correct.  First, is $w_i$ defined independently for each sample or is it defined that $w_i * G_i^m$ is coming from the same normal distribution for every $m$ where m is $m$-th sample? If we define latter - it is impossible to do (by doing math one can check this). If we define independently from sample then $w_i$ and $v_i$ are normal variables which mean and std depend on the $G_i$ and $H_i$. In this case both $G_e$ and $H_e$ will be normally distributed variables which mean and std do not depend on $G_i$ and $H_i$ at all (e.g. $G_e$ is coming from sum of the same normal distributions $N(0.05, 0.01)$). The latter leads to random splitting criteria for the tree construction which is different from what authors state as contribution.\n  - Algorithm 3 does not make sense to me: it is doing global scaling of optimal weights set in the leaves. Why do we need this scaling if optimal weights are assigned already in the leaves? There is no any empirical evaluation in the paper why scaling with Algorithm 3 is needed or helpful. Moreover, in the text authors say that \"Therefore set the corr as a constant equal 1.0 might be better on some complex datasets which means the updating rate is same as the traditional single task GBDT\" and \"Trees get inconsistent paces if we set different rates for different tasks, which leads to bad results.\". Thus, I think, the stated contribution \"We propose a new learning algorithm that adaptively adjusts the learning rate for each task by taking the scale of residual errors into consideration. It stabilizes training on arbitrarily complex problems.\" is not supported by theory or experiments and could be even wrong. \n \n*Empirical correctness*\n- Absence of baseline from (Chapelle et al., 2011) \n- There is no any details and ablations on how and why $corr$, $w_i$, $v_i$ are defined in Algorithms 2 and 3\n- Comparison with neural networks is artificial as neural network baseline is far behind the boosting baselines, so there is no any sense to compare with it even in multi-task training\n- There is lack of significance testing for the first dataset\n- No any experiments with several main tasks so that they are not sub-tasks of each other\n \n**Additional questions**\n- Why not simply consider loss as a weighted sum (weights could be estimated from the task correlations on original labels) of tasks losses, compute its gradient and hessian and use Eq. (3) to construct the tree but still use multi-label assigning in the leaves (each task loss is considered here separately)? This should be considered as one of the multi-task baselines\n- What is the performance of all models on the sub-tasks? This analysis would be helpful to understand that in multi-task training we improve all sub-tasks too\n- Ideally there should be implementation of MT-GBM on top of XGBoost and Catboost to demonstrate that for several different boosting algorithms proposed multi-task training improves one-task training\n\n**Clarity**\n\nThe paper needs additional work on fixing English language (a lot of sentences lacks the correct verb forms which makes hard to read and understand meaning of important sentences). Some comments on text improvements please find below:\n- Related works, first paragraph: rewrite the paragraph so that it is readable\n- page 3, last paragraph: rewrite the paragraph so that it is readable\n- notation on supervised data: would be better to reflect the labels coming from different tasks), e.g. introduce $(x_i, y_i^1, .. y_i^n)$, thus it is clear what data and labels are given\n- Eq. (1), (2) - index $t$ is absent in the right hand, but should be there\n- Algorithm 1: i) $n$ should be the number of classes, not tasks (I found some ambiguity in multi-class and multi-task terms usage in Sec. 2 and in Algorithm 1); ii)  row with index 2 does not make sense if $p_{i,t}$ is probability (as it should be); why do we do exponentiation and normalization again? iii) does groundtruth label is vector with zeros and only one \"1\" in case we consider multi-class classification problem?\n- Eq. (3) $T$ is not defined\n- I don't understand this statement \"while realistic datasets above always have dense outputs and multi-class which can be positive at the same time\"\n- \"H equals constant\" -> \"H is constant\"\n- In Sec. 4 I think it is good to recap how Eq. (3) is obtained for gradient boosting\n- Algorithm 2 uses $L$ but should use $L_i$, also $dp$ -> $dp_i$; there is no clear usage of $w$ and $\\text{w}$,  $v$ and $\\text{v}$\n- Algorithm 3 uses $L$ but should use $L_i$, also $dp$ -> $dp_i$; How is the $corr$ computed? What is function $f$?\n- There should be exact formula how authors assign weights/values in each leaf for multi-labels to be clear on this. Overall, would be helpful for any reader to have a quick overview of XGBoost / gradient boosting in the introduction with necessary formulas including Eq. (3) and then demonstrate all new changes introduced by authors by referring to the original gradient boosting formulas\n- \"When optimizing end-to-end neural networks, it is natural to sum up the gradients from all losses. However, for tree-based models, the splits that have maximal information gain for the losses can differ a lot, especially when training classification and regression tasks at the same time.\" I do not agree with statement on neural networks as depending on loss magnitude of its gradient can be different and we will be learning only one task. Often special loss weighting or even tricks for optimization (lr, special scheduling, clipping) are needed. I would say problems for both neural networks and GBDT, but the source/solution of the problems is different as models structure is different.\n- \"It seems to be a bit complex and obsolete and is beaten by the existing implementations of gradient boosted decision trees such as XGBoost and LightGBM.\" Do authors mean that if we do splitting criteria to optimize sum of all tasks losses (which is more computations than MT-GBM approach) on top of XGBoost then vanilla XGBoost for one main task wins?\n- From presentation perspective it would be great to have (Chapelle et al., 2011) as a baseline to demonstrate how MT-GBM faster in terms of training (as simpler splitting criteria) and better in final accuracy\n- Could authors confirm that MT-GBM has exactly the same input data features and all gradient boosting hyper-parameters as XGBoost, so that the only difference is the usage of Algorithm 2 and 3?\n\n**Related works**\n\n- I think it is worth to mention https://www.diva-portal.org/smash/get/diva2:1309070/FULLTEXT01.pdf and http://www.ms.k.u-tokyo.ac.jp/sugi/2014/TreeMTL.pdf and compare with them.\n- \"Therefore, it cannot learn a shared latent feature representation, which is essential for multi-task learning. ... First and foremost, the core of efficient multi-task learning is to extract valuable hidden representation that shares across all tasks, which is non-trivial for tree-based ensemble learning methods like GBDT. \" Could authors refer to any work to support this statement? Otherwise, more smooth formulation like \"In some works ... it was shown that share latent feature representation is essential. As soon as for neural networks shared representation works great we could inherit this idea for GBDT and model it via the shared tree structure (as a tree can be seen as latent representation).\" is preferable. Maybe we don't need to have shared latent representation and other possible solutions exist which can be modeled by GBDT too.\n", "summary_of_the_paper": "Recent developments in deep learning and neural networks showed impressive results on structured data, like audio, text and images. However, there are a lot of tabular data, like finance, search-engine machines, recommendation systems, telecommunications, physics and astrophysics characteristics, etc. For problems and applications with these tabular data random forest and gradient boosting (over decision trees) are the mostly used algorithms (most kaggle competitions were won by them). \n\nMutli-task learning is very active area nowadays as solving tasks together could improve each task (less over-fitting, better representation and generalization) and moreover we could have one general model (faster inference). It is studied in the context of deep learning a lot: computational graph mostly is shared across tasks and we can natively optimize combination of loss functions correspondent to each particular task. On the other hand, for gradient boosting (over decision trees) multi-task learning is still challenging. One of the ways is to learn shared tree structure across tasks at each gradient boosting iteration. Authors are focusing on this idea: they share same tree structure and leaves. Then authors propose the splitting criteria based on multi-task losses (instead of one-task loss as done in classic gradient boosting): gradients and hessians of tasks losses are aggregated to provide the final loss which is used as splitting criteria in the greedy algorithm of tree construction. Further, authors propose a special scaling for gradients based on correlation between the tasks (computed on losses gradients) to assign multi-label in each leaf. Proposed algorithm, called MT-GBM (multi-task gradient boosting machine) is empirically estimated on 3 datasets where subtasks are selected from the main task and should improve the main task via multi-task learning. Implementation is done on top of XGBoost and with experiments it is shown that MT-GBM improves upon the main task learning.", "summary_of_the_review": "The problem considered in the paper is important and actual as I describe in the paper summary. The idea on developing algorithm with the shared tree construction based on tasks losses makes sense and in some sense parallel to the things done in deep learning. However, there are a lot of ambiguous, not clear places (even I think some places are wrong, see the main review) in the paper that does not allow to conclude on the correctness and applicability of the proposed approach. Moreover, the paper lacks of the proper baselines and comparisons, statistical significance testing for one of the datasets, and ablations on the parameters introduced in the algorithm 2, 3 (details in the main review). For the current paper state my recommendation is to reject due to all aforementioned things.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T2byMCYXyax", "reviewer": "Reviewer_tQ17", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- (S1) Trying to adapt multi-task learning (MTL) approaches to GBMs is a natural and timely topic\n\nWeaknesses:\n- (W1) Comparison to related work: The comparisons to prior work like Chapelle et al 2011 and Fan et al 2017 are presented with very high-level, hand-wavey arguments for why these prior approaches are insufficient.  A combination of clearer arguments and actual comparison in the experimental section would be necessary to properly compare here.\n- (W1a) Also, the authors miss some more recent works that should probably be included, e.g. http://proceedings.mlr.press/v108/han20a/han20a.pdf\n- (W2) Imprecise arguments: Overall, the paper is written with many heuristic claims throughout that are not tied to specific formal or empirical justification.\n- (W3) Insufficient empirical evaluation:\n  * (W3a) The experiments, as noted in #W1, don't compare to prior approaches in this area, and should have done this.\n  * (W3b) The experiments compare to one very simple DNN MTL model, and two single-task GBM models.  This is both overall an insufficient set of baselines (given the many very sophisticated and competitive models that could be used here), and also does not properly disentangle/ablate the effect of multi- vs. single-task from the specific advantages of the particular MTL-GBM approach considered here.\n  * (W3c) Insufficient ablations of method\n  * (W3d) Overall, the experiments section in particular is written in a very imprecise and unclear fashion.", "summary_of_the_paper": "This paper proposes a new multi-task variant of Gradient Boosted Machines (GBMs), a corresponding learning algorithm that adaptively adjusts the per-task learning rate, and implements a version on the LightGBM framework to show empirical results.", "summary_of_the_review": "Overall, the paper proposes an interesting new method, but has an insufficient empirical evaluation and is additionally quite imprecise throughout.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a multi-task version of Gradient Boosted Machines (GBMs). The paper proposes a learning algorithm that adaptively adjusts the learning rate per task. Empirical evaluation is carried out on two datasets with the method implemented in the LightGBM framework.

The reviewers thought that the paper is not very clear. They were not ready to accept the paper claims based on the current version. In particular, the algorithms are hard to follow, the empirical evaluation is not easy to follow and there are missing comparisons to related work. The authors did not offer a response to the reviews.
Decision: Reject