MT-GBM: A MULTI-TASK GRADIENT BOOSTING MACHINE WITH SHARED DECISION TREES
Year: 2021

Abstract:
Despite the success of deep learning in computer vision and natural language processing, Gradient Boosted Decision Tree (GBDT) is yet one of the most powerful tools for applications with tabular data such as e-commerce and FinTech. However, applying GBDT to multi-task learning is still a challenge. Unlike deep models that can jointly learn a shared latent representation across multiple tasks, GBDT can hardly learn a shared tree structure. In this paper, we propose Multi-task Gradient Boosting Machine (MT-GBM), a GBDT-based method for multi-task learning. The MT-GBM can find the shared tree structures and split branches according to multi-task losses. First, it assigns multiple outputs to each leaf node. Next, it computes the gradient corresponding to each output (task). Then, we also propose an algorithm to combine the gradients of all tasks and update the tree. Finally, we apply MT-GBM to LightGBM. Experiments show that our MT-GBM improves the performance of the main task significantly, which means the proposed MT-GBM is efficient and effective.

heading: 1 INTRODUCTION
text: Gradient boosting is a powerful machine-learning technique that achieves state-of-the-art results in a variety of applications. It is a process of constructing a strong predictor by ensembling weak predictors and performing gradient descent in a functional space. Gradient boosting has achieved tremendous success in a wide range of machine learning tasks. It is popular in applications with heterogeneous features, noisy tabular data, and complex dependencies, such as recommender systems (Wang et al., 2016; Wen et al., 2019), web search (Agrawal & Shanahan, 2010), FinTech (Wang et al., 2018; Mei & Li-ren, 2015; Yu-Zhuo & Xiao-Ni, 2006), online advertising, etc (Roe et al., 2004; Zhang & Haghani). Among all gradient boosting algorithms, XGBoost and LightGBM are the two most famous algorithms. They have contributed to countless top solutions in machine learning and data mining competitions on tabular data.
Despite the considerable success of gradient boosting, applying GBDT models to multi-task learning is still challenging. The GBDT model aims to train a high-performance discriminative model towards a single loss function of a single task. Therefore, it cannot learn a shared latent feature representation, which is essential for multi-task learning.
We list three things that limit the applicability of GBDT in multi-task learning.
First and foremost, the core of efficient multi-task learning is to extract valuable hidden representation that shares across all tasks, which is non-trivial for tree-based ensemble learning methods like GBDT. For example, in news recommendation, user response prediction tasks include the prediction of click-through rate, dwell-time, thumbs, reviews, shares, and so on. These objects correspond to the same user; thus an informative representation of the user is the key to such a multi-task learning problem. The recent development of deep learning can achieve this naturally using hidden embeddings and layers in neural networks (Ruder, 2017; Huang et al., 2018; Gao et al., 2019), thus has shown its dominance for multi-task learning in computer vision and natural language processing (Su et al., 2015; Radford et al., 2019; Liu et al., 2019). However, it is known that neural networks sometimes fail in applications with tabular data.
Second, the splitting mechanism for decision trees aims to separate the input space as well as the training samples according to a single loss function, which could not be applied directly to multi-
task learning where there are multiple losses. When optimizing end-to-end neural networks, it is natural to sum up the gradients from all losses. However, for tree-based models, the splits that have maximal information gain for the losses can differ a lot, especially when training classification and regression tasks at the same time. Therefore, it is challenging to find tree splits that are suitable for all tasks.
Third, in multi-task learning, the model should keep similar paces for each task for stable learning. For neural networks, current gradient descent methods with adaptive step size can adjust the learning rate for different tasks. However, the learning speed of GBDT is related to the residual errors. Therefore, when there are multiple tasks, if a fixed learning rate is applied to all tasks, the model tends to overfit certain tasks while underfit the others. Therefore, how to adjust learning rates for multiple tasks remains a challenge.
In this paper, we propose a new algorithmic technique for multi-task learning based on GBDT, namely multi-task Gradient Boosting Machine (MT-GBM). The key novelty of MT-GBM is that it learns shared decision tree structures for multiple tasks, which can be understood as the hidden knowledge across all tasks. As the decision trees are shared, it requires almost the same computational overhead comparing to a single task GBDT. Therefore, MT-GBM can be efficiently trained. We verified MT-GBM on two financial datasets. Experimental results show that MTGBM can achieve significantly higher performance than existing state-of-the-art gradient boosted decision trees methods, XGBoost (Roe et al., 2004), LightGBM (Ke et al., 2017) and Catboost (Prokhorenkova et al., 2018) .
Our contribution can be summarized as follows:
• We propose MT-GBM that learns an ensemble of shared decision trees learned from multiple losses simultaneously, which significantly improves the efficiency and effectiveness of GBDTs for multi-task learning.
• We propose a new learning algorithm that adaptively adjusts the learning rate for each task by taking the scale of residual errors into consideration. It stabilizes training on arbitrarily complex problems.
• Our method is easy to implement upon existing GBDT models such as XGBoost and LightGBM. We tested our method by modifying LightGBM and showed significant performance improvements.

heading: 2 PRELIMINARIES

heading: 2.1 MULTI-TASK LEARNING WITH DECISION TREES
text: It is widely known that multi-task learning could improve the performance of every single task by sharing knowledge from all tasks. When using max-margin machine learning (i.e., SVMs) or deep learning (i.e., feed-forward neural networks), multi-task learning has achieved significant success (Caruana, 1997; Evgeniou & Pontil, 2004; Bingel & Søgaard, 2017). However, with decision-treebased models, it remains unclear how to learn shared knowledge from multiple tasks.
There are many potential reasons why adding extra outputs to a model might improve generalization performance. For example, to the extent that tasks are uncorrelated, their contribution to the aggregated gradient, which guides the node splitting in GBDT, can appear as noise to other tasks. Thus uncorrelated tasks might improve generalization by acting as a source of the noise. Figure 1 (a) shows a situation of data samples with a noisy label. One single sample contains two rectangles while the left represents label 1 and the right represents label 2. We can assume that task 1 clicks behavior and task 2 is shares behavior in the recommendation system. It makes it clear that two tasks fix the splitting line from dotted line to solid line. Figure 1 (b) shows that Sub tasks also help the sample splitting when dataset has the class imbalance problem.

heading: 2.2 PROBLEM DEFINITION
text: GBDT is an ensemble model of decision trees, which are trained in sequence. In each iteration, GBDT learns a new decision tree by minimizing the negative gradients (also known as the residual
errors) of previously trained trees. Specifically, to construct the decision trees, the algorithm finds the split points in the feature space according to the residual errors.
Assume we have n learning tasks t ∈ {1, 2, ..., n}, each with a set of supervised data {(x1, y1), ..., (xm, ym)}. For simplicity, we assume that the number of data points m in each task is the same, and all the tasks share the same input space. For the t-th task, there is a predefined learning objective Lt. In regression, we might consider the mean squared error:
Lt = 1
m m∑ i=1 (yi − pi)2 (1)
And we use the negative log-likelihood as the classification loss:
Lt = − 1
m m∑ i=1 (yi log pi + (1− yi) log(1− pi)) (2)
Finally, we assume the main task objective function L1 as the metric to evaluate the model.

heading: 2.3 RELATED WORK
text: In this section, we ignore the related work of neural network because most of neural network work well in specific tabular dataset (Zhang & Yang, 2018; Ruder, 2017) .
To achieve a multi-task tree, there were some works about multi-classifier tree on Adaboost (Faddoul et al., 2012) . This paper revises the information gain rule for learning decision trees and modified MT-Adaboost to combine multi-task Decision Trees as weak learners. Since its algorithm depends on their new equation of Information Gain, the model does not support doing the regression task yet. Obviously, it fails when it comes to residual errors in GBDT cause the loss is float number even in classifier tasks.
Another approach to multi-task learning is the work sharing tree structure for different tasks (Chapelle et al., 2011). This paper presents a general solution of boosting for different task types by minimizing the sum of losses ∑n i=1 Li. It uses a relative value to control the strength of the connection between the tasks. We approve of the way about exploring the general solution. But the part of optimization in which splitting the node would be hard and slow when consider minimizing all loss functions. It seems to be a bit complex and obsolete and is beaten by the existing implementations of gradient boosted decision trees such as XGBoost and LightGBM.
However, the GBDTs has some tricks to deal with multi-class task (Fan et al., 2017), i.e. training n trees in one iteration for n classes. When inferring, the trees belong to the specified class will be activated.
Algorithm 1 Traditional Multi-Class GBDT Input: n the number of tasks, m the number of samples. Accumulative prediction of previous
rounds:{p1,t, . . . , pm,t}nt=1; Goundtruth label:{y1,t, . . . , ym,t}nt=1 Output: New decision trees T1, T2, ..., Tn in this iteration.
1: for i = 1, . . . ,m do 2: pi,t =
exp(pi,t)∑n t=1 exp(pi,t)
, t = 1, . . . , n 3: for t = 1, . . . , n do 4: // J is the number of nodes in the tree 5: // N is the number of samples in the node j 6: ỹj,t = yj,t − pj,t, j = 1, . . . , N 7: // Rjti means that the ith sample belongs to the j node at the tth task 8: // γjti means the prediction of i sample at the tth task 9: {Rjti}Jj=1 = J − terminalNodeTree({ỹlt, xl}Nl )
10: γjti = K−1K ∑ ỹlt∑ |ỹlt|(1−|ỹlt|) , j = 1, . . . , J 11: end for 12: end for
But the trees for different classes have no interaction when training except the last softmax transformation. The trees for the label with few positive samples would have a terrible training, since it can not learn things from the other labels.
Moreover, it has a big problem that it costs n times of resources when training and inferring. There were papers that implied the multi-class by solving the sparse outputs problem (Si et al., 2017) and achieving an order of magnitude improvements in model size and prediction time over existing methods. But it still doesn’t seem to be a multi-task solution, while realistic datasets above always have dense outputs and multi-class which can be positive at the same time. Therefore we call it a multi-class solution, not multi-task.
To address the limitations of previous works, we propose two new novel techniques called multioutput tree and multi-task splitting.

heading: 3 MULTI-OUTPUT TREE
text: The multi-task demands a new structure called multi-output tree to save the intermediate results. The new model should manage to specify weights for each label, and update leaf outputs with specified rates, which make it necessary to multiply gradients of the samples with the number of labels.
In this section, we propose a new structure tree by modifying the traditional decision tree, which almost has no disruptive innovation to the GBDT training procedure. Multi-output Tree shares same structure and same leaf instead of sharing other parameters. And we enrich the node and leaf information which contains Gradients, Hessians, mean value of the data at the node and etc. There will be a discussion about them below:
Firstly, each sample in the dataset should have n predictions, n Gradients and n Hessians, where the n is the number of labels. They should be calculated at the beginning of each iteration like old GBDT. And it causes a problem that the histogram object(Ke et al., 2017) which is used to speed up splitting could only be calculated by one Gradient and one Hessian for each sample.
Then, we separate the concept of Gradients and Hessians. When training the tree, old Gradients and Hessians were designed both for splitting the node and updating the output. In MT-GBM, we denote the Gradients and Hessians used for updating as Gu and Hu, respectively. The Gradients and Hessians used for splitting the node are called ensemble Ge and He, while it has one value for each sample, which determines how the tree splits. And more discussion about the ensemble Ge & He and updating Gu & Hu will be shown in the next sections.
At last, we only need to save the mean of residual errors of all the data at each leaf for each label, which was used to calculated residual errors in iterations below. It’s easy to support the model to save part of trees for every single label. Therefore the dump model file will be the same size as the traditional GBDT and single label can be inferred as the same speed.

heading: 4 MULTI-TASK SPLITTING
text: In this section, we define how to calculate the two types of G and H and their usages in MT-GBM.
XGBoost and LightGBM have done a lot of work to make the GBDT efficient by proposing new loss function and parallel computing when splitting. Their loss functions were based on Gradients and Hessians of the data:
L∗ = −1 2 T∑ j=1 G2j Hj + λ + γT (3)
The loss serves as the most principle of the splitting selection, although XGBoost and LightGBM both have some techniques modify or simplify them in computational procedure. At each leafs, the H and G means the sum of the Gradients and Hessians of all the data on it.

heading: 4.1 GRADIENTS FOR SPLITTING
text: The tree splits depending on the loss function, while the sum of two leafs’ losses should be minimum at the split position. It would be a large amount of computation if model try to minimize all the losses. Therefore we propose a new Ensemble Gradient Ge and a new Ensemble Hessian He in the new equation:
L∗ = −1 2 T∑ j=1 G2ej Hej + λ + γT (4)
To consider all the G and H of all type of labels, there would be two functions that calculate the Ge and He: Ge = fG(G1,G2,G3,...,Gn), He = fH (H1,H2,H3,...,Hn).
Different tasks may bring different distributions of gradients to model. Set a normalization factor w to normalize all the losses of tasks would be beneficial to the ensemble. We normalize them to a distribution with 0.05 mean and 0.01 standard deviation.
We realizes the fG and fH have to be designed to outstand only one type of label or several relative labels with positive correlation in one iteration by trials. That is because different labels will be learned or fitted at different rates and the G and H will have a completely different order of magnitude, which is hard to ensemble. Thus we define the fG end fH by sum them with weights
Algorithm 2 The calculations of Ge and He for one sample before an iteration Input: predication of trees above of each task:p1,p2,p3,...,pn;true label of each task:l1,l2,l3,...,ln Output: Ge,He
1: n = number of tasks 2: for i = 1, . . . , n do 3: Gi = d dpL(li, pi) 4: Hi = d2
dp2L(li, pi)
5: end for 6: Randomly initialize w1,w2,w3, ...,wn to make wi ∗Gi with 0.05 mean and standard deviation
0.01 7: Randomly choice one or more indexes of tasks from 1 to n as k1 to km 8: for k = k1 , . . . , km do 9: wk = γwk, which γ is a parameter from 10 to 100
10: end for 11: Ge = ∑n j=1(wj ·Gj) 12: Randomly initialize v1, v2, v3, ..., vn to make vi ·Hi with 1.0 mean and standard deviation 0.1 13: He = ∑n j=1(vj ·Hj) 14: return Ge, He
which will randomly concentrate on one of few tasks. The probability of each task be chosen can be assigned which represents the importance of tasks.
The parameter γ is used as a factor to outstanding the chosen tasks. It could be verified from 10 to 100 by experiments from case by case. The smaller correlation, which means that there are strong conflicts between tasks, the better result with big γ. From the algorithm we could find it is complex to keep different G and H ensemble balance. We propose a common one above, and believe there might be other better ensemble algorithms for particular problems and tasks.
When it comes to a problem with main task and sub tasks, k1 to km should not be selected randomly. We suggest that to always choice the main task label as the k. And there is a helpful method to improve the learning process, which is to calculate or evaluate the correlation of the main task with others and give a bigger w and smaller v for the task of positive correlation.

heading: 4.2 GRADIENTS FOR UPDATING LEAF VALUE
text: After defined the Ge and He, the GBDT tree learner could build trees for each iteration. The length of Ge and He should be same as the length of the number of samples, while the length of Updating Gradient Gun and Updating Hessian Hun same as the length of the number of samples multiplying the number of labels.
It is acceptable for the MT-GBM to set the same rate for each label with traditional GBDT. However, we still have some explorations about the Gun and Hun, and propose the algorithm below with some tricks.
The function f we defined is used to calculate correlation of the gradients vector of each label in current iteration, which means the updating speed should be fast if all the gradients have the similar distribution. Actually whether the calculation of correlation works or not depends on many things such as type of tasks and meaning of tasks. Therefore set the corr as a constant equal 1.0 might be better on some complex datasets, which means the updating rate is same as the traditional single task GBDT. And noticed that we multiply the same corr for each Gui in one iteration, it is because that the GBDT learns things from residual errors. Trees get inconsistent paces if we set different rates for different tasks, which leads to bad results.

heading: 5 EXPERIMENTS

heading: 5.1 EXPERIMENT SETUP
text: Through experiments, we seek to answer the following questions:
Algorithm 3 The calculations of Gun and Hun for one sample before an iteration Input: predication of trees above of each task :p1,p2,p3,...,pn;true label of each task:l1,l2,l3,...,ln Output: Gu1,Gu2,Gu3,...,Gun,Hu1,Hu2,Hu3,...,Hun
1: n = number of tasks 2: for i = 1, . . . , n do 3: Gui = d dpL(li, pi) 4: Hui = d2
dp2L(li, pi)
5: concatenate the Gui of all the samples as vector Gsui 6: end for 7: calculate the correlation corr with n vectors (Gsu1, Gsu2, Gsu3, ..., Gsun): corr = f(Gsu1, Gsu2, Gsu3, ..., Gsun) 8: for i = 1, . . . , n do 9: Gui∗ = clip(corr, 0.5, 1)
10: end for 11: return Gu1,Gu2,Gu3,...,Gun,Hu1,Hu2,Hu3,...,Hun
1. Can MT-GBM achieve better performance state-of-the-art implementations of gradient boosted decision trees (GBDTs) XGBoost, LightGBM and Catboost?
2. Does neural networks of multi-task achieves same performance as MT-GBM?
In this section, we report the experimental results regarding our proposed MT-GBM algorithm which was developed on the LightGBM(The code1https://github.com/Microsoft/LightGBM is available at GitHub). Particularly, the details of experimental setup, including data description, baseline models, and some specific experiments settings, will be presented below. All experiments are done with the same features in GBDTs, Neural Network, and MT-GBM, and the sub tasks in MT-GBM are derived from the main task. Our experimental environment is a Linux server with two i9-7980 CPUs (in total 36 cores) and 128GB memories.

heading: 5.2 DATASETS AND BASELINES
text: We tested the methods on two different real-world datasets. The first one is part of China Foreign Currency Volume from 2017 to 2019. The dataset is about the currency exchange volume between one country and China by common people. The target is simple: use the historic data to predict next day volume and we make the last 30 days value as the testset. It is a time-series problem, and it only contains one amount in every workday from 2017 to 2019 and has no features. Rmse and mape are used to evaluate the models.
GBDT is proved as a great model in time-series cases and features are often be constructed by sliding window (Lee et al., 2018; Liu et al., 2018; Xia & Chen, 2017). We count the minimum, maximum, average, and variance of the last three days, one week, two weeks, and one month as the sliding window features. With these, a baseline is finished and we run XGBoost and LightGBM on it.
And the second dataset is Kaggle Competition:IEEE-CIS Fraud Detection (2), which is a credit fraud detection case from Vesta Corporation. The target of model is to predict whether this transaction is fraud or not. There is about 3.5% fraud transactions in the dataset while ROC-AUC is a good metric for the imbalance data. There are 600,000 samples in the trainset and 500,000 in the testset. Since we can not get the true label of testset, we cut the last 20% samples in the trainset by datetime as the validation data, exactly same as the reality.
About 400+ original features, contains user information, shop information and transactions details and anonymous features, are provided by the official. As a supplement, 100+ features are mined from raw features in a baseline solution of the golden section. We ensure that the baseline is a top single model by the leaderboard. We run XGBoost, LightGBM, CatBoost and MT-GBM with the setting copied from the open source code.
1https://github.com/Microsoft/LightGBM 2https://www.kaggle.com/c/ieee-fraud-detection

heading: 5.3 RESULTS
text: In the dataset of Currency Volume, the trend of amount may have some regular patterns. Only the label of amount itself is not enough. We propose that a sub label is the volume of next day divided by current day, while the main label is the volume of next day. Two labels have different orders of magnitude while the sub label is a number floating around 1.0 and the main label is great than 108.
All tree models in experiment 1 perform not badly as Table 1 shown. XGBoost is better than LightGBM at the same parameters on both metrics.
The result of first experiment shows the effects of ensemble numerical labels with different orders of magnitude. We compare the LightGBM and MT-GBM as the improvement of multi-task model, as we add the multi-task algorithm on the LightGBM repository. From Table 1, the best mape is reduced from 4.3% to 3.9% and the best rmse is reduced from 400.32 to 308.21.
We conclude the results that multi-task teach model not only the volume of currency but also the rise and fall which make model robust at small dataset by splitting at a better position, meanwhile slowing down convergence speed of the model.
However, the second dataset has not provided multi labels too. In the risk management, fraud transaction is often divided into many classes by the type of fraud. Table 2 shows the details of sub tasks.
The dataset is divided into 3 folds in order to reduce the influence of randomness. The final result contains the average of 3 folds and single fold versions. Since we have 4 sub tasks in the case, some combinations have been tested. From Table 3, most of the multi-task experiments behave better than a single task on the ROC-AUC of the main task. We can find that 2-tasks models with sub task 1 or 3 have presentable results and the 4-tasks model has the best performance through comparative analyses between LightGBM and MT-GBM. Low correlation task means irrelevant task which is helpless to the main task and high correlation seem to have less supplement for the main task, so the task 1 and 3 are the best for this case. And the Catboost and XGBoost with single task can’t reach the metric of MT-GBM. It is worthwhile mentioning that this experiment is training as the outstanding parameter γ equals 50, cause the correlation between sub task and main task seems not so strong. And the improvement will be descended when γ equals 10. We regard this case as an example that our algorithm makes a balance on the conflicting tasks.
Somehow the more tasks do not bring the better result. We concluded it the ensemble method of Gradients and Hessians is not perfect enough, failing to express too many tasks.

heading: 5.4 COMPARISON WITH NEURAL NETWORKS
text: We also try neural networks because it is a natural multi-task model. There are lots of big numerical numbers in the features disturbing the parameters in the neural cells. Therefore we have to transform them through F (x):
F (x) = log10(x+ 1) (5)
On the other hand, we fill the NaN value in the features with 0. And after this data preprocessing, we create a simple Multi Layer Perceptron network structure with 2 hidden layers and GELUs activation function (Hendrycks & Gimpel, 2016) to fit the dataset. The result is shown in Tables 1 and 3.
The neural network performs badly in both experiments as we explain before. It is not an unexpected result while most of these problems are solved with GBDT in the industry.

heading: 6 CONCLUSION
text: In this paper, we have proposed a new general algorithm, multi-task gradient boosted tree, basing on gradient boosted decision tree, which provides powerful features to the tree model. We have performed both theoretical analysis and experimental studies on the implementation. The experimental results are consistent with the theory and show that with the help of multi-task, MT-GBM can significantly outperform traditional GDBTs. And the MT-GBM has great extensibility as it didn’t place restrictions on the datasets and the type of task.

heading: B ADDITIONAL EXPERIMENT RESULTS
text: For clearness, we only show the final metric in the main text. There are some visualizations of intermediate process here.
Figures 3 and 4 show the metric during training in the Currency dataset. The lines of single task(blue and yellow) are declined fast and overfit at about iteration 150. The lines of Multi-Task(green and red) behaves better at both metrics. However, the valid metric of Multi-Task overfits finally too, as a result of a normal phenomenon in the tree model.
In Ieee-Fraud experiment we have defined 4 sub tasks for the main task. The first three of them are the different types of fraud extracted from the main task. Take the label of main task and first three tasks as ygt1 to ygt4, the predictions of single task LGB model as ysingle, and the predictions of MT-GBM as ymt1 to ymt4. Figure 5 shows the Roc-auc curves of different outputs.
3https://www.kaggle.com/whitebird/ieee-internal-blend

Review summaries:
Authors propose a way to perform multi-task learning with gradient boosted decision trees. Multitask learning has shown promise in deep learning and other machine learning systems. It is hard to perform multitask learning in decision trees. The main contributions from the paper are as follows Authors propose a multitasking gradient boosted decision tree Authors further propose a way to dynamically adjust the learning rate for each task so the learner does not overfit any specific task. They evaluate their approach on two public datasets. 

This paper focuses on the multi-task learning for the popular gradient boosted decision trees (GBDTs) for tabular data. This paper proposes two novel techniques to the learning of GBDTs -- the multi-output tree and the mutli-task splitting -- that can be easily incorporated to the existing implementations of GBDTs without significant computational overhead. The multi-output tree shares the tree structure (split dimensions and thresholds) across all tasks but maintains different leaf statistics for each task. The multi-task splitting utilizes the per-task statistics in an internal node to generate a single statistic (gradient and hessian) that incorporates information from a small number of tasks to find the optimal split. The leaf value updating procedure is also modified to handle the multi-task setup. The proposed MT-GBM scheme is evaluated on two data sets where there is a single "main" tasks and auxilliary tasks are generated and used in the multi-task learning of MT-GBM. The empirical results indicate that the proposed MT-GBM is able to marginally outperform XGBoost and LightGBM trained on a single task, and a neural network trained with multiple tasks.  

Recent developments in deep learning and neural networks showed impressive results on structured data, like audio, text and images. However, there are a lot of tabular data, like finance, search-engine machines, recommendation systems, telecommunications, physics and astrophysics characteristics, etc. For problems and applications with these tabular data random forest and gradient boosting (over decision trees) are the mostly used algorithms (most kaggle competitions were won by them).   Mutli-task learning is very active area nowadays as solving tasks together could improve each task (less over-fitting, better representation and generalization) and moreover we could have one general model (faster inference). It is studied in the context of deep learning a lot: computational graph mostly is shared across tasks and we can natively optimize combination of loss functions correspondent to each particular task. On the other hand, for gradient boosting (over decision trees) multi-task learning is still challenging. One of the ways is to learn shared tree structure across tasks at each gradient boosting iteration. Authors are focusing on this idea: they share same tree structure and leaves. Then authors propose the splitting criteria based on multi-task losses (instead of one-task loss as done in classic gradient boosting): gradients and hessians of tasks losses are aggregated to provide the final loss which is used as splitting criteria in the greedy algorithm of tree construction. Further, authors propose a special scaling for gradients based on correlation between the tasks (computed on losses gradients) to assign multi-label in each leaf. Proposed algorithm, called MT-GBM (multi-task gradient boosting machine) is empirically estimated on 3 datasets where subtasks are selected from the main task and should improve the main task via multi-task learning. Implementation is done on top of XGBoost and with experiments it is shown that MT-GBM improves upon the main task learning.