MT-GBM: A Multi-Task Gradient Boosting Machine  with Shared Decision Trees

1 INTRODUCTION . Gradient boosting is a powerful machine-learning technique that achieves state-of-the-art results in a variety of applications . It is a process of constructing a strong predictor by ensembling weak predictors and performing gradient descent in a functional space . Gradient boosting has achieved tremendous success in a wide range of machine learning tasks . It is popular in applications with heterogeneous features , noisy tabular data , and complex dependencies , such as recommender systems ( Wang et al. , 2016 ; Wen et al. , 2019 ) , web search ( Agrawal & Shanahan , 2010 ) , FinTech ( Wang et al. , 2018 ; Mei & Li-ren , 2015 ; Yu-Zhuo & Xiao-Ni , 2006 ) , online advertising , etc ( Roe et al. , 2004 ; Zhang & Haghani ) . Among all gradient boosting algorithms , XGBoost and LightGBM are the two most famous algorithms . They have contributed to countless top solutions in machine learning and data mining competitions on tabular data . Despite the considerable success of gradient boosting , applying GBDT models to multi-task learning is still challenging . The GBDT model aims to train a high-performance discriminative model towards a single loss function of a single task . Therefore , it can not learn a shared latent feature representation , which is essential for multi-task learning . We list three things that limit the applicability of GBDT in multi-task learning . First and foremost , the core of efficient multi-task learning is to extract valuable hidden representation that shares across all tasks , which is non-trivial for tree-based ensemble learning methods like GBDT . For example , in news recommendation , user response prediction tasks include the prediction of click-through rate , dwell-time , thumbs , reviews , shares , and so on . These objects correspond to the same user ; thus an informative representation of the user is the key to such a multi-task learning problem . The recent development of deep learning can achieve this naturally using hidden embeddings and layers in neural networks ( Ruder , 2017 ; Huang et al. , 2018 ; Gao et al. , 2019 ) , thus has shown its dominance for multi-task learning in computer vision and natural language processing ( Su et al. , 2015 ; Radford et al. , 2019 ; Liu et al. , 2019 ) . However , it is known that neural networks sometimes fail in applications with tabular data . Second , the splitting mechanism for decision trees aims to separate the input space as well as the training samples according to a single loss function , which could not be applied directly to multi- task learning where there are multiple losses . When optimizing end-to-end neural networks , it is natural to sum up the gradients from all losses . However , for tree-based models , the splits that have maximal information gain for the losses can differ a lot , especially when training classification and regression tasks at the same time . Therefore , it is challenging to find tree splits that are suitable for all tasks . Third , in multi-task learning , the model should keep similar paces for each task for stable learning . For neural networks , current gradient descent methods with adaptive step size can adjust the learning rate for different tasks . However , the learning speed of GBDT is related to the residual errors . Therefore , when there are multiple tasks , if a fixed learning rate is applied to all tasks , the model tends to overfit certain tasks while underfit the others . Therefore , how to adjust learning rates for multiple tasks remains a challenge . In this paper , we propose a new algorithmic technique for multi-task learning based on GBDT , namely multi-task Gradient Boosting Machine ( MT-GBM ) . The key novelty of MT-GBM is that it learns shared decision tree structures for multiple tasks , which can be understood as the hidden knowledge across all tasks . As the decision trees are shared , it requires almost the same computational overhead comparing to a single task GBDT . Therefore , MT-GBM can be efficiently trained . We verified MT-GBM on two financial datasets . Experimental results show that MTGBM can achieve significantly higher performance than existing state-of-the-art gradient boosted decision trees methods , XGBoost ( Roe et al. , 2004 ) , LightGBM ( Ke et al. , 2017 ) and Catboost ( Prokhorenkova et al. , 2018 ) . Our contribution can be summarized as follows : • We propose MT-GBM that learns an ensemble of shared decision trees learned from multiple losses simultaneously , which significantly improves the efficiency and effectiveness of GBDTs for multi-task learning . • We propose a new learning algorithm that adaptively adjusts the learning rate for each task by taking the scale of residual errors into consideration . It stabilizes training on arbitrarily complex problems . • Our method is easy to implement upon existing GBDT models such as XGBoost and LightGBM . We tested our method by modifying LightGBM and showed significant performance improvements . 2 PRELIMINARIES . 2.1 MULTI-TASK LEARNING WITH DECISION TREES . It is widely known that multi-task learning could improve the performance of every single task by sharing knowledge from all tasks . When using max-margin machine learning ( i.e. , SVMs ) or deep learning ( i.e. , feed-forward neural networks ) , multi-task learning has achieved significant success ( Caruana , 1997 ; Evgeniou & Pontil , 2004 ; Bingel & Søgaard , 2017 ) . However , with decision-treebased models , it remains unclear how to learn shared knowledge from multiple tasks . There are many potential reasons why adding extra outputs to a model might improve generalization performance . For example , to the extent that tasks are uncorrelated , their contribution to the aggregated gradient , which guides the node splitting in GBDT , can appear as noise to other tasks . Thus uncorrelated tasks might improve generalization by acting as a source of the noise . Figure 1 ( a ) shows a situation of data samples with a noisy label . One single sample contains two rectangles while the left represents label 1 and the right represents label 2 . We can assume that task 1 clicks behavior and task 2 is shares behavior in the recommendation system . It makes it clear that two tasks fix the splitting line from dotted line to solid line . Figure 1 ( b ) shows that Sub tasks also help the sample splitting when dataset has the class imbalance problem . 2.2 PROBLEM DEFINITION . GBDT is an ensemble model of decision trees , which are trained in sequence . In each iteration , GBDT learns a new decision tree by minimizing the negative gradients ( also known as the residual errors ) of previously trained trees . Specifically , to construct the decision trees , the algorithm finds the split points in the feature space according to the residual errors . Assume we have n learning tasks t ∈ { 1 , 2 , ... , n } , each with a set of supervised data { ( x1 , y1 ) , ... , ( xm , ym ) } . For simplicity , we assume that the number of data points m in each task is the same , and all the tasks share the same input space . For the t-th task , there is a predefined learning objective Lt . In regression , we might consider the mean squared error : Lt = 1 m m∑ i=1 ( yi − pi ) 2 ( 1 ) And we use the negative log-likelihood as the classification loss : Lt = − 1 m m∑ i=1 ( yi log pi + ( 1− yi ) log ( 1− pi ) ) ( 2 ) Finally , we assume the main task objective function L1 as the metric to evaluate the model . 2.3 RELATED WORK . In this section , we ignore the related work of neural network because most of neural network work well in specific tabular dataset ( Zhang & Yang , 2018 ; Ruder , 2017 ) . To achieve a multi-task tree , there were some works about multi-classifier tree on Adaboost ( Faddoul et al. , 2012 ) . This paper revises the information gain rule for learning decision trees and modified MT-Adaboost to combine multi-task Decision Trees as weak learners . Since its algorithm depends on their new equation of Information Gain , the model does not support doing the regression task yet . Obviously , it fails when it comes to residual errors in GBDT cause the loss is float number even in classifier tasks . Another approach to multi-task learning is the work sharing tree structure for different tasks ( Chapelle et al. , 2011 ) . This paper presents a general solution of boosting for different task types by minimizing the sum of losses ∑n i=1 Li . It uses a relative value to control the strength of the connection between the tasks . We approve of the way about exploring the general solution . But the part of optimization in which splitting the node would be hard and slow when consider minimizing all loss functions . It seems to be a bit complex and obsolete and is beaten by the existing implementations of gradient boosted decision trees such as XGBoost and LightGBM . However , the GBDTs has some tricks to deal with multi-class task ( Fan et al. , 2017 ) , i.e . training n trees in one iteration for n classes . When inferring , the trees belong to the specified class will be activated . Algorithm 1 Traditional Multi-Class GBDT Input : n the number of tasks , m the number of samples . Accumulative prediction of previous rounds : { p1 , t , . . . , pm , t } nt=1 ; Goundtruth label : { y1 , t , . . . , ym , t } nt=1 Output : New decision trees T1 , T2 , ... , Tn in this iteration . 1 : for i = 1 , . . . , m do 2 : pi , t = exp ( pi , t ) ∑n t=1 exp ( pi , t ) , t = 1 , . . . , n 3 : for t = 1 , . . . , n do 4 : // J is the number of nodes in the tree 5 : // N is the number of samples in the node j 6 : ỹj , t = yj , t − pj , t , j = 1 , . . . , N 7 : // Rjti means that the ith sample belongs to the j node at the tth task 8 : // γjti means the prediction of i sample at the tth task 9 : { Rjti } Jj=1 = J − terminalNodeTree ( { ỹlt , xl } Nl ) 10 : γjti = K−1K ∑ ỹlt∑ |ỹlt| ( 1−|ỹlt| ) , j = 1 , . . . , J 11 : end for 12 : end for But the trees for different classes have no interaction when training except the last softmax transformation . The trees for the label with few positive samples would have a terrible training , since it can not learn things from the other labels . Moreover , it has a big problem that it costs n times of resources when training and inferring . There were papers that implied the multi-class by solving the sparse outputs problem ( Si et al. , 2017 ) and achieving an order of magnitude improvements in model size and prediction time over existing methods . But it still doesn ’ t seem to be a multi-task solution , while realistic datasets above always have dense outputs and multi-class which can be positive at the same time . Therefore we call it a multi-class solution , not multi-task . To address the limitations of previous works , we propose two new novel techniques called multioutput tree and multi-task splitting .

Summary: This paper focuses on the multi-task learning for the popular gradient boosted decision trees (GBDTs) for tabular data. This paper proposes two novel techniques to the learning of GBDTs -- the multi-output tree and the mutli-task splitting -- that can be easily incorporated to the existing implementations of GBDTs without significant computational overhead. The multi-output tree shares the tree structure (split dimensions and thresholds) across all tasks but maintains different leaf statistics for each task. The multi-task splitting utilizes the per-task statistics in an internal node to generate a single statistic (gradient and hessian) that incorporates information from a small number of tasks to find the optimal split. The leaf value updating procedure is also modified to handle the multi-task setup. The proposed MT-GBM scheme is evaluated on two data sets where there is a single "main" tasks and auxilliary tasks are generated and used in the multi-task learning of MT-GBM. The empirical results indicate that the proposed MT-GBM is able to marginally outperform XGBoost and LightGBM trained on a single task, and a neural network trained with multiple tasks.