MT-GBM: A Multi-Task Gradient Boosting Machine  with Shared Decision Trees

1 INTRODUCTION . Gradient boosting is a powerful machine-learning technique that achieves state-of-the-art results in a variety of applications . It is a process of constructing a strong predictor by ensembling weak predictors and performing gradient descent in a functional space . Gradient boosting has achieved tremendous success in a wide range of machine learning tasks . It is popular in applications with heterogeneous features , noisy tabular data , and complex dependencies , such as recommender systems ( Wang et al. , 2016 ; Wen et al. , 2019 ) , web search ( Agrawal & Shanahan , 2010 ) , FinTech ( Wang et al. , 2018 ; Mei & Li-ren , 2015 ; Yu-Zhuo & Xiao-Ni , 2006 ) , online advertising , etc ( Roe et al. , 2004 ; Zhang & Haghani ) . Among all gradient boosting algorithms , XGBoost and LightGBM are the two most famous algorithms . They have contributed to countless top solutions in machine learning and data mining competitions on tabular data . Despite the considerable success of gradient boosting , applying GBDT models to multi-task learning is still challenging . The GBDT model aims to train a high-performance discriminative model towards a single loss function of a single task . Therefore , it can not learn a shared latent feature representation , which is essential for multi-task learning . We list three things that limit the applicability of GBDT in multi-task learning . First and foremost , the core of efficient multi-task learning is to extract valuable hidden representation that shares across all tasks , which is non-trivial for tree-based ensemble learning methods like GBDT . For example , in news recommendation , user response prediction tasks include the prediction of click-through rate , dwell-time , thumbs , reviews , shares , and so on . These objects correspond to the same user ; thus an informative representation of the user is the key to such a multi-task learning problem . The recent development of deep learning can achieve this naturally using hidden embeddings and layers in neural networks ( Ruder , 2017 ; Huang et al. , 2018 ; Gao et al. , 2019 ) , thus has shown its dominance for multi-task learning in computer vision and natural language processing ( Su et al. , 2015 ; Radford et al. , 2019 ; Liu et al. , 2019 ) . However , it is known that neural networks sometimes fail in applications with tabular data . Second , the splitting mechanism for decision trees aims to separate the input space as well as the training samples according to a single loss function , which could not be applied directly to multi- task learning where there are multiple losses . When optimizing end-to-end neural networks , it is natural to sum up the gradients from all losses . However , for tree-based models , the splits that have maximal information gain for the losses can differ a lot , especially when training classification and regression tasks at the same time . Therefore , it is challenging to find tree splits that are suitable for all tasks . Third , in multi-task learning , the model should keep similar paces for each task for stable learning . For neural networks , current gradient descent methods with adaptive step size can adjust the learning rate for different tasks . However , the learning speed of GBDT is related to the residual errors . Therefore , when there are multiple tasks , if a fixed learning rate is applied to all tasks , the model tends to overfit certain tasks while underfit the others . Therefore , how to adjust learning rates for multiple tasks remains a challenge . In this paper , we propose a new algorithmic technique for multi-task learning based on GBDT , namely multi-task Gradient Boosting Machine ( MT-GBM ) . The key novelty of MT-GBM is that it learns shared decision tree structures for multiple tasks , which can be understood as the hidden knowledge across all tasks . As the decision trees are shared , it requires almost the same computational overhead comparing to a single task GBDT . Therefore , MT-GBM can be efficiently trained . We verified MT-GBM on two financial datasets . Experimental results show that MTGBM can achieve significantly higher performance than existing state-of-the-art gradient boosted decision trees methods , XGBoost ( Roe et al. , 2004 ) , LightGBM ( Ke et al. , 2017 ) and Catboost ( Prokhorenkova et al. , 2018 ) . Our contribution can be summarized as follows : • We propose MT-GBM that learns an ensemble of shared decision trees learned from multiple losses simultaneously , which significantly improves the efficiency and effectiveness of GBDTs for multi-task learning . • We propose a new learning algorithm that adaptively adjusts the learning rate for each task by taking the scale of residual errors into consideration . It stabilizes training on arbitrarily complex problems . • Our method is easy to implement upon existing GBDT models such as XGBoost and LightGBM . We tested our method by modifying LightGBM and showed significant performance improvements . 2 PRELIMINARIES . 2.1 MULTI-TASK LEARNING WITH DECISION TREES . It is widely known that multi-task learning could improve the performance of every single task by sharing knowledge from all tasks . When using max-margin machine learning ( i.e. , SVMs ) or deep learning ( i.e. , feed-forward neural networks ) , multi-task learning has achieved significant success ( Caruana , 1997 ; Evgeniou & Pontil , 2004 ; Bingel & Søgaard , 2017 ) . However , with decision-treebased models , it remains unclear how to learn shared knowledge from multiple tasks . There are many potential reasons why adding extra outputs to a model might improve generalization performance . For example , to the extent that tasks are uncorrelated , their contribution to the aggregated gradient , which guides the node splitting in GBDT , can appear as noise to other tasks . Thus uncorrelated tasks might improve generalization by acting as a source of the noise . Figure 1 ( a ) shows a situation of data samples with a noisy label . One single sample contains two rectangles while the left represents label 1 and the right represents label 2 . We can assume that task 1 clicks behavior and task 2 is shares behavior in the recommendation system . It makes it clear that two tasks fix the splitting line from dotted line to solid line . Figure 1 ( b ) shows that Sub tasks also help the sample splitting when dataset has the class imbalance problem . 2.2 PROBLEM DEFINITION . GBDT is an ensemble model of decision trees , which are trained in sequence . In each iteration , GBDT learns a new decision tree by minimizing the negative gradients ( also known as the residual errors ) of previously trained trees . Specifically , to construct the decision trees , the algorithm finds the split points in the feature space according to the residual errors . Assume we have n learning tasks t ∈ { 1 , 2 , ... , n } , each with a set of supervised data { ( x1 , y1 ) , ... , ( xm , ym ) } . For simplicity , we assume that the number of data points m in each task is the same , and all the tasks share the same input space . For the t-th task , there is a predefined learning objective Lt . In regression , we might consider the mean squared error : Lt = 1 m m∑ i=1 ( yi − pi ) 2 ( 1 ) And we use the negative log-likelihood as the classification loss : Lt = − 1 m m∑ i=1 ( yi log pi + ( 1− yi ) log ( 1− pi ) ) ( 2 ) Finally , we assume the main task objective function L1 as the metric to evaluate the model . 2.3 RELATED WORK . In this section , we ignore the related work of neural network because most of neural network work well in specific tabular dataset ( Zhang & Yang , 2018 ; Ruder , 2017 ) . To achieve a multi-task tree , there were some works about multi-classifier tree on Adaboost ( Faddoul et al. , 2012 ) . This paper revises the information gain rule for learning decision trees and modified MT-Adaboost to combine multi-task Decision Trees as weak learners . Since its algorithm depends on their new equation of Information Gain , the model does not support doing the regression task yet . Obviously , it fails when it comes to residual errors in GBDT cause the loss is float number even in classifier tasks . Another approach to multi-task learning is the work sharing tree structure for different tasks ( Chapelle et al. , 2011 ) . This paper presents a general solution of boosting for different task types by minimizing the sum of losses ∑n i=1 Li . It uses a relative value to control the strength of the connection between the tasks . We approve of the way about exploring the general solution . But the part of optimization in which splitting the node would be hard and slow when consider minimizing all loss functions . It seems to be a bit complex and obsolete and is beaten by the existing implementations of gradient boosted decision trees such as XGBoost and LightGBM . However , the GBDTs has some tricks to deal with multi-class task ( Fan et al. , 2017 ) , i.e . training n trees in one iteration for n classes . When inferring , the trees belong to the specified class will be activated . Algorithm 1 Traditional Multi-Class GBDT Input : n the number of tasks , m the number of samples . Accumulative prediction of previous rounds : { p1 , t , . . . , pm , t } nt=1 ; Goundtruth label : { y1 , t , . . . , ym , t } nt=1 Output : New decision trees T1 , T2 , ... , Tn in this iteration . 1 : for i = 1 , . . . , m do 2 : pi , t = exp ( pi , t ) ∑n t=1 exp ( pi , t ) , t = 1 , . . . , n 3 : for t = 1 , . . . , n do 4 : // J is the number of nodes in the tree 5 : // N is the number of samples in the node j 6 : ỹj , t = yj , t − pj , t , j = 1 , . . . , N 7 : // Rjti means that the ith sample belongs to the j node at the tth task 8 : // γjti means the prediction of i sample at the tth task 9 : { Rjti } Jj=1 = J − terminalNodeTree ( { ỹlt , xl } Nl ) 10 : γjti = K−1K ∑ ỹlt∑ |ỹlt| ( 1−|ỹlt| ) , j = 1 , . . . , J 11 : end for 12 : end for But the trees for different classes have no interaction when training except the last softmax transformation . The trees for the label with few positive samples would have a terrible training , since it can not learn things from the other labels . Moreover , it has a big problem that it costs n times of resources when training and inferring . There were papers that implied the multi-class by solving the sparse outputs problem ( Si et al. , 2017 ) and achieving an order of magnitude improvements in model size and prediction time over existing methods . But it still doesn ’ t seem to be a multi-task solution , while realistic datasets above always have dense outputs and multi-class which can be positive at the same time . Therefore we call it a multi-class solution , not multi-task . To address the limitations of previous works , we propose two new novel techniques called multioutput tree and multi-task splitting .

Summary: Recent developments in deep learning and neural networks showed impressive results on structured data, like audio, text and images. However, there are a lot of tabular data, like finance, search-engine machines, recommendation systems, telecommunications, physics and astrophysics characteristics, etc. For problems and applications with these tabular data random forest and gradient boosting (over decision trees) are the mostly used algorithms (most kaggle competitions were won by them).   Mutli-task learning is very active area nowadays as solving tasks together could improve each task (less over-fitting, better representation and generalization) and moreover we could have one general model (faster inference). It is studied in the context of deep learning a lot: computational graph mostly is shared across tasks and we can natively optimize combination of loss functions correspondent to each particular task. On the other hand, for gradient boosting (over decision trees) multi-task learning is still challenging. One of the ways is to learn shared tree structure across tasks at each gradient boosting iteration. Authors are focusing on this idea: they share same tree structure and leaves. Then authors propose the splitting criteria based on multi-task losses (instead of one-task loss as done in classic gradient boosting): gradients and hessians of tasks losses are aggregated to provide the final loss which is used as splitting criteria in the greedy algorithm of tree construction. Further, authors propose a special scaling for gradients based on correlation between the tasks (computed on losses gradients) to assign multi-label in each leaf. Proposed algorithm, called MT-GBM (multi-task gradient boosting machine) is empirically estimated on 3 datasets where subtasks are selected from the main task and should improve the main task via multi-task learning. Implementation is done on top of XGBoost and with experiments it is shown that MT-GBM improves upon the main task learning.