Package: treemodel2sql
XGBoost和LightGBM模型转sql语句工具包现在是大数据量的时代，我们开发的模型要应用在特别大的待预测集上，使用单机的python，需要预测2、3天，甚至更久，中途很有可能中断。因此需要通过分布式的方式来预测。这个工具包就是实现了将训练好的python模型，转换成sql语句。将生成的sql语句可以放到大数据环境中进行分布式执行预测，能比单机的python预测快好几个量级思想碰撞微信微信公众号干饭人魔都数据干饭人仓库地址：https://github.com/ZhengRyan/treemodel2sql微信公众号文章：https://mp.weixin.qq.com/s/u8Nsp5M93WIGL2M0tU4U_gpipy包：https://pypi.org/project/treemodel2sql/环境准备可以不用单独创建虚拟环境，因为对包的依赖没有版本要求treemodel2sql安装pip install（pip安装）pipinstalltreemodel2sql# to installpipinstall-Utreemodel2sql# to upgradeSource code install（源码安装）pythonsetup.pyinstall运行样例###【注意：：：核验对比python模型预测出来的结果和sql语句预测出来的结果是否一致请查看教程代码】"https://github.com/ZhengRyan/treemodel2sql/tree/master/examples"路径处的tutorial_code_lightgbm_model_transform_sql.ipynb和tutorial_code_xgboost_model_transform_sql.ipynbxgboost model transform sql导入相关依赖importxgboostasxgbfromsklearn.datasetsimportmake_classificationfromsklearn.model_selectionimporttrain_test_split训练1个xgboost二分类模型X,y=make_classification(n_samples=10000,n_features=10,n_informative=3,n_redundant=2,n_repeated=0,n_classes=2,weights=[0.7,0.3],flip_y=0.1,random_state=1024)X_train,X_test,y_train,y_test=train_test_split(X,y,random_state=1024)###训练模型model=xgb.XGBClassifier(n_estimators=3)model.fit(X_train,y_train)#xgb.to_graphviz(model)使用treemodel2sql工具包将模型转换成的sql语句fromtreemodel2sqlimportXGBoost2Sqlxgb2sql=XGBoost2Sql()sql_str=xgb2sql.transform(model)将sql语句保存xgb2sql.save()将sql语句打印出来print(sql_str)selectkey,1/(1+exp(-(tree_1_score+tree_2_score+tree_3_score)+(-0.0)))asscorefrom(selectkey,--tree1casewhen(f9<-1.64164519orf9isnull)thencasewhen(f3<-4.19117069orf3isnull)thencasewhen(f2<-1.31743848orf2isnull)then-0.150000006else-0.544186056endelsecasewhen(f3<1.23432565orf3isnull)thencasewhen(f7<-2.55682254orf7isnull)then-0.200000018elsecasewhen(f5<0.154983491orf5isnull)then0.544721723elsecasewhen(f3<0.697217584orf3isnull)then-0.150000006else0.333333373endendendelsecasewhen(f5<-1.0218116orf5isnull)thencasewhen(f0<-0.60882163orf0isnull)thencasewhen(f2<0.26019755orf2isnull)then0.0666666701else-0.300000012endelse-0.520000041endelse0.333333373endendendelsecasewhen(f9<1.60392439orf9isnull)thencasewhen(f3<0.572542191orf3isnull)thencasewhen(f5<0.653370142orf5isnull)thencasewhen(f7<-0.765973091orf7isnull)thencasewhen(f3<-0.432390809orf3isnull)then0.204000011else-0.485454559endelsecasewhen(f3<-1.20459461orf3isnull)then-0.5104478else0.441509455endendelsecasewhen(f7<0.133017987orf7isnull)thencasewhen(f8<0.320554674orf8isnull)then-0.290322572else0.368339777endelsecasewhen(f8<-0.211985052orf8isnull)then0.504000008else-0.525648415endendendelsecasewhen(f7<2.22314501orf7isnull)thencasewhen(f8<-0.00532855326orf8isnull)thencasewhen(f8<-0.204920739orf8isnull)then-0.533991575else-0.200000018endelse0.428571463endelsecasewhen(f3<1.33772755orf3isnull)thencasewhen(f0<-0.975171864orf0isnull)then0.163636371else0.51818186endelse-0endendendelsecasewhen(f3<1.77943277orf3isnull)thencasewhen(f7<-0.469875157orf7isnull)thencasewhen(f3<-0.536645889orf3isnull)thencasewhen(f9<1.89841866orf9isnull)then-0else0.333333373endelsecasewhen(f4<-2.43660188orf4isnull)then0.150000006else0.551020443endendelsecasewhen(f1<-0.0788691565orf1isnull)then0.150000006else-0.375endendelsecasewhen(f4<-1.73232496orf4isnull)then-0.150000006elsecasewhen(f6<-1.6080606orf6isnull)then-0.150000006elsecasewhen(f7<-0.259483218orf7isnull)then-0.558620751else-0.300000012endendendendendendastree_1_score,--tree2casewhen(f9<-1.64164519orf9isnull)thencasewhen(f3<-4.19117069orf3isnull)thencasewhen(f0<0.942570388orf0isnull)then-0.432453066else-0.128291652endelsecasewhen(f3<1.23432565orf3isnull)thencasewhen(f7<-2.55682254orf7isnull)then-0.167702854elsecasewhen(f5<0.154983491orf5isnull)thencasewhen(f1<2.19985676orf1isnull)then0.41752997else0.115944751endelse0.115584135endendelsecasewhen(f5<-1.0218116orf5isnull)thencasewhen(f0<-0.60882163orf0isnull)then-0.119530827else-0.410788596endelse0.28256765endendendelsecasewhen(f9<1.60392439orf9isnull)thencasewhen(f3<0.460727394orf3isnull)thencasewhen(f5<0.653370142orf5isnull)thencasewhen(f7<-0.933565617orf7isnull)thencasewhen(f3<-0.572475374orf3isnull)then0.182491601else-0.377898693endelsecasewhen(f3<-1.20459461orf3isnull)then-0.392539263else0.352721155endendelsecasewhen(f7<0.207098693orf7isnull)thencasewhen(f8<0.498489976orf8isnull)then-0.193351224else0.29298231endelsecasewhen(f8<-0.117464997orf8isnull)then0.400667101else-0.402199954endendendelsecasewhen(f7<1.98268723orf7isnull)thencasewhen(f8<-0.00532855326orf8isnull)thencasewhen(f7<1.36281848orf7isnull)then-0.408002198else-0.236123681endelsecasewhen(f5<1.14038813orf5isnull)then0.404326111else-0.110877581endendelsecasewhen(f3<1.56952488orf3isnull)thencasewhen(f5<2.14646816orf5isnull)then0.409404457else0.0696995854endelse-0.32059738endendendelsecasewhen(f3<1.77943277orf3isnull)thencasewhen(f7<-0.469875157orf7isnull)thencasewhen(f3<-0.536645889orf3isnull)thencasewhen(f9<1.89841866orf9isnull)then-0else0.28256765endelse0.419863999endelsecasewhen(f3<0.444227457orf3isnull)then-0.34664312else0.0693304539endendelsecasewhen(f4<-1.10089087orf4isnull)thencasewhen(f3<2.3550868orf3isnull)then0.0147894565else-0.331404865endelse-0.421277165endendendendastree_2_score,--tree3casewhen(f9<-1.64164519orf9isnull)thencasewhen(f3<-4.19117069orf3isnull)thencasewhen(f4<-1.30126143orf4isnull)then-0.0772174299else-0.374165356endelsecasewhen(f3<1.23432565orf3isnull)thencasewhen(f7<-2.55682254orf7isnull)then-0.142005175elsecasewhen(f5<0.154983491orf5isnull)thencasewhen(f7<3.59379435orf7isnull)then0.352122813else0.132789165endelse0.0924336985endendelsecasewhen(f5<-1.0218116orf5isnull)thencasewhen(f0<-0.60882163orf0isnull)then-0.0954768136else-0.351594836endelse0.245992288endendendelsecasewhen(f9<1.60392439orf9isnull)thencasewhen(f3<0.347133756orf3isnull)thencasewhen(f5<0.661561131orf5isnull)thencasewhen(f7<-0.933565617orf7isnull)thencasewhen(f3<-0.472413659orf3isnull)then0.116336405else-0.313245147endelsecasewhen(f3<-1.5402329orf3isnull)then-0.352897167else0.311400592endendelsecasewhen(f7<0.275665522orf7isnull)thencasewhen(f8<0.403402805orf8isnull)then-0.292606086else0.220064178endelsecasewhen(f8<-0.0442957953orf8isnull)then0.350784421else-0.336107522endendendelsecasewhen(f7<1.77503061orf7isnull)thencasewhen(f8<0.196157426orf8isnull)thencasewhen(f7<1.36281848orf7isnull)then-0.3376683else-0.0711223111endelsecasewhen(f7<-0.661211252orf7isnull)then0.434363276else-0.219307661endendelsecasewhen(f3<1.37940335orf3isnull)thencasewhen(f6<1.34894884orf6isnull)then0.367155522else0.124757253endelse-0.293739736endendendelsecasewhen(f3<1.77943277orf3isnull)thencasewhen(f7<-0.469875157orf7isnull)thencasewhen(f3<-0.536645889orf3isnull)thencasewhen(f9<1.89841866orf9isnull)then-0else0.245992288endelsecasewhen(f0<1.60565615orf0isnull)then0.357973605else0.193993196endendelsecasewhen(f9<1.89456153orf9isnull)then-0.276471078else0.111896731endendelsecasewhen(f1<1.35706067orf1isnull)thencasewhen(f4<-1.10089087orf4isnull)thencasewhen(f3<2.3550868orf3isnull)then0.0119848112else-0.284813672endelse-0.376859784endelsecasewhen(f2<-0.25748384orf2isnull)then0.0723158419else-0.253415495endendendendendastree_3_scorefromdata_table)lightgbm model transform sql导入相关依赖importlightgbmaslgbfromsklearn.datasetsimportmake_classificationfromsklearn.model_selectionimporttrain_test_split训练1个lightgbm二分类模型X,y=make_classification(n_samples=10000,n_features=10,n_informative=3,n_redundant=2,n_repeated=0,n_classes=2,weights=[0.7,0.3],flip_y=0.1,random_state=1024)X_train,X_test,y_train,y_test=train_test_split(X,y,random_state=1024)###训练模型model=lgb.LGBMClassifier(n_estimators=3)model.fit(X_train,y_train)lgb.create_tree_digraph(model._Booster,tree_index=0)使用treemodel2sql工具包将模型转换成的sql语句fromtreemodel2sqlimportLgb2Sqllgb2sql=Lgb2Sql()sql_str=lgb2sql.transform(model)将sql语句保存lgb2sql.save()将sql语句打印出来print(sql_str)selectkey,1/(1+exp(-((tree_0_score+tree_1_score+tree_2_score)+(-0.0))))asscorefrom(selectkey,--tree0casewhen(Column_9isnullandtrue==trueorColumn_9<=-1.6632177658322267)thencasewhen(Column_3isnullandtrue==trueorColumn_3<=-1.5495347315228287)then-0.8762269835062461elsecasewhen(Column_3isnullandtrue==trueorColumn_3<=1.3809327191663714)thencasewhen(Column_7isnullandtrue==trueorColumn_7<=-0.9295802204706084)then-0.5897045622608377else-0.45043755217654896endelse-0.8259039361137315endendelsecasewhen(Column_9isnullandtrue==trueorColumn_9<=1.6094262296522934)thencasewhen(Column_3isnullandtrue==trueorColumn_3<=0.5872311864187175)thencasewhen(Column_5isnullandtrue==trueorColumn_5<=0.6614392939376948)thencasewhen(Column_7isnullandtrue==trueorColumn_7<=-0.7719746626173067)thencasewhen(Column_3isnullandtrue==trueorColumn_3<=-0.47247338144095047)thencasewhen(Column_8isnullandtrue==trueorColumn_8<=1.366961884973301)then-0.4786799717862057else-0.7246405208927198endelsecasewhen(Column_5isnullandtrue==trueorColumn_5<=0.33583497995253037)then-0.8576174849758683elsecasewhen(Column_8isnullandtrue==trueorColumn_8<=0.725660478148893)then-0.45255653873335094else-0.7268525857883245endendendelsecasewhen(Column_3isnullandtrue==trueorColumn_3<=-1.1870271596234325)then-0.8614608311023392elsecasewhen(Column_3isnullandtrue==trueorColumn_3<=0.3450499438974393)thencasewhen(Column_9isnullandtrue==trueorColumn_9<=0.719805683840674)then-0.452301345707117elsecasewhen(Column_3isnullandtrue==trueorColumn_3<=0.24246683106041103)then-0.5283016889821918else-0.7127023928846948endendelsecasewhen(Column_8isnullandtrue==trueorColumn_8<=-0.41161510713991156)then-0.4296985348121034elsecasewhen(Column_5isnullandtrue==trueorColumn_5<=0.10191127467897389)then-0.8660786096722881else-0.7213351365659543endendendendendelsecasewhen(Column_7isnullandtrue==trueorColumn_7<=0.1335285190014809)thencasewhen(Column_8isnullandtrue==trueorColumn_8<=0.3151386351229311)then-0.7813601336005307elsecasewhen(Column_8isnullandtrue==trueorColumn_8<=0.797889562545179)thencasewhen(Column_7isnullandtrue==trueorColumn_7<=-0.6119464230811894)then-0.47541454265459887elsecasewhen(Column_9isnullandtrue==trueorColumn_9<=1.036435633602195)then-0.5335985526359568else-0.7562414479727859endendelse-0.4911879020856353endendelsecasewhen(Column_8isnullandtrue==trueorColumn_8<=-0.2260123021962532)then-0.4495750599610143elsecasewhen(Column_5isnullandtrue==trueorColumn_5<=1.3421825678767703)then-0.8610329200059168elsecasewhen(Column_7isnullandtrue==trueorColumn_7<=0.9404699950203693)then-0.8282483467723211else-0.6811365779458289endendendendendelsecasewhen(Column_7isnullandtrue==trueorColumn_7<=2.2029620939306103)thencasewhen(Column_8isnullandtrue==trueorColumn_8<=-0.20972527484810635)then-0.8620295284631452else-0.6406954940851597endelse-0.49244599655670496endendelsecasewhen(Column_3isnullandtrue==trueorColumn_3<=1.7659981730252496)thencasewhen(Column_7isnullandtrue==trueorColumn_7<=-0.709846883702808)then-0.44913417585419935else-0.7039945818670766endelse-0.8609816276658351endendendastree_0_score,--tree1casewhen(Column_9isnullandtrue==trueorColumn_9<=-1.6632177658322267)thencasewhen(Column_3isnullandtrue==trueorColumn_3<=-1.5495347315228287)then-0.1304300334944157elsecasewhen(Column_3isnullandtrue==trueorColumn_3<=1.084128912605023)then0.23707714508016445else-0.0285157252239888endendelsecasewhen(Column_9isnullandtrue==trueorColumn_9<=1.6094262296522934)thencasewhen(Column_3isnullandtrue==trueorColumn_3<=0.5872311864187175)thencasewhen(Column_5isnullandtrue==trueorColumn_5<=0.6614392939376948)thencasewhen(Column_7isnullandtrue==trueorColumn_7<=-0.9295802204706084)thencasewhen(Column_3isnullandtrue==trueorColumn_3<=-0.47247338144095047)thencasewhen(Column_9isnullandtrue==trueorColumn_9<=0.6831935640096413)then0.19417988489655402else-0.02440020590879623endelsecasewhen(Column_9isnullandtrue==trueorColumn_9<=1.036435633602195)then-0.11286120415008301else0.08687819192210895endendelsecasewhen(Column_3isnullandtrue==trueorColumn_3<=-1.1870271596234325)then-0.11564799847528345elsecasewhen(Column_9isnullandtrue==trueorColumn_9<=0.3004389492550987)then0.2405259765006961elsecasewhen(Column_3isnullandtrue==trueorColumn_3<=0.2158166423973093)thencasewhen(Column_7isnullandtrue==trueorColumn_7<=-0.6547653704267197)then0.11054725461728983else0.2269735323256574endelsecasewhen(Column_5isnullandtrue==trueorColumn_5<=0.10191127467897389)then-0.1208387319652229elsecasewhen(Column_1isnullandtrue==trueorColumn_1<=0.6151636033969191)thencasewhen(Column_8isnullandtrue==trueorColumn_8<=-0.018071874612147206)then0.0016586271011360867else0.1332964744950624endelse-0.04300639955361431endendendendendendelsecasewhen(Column_7isnullandtrue==trueorColumn_7<=0.1335285190014809)thencasewhen(Column_8isnullandtrue==trueorColumn_8<=0.27824194846895633)then-0.0613742224255702elsecasewhen(Column_3isnullandtrue==trueorColumn_3<=-0.37785910715343846)then0.23646592246665976elsecasewhen(Column_7isnullandtrue==trueorColumn_7<=-0.3752232334694377)thencasewhen(Column_8isnullandtrue==trueorColumn_8<=1.2531231065059638)then0.20734413117883457else0.08650249777206469endelsecasewhen(Column_6isnullandtrue==trueorColumn_6<=0.1864449772206759)then0.0014253268716679143else0.19695058122999737endendendendelsecasewhen(Column_8isnullandtrue==trueorColumn_8<=-0.2260123021962532)then0.23847961174940435elsecasewhen(Column_5isnullandtrue==trueorColumn_5<=1.3421825678767703)then-0.115222500730401elsecasewhen(Column_8isnullandtrue==trueorColumn_8<=0.1800136296190439)then0.060703267253890685else-0.08603393258665121endendendendendelsecasewhen(Column_7isnullandtrue==trueorColumn_7<=2.0654353175567177)thencasewhen(Column_8isnullandtrue==trueorColumn_8<=-0.20972527484810635)then-0.11660912729818569else0.08556190442726179endelse0.19286044838453295endendelsecasewhen(Column_3isnullandtrue==trueorColumn_3<=1.7659981730252496)thencasewhen(Column_7isnullandtrue==trueorColumn_7<=-0.709846883702808)then0.23881782889986833else0.031196179358762684endelse-0.11517151290097437endendendastree_1_score,--tree2casewhen(Column_9isnullandtrue==trueorColumn_9<=-1.6632177658322267)thencasewhen(Column_3isnullandtrue==trueorColumn_3<=-1.5495347315228287)then-0.1246790381676725elsecasewhen(Column_3isnullandtrue==trueorColumn_3<=1.3809327191663714)thencasewhen(Column_3isnullandtrue==trueorColumn_3<=0.7880730732430219)then0.20644395671242038else0.1189732213687583endelse-0.0788791406092519endendelsecasewhen(Column_9isnullandtrue==trueorColumn_9<=1.6094262296522934)thencasewhen(Column_3isnullandtrue==trueorColumn_3<=0.6038877970616275)thencasewhen(Column_5isnullandtrue==trueorColumn_5<=0.6614392939376948)thencasewhen(Column_7isnullandtrue==trueorColumn_7<=-0.9295802204706084)thencasewhen(Column_3isnullandtrue==trueorColumn_3<=-0.47247338144095047)thencasewhen(Column_7isnullandtrue==trueorColumn_7<=-1.6012773225012003)thencasewhen(Column_9isnullandtrue==trueorColumn_9<=0.5166566313259823)then0.14609631025061118else-0.08519567041375777endelse0.20775467159820815endelsecasewhen(Column_9isnullandtrue==trueorColumn_9<=1.013669661144344)then-0.10698219135389433else0.06871478156256401endendelsecasewhen(Column_3isnullandtrue==trueorColumn_3<=-1.3968501345470952)then-0.11575647257046306elsecasewhen(Column_3isnullandtrue==trueorColumn_3<=0.3381305759653203)thencasewhen(Column_9isnullandtrue==trueorColumn_9<=0.6711065070274237)then0.2055124396523362elsecasewhen(Column_3isnullandtrue==trueorColumn_3<=0.08416177325543671)then0.18486979450427765elsecasewhen(Column_5isnullandtrue==trueorColumn_5<=0.4068544493018646)then-0.04785536117613759else0.11879752654423877endendendelsecasewhen(Column_8isnullandtrue==trueorColumn_8<=-0.5159476948660279)then0.22110765206038635elsecasewhen(Column_9isnullandtrue==trueorColumn_9<=0.7486693174493504)then-0.07426890908854177else0.12667195625568606endendendendendelsecasewhen(Column_7isnullandtrue==trueorColumn_7<=0.1335285190014809)thencasewhen(Column_8isnullandtrue==trueorColumn_8<=0.24202650482335777)then-0.07289970951054063elsecasewhen(Column_8isnullandtrue==trueorColumn_8<=0.797889562545179)thencasewhen(Column_7isnullandtrue==trueorColumn_7<=-0.6119464230811894)then0.19832855719431475else0.05525269245438189endelse0.17963780553305242endendelsecasewhen(Column_8isnullandtrue==trueorColumn_8<=-0.25745431183810524)then0.20767489164465192elsecasewhen(Column_5isnullandtrue==trueorColumn_5<=1.3421825678767703)then-0.10914096073415368elsecasewhen(Column_9isnullandtrue==trueorColumn_9<=1.1239281866391186)then0.04590138464892762else-0.08877584425107066endendendendendelsecasewhen(Column_7isnullandtrue==trueorColumn_7<=2.2516563501714586)thencasewhen(Column_8isnullandtrue==trueorColumn_8<=-0.20972527484810635)then-0.11032211105487942else0.06868549293845214endelse0.18271175459060623endendelsecasewhen(Column_3isnullandtrue==trueorColumn_3<=1.7659981730252496)thencasewhen(Column_7isnullandtrue==trueorColumn_7<=-0.709846883702808)then0.2062125352630051else0.027771830564502133endelse-0.10919849076791249endendendastree_2_scorefromdata_table)