Tree-structure segmentation for logistic regression
Authors: Adrien Ehrhardt
Venue: Submitted to ICLR 2023 (2023)
Keywords: logistic regression
decision tree
credit scoring
segmentation

Abstract: The decision for a financial institution to accept or deny a loan is based on the probability of a client paying back their debt in time. This probability is given by a model such as a logistic regression, and estimated based on, e.g., the clients’ characteristics, their credit history, the repayment performance. Historically, different models have been developed on different markets and/or credit products and/or addressed population. We show that this amounts to modelling default as a mixture model composed of a decision tree and logistic regression on its leaves (thereafter “logistic regression tree”). We seek to optimise this practice by considering the population to which a client belongs as a latent variable, which we will estimate. After exposing the context, the notations and the problem formalisation, we will conduct estimation using a Stochastic-Expectation-Maximisation (SEM) algorithm. We will finally show the performance on simulated data, and on real retail credit data from [COMPANY], as well as real open-source data.

Under review as a conference paper at ICLR 2023
TREE-STRUCTURE SEGMENTATION FOR LOGISTIC RE- GRESSION
Anonymous authors Paper under double-blind review
ABSTRACT
The decision for financial institution to accept deny loan is based the probability of a client paying back their debt in time. This probability given by model such as a logistic regression, and estimated based on, e.g., the clients' characteristics, their credit history, the repayment performance. Historically, dif- ferent models have been developed different markets and/or credit products and/or addressed population. We show that this amounts to modelling default as mixture model composed decision tree and logistic regression its leaves (thereafter "logistic regression tree"). We seek to optimise this practice by consid- ering the population which a client belongs as a latent variable, which we will estimate. After exposing the context, the notations and the problem formalisation. we will conduct estimation using a Brechastii-Expectttiie Expetaatooo-Maiiiisttionn (SEM) algorithm. We will finally show the performance retail credit data from [COMPANY], as well as real open-source data.
Maximisation
simulated data, and real
INTRODUCTION AND NOTATIONS
1.1 CONTEXT
[COMPANY], like most financial institutions, has a relatively automatic procedure to accept or deny loans and estimate its capital requirements. The procedure is based on credit scores. client fills in questionnaire with socio-demographic information and banking behavioural questions, which answers are used compute a score. This score determines the financing of the client and the necessary impairment for the bank to be ready in case of potential default. The score is learned on past clients' characteristics (from the questionnaire), which we denote by æ, and the repayment in time, or not, of their loan which we denote by y E {0, 1 (where represents the default). The score is directly proportional to the probability p(1 (ax) of the client not paying back the loan in time, which is estimated with model {po [)) Jese. parametric family θ is chosen (usually logistic regression) and the optimal parameter Ô in this family (Ō E +) is estimated from n-sample (x = (æ;, y usually using a maximum likelihood approach. Such model is relatively weak, in the sense that the hypothesis space is too restricted fit the whole clientele of big financial institutions.
1.2 A MODEL FOR EACH SEGMENT OF CLIENTS
Most financial institutions address multiple markets, egg. automobile, home appliances, or partners who sell such products, and different populations of clients (professionals, organisations, agriculture, private clients). We call "segment" such a populattul,,n and denote it by C f1...., K), where K denotes the total number of segments.
Formally, we have a vector of customer characteristics X (X)) made of d features, either continuous (i.e. valued in R) or categorical (i.e. valued in 11...,mm) without order). The aim is to predict the default Y E {0, 1} from an observation æ. These features differ depending on the segment of the population, for instance "time since creation of the company" is feature which does not apply to private clients. However, for simplicity, in the rest of the paper, we will assume that all d features shared by all segments. This leads to little loss in generality since continuous features can be discretized and a "Not Relevant" level can be introduced for categorical features; additionally can resort to feature selection at the segment level (see Section 4.6).

---

Under review as a conference paper at ICLR 2023
Clients
Private
Professional
Natural person
Group of persons
Automobile
Real Estate
Industry
Pos (y/=:; 1)
Pap (y)o: 2)
Pes w
P64 (y/a;;
4)
Poo (M) [x; 5)
Figure 1: Example of model segmentation.
Subsequently, financial institutions create different predictive models {pa- (yl=(y[))) for each popu- lation where 0 denotes the coefficient used for segment (with potential null entries), shown in Figure D which leads to models. This means we learn "expert" logistic regression models on separate "semments" of clients arranged in tree.
Since this structure is inherited from past a priori decisions, it is likely to be sub-optimal: hence seek to optimise the performance on the whole population. To this end, we formalise the data generating process in the next section.
1.3 FORMALISATION OF THE DATA GENERATING PROCESS
We assume that the model in Figure used by financial institutions accurately depicts the data generation, i.e. for given client æ, there exists segment and logistic regression parameter d for which the default is drawn from (-l:--);)). c) In other words, we assume that this model is well- specified. We denote by C p(-) the random variable valued in (1,-.., ; K) which corresponds to the assignment to a group (the tree's 's leaves in Figure []]. C' specifies both the distribution of the
predicting variables, i.e. æ|c p(-|c) and the default law for each group, which we suppose to be logistic, i.e. Y| c po- (-),(-)))) c).
Just like for Gaussian mixture models, we seek estimate p(c|z) simple proportion in the latter example), which will subsequently allow estimation of pa- ()) Current procedures are de- scribed the next section.
1.4
AD HOC "TWO-STAGEE STAGES" PRACTICE
The ad-hoc methods rely on "two-stages" procedures: first optimising the segmentation, then learn- ing the separate logistic regression each segment. The segmentation is done by practitioners using simple unsupervised "clustering" techniques such as Principal Component Analysis (PCA) and its refinements. In presence of (possibly only) categorical features, the Multiple Correspondence Anal- ysis (MCA), by Lebart et al. 1995) or the Factor Analysis of Mixed Data (FAMD), by Pagès (2014) can be more appropriate. Practitioners then visually assess whether clusters appear the projec- tion of samples onto the 2-3 first Principal Components like in the examples of Appendix A thus resulting in qualitative, clustering technique, which often performs poorly.
In Section 2 we review the existing approaches to create logistic regression trees. In Section formalise the problem of determining the best logistic regression tree mixture model and propose an estimation strategy in Section We devote Section 5 to numerical experiments on simulated data, and Section to experiments real data.
LITERATURE REVIEW OF EXISTING DIRECT APPROACHES: LOGISTIC REGRESSION TREES
The first research work focusing on similar problem than the present one seems to be LOTUS, by Chan & Loh (2004), where logistic regression trees are constructed so as to select features to split the data on the tree's nodes which break the linearity assumption of logistic regression. Its authors' motivation is that logistic regression has a fixed parameter space, defined by the number of input features, whereas trees adapt their flexibility (i.e. depth) to the sample size Thus, they search for trees which leaves are logistic regressions with a few continuous features and which intermediate nodes (found via appropriate x² test) split the population based categorical or
2

---

Under review as a conference paper at ICLR 2023
continuous features which relationship to the log-odd ratio of y is not linear (i.e. features that would perform poorly in a logistic regression). Their optimised criterion is the sum of the log-likelshoods of the logistic regression on the tree's leaves. This leads to overfitting which requires the tree to be pruned (as is classical for decision trees) using a method closely related to the one developed in the classical CART algorithm by Breiman al. (1984).
The second approach closely related to our industrial problem is named LMT, by Landwehr et al. (2005). Its authors' approach differs however from LOTUS in that they rely on boosting approach derived from the LogitBoost algorithm by Friedman et al. 2000) to estimate the logistic regression, and an adaptation of the classical C4.5 algorithm by Quinlan (2014) to grow the tree. The two cen- tral ideas behind their usage of the LogitBoost algorithm are that: it allows (1) to perform feature selection via stagewwisellike process where one feature enters the model each step, and (2) to recursively "refine" the logistic regression by boosting the logistic regression fitted at node's par- ent. Indeed, a first logistic regression is fitted at the tree's root via LogitBoost using all observations (x. y), which is further boosted separately at its subsequent children nodes sub-populations, say ((x), y (x²,y") and so on. This most probably induces less parameter estimation variance in each leaf since they partly benefit from samples not in their leaf but used to fit the parents' logistic regression, and is fast. The resulting tree must also be pruned and either a tactic similar to the clas- sical tree algorithm CART, cross-validation, the AIC criterion (in a refinement of the method proposed by (Sumner et al. )2005) are used.
Lastly, third approach is MOB, by Zeileis et al. (2008). Their algorithm consists in fitting the chosen model (in our case, logistic regression) for all observations at the current node and decide split these into subsets based on correlation measure (several such measures are proposed) of the residuals of the current model (cor(x), y Pg. (x))x))). The procedure is repeated until no significant "correlation" is detected. Similarly to LOTUS and contrary to C4.5, MOB performs, for binary splits and when confronted to a categorical feature having m, levels, 2mn tests. Finally, the number of segments per split searched exhaustively. Thus, it is computation intensive.
To sum up, these direct approaches produce the sought tree-structure of Figure | with different algorithms: LOTUS only considers continuous features in the leaves and relies on a x test to select the splits. LMT relies C4.5 and boosting to grow the tree and estimate the logistic regression respectively. MOB estimates logistic regression at each node and chooses splits according to a correlation its residuals. In addition to MOB and LMT, there is a vast literature on Hierarchical Mixtures of Experts from Jordan & Jacobs (1994) which have a "tree-like" structure but for which all "experts" (our logistic regressions) output predictions which are thereafter weighted by the tree structure. Despite "looking" similar,, the approaches differ because our desired segmentation means want to rely classical decision tree (in particular, univariate splits at each node). We formalise now the problem as model selection problem.
single expert for each instance, and wish that this expert is chosen by
3 LOGISTIC REGRESSION TREES: A DIFFICULT OPTIMISATION PROBLEM
In the ad-hoc method, the segments (ci)f( were determined a priori using historical or practical reasons shown in Figure As we aim optimising the segmentation, it desirable find the probability of belonging to each segment c, and to fit the model pe (y) æ c) on each segment. The total number of segments K is also to be determined. This amounts mixture model:
p(y|z) = Σ p c)lp,c))))
(I)
The approach we take considers the real segment C* as latent random feature. Each observation belongs to one segment only, thus p(c|z) is non-zero only for c*. Subsequently, denoting by x" the subset of observations for which c c* and by of the segment of an observation æ; we have:
3

---

Under review as a conference paper at ICLR 2023
y(x,)) Σ plyls<cp(<<>)p(03
plu ma; c) p(clar)) p(ax)
for citic
Il c = II
ply |x" ;x=;"")"" ;<*)p(x)
=1=1
Pgr. (y Ix" "p(0 "p(rr""c"") (c")d0"pp p(x).
c'=1 griet
K
Inp(x,
Σ
In Par.ct (y 1x" )p(0r= (c")drr + +ln((x)
c'=l 0r
K
BIC(O (/2(0~>))/2+((33)) Inp((x)
where BIC stands for the Bayesian Information Criterion (similar to AIC, see Neath & Cavanaugh (2012). Since in our application, the number of sample n ze 10 is large and the number of desired segments K x 10 is low, we use the following criterion to select segmentation:
K
(K arg min Σ BIC(O"). K.E
The difficulty in optimising Equation directly lies in the discrete nature of e given æ. This highly- combinatorial discrete problem relaxed by approximating door functions p(c|æ) with "smooth" proxy ps(c)z) x) and relying on Markov Chain Monte Carlo (MCMC) methods. As we will rely on decision trees, the "parameter" will stand for the tree's 's split features and cut points.
4 ESTIMATING LOGISTIC REGRESSION TREES
As Equation D mixture model with latent variable, it seems natural resort Expectation- Maximization (EM) algorithm, which will be exposed in the next section. We will then mitigate two downsides of this approach by relying on a Stochastic- Stochastic-Eppectation pnpectatioo-Maxiiization Maximization (SEM) algorithm in the subsequent section, and finally discuss how we obtain a logistic regression tree from all the candidates that these two algorithms provide.
4.1 A CLASSICAL EM ESTIMATION STRATEGY
We would like to maximise the following likelihood, derived from Equation [ both in terms of the segmentation and the resulting logistic regressions:
(B, xx xx.
In po- (p: c)pa())=.))
The EM algorithm from Dempster et al. (1977 is an iterative method that can be used to estimate the maximum posteriori (MAP) of (( v), since e is latent, and alternates between the expectation (E-)step, which computes the relative membership of the observations into each segment, and maximisation (M-)step, which computes the maximum likelihood estimate (MLE) of the parameters of the log- ligeeiikossoos of each segment's logistic regression and the tree structure. These new logistic regression and tree estimates are then used to determine the distribution of the latent variables in the next E-step. Considering the number of segments K fixed, the E and M-steps the EM can be derived as follows.
4

---

Under review as a conference paper at ICLR 2023
E-step- At iteration (s 1), the partial membership of observation i to segment e is:
Pactol (yr (;; PP (c/a:)
p(aiaan, ye; oc(+), B(o))
(s+1)
1,
-o Pao (1i (z) Pppc) (c' (as)
For notational convenience, we denote the matrix of partial membership of all observations to all segments ((u))ssss
MI-step- The previous E-step allows to derive the new MLE the logistic regression parameters of each segment e as:
go(s+1) = arg max E|e(3, (0
xx; (++1)) ((( 3(0) K
arg max
i.
(e+1)
In Inpo- (y.lox).
M2-step-
Similarly, a new tree structure can be derived by the new MLE of its parameter 3: g(++1) arg max [[(( (0 y; K, ((ot1))/00oo)) 3(+), K]
9
K
(++1) 3e
arg max
3
[c_(e]| x R where
where pa(c(ax,) is estimated by relative frequency in each leaf, such that Pa(c|æ) C(x) denotes the leaf in which falls. In this M2-step, could argue that 0c(s+1) could be used, since is computed in the MI-step, which could improve convergence. However, this would require recalculating the partial memberships (a+11, Hence unclear if this would be beneficial to the algorithm's 's runtime.
Additionally, tree induction methods like CART or C4.5 do not follow maximum likelihood ap- proach, that they rather try to minimise so-called impurity measure, the Gini index or the entropy, respectively. However, since is hoped that segments c* are "peaks" of the distribution (a(e))) assume the ligeliikl can be approximated by the entropy:
n
K
g(++1) arg max
(++1)
Pa pa(c)
Inpa(c|x,).
forc=
otherwise.
This last formulation allows obtain ((s) from simple application of the C4.5 algorithm, with observations properly weighted by fi However, this approach suffers from two main drawbacks: first, all observations are used in all logistic regression Pø- which might hinder runtime; second, all possible values of K must be iterated through since the EM algorithm does not allow for the disappearance of segment contrary the SEM approach developed hereafter.
4.2
AN SEM ESTIMATION STRATEGY
Using an MCMC approach, straightforward way of building logistic regression trees is to propose a tree structure, fit logistic regressions its leaves, and evaluate the goodness-of-fit using Equation 3 of the resulting logistic regression tree. This is somehow the way LMT works: tree structure proposed based on C4.5, logistic regressions are fitted using the LogitBoost algorithm, and the tree pruned back using goodness-of-fit criterion. Doing for all possible tree structures being intractable, we design a way of generating "good" candidates by relying on an SEM algorithm, which we call [MODEL). The E-step of the previous section is thus replaced by Stochastic (S-) step which has some consequences on the M-steps.
S-step The "soft" assignment of the EM algorithm of the previous section is hereby replaced by a "hard" stochastic assignment such that:
(++1) Po cn (11 pm:) (- (-120:).

---

Under review as a conference paper at ICLR 2023
M1-step- Thanks to the previous step, the segments are assigned such that the logistic gressions can be estimated using only observations affected to their segment:
(c(s+1) arg max ((0; x clon) )
0=
arg lolc In por
0e
M2-step This is again approximated by C4.5's (unweighted) impurity measure, the entropy, using only observations affected to each segment.
4.3
GOING BACK TO "HARD" SEGMENTS
4.3.1
MAP ESTIMATE
In the previous sections, we relaxed the discrete problem into "soft" assignments Pa(cyl-)- This allows observations "partly" belong to each segment, which can be interpreted mixture of logistic regressions: all observations are scored by all models which are subsequently weighted. This arguably not interpretable, nor the initial goal to retrieve a tree such as in Figure I An assignment of each sample i to a single most appropriate model, i.e. to a leaf of the segmentation tree, is achieved in parallel from the FS)EM algorithm(s) by MAP step such that:
arg max Pott) (c/xx.)
4.3.2 LEAVES AS SEGMENTS
Alternatively, we can simply consider the leaves of the estimated tree Pat (c/an:) as segments. In
other words, if we number the terminal nodes of the tree (e.g. left right), a(*) becomes the number of the Ieaf where æ; lands. There is no obvious reason why this would work better than the MAP estimation, nor a theoretical justification. However, experiments on simulated data Section suggest it performs better.
4.4 CHOOSING THE BEST SEGMENTATION CANDIDATE
The EM and SEM strategies introduced in the two previous sections for segmentation are merely "segments providers". Indeed, through the iterations 1 to S, as argued in the two preceding para- graphs, segmentations ε(1) ..... c( are proposed through MAP or leaves as segments rule parallel these algorithms. The best performing segmentation is then chosen using Equation (where the search space is restricted to the proposed segmentations).
4.5 CONVERGENCE PROPERTY: EXPLORING THE NUMBER OF SEGMENTS
In the preceding sections, the number of segments K was assumed be fixed. However, the MAP scheme introduced in this section allows us, when going from "soft" Pa Pa(c;l-) to "hard" segment assignment, to explore a number of segments potentially way lower than K: for fixed segment if there is no observation such that pa(c(a;;) > pa(ee(m)) (m:) for d' * than the segment is empty, which is equivalent to producing segmentation K segments. Supplemental this thresholding effect, the use of an SEM algorithm makes possible to enforce this phenomenon: as e is drawn in the S-step, there a non-zero probability not drawing particular segment c at given step (s). When
long enough, the chain will stop with K 1. This can be seen strength since it does not require to loop over the number of segments K which would be required for EM algorithm, which is why focus is given the SEM algorithm in what follows.
For the leaves as segments approach, the number of segments K entirely depends on the form (in particular the depth) of the tree. Little can be said about the quality of the exploration.

---

Under review as a conference paper at ICLR 2023
It
2
2
~
N))
0
>
>
c 8. 0 ~ (O0,5) 5) expit((0) (0) + 0/2²²)
8g 02 (0,5) expi Ctppitt/00 t00 + 65
(0,55 5) expit (0% + 0z
6b. 62 (0,55) 5)
expit (0g 0g 03
Figure 2: Data simulation procedure.
4.6 CATEGORICAL VARIABLES, DISCRETIZATION OF CONTINUOUS VARIABLES ON EACH SEGMENT AND VARIABLE SELECTION
As logistic regression assumes lincarity of the log-odd ratio w.r.t. continuous features and conversely might estimate coefficient relative categorical level taken by few samples with a lot of variance, practitioners often discretize continuous features and regroup categorical levels to obtain the best model. In parallel, the goal of the SEM algorithm is split the population into segments that "behave" differently. Thus, to achieve better performance, both the discretization of continuous features and the grouping of levels of categorical features must be segment dependent. The variables selected in each segment by the logistic regression (via an LI -eggullariistinn) will also be different. We therefore add a "processing" step to our SEM algorithm.
P-step- Discretization continuous features We discretize continuous features using a Mini- mum Description Length Principle method (MDLP, see Fayyad & Irani (1993), which consists in trying different cutting values (midpoints of distinct consecutive values), selecting the best based on entropy, and deciding whether worth continuing further discretize with the MDLPC criterion.
Merging levels of categorical features We merge levels of categorical features using a x² method, where we compute the X² contingency of every unique pair of categories, merge the pair with the highest contingency if the contingency is above a certain value, and repeat (while keeping a mini- mum of two categories). In the following section, we generate continuous features and show empir- ical consistence of the proposed method.
PERFORMANCE ON SIMULATED DATA
5.1 DATA SIMULATION MECHANISM
1.3 drawing features X j e (1,2, 3,,, from (0, 1), forming decision tree by choosing splits and x=2 0, which yields K = 4 segments, and drawing distinct logistic regression coefficients 0 from (0,5) on each of the leaves of the tree. Default y is then drawn from pø- (-12;))). see Figure 2
We generate the data from a logistic regression tree (see Section
5.2 IMPORTANCE OF THE HYPER-PARAMETERS & EMPIRICAL CONVERGENCE
The SEM algorithm, which pseudo-code can be found in Appendix C has multiple parameters: using MAP or leaves as segments, the number of iterations S, the initial number of segments K, and, being an MCMC method, the number of initialisations. Indeed, to avoid risking a bad performance because of an unlucky intiialisation, we randomly initialize the algorithm multiple times in parallel, run it and return the best model found among the parallel runs.
Figure 3 displays the results: its first row represents Equation R w.r.t. the number of initialisations, the number of iterations and the number of samples (which were resp. fixed at 5, 100 and 6,000 when another parameter was tested). As the BIC criterion is an information criterion, it is computed on the training set. There is no test set (as it would not make sense to penalize the likelihood test set). The second row displays the percentage of experiments where the correct tree (its depth, the features chosen to split and the splits themselves within the [-0.1, 0.1] range). In both cases, the option to use a MAP estimation or leaves to obtain segments are plotted against each other. To obtain a 1-standard deviation, i.e. 68 %, confidence interval, each experiment
200 times.
7

---

Under review as a conference paper at ICLR 2023
4,000
4,000
1.5
3,000
RCC
3,000
0.5
Leandi za agganest MAP sepmerro
TZEARS segnanni MAP ergrent
2.000
segments
Drack
Onccke
30
100
150
200
25
Nchaire
Aevessioes
Aisrppis
Comeen Ceph. Bonners & spit (8)
0.9
0.6
0.5
0.6
Cornetdeph
07-
0.1
depth.
0.6
aegteens
0.2
Leooss G0pRIZzEs MAP segments Orodle
segments
Undle
05-
50
1001
150
900
Achaira
Homplex
404
Figure 3: Top row: achieved sum of BIC (Equation B) Bottom row: proportion of correct trees and models retrieved.
Table 1: Comparison of the proposed approach [MODEL]-SER] and other classical algorithms
three open-source datasets and our closed-source dataset.
AUC
Logistic regression
Decision
[MODEL]
Gradient
Random
Tree
SEM
Boosting
Forest
Statlog (german)
62.1 (3.6)
56.6 (4.4)
68.0 (2.4)
63.3 (2.8)
62.7 (3.1)
Adult
84.1 (0.7)
81.7 (0.9)
85.3 (0.3)
86.9 (0.2)
84.8 (0.3)
Fraud
96.9 (0.6)
93.6 (0.4)
95.9 (0.9)
74.3 (1.8)
97.3 (0.2)
In-house (+: vs current method)
-3.02
-2.66
-1.78
-0.17
+0.36
When increasing the size of the data set, we asymptotically get the model which we used sim- ulate the data more frequently and with greater confidence. Thus, this empirical convergence and consistency to the data generating mechanism allow us to be confident that the approach correct on well-separated data. We will now apply this algorithm to real data.
PERFORMANCE ON REAL DATA
6.1 BENCHMARK OPEN-SOURCE DATASETS
As our in-house data, used in the following section, cannot be openly shared, we resort to some experiments on open-source datasets. The statlog (german) and adult datasets from UCI (Dua & Graff (2017) are used they both have mixed-type data, few features (20 and 14 resp.) and many observations (w.r.t. the number of features 1,000 and 48,842 resp.), well as the Credit Card Fraud Detection from Kaggle (Le Borgne et al. (2022) 29 features and 284,807 observations).
6.2 COMPARISON TO THE CURRENT METHOD AND CLASSIC ALGORITHMS
In-house data We now use representative sample of n 100,000 [COMPANY] clients (pro- portionally taken from the population, i.e. from every existing segment), for which we know the repayment performance, initially preprocessed (see Section 4.6) for the whole population and not for each segment, as the computational cost increases dramatically.
training sample. We use 14,600,000 test sample. The data
8

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of the proposed approach [MODEL]-SEM and other logistic regression trees algorithms.
SEM
LMT
MOB
segment (current: 9) AUC (#: vs current method)
11
-1.52
-7.70
-5.21
Benchmark models As benchmark models, we rely on one hand on "weak" but explainable learn- ers which form the basis of our approach: decision trees and logistic regression. On the other hand, we choose also gradient boosting and random forests which traditionally perform quite well on tab- ular data, in particular on credit risk data, but are not explainable enough to be used in production on credit risk use cases of now.
Experimental setup Five to twenty (depending on the dataset's size) 70/30 training/test splits are drawn so as to give an idea the variance of the approach. AUC (and its standard deviation in compared in Table I in absolute terms of
parentheses) are given in Table T All the algorithms Area Under the ROC Curve (AUC), a common metric in credit risk, for open-source datasets, and in relative terms (to the current method), for the in-house dataset. The segment-dependent processing step described in Section 4.6 is not performed.
Hyperparameter tuning A grid-search performed for our model as well as gradient boosting and random forest. For our model, for computational reasons, we split the training dataset further into a training and validation set. The best set of hyperparameters is then chosen w.r.t. the highest AUC on the validation set. For standalone logistic regression, gradient boosting and random forest, cross-validation grid-search is performed. Pseudo-code as well as the hyperparameter grids are available in Appendix
Results First, as expected, [MODEL]-SEM performs better than both the logistic regression or the decision tree alone. Other methods such as Gradient Boosting or Random Forest have similar or better results than the current method, but when comparing the results on each existing segment we that they don't 't perform well on all segments universally. Because of that, even with a slightly better overall performance, we don't I consider them a more satisfying method.
see that its performance is satisfactory. However, the current method has a "wider" hypothesis space, since the data preprocessing (dis- cretization, grouping categories for categorical variables) is done separately for every segment and involved a lot of manual fine-tuning. Additionally, our "small" sample of 100k observations (w.rt. the test sample of 14.6M observations) cannot capture "very small", manually-crafted segments (several segments with approx. 50-100k observations in the full sample, e.g. voluntary associa- tions/unions). The current method thus achieves higher performance.
Comparing [MODEL]-SEM to the current method,
Our SEM algorithm nevertheless creates fewer segments than the current method (the result depends on the initialisation, but we usually find 2 to 4 segments, compared to the segments of the current method). Thus, the proposed model is less complex, requires much less of practitioners' time, but yields lower performance.
Comparison to existing logistic regression trees algorithms We compare the SEM algorithm to the other logistic regression tree approaches discussed in Section 2 except for LOTUS which does not have any available implementation. We now use the version of SEM which incorporates the data preprocessing in each segment (see Section 4.6). For 10 segments, 100 iterations and 5 different initialisations, this amounts to discretizing and merging levels 5000 times. Adding this segment-dependent data preprocessing does slightly increase performance the cost if increased computation time. The small number of segments is limiting the performance, but nevertheless superior to existing approaches (see Table 2).

---

Under review as a conference paper at ICLR 2023
7 CONCLUSION
This paper aims at formalising an old problem in Credit Scoring, namely client segmentation, by providing literature review as well as new algorithmic approach. As is often the case, practition- ers have had good intuitions to deal with practical and theoretical requirements, such as performing clustering techniques, choosing segments empirically from the resulting visualisation and fitting lo- gistic regression on these. However, situations can easily be imagined where such practices can fail, which is why other methods, which take into account the predictive task, shall be preferred. To this end, method is proposed, based
SEM algorithm.
On simulated data, shows good results which demonstrate empirical consistency of the approach. On open-source real data, it shows superior performance than "plain" logistic regression and deci- sion tree. On real data from [COMPANY], the automatic [MODEL]-SEM almost competes with the current performance which requires lot of human time and expert knowledge, and performs better than the other existing logistic regression trees approaches. The [MODEL] method is available as a package from [MASK], well as scripts reproduce results from Sections? 5 6.
REFERENCES
Leo Breiman, J. H. Friedman, R. A. Olshen, and C. Stone. Classification and Regression Trees. StatisticsProoabbiiity Series. Wadsworth Publishing Company, Belmont, California, U.S.A., 1984.
Kin-Yee Chan and Wei- Yin Loh. Lotus: An algorithm for building accurate and comprehensible logistic regression trees. Journal of Computational and Graphical Statistics, 13(4):826-852, 2004. doi: 10. NEN110611604XXX
A. Dempster, N. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-22, 1977. doi: 10.1111/j.217-6666 1w...1111.......... 997..b616600.x
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. http:///aarch ics. uci. lecimedu/ml
Usama M. Fayyad and Keki B. Irani. Multi-interval discretization of continuous-valued attributes for classification learning. In IJCAI, 1993.
Jerome Friedman, Trevor Hastie, Robert Tibshirani, al. Additive logistic regression: statistical view of boosting (with discussion and a rejoinder by the authors). The Annals of Statistics, 28(2): 337-407, 2000.
doi: 10. 10.221/aas/0166112233
Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214 1994.
Niels Landwehr, Mark Hall, and Eibe Frank. Logistic model trees. Machine learning, 59(1-2): 161-205, 2005. doi: 10.
Yann-Aël Le Borgne, Wissam Siblini, Bertrand Lebichot, and Gianluca Bontempi. Reproducible Machine Learning for Credit Card Fraud Detection Practical Handbook. Université Li- bre de Bruxelles, 2022. URL htt.tttt////tttbbbbbbbbb Fraud-detection iotiteecci.on-aaaaboook
o Handbooookk/
Ludovic Lebart, Alain Morineau, and Marie Piron. Statistique exploratoire multidimmnnsionnell.. volume 3. Dunod Paris, 1995.
Andrew A Neath and Joseph E Cavanaugh. The bayesian information criterion: background, deriva- tion, and applications. Wiley Interdisciplinary Reviews: Computational Statistics, 1(2):199-203, 2012.
Jérôme Pagès. Multiple factor analysis by example using R. Chapman and Hall/CRC, 2014.
J Ross Quinlan. C4, 5: programs for machine learning. Elsevier, 2014.
10

---

Under review as a conference paper at ICLR 2023
Indebtedness
Wages
(a) Multi-modal wages and indebtedness data generating mechanism with y {0, 1} classes displayed in red and green respectively.
Indebtednesss
Indebtedness
Indebtedness
Wages
Wages
Wages
(I) Renters
(2) Salaried workers
(3) Self-employed
(b) Uni-mo modal wages and indebtedness data generating mechanism with y {0, 1) classes displayed in green and red respectively w.r.t a "source of wages" categorical feature.
Figure 4: Ad hoc practices failing situations.
Mare Sumner, Eibe Frank, and Mark Hall. Speeding up logistic model tree induction. In Euro- pean Conference on Principles of Data Mining and Knowledge Discovery, pp. 675-683. Springer, 2005. doi: 10.1007/11560 1564126 72.
Achim Zeileis, Torsten Hothorn, and Kurt Hornik. Model-based recursive partitioning. Jour- nal of Computational and Graphical Statistics, 17(2):492-514, 2008. doi: 10.1198/ 106186008X319331.
A AD-HOC PRACTICES CAN FAIL
appears clearly that this approach does not directly optimise a predictive goal. The segmentation and the logistic regression are optimised independently, according to different error losses. When those goals don't align, theses practises can fail.
The first of these failing situations is when the probability density function of covariates (suppose for simplicity that all of them are continuous) p(x) is multi-modal as on Figure 4a where we distinguish the lower, middle and upper-classes of respective low, average and high wages and indebtedness. An unsupervised generative approach like PCA would urge the practitioner to construct 3 models (one for each of the aforementioned classes). However, displaying y (default) as red and y 0 as green, we can see that perfect separation can be achieved: it depends solely on the indebtedness level (the ratio of wages over indebtedness). Thus, the resulting models would be asymptotically the same. Since each of the 3 models would have three times less samples to learn from, it would amount to increasing the estimation variance of the coefficients, and ultimately result in lower performance. On practical note, one could argue that it reduces interpretability by adding an avoidable complexity to the decision system.
The second failing situation is the counterpart of the first tailored data generating mechanism and is displayed in Figure 4bb This time, suppose the covariates are uniformly sampled. Suppose there is a third categorical feature "wages source" which is drawn from three levels: renters, salaried workers and employed. Suppose that renters' risk level do not depend on their indebtedness, which is typically low, salaried workers' risk level is positively correlated with their indebtedness ratio as was the case for the first introductory example and self-employed people's 's risk level is negatively correlated with this indebtedness ratio. In this situation, unsupervised generative "clustering"
11

---

Under review as a conference paper at ICLR 2023
algorithm like the projection of the data the two first PCA axes would not partition the data and we would construct only one scorecard. This scorecard would have high model bias since is too simple to accommodate for the variety of the data generating mechanism.
B HYPERPARAMETER SEARCH
Standalone Logistic Regression An Elastic-net penalty is applied. The best set of hyperparame- ters, consisting in the regularization strength and the balance between L1 and L2 regularizations found via 5-fold crass-vvli with scikit-learn.
[MODEL]-SEM Logistic Regression for e An L2-penalty is applied with (small) regularization parameter 0.01. The rationale behind this choice is to avoid overfitting.
[MODEL]-SEM Logistic Regression for è An -ipennalty is applied with (small) regularization parameter of 0.01. The rationale behind this choice that some features might be relevant for some segments.
Gradient Boosting Grid-search is performed with scikit-learn via 5-fold Cross-Validation Validation with the following typer-parameters:
learning rate: [0.01, 0.1, 0.5, 2.0],
n.estimators: [100, 300, 1000].
subsample: [0.5, 0.75, 1.0],
min min.samples.split: [10, 30),
min min.samples.Jeat [1, 5, 20),
max.depth: [2, 5, 10],
max.features: [' log2', 'sqrt', None]
Random Forest
Grid-search is performed with scikit-learn via 5-fold Vooss-Vaidatiion with the following hyper-parameters:
n_estimators: [10, 100, 1000],
mi i_samples.sppli it: [10, 301,
min mdi.samples.lett [1, 5, 20].
max_depth: [2, 5, 10).
max_features: [' log2', 'qgr'', None]
ccp alpha: [0.0, 0.1, 1.0]
12

---

Under review as a conference paper at ICLR 2023
C [MODEL]-SEM: PSEUDO-CODE
Data: Features, targets, number of segments, number of iterations: x, y, K, max_iter Final number of segments, assignments of each instance to segment, logistic
Result:
regression parameters of each segment, segmentation tree: K c', {0 h
Initialize "soft" (c(")) and "hard" (e(1) segments at random: c; Q ~ [_...., K]; Track best iteration far with bter, best_BIC 0, oc;
while S max do
for 1 c K do
(x((s), , yd(s) is the dataset of instances for which e; c;
(M1) Estimate (r(*) by fitting logistic regression to (xx(*), ( .y(()));
Estimate 0c(*) by fitting logistic regression to (x((s), yr(+));
end
(M2) Estimate ((0) by fitting decision tree to x, cl*);
Draw new c from a row-wise normalization of po
(note: some segments might not be drawn at all, thus K +- distinct values c(++1));
Calculate "hard" segments e(++1) as the row-wise MAP of po (ylx)ppt)) (-Ix);
If sum of BICs (Equation 3 S best BIC, update best iter, best BIC to current values;
end
Output K*, c*, (0") h which correspond to the lowest of BICs (as given by
Algorithm 1: [MODEL]-SEM pseudo-code.
13


Reviews:
[{"review_id": "LGJetttbG-L", "reviewer": "Reviewer_tW1b", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper deals with the credit scoring problem. The paper formulates the problem as a logistic regression tree problem, where the data points are split into segments and each segment has a logistic regression model to model the probability of default. The paper then proposes an algorithm that can jointly learn segment assignments and the logistic regression model of each segment in an expectation maximization fashion. Experiments are conducted on both synthetic data and real-world data to evaluate the performance of the proposed method.", "summary_of_the_review": "Overall, while I think the paper deals with an important and interesting real-world problem, I believe the presentation of the paper can be improved and experiments can be more exhaustive to reach a publishable stage. Therefore, I recommend rejection.", "strength_and_weaknesses": "Pros:\n1. the paper is concerned with an important problem in the financial industry.\n2. the proposed method is interesting, where the joint learning of both the segment and the model of each segment seems to be a more principled approach compared to the more prevalent naive two-stage practice.\n3. The paper provides a review of several existing methods for this problem.\n\nCons:\n1. Many mathematical derivations seem to be omitted. I cannot find a supplemental file with a more detailed derivation. For example, in p(x,y) on page 3, why the second equality is true? In the first equation of section 4.1, it is not clear to me how this equation can be derived from equation 4.1. The derivation of the EM algorithm is also not clear, despite the EM algorithm being the key part of the paper. I would suggest the authors to provide as many derivation details as possible to better justify the correctness of their approach. As it stands, it is difficult to evaluate with high certainty that the mathematics of the paper is correct.\n\n2.  the notation of the paper can be improved. For example, I cannot find what beta stands for. The definition of BIC is also not specified. \n\n3. the experimental results are some what limited and unconvincing. For example, experiments are only run on four real world datasets. It is not clear whether the proposed method has better performance or not compared to alternatives. Even though the authors argue that the proposed method provides more consistency across segments. It is not clear to me whether this is true across all datasets or is true only about the in-house dataset.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "JH2szz_V1A8", "reviewer": "Reviewer_heWb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission presents an algorithm for learning decision trees with logistic regression models at the leaf nodes. It initially considers soft splits and estimation of the model using expectation maximization before proceeding to more interpretable hard splits and a stochastic expectation maximization algorithm for learning the trees. An important parameter of the algorithm is the maximum number of leaf nodes in the resulting tree. AUROC is estimated for three UCI datasets and one private credit-rating dataset. On the private dataset, the proposed method outperforms LMT and MOB, which both also grow decision trees with logistic regression models.", "summary_of_the_review": "The proposed method may have merit, but the submission does not present sufficient empirical evidence to be convincing. There is no discussion of related work on mixtures of experts. Limitations of the proposed method, particularly considering runtime, are not discussed.", "strength_and_weaknesses": "\nThe empirical comparison is unsatisfactory. LMT and MOB, the most direct competitors, are only compared to on the private data, not the public datasets.\n\nThe number of datasets included in the experiments is unnecessarily small. \n\nIn the evaluation of the proposed method, discretization and a method for merging categorical values are applied. It is not stated whether these methods are applied in conjunction with the other learning algorithms included in the experiments. It seems important, in order to make the comparison as fair as possible, to evaluate whether these pre-processing methods also help the other learning algorithms (or are detrimental to their performance if they are applied).\n\nThe proposed method appears to be computationally quite expensive, but this is not quantified in the paper. In particular, it seems the maximum number of leaves will have to be quite limited to apply the proposed algorithm in practical applications.\n\nIt seems important to relate the proposed method to the classic work on mixtures of experts.\n\nHyperparameter settings need to be clearly stated. For example, the poor results for gradient boosting are a strong indication that it was inappropriately configured. Also, it appears that L_1 regularization may have been used for logistic regression in the proposed method (see Section 4.6). How was the strength of regularization determined? And was similar regularization applied in stand-alone logistic regression?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Hi_NNRZFcd4", "reviewer": "Reviewer_hAgH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a tree structured model for regression for applications in finance. The authors describe an inference procedure for fitting the structure and parameters. The authors provide empirical analysis of proposed method and its variants on their own corporate data and academic datasets.", "summary_of_the_review": "While the proposed approach is an interesting and elegant model, there are concerns about where it fits into a broader landscape of related work. Further, better understanding when and why the proposed approach is effective would greatly strengthen the paper. ", "strength_and_weaknesses": "This paper presents an interesting approach for regression with the following strengths and weaknesses:\n\n**Strengths**\n* The paper presents an interesting methodological approach, it is a clear an elegant model, which has a rich and challenging inference task. \n* The proposed approach appears to be quite effective, especially on the corporate dataset.\n\n**Weaknesses**\n* I feel that there is a wide landscape of related approaches related to mixture of experts, especially tree-structured mixture of experts [1,2, inter alia] that is not explored. There is also much work on tree structured methods that do not split on specific features e.g. [3,4,5 inter alia] I think that it would be very helpful to consider a wider landscape of related method.\n* I wonder if the authors could clarify the selection of baseline algorithms? Would it make sense to try a simple few layer MLP? Would it make sense to use xgboost? Kernel ridge regression?\n* Empirical analysis - The results are most impressive on the corporate dataset, it would be helpful to understand why the proposed approach does not work as well and what characteristics of data would lead someone to select the proposed approach compared to the other methods.\n* The proposed approach is a nice model, but it leaves many open questions. For instance, What are its theoretical properties? When should someone prefer it to other models? How does the quality of the structure effect performance? When is the structure learnable? I think without a deeper understanding of the proposed approach. It is not clear to me whether we can justify the modeling approach on its own without such understanding or more complete empirical analysis.\n\n[1] Bishop, Christopher M., and Markus Svensén. \"Bayesian hierarchical mixtures of experts.\" UAI, 2003.\n\n[2] Zhao, Wenbo, Yang Gao, Shahan Ali Memon, Bhiksha Raj, and Rita Singh. \"Hierarchical routing mixture of experts.\" In 2020 25th International Conference on Pattern Recognition (ICPR), 2021.\n\n[3] Choromanska, Anna E., and John Langford. \"Logarithmic time online multiclass prediction.\" Advances in neural information processing systems 28 (2015).\n\n[4] Daumé III, Hal, Nikos Karampatziakis, John Langford, and Paul Mineiro. \"Logarithmic time one-against-some.\" In International Conference on Machine Learning, pp. 923-932. PMLR, 2017.\n\n[5] Yu, Hsiang-Fu, Kai Zhong, Jiong Zhang, Wei-Cheng Chang, and Inderjit S. Dhillon. \"PECOS: Prediction for enormous and correlated output spaces.\" Journal of Machine Learning Research 23, no. 98 (2022): 1-32.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject