Package: glmdisc
Feature quantization for parsimonious and interpretable modelsTable of ContentsDocumentationInstallation instructionsTheorySome examplesOpen an issueReferencesContributeMotivationCredit institutions are interested in the refunding probability of a loan given the applicant’s characteristics in order to assess the worthiness of the credit. For regulatory and interpretability reasons, the logistic regression is still widely used to learn this probability from the data. Although logistic regression handles naturally both quantitative and qualitative data, three pre-processing steps are usually performed: firstly, continuous features are discretized by assigning factor levels to pre-determined intervals; secondly, qualitative features, if they take numerous values, are grouped; thirdly, interactions (products between two different predictors) are sparsely introduced. By reinterpreting discretized (resp. grouped) features as latent variables, we are able, through the use of a Stochastic Expectation-Maximization (SEM) algorithm and a Gibbs sampler to find the best discretization (resp. grouping) scheme w.r.t. the logistic regression loss. For detecting interacting features, the same scheme is used by replacing the Gibbs sampler by a Metropolis-Hastings algorithm. The good performances of this approach are illustrated on simulated and real data from Credit Agricole Consumer Finance.This repository is the implementation of Ehrhardt Adrien, et al.Feature quantization for parsimonious and interpretable predictive models, preprint arXiv:1903.08920 (2019).Getting startedThese instructions will get you a copy of the project up and running on your local machine for development and testing purposes.PrerequisitesThis code is supported on Python 3.7, 3.8, 3.9 and 3.10 (seetox file).Installing the packageInstalling the development versionIfgitis installed on your machine, you can use:pipinstallgit+https://github.com/adimajo/glmdisc_python.gitIfgitis not installed, you can also use:pipinstall--upgradehttps://github.com/adimajo/glmdisc_python/archive/master.tar.gzInstalling through thepipcommandYou can install a stable version fromPyPiby using:pipinstallglmdiscInstallation guide for AnacondaThe installation with thepipcommandshouldwork. If not, please raise an issue.For people behind proxy(ies)...A lot of people, including myself, work behind a proxy at work...A simple solution to get the package is to use the--proxyoption ofpip:pip--proxy=http://username:password@server:portinstallglmdiscwhereusername,password,serverandportshould be replaced by your own values.If environment variableshttp_proxyand / orhttps_proxyand / or (unfortunately depending on applications...)HTTP_PROXYandHTTPS_PROXYare set, the proxy settings should be picked up bypip.Over the years, I've foundCNTLMto be a great tool in this regard.What follows is a quick introduction to the problem of discretization and how this package answers the question.Use case exampleFor a thorough explanation of the approach, seethis blog postorthis article.If you're interested in directly using the package, you can skip this part and go tothis part below.In practice, the statistical modeler has historical data about each customer's characteristics. For obvious reasons, only data available at the time of inquiry must be used to build a future application scorecard. Those data often take the form of a well-structured table with one line per client alongside their performance (did they pay back their loan or not?) as can be seen in the following table:JobHabitationTime in jobChildrenFamily statusDefaultCraftsmanOwner100DivorcedNoTechnicianRenterMissing1WidowerNoMissingStarter52SingleYesOffice employeeBy family23MarriedNoNotationsIn the rest of the vignette, the random vectorwill designate the predictive features, i.e. the characteristics of a client. The random variablewill designate the label, i.e. if the client has defaulted () or not ().We are provided with an i.i.d. sampleconsisting inobservations ofand.Logistic regressionThe logistic regression model assumes the following relation betweenand:whereare estimated using(anddenotes the coefficients associated with a categorical featurebeing equal to).Clearly, for continuous features, the model assumes linearity of the logit transform of the responsewith respect to.
On the contrary, for categorical features, it might overfit if there are lots of levels (). It does not handle missing values.Common problems with logistic regression on "raw" dataFitting a logistic regression model on "raw" data presents several problems, among which some are tackled here.Feature selectionFirst, among all collected information on individuals, some are irrelevant for predicting. Their coefficientshould be 0  which might (eventually) be the case asymptotically (i.e.).Second, some collected information are highly correlated and affect each other's coefficient estimation.As a consequence, data scientists often perform feature selection before training a machine learning algorithm such as logistic regression.There already exists methods and packages to perform feature selection, see for example thefeature_selectionsubmodule in thesklearnpackage.glmdiscis not a feature selection tool but acts as such as a side-effect: when a continuous feature is discretized into only one interval, or when a categorical feature is regrouped into only one value, then this feature gets out of the model.For a thorough reference on feature selection, see e.g. Guyon, I., & Elisseeff, A. (2003).An introduction to variable and feature selection.Journal of machine learning research, 3(Mar), 1157-1182.LinearityWhen provided with continuous features, the logistic regression model assumes linearity of the logit transform of the responsewith respect to. This might not be the case at all.For example, we can simulate a logistic model with an arbitrary power ofand then try to fit a linear logistic model:Show the Python codeGet this graph onlineOf course, providing thesklearn.linear_model.LogisticRegressionfunction with a dataset containingwould solve the problem. This can't be done in practice for two reasons: first, it is too time-consuming to examine all features and candidate polynomials; second, we lose the interpretability of the logistic decision function which was of primary interest.Consequently, we wish to discretize the input variableinto a categorical feature which will "minimize" the error with respect to the "true" underlying relation:Show the Python codeGet this graph onlineToo many values per categorical featureWhen provided with categorical features, the logistic regression model fits a coefficient for all its values (except one which is taken as a reference). A common problem arises when there are too many values as each value will be taken by a small number of observationswhich makes the estimation of a logistic regression coefficient unstable:Show the Python codeGet this graph onlineIf we divide the training set in 10 and estimate the variance of each coefficient, we get:Show the Python codeGet this graph onlineAll intervals crossing 0 are non-significant! We should group factor values to get a stable estimation and (hopefully) significant coefficient values.Discretization and grouping: theoretical backgroundNotationsLetbe the latent discretized transform of, i.e. taking values inwhere the number of values of each covariateis also latent.The fitted logistic regression model is now:Clearly, the number of parameters has grown which allows for flexible approximation of the true underlying model.Best discretization?Our goal is to obtain the modelwith best predictive power. Asandare both optimized, a formal goodness-of-fit criterion could be:where AIC stands for Akaike Information Criterion.CombinatoricsThe problem seems well-posed: if we were able to generate all discretization schemes transformingto, learnfor each of them and compare their AIC values, the problem would be solved.Unfortunately, there are way too many candidates to follow this procedure. Suppose we want to construct k intervals ofgiven n distinct. There ismodels. The true value of k is unknown, so it must be looped over. Finally, as logistic regression is a multivariate model, the discretization ofcan influence the discretization of,.As a consequence, existing approaches to discretization (in particular discretization of continuous attributes) rely on strong assumptions to simplify the search of good candidates as can be seen in the review of Ramírez‐Gallego, S. et al. (2016) - seeReferences section.Discretization and grouping: estimationLikelihood estimationcan be introduced in:First, we assume that all information aboutinis already contained inso that:Second, we assume the conditional independence ofgiven, i.e. knowing, the discretizationis independent of the other featuresandfor all:The first equation becomes:As said earlier, we consider only logistic regression models on discretized data. Additionnally, it seems like we have to make further assumptions on the nature of the relationship ofto. We chose to use polytomous logistic regressions for continuousand contengency tables for qualitative. This is an arbitrary choice and future versions will include the possibility of plugging your own model.The first equation becomes:The SEM algorithmIt is still hard to optimize overas the number of candidate discretizations is gigantic as said earlier.However, calculatingis easy:As a consequence, we will draw random candidatesapproximately at the mode of the distributionusing an SEM algorithm (see seeReferences section).Gibbs samplingTo update, at each random draw, the parametersandand propose a new discretization, we use the following equation:Note that we drawknowing all other variables, especiallyso that we introduced a Gibbs sampler (see References section).TheglmdiscpackageTheglmdiscclassThe documentation is available as aGithub Page.Theglmdiscclass implements the algorithm described in the previous section. Its parameters are described first, then its internals are briefly discussed. We finally focus on its ouptuts.ParametersThe number of iterations in the SEM algorithm is controlled through theiterparameter. It can be useful to first run theglmdiscfunction with a low (10-50)iterparameter so you can have a better idea of how much time your code will run.Thevalidationandtestboolean parameters control if the provided dataset should be divided into training, validation and/or test sets. The validation set aims at evaluating the quality of the model fit at each iteration while the test set provides the quality measure of the final chosen model.Thecriterionparameters lets the user choose between standard model selection statistics likeaicandbicand theginiindex performance measure (proportional to the more traditional AUC measure). Note that ifvalidation=TRUE, there is no need to penalize the log-likelihood andaicandbicbecome equivalent. On the contrary ifcriterion="gini"andvalidation=FALSEthen the algorithm may overfit the training data.Them_startparameter controls the maximum number of categories offorcontinuous. The SEM algorithm will start with randomtaking values in. For qualitative features,is initialized with as many values asso thatm_starthas no effect.Empirical studies show that with a reasonably small training dataset (< 10,000 rows) and a smallm_startparameter (< 20), approximately 500 to 1500 iterations are largely sufficient to obtain a satisfactory model.>>>importglmdisc>>>logreg_disc=glmdisc.Glmdisc(iter=100,validation=True,test=True,criterion="bic",m_start=10)2020-07-1618:11:03.087|WARNING|glmdisc:__init__:216-Noneedtopenalizethelog-likelihoodwhenavalidationsetisused.Usinglog-likelihoodinstead.ThefitfunctionThefitfunction of theglmdiscclass is used to run the algorithm over the data provided to it. Subsequently, its parameters are:predictors_contandpredictors_qualwhich represent respectively the continuous features to be discretized and the categorical features which values are to be regrouped. They must be of type numpy array, filled with numeric and strings respectively. The last parameter is the classlabels, of type numpy array as well, in binary form (0/1).>>>n=100>>>d=2>>>x,y,_=glmdisc.Glmdisc.generate_data(n,d)>>>logreg_disc.fit(predictors_cont=x,predictors_qual=None,labels=y)Thebest_formulafunctionThebest_formulafunction prints out in the console: the cut-points found for continuous features, the regroupments made for categorical features' values. It also returns it in a list.>>>logreg_disc.best_formula()2020-07-1618:13:29.921|INFO|glmdisc._bestFormula:best_formula:29-Cut-pointsfoundforcontinuousvariable0[0.9568289154869697,0.6661178585993954,0.49039089060451335,0.33038638461067193,0.7152644679549544]2020-07-1618:13:29.922|INFO|glmdisc._bestFormula:best_formula:29-Cut-pointsfoundforcontinuousvariable1[0.48684331022166916,0.17904111281801316,0.6603144758481163,0.03838803248009037]Thediscrete_datafunctionThediscrete_datafunction returns the discretized / regrouped version of thepredictors_contandpredictors_qualarguments using the best discretization scheme found so far.>>>logreg_disc.discrete_data()2020-07-1618:14:57.261|INFO|glmdisc._discreteData:discrete_data:44-Returningdiscretizedtestset.<20x11sparsematrixoftype'<class 'numpy.float64'>'with40storedelementsinCompressedSparseRowformat>>>>logreg_disc.discrete_data().toarray()2020-07-1618:15:31.041|INFO|glmdisc._discreteData:discrete_data:44-Returningdiscretizedtestset.array([[1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],[0.,0.,0.,0.,1.,0.,1.,0.,0.,0.,0.],[0.,1.,0.,0.,0.,0.,1.,0.,0.,0.,0.],[0.,0.,0.,1.,0.,0.,0.,1.,0.,0.,0.],[...]ThediscretizefunctionThediscretizefunction discretizes a new input dataset in thepredictors_cont,predictors_qualformat using the best discretization scheme found so far. The result is a numpy array of the size of the original data.>>>n_new=100>>>x_new,_,_=glmdisc.Glmdisc.generate_data(n_new,d)>>>logreg_disc.discretize(predictors_cont=x_new,predictors_qual=None)array([[4., 1.],[5.,2.],[4.,3.],[4.,4.],[3.,4.],[0.,2.],[...]Thediscretize_dummyfunctionThediscretize_dummyfunction discretizes a new input dataset in thepredictors_cont,predictors_qualformat using the best discretization scheme found so far. The result is a dummy (0/1) numpy array  corresponding to the One-Hot Encoding of the result provided by thediscretizefunction.>>>logreg_disc.discretize_dummy(predictors_cont=x_new,predictors_qual=None)<100x11sparsematrixoftype'<class 'numpy.float64'>'with200storedelementsinCompressedSparseRowformat>>>>logreg_disc.discretize_dummy(predictors_cont=x_new,predictors_qual=None).toarray()array([[0., 0., 0., ..., 0., 0., 0.],[0.,0.,0.,...,1.,0.,0.],[0.,0.,0.,...,0.,1.,0.],...,[1.,0.,0.,...,1.,0.,0.],[1.,0.,0.,...,0.,0.,0.],[1.,0.,0.,...,0.,0.,0.]])ThepredictfunctionThepredictfunction discretizes a new input dataset in thepredictors_cont,predictors_qualformat using the best discretization scheme found so far through thediscretizeDummyfunction and then applies the corresponding best Logistic Regression modelfound so far.>>>logreg_disc.predict(predictors_cont=x_new,predictors_qual=None)array([[9.99394254e-01,6.05745839e-04],[9.99694576e-01,3.05424466e-04],[9.99817560e-01,1.82439609e-04],[9.99967791e-01,3.22085041e-05],[9.92296119e-01,7.70388116e-03],[...]The attributesAll parameters are stored as attributes:test,validation,criterion,iter,m_startas well as:criterion_iter: list of values of the criterion chosen;>>>logreg_disc.criterion_iter[-30.174443117243992,-26.182075441528603,-31.61227858514535,-19.70369464830396,-31.61997286396158,-25.99964499964587,...]best_link: link function of the best quantization;>>>logreg_disc.best_link[LogisticRegression(C=1e+40,max_iter=25,multi_class='multinomial',solver='newton-cg',tol=0.001),LogisticRegression(C=1e+40,max_iter=25,multi_class='multinomial',solver='newton-cg',tol=0.001)]best_reglog: logistic regression function of the best quantization;>>>logreg_disc.best_reglogLogisticRegression(C=1e+40,max_iter=25,solver='liblinear',tol=0.001)affectations: list of label encoders for categorical features;>>>logreg_disc.affectations[None, None]best_encoder_emap: one hot encoder of the best quantization;>>>logreg_disc.best_encoder_emapOneHotEncoder(handle_unknown='ignore')performance: value of the chosen criterion for the best quantization;>>>logreg_disc.performance-14.924603930263428train: array of row indices for training samples;>>>logreg_disc.trainarray([97,39,94,5,16,77,88,54,80,99,46,43,52,37,28,0,18,...validate: array of row indices for validation samples;>>>logreg_disc.validatearray([36,45,29,62,8,82,76,96,41,83,17,49,57,31,60,64,65,...test_rows: array of row indices for test samples;>>>logreg_disc.test_rowsarray([3,75,51,27,21,48,4,44,72,68,34,22,23,50,47,6,42,...To see the package in action, please refer tothe accompanying Jupyter Notebook.Do a notebookAuthorsAdrien EhrhardtVincent VandewallePhilippe HeinrichChristophe BiernackiLicenseThis project is licensed under the MIT License - see theLICENSEfile for details.AcknowledgmentsThis research has been financed byCrédit Agricole Consumer Financethrough a CIFRE PhD.This research was supported byInria Lille - Nord-EuropeandLille Universityas part of a PhD.ReferencesEhrhardt, A. (2019),Formalization and study of statistical problems in Credit Scoring: Reject inference, discretization and pairwise interactions, logistic regression trees(PhD thesis).Ehrhardt, A., et al.Feature quantization for parsimonious and interpretable predictive models. arXiv preprint arXiv:1903.08920 (2019)].Celeux, G., Chauveau, D., Diebolt, J. (1995),On Stochastic Versions of the EM Algorithm. [Research Report] RR-2514, INRIA. 1995.Agresti, A. (2002)Categorical Data. Second edition. Wiley.Ramírez‐Gallego, S., García, S., Mouriño‐Talín, H., Martínez‐Rego, D., Bolón‐Canedo, V., Alonso‐Betanzos, A. and Herrera, F. (2016).Data discretization: taxonomy and big data challenge.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 6(1), 5-21.Future development: integration of interaction discoveryVery often, predictive features $X$ "interact" with each other with respect to the response feature. This is classical in the context of Credit Scoring or biostatistics (only the simultaneous presence of several features - genes, SNP, etc. is predictive of a disease).With the growing number of potential predictors and the time required to manually analyze if an interaction should be added or not, there is a strong need for automatic procedures that screen potential interaction variables. This will be the subject of future work.Future development: possibility of changing model assumptionsIn the third section, we described two fundamental modelling hypotheses that were made:The real probability density function $p(Y|X)$ can be approximated by a logistic regression $p_\theta(Y|E)$ on the discretized data $E$.The nature of the relationship of $\mathfrak{q}_j$ to $X_j$ is:A polytomous logistic regression if $X_j$ is continuous;A contengency table if $X_j$ is qualitative.These hypotheses are "building blocks" that could be changed at the modeller's will: discretization could optimize other models.To delete when done withResultsFirst we simulate a "true" underlying discrete model:x = matrix(runif(300), nrow = 100, ncol = 3)
cuts = seq(0,1,length.out= 4)
xd = apply(x,2, function(col) as.numeric(cut(col,cuts)))
theta = t(matrix(c(0,0,0,2,2,2,-2,-2,-2),ncol=3,nrow=3))
log_odd = rowSums(t(sapply(seq_along(xd[,1]), function(row_id) sapply(seq_along(xd[row_id,]),
function(element) theta[xd[row_id,element],element]))))
y = rbinom(100,1,1/(1+exp(-log_odd)))Theglmdiscfunction will try to "recover" the hidden true discretizationxdwhen provided only withxandy:library(glmdisc)
discretization <- glmdisc(x,y,iter=50,m_start=5,test=FALSE,validation=FALSE,criterion="aic",interact=FALSE)library(glmdisc)
discretization <- glmdisc(x,y,iter=50,m_start=5,test=FALSE,validation=FALSE,criterion="aic",interact=FALSE)How well did we do?To compare the estimated and the true discretization schemes, we can represent them with respect to the input "raw" datax:plot(x[,1],xd[,1])
plot(discretization@cont.data[,1],discretization@disc.data[,1])ContributeYou can clone this project using:gitclonehttps://github.com/adimajo/glmdisc_python.gitYou can install all dependencies, including development dependencies, using (note that
this command requirespipenvwhich can be installed by typingpip install pipenv):pipenvinstall-dYou can build the documentation by going into thedocsdirectory and typingmake html.NOTE: you need to have a separate folder namedglmdisc_python_docsin the same directory as this repository,
as it will build the docs there so as to allow me to push this other directory as a separategh-pagesbranch.You can run the tests by typingcoverage run -m pytest, which relies on packagescoverageandpytest.To run the tests in different environments (one for each version of Python), installpyenv(seethe instructions here),
install all versions you want to test (seetox.ini), e.g. withpyenv install 3.7.0and runpipenv run pyenv local 3.7.0 [...](and all other versions) followed bypipenv run tox.