Package: penlm
penlm - Penalized Linear Modelspenlmis a Python package that implements a few penalty based linear models that aren't (currently) available inscikit-learn. All the models are implemented with the familiarfit/predict/predict_proba/scoreinterface.The supported estimators are:Smoothly Adaptively Centered Ridge (SACR)Ridge penalty on first/second derivatives (Functional Linear Model)Adaptive LassoRelaxed LassoBroken Adaptive Ridge (BAR)Non-negative Garrote (NNG)All the estimators havefit_interceptandscalearguments (scaling is done withsklearn.preprocessing.StandardScaler) , and work for the following tasks:Linear RegressionBinary Classification (Logistic Regression)Multiclass Classification (One-vs-Rest)A customcross-validationobject is provided in order to perform grid search hyperparameter tuning (with any splitter fromscikit-learn), and usesmultiprocessingfor parallelization (defaultn_jobs = -1).
Multiclass fitting isnotparallelized in this version (that would be beneficial when a high number of cores is available, or when refitting the best estimator in the grid search object).InstallationThe package can be installed from terminal with the commandpip install penlm. Some of the estimators inpenlmare obtained by directly wrappingscikit-learnclasses, while the SACR, FLMs, and NNG are formulated usingPyomo, which in turn needs asolverto interface with. Depending on the estimator, the optimization problems are quadratic with equality and/or inequality constraints, and all the code was tested using the solverIpopt. You just need to download theexecutable binary, and then add the folder that contains it to your path (tested on Ubuntu).UsageThe following lines show how to fit an estimator with its own parameters and grid search object, by using aStratifiedKFoldsplitter:importnumpyasnp
importpenlm.grid_searchasgs
frompenlm.smoothly_adaptively_centered_ridgeimportSACRClassifier
fromsklearn.model_selectionimportStratifiedKFold
fromsklearn.datasetsimportload_wine
fromsklearn.metricsimportbalanced_accuracy_score

X,Y=load_wine(return_X_y=True)train_index=[iforiinrange(100)]test_index=[iforiinrange(100,len(X))]lambda_list=np.logspace(-5,5,10,base=2)phi_list=np.linspace(0,1,10)[1:]estimator=SACRClassifier(solver='ipopt',scale=True,fit_intercept=True)parameters={'phi':phi_list,'lambda':lambda_list}cv=StratifiedKFold(n_splits=2,random_state=46,shuffle=True)grid_search=gs.GridSearchCV(estimator,parameters,cv,scoring=balanced_accuracy_score,verbose=False,n_jobs=-1)grid_search.fit(X[train_index],Y[train_index])score=grid_search.score(X[test_index],Y[test_index])The test folder in thegithubrepo contains two sample scripts that show how to use all the estimators (in both classification and regression tasks). In particular, for the Adaptive Lasso and the NNG you need to provide an initial coefficient vector as anp.ndarray, with the same shape as the one found inscikit-learnestimators (the test scripts fit a LogisticRegression/Ridge estimator and use theestimator.coef_object).
Moreover, regarding thescoring, all the estimators and the grid search class useaccuracy/R^2as default scores (when the argumentscoring = None), but you can provide anyCallablescoring function found insklearn.metrics. Beware that higher is better, and therefore when scoring with errors likesklearn.metrics.mean_squared_error, you need to wrap that in a custom function that changes its sign.CitingWe encourage the users to cite the original papers of the implemented estimators.
In particular, the code published in this package has been used in the case studies ofthispaper.