Package: neo-ls-svm
Neo LS-SVMNeo LS-SVM is a modernLeast-Squares Support Vector Machineimplementation in Python that offers several benefits over sklearn's classicsklearn.svm.SVCclassifier andsklearn.svm.SVRregressor:⚡ Linear complexity in the number of training examples withOrthogonal Random Features.🚀 Hyperparameter free: zero-cost optimization of the regularisation parameter γ and kernel parameter σ.🏔️ Adds a new tertiary objective that minimizes the complexity of the prediction surface.🎁 Returns the leave-one-out residuals and error for free after fitting.🌀 Learns an affine transformation of the feature matrix to optimally separate the target's bins.🪞 Can solve the LS-SVM both in the primal and dual space.🌡️ Isotonically calibratedpredict_probabased on the leave-one-out predictions.🎲 Asymmetric conformal Bayesian confidence intervals for classification and regression.UsingInstallingFirst, install this package with:pipinstallneo-ls-svmClassification and regressionThen, you can importneo_ls_svm.NeoLSSVMas an sklearn-compatible binary classifier and regressor. Example usage:fromneo_ls_svmimportNeoLSSVMfrompandasimportget_dummiesfromsklearn.datasetsimportfetch_openmlfromsklearn.model_selectionimporttrain_test_split# Binary classification example:X,y=fetch_openml("churn",version=3,return_X_y=True,as_frame=True,parser="auto")X_train,X_test,y_train,y_test=train_test_split(get_dummies(X),y,test_size=0.15,random_state=42)model=NeoLSSVM().fit(X_train,y_train)model.score(X_test,y_test)# 93.1% (compared to sklearn.svm.SVC's 89.6%)# Regression example:X,y=fetch_openml("ames_housing",version=1,return_X_y=True,as_frame=True,parser="auto")X_train,X_test,y_train,y_test=train_test_split(get_dummies(X),y,test_size=0.15,random_state=42)model=NeoLSSVM().fit(X_train,y_train)model.score(X_test,y_test)# 82.4% (compared to sklearn.svm.SVR's -11.8%)Confidence intervalsNeo LS-SVM implements conformal prediction with a Bayesian nonconformity estimate to compute confidence intervals for both classification and regression. Example usage:fromneo_ls_svmimportNeoLSSVMfrompandasimportget_dummiesfromsklearn.datasetsimportfetch_openmlfromsklearn.model_selectionimporttrain_test_split# Load a regression problem and split in train and test.X,y=fetch_openml("ames_housing",version=1,return_X_y=True,as_frame=True,parser="auto")X_train,X_test,y_train,y_test=train_test_split(get_dummies(X),y,test_size=50,random_state=42)# Fit a Neo LS-SVM model.model=NeoLSSVM().fit(X_train,y_train)# Predict the house prices and confidence intervals on the test set.ŷ=model.predict(X_test)ŷ_conf=model.predict_proba(X_test,confidence_interval=True,confidence_level=0.95)# ŷ_conf[:, 0] and ŷ_conf[:, 1] are the lower and upper bound of the confidence interval for the predictions ŷ, respectivelyLet's visualize the confidence intervals on the test set:Expand to see the code that generated the above graph.importmatplotlib.pyplotaspltimportmatplotlib.tickerastickerimportnumpyasnpidx=np.argsort(-ŷ)y_ticks=np.arange(1,len(X_test)+1)plt.figure(figsize=(4,5))plt.barh(y_ticks,ŷ_conf[idx,1]-ŷ_conf[idx,0],left=ŷ_conf[idx,0],label="95% Confidence interval",color="lightblue")plt.plot(y_test.iloc[idx],y_ticks,"s",markersize=3,markerfacecolor="none",markeredgecolor="cornflowerblue",label="Actual value")plt.plot(ŷ[idx],y_ticks,"s",color="mediumblue",markersize=0.6,label="Predicted value")plt.xlabel("House price")plt.ylabel("Test house index")plt.yticks(y_ticks,y_ticks)plt.tick_params(axis="y",labelsize=6)plt.grid(axis="x",color="lightsteelblue",linestyle=":",linewidth=0.5)plt.gca().xaxis.set_major_formatter(ticker.StrMethodFormatter('${x:,.0f}'))plt.gca().spines["top"].set_visible(False)plt.gca().spines["right"].set_visible(False)plt.legend()plt.tight_layout()plt.show()BenchmarksWe select all binary classification and regression datasets below 1M entries from theAutoML Benchmark. Each dataset is split into 85% for training and 15% for testing. We applyskrub.TableVectorizeras a preprocessing step forneo_ls_svm.NeoLSSVMandsklearn.svm.SVC,SVRto vectorize the pandas DataFrame training data into a NumPy array. Models are fitted only once on each dataset, with their default settings and no hyperparameter tuning.Binary classificationROC-AUC on 15% test set:datasetLGBMClassifierNeoLSSVMSVCada🥈 90.9% (0.1s)🥇 90.9% (0.8s)83.1% (1.0s)adult🥇 93.0% (0.5s)🥈 89.1% (6.0s)/amazon_employee_access🥇 85.6% (0.5s)🥈 64.5% (2.8s)/arcene🥈 78.0% (0.6s)70.0% (4.4s)🥇 82.0% (3.4s)australian🥇 88.3% (0.2s)79.9% (0.4s)🥈 81.9% (0.0s)bank-marketing🥇 93.5% (0.3s)🥈 91.0% (4.1s)/blood-transfusion-service-center62.0% (0.1s)🥇 71.0% (0.5s)🥈 69.7% (0.0s)churn🥇 91.7% (0.4s)🥈 81.0% (0.8s)70.6% (0.8s)click_prediction_small🥇 67.7% (0.4s)🥈 66.6% (3.3s)/jasmine🥇 86.1% (0.3s)79.5% (1.2s)🥈 85.3% (1.8s)kc1🥇 78.9% (0.2s)🥈 76.6% (0.5s)45.7% (0.2s)kr-vs-kp🥇 100.0% (0.2s)99.2% (0.8s)🥈 99.4% (0.6s)madeline🥇 93.1% (0.4s)65.6% (0.8s)🥈 82.5% (4.5s)ozone-level-8hr🥈 91.2% (0.3s)🥇 91.6% (0.7s)72.8% (0.2s)pc4🥇 95.3% (0.3s)🥈 90.9% (0.5s)25.7% (0.1s)phishingwebsites🥇 99.5% (0.3s)🥈 98.9% (1.3s)98.7% (2.6s)phoneme🥇 95.6% (0.2s)🥈 93.5% (0.8s)91.2% (0.7s)qsar-biodeg🥇 92.7% (0.2s)🥈 91.1% (1.2s)86.8% (0.1s)satellite🥈 98.7% (0.2s)🥇 99.5% (0.8s)98.5% (0.1s)sylvine🥇 98.5% (0.2s)🥈 97.1% (0.8s)96.5% (1.0s)wilt🥈 99.5% (0.2s)🥇 99.8% (0.9s)98.9% (0.2s)RegressionR² on 15% test set:datasetLGBMRegressorNeoLSSVMSVRabalone🥈 56.2% (0.1s)🥇 59.5% (1.1s)51.3% (0.2s)boston🥇 91.7% (0.2s)🥈 89.3% (0.4s)35.1% (0.0s)brazilian_houses🥈 55.9% (0.4s)🥇 88.3% (1.5s)5.4% (2.0s)colleges🥇 58.5% (0.4s)🥈 43.7% (4.1s)40.2% (5.1s)diamonds🥇 98.2% (0.7s)🥈 95.2% (4.5s)/elevators🥇 87.7% (0.4s)🥈 82.6% (2.6s)/house_16h🥇 67.7% (0.3s)🥈 52.8% (2.4s)/house_prices_nominal🥇 89.0% (0.6s)🥈 78.2% (1.3s)-2.9% (0.3s)house_sales🥇 89.2% (1.3s)🥈 77.8% (2.2s)/mip-2016-regression🥇 59.2% (0.4s)🥈 34.9% (2.6s)-27.3% (0.1s)moneyball🥇 93.2% (0.2s)🥈 91.2% (0.6s)0.8% (0.1s)pol🥇 98.7% (0.3s)🥈 75.2% (1.7s)/quake-10.7% (0.2s)🥇 -0.1% (0.5s)🥈 -10.7% (0.0s)sat11-hand-runtime-regression🥇 78.3% (0.5s)🥈 61.7% (1.0s)-56.3% (1.0s)sensory🥇 29.2% (0.2s)3.8% (0.4s)🥈 16.4% (0.0s)socmob🥇 79.6% (0.2s)🥈 72.5% (1.5s)30.8% (0.0s)space_ga🥇 70.3% (0.2s)🥈 43.7% (0.6s)35.9% (0.1s)tecator🥈 98.3% (0.1s)🥇 99.4% (0.2s)78.5% (0.0s)us_crime🥈 62.8% (0.4s)🥇 63.0% (0.8s)6.7% (0.2s)wine_quality🥇 45.6% (0.6s)-8.0% (0.9s)🥈 16.4% (0.5s)ContributingPrerequisites1. Set up Git to use SSHGenerate an SSH keyandadd the SSH key to your GitHub account.Configure SSH to automatically load your SSH keys:cat<< EOF >> ~/.ssh/configHost *AddKeysToAgent yesIgnoreUnknown UseKeychainUseKeychain yesEOF2. Install DockerInstall Docker Desktop.EnableUse Docker Compose V2in Docker Desktop's preferences window.Linux only:Export your user's user id and group id so thatfiles created in the Dev Container are owned by your user:cat<< EOF >> ~/.bashrcexport UID=$(id --user)export GID=$(id --group)EOF3. Install VS Code or PyCharmInstall VS CodeandVS Code's Dev Containers extension. Alternatively, installPyCharm.Optional:install aNerd Fontsuch asFiraCode Nerd Fontandconfigure VS Codeorconfigure PyCharmto use it.Development environmentsThe following development environments are supported:⭐️GitHub Codespaces: click onCodeand selectCreate codespaceto start a Dev Container withGitHub Codespaces.⭐️Dev Container (with container volume): click onOpen in Dev Containersto clone this repository in a container volume and create a Dev Container with VS Code.Dev Container: clone this repository, open it with VS Code, and runCtrl/⌘+⇧+P→Dev Containers: Reopen in Container.PyCharm: clone this repository, open it with PyCharm, andconfigure Docker Compose as a remote interpreterwith thedevservice.Terminal: clone this repository, open it with your terminal, and rundocker compose up --detach devto start a Dev Container in the background, and then rundocker compose exec dev zshto open a shell prompt in the Dev Container.DevelopingThis project follows theConventional Commitsstandard to automateSemantic VersioningandKeep A ChangelogwithCommitizen.Runpoefrom within the development environment to print a list ofPoe the Poettasks available to run on this project.Runpoetry add {package}from within the development environment to install a run time dependency and add it topyproject.tomlandpoetry.lock. Add--group testor--group devto install a CI or development dependency, respectively.Runpoetry updatefrom within the development environment to upgrade all dependencies to the latest versions allowed bypyproject.toml.Runcz bumpto bump the package's version, update theCHANGELOG.md, and create a git tag.