Package: peak-engines
peak-enginespeak-enginesis machine learning framework that focuses on applying advanced optimization algorithms
to build better models.Installationpip install peak-enginesFit Logistic Regression HyperparametersThe leave-one-out cross-validation of logistic regression can be efficiently approximated. At a
high level, this is how it works: For given hyperparametersC, weFind the parametersbthat optimize logistic regression for the givenC.For each data indexi, we compute the hessianH_{-i}and gradientg_{-i}of the
log-likelihood with the ith data entry removed. (We can reuse the hessian computed from (1) to do this
with a minimal amount of work.)We apply theMatrix Inversion Lemmato
efficiently compute the inverseH_{-i}^{-1}.We useH_{-i}^{-1}andg_{-i}to take a single step ofNewton's methodto approximate the logistic regression coefficients with the ith entry removedb_{-i}.And finally, we used theb_{-i}'s to approximate the out-of-sample predictions and estimate the leave-one-out cross-validation.See the paperA scalable estimate of the out-of-sample prediction error via approximate leave-one-outby Kamiar Rad and Arian Maleki for more details.We can, furthermore, differentiate the Approximate Leave-One-Out metric with respect to the hyperparameters and quickly climb to the best performingC. Here's how to do it withpeak-engines:Load an example datasetfromsklearn.datasetsimportload_breast_cancerfromsklearn.preprocessingimportStandardScalerX,y=load_breast_cancer(return_X_y=True)X=StandardScaler().fit_transform(X)Find the best performing Cmodel=peak_engines.LogisticRegressionModel()model.fit(X,y)print('C =',model.C_[0])printsC = 0.66474879If we compute the LOOCV by brute force and compare to the ALOOCV, we can see how accurate the approximation isFit Ridge Regression HyperparametersBy expressing cross-validation as an optimization objective and computing derivatives,peak-enginesis able to efficiently find regularization parameters that lead to the best
score on a leave-one-out or generalized cross-validation. It, futhermore, scales to handle
multiple regularizers. Here's an example of how it worksimportnumpyasnpfromsklearn.datasetsimportload_bostonX,y=load_boston(return_X_y=True)frompeak_enginesimportRidgeRegressionModelmodel=RidgeRegressionModel(normalize=True)# Fit will automatically find the best alpha that minimizes the Leave-one-out Cross-validation.# when you call fit. There's no need to provide a search space because peak_engines optimizes the# LOOCV directly. It the computes derivatives of the LOOCV with respect to the hyperparameters and# is able to quickly zero in on the best alpha.model.fit(X,y)print('alpha =',model.alpha_)printsalpha = 0.009274259071634289Fit Warped Linear RegressionLetXandydenote the feature matrix and target vector of a regression dataset. Under the
assumption of normally distributed errors, Ordinary Least Squares (OLS) finds the linear model
that maximizes the likelihood of the datasetWhat happens when errors aren't normally distributed? Well, the model will be misspecified and
there's no reason to think its likelihood predictions will be accurate. This is where
Warped Linear Regression can help. It introduces an extra step to OLS where it transforms the
target vector using a malleable, monotonic functionfparameterized byψand adjusts the parameters to
maximize the likelihood of the transformed datasetBy introducing the additional transformation step, Warped Linear Regression is more general-purpose
than OLS while still retaining the strong structure and interpretability.  Here's how you use itLoad an example datasetfromsklearn.datasetsimportload_bostonX,y=load_boston(return_X_y=True)Fit a warped linear regression modelimportpeak_enginesmodel=peak_engines.WarpedLinearRegressionModel()model.fit(X_train,y_train)Visualize the warping functionimportnumpyasnpimportmatplotlib.pyplotasplty_range=np.arange(np.min(y),np.max(y),0.01)z=model.warper_.compute_latent(y_range)plt.plot(y_range,z)plt.xlabel('Median Housing Value in $1000s')plt.ylabel('Latent Variable')plt.scatter(y,model.warper_.compute_latent(y))TutorialsRidge RegressionWarped Linear RegressionArticlesHow to Do Ridge Regression BetterWhat Form of Cross-Validation Should You Use?What to Do When Your Model Has a Non-Normal Error DistributionExamplesexample/ridge_regression/california_housing.ipynb:
Measure the performance of different ridge regression models for predicting housing values.example/ridge_regression/pollution.ipynb:
Build ridge regression models to predict mortality rates from air quality.example/warped_linear_regression/boston_housing.ipynb:
Build a warped linear regression model to predict housing values.example/warped_linear_regression/abalone.ipynb:
Predict the age of sea snails using warped linear regression.DocumentationSeedoc/Reference.pdf