Package: gpie
GPieGaussianProcess tinyexplorersimple: an intuitive syntax inspired by scikit-learnpowerful: a compact core of expressive abstractionsextensible: a modular design for effortless compositionlightweight: a minimal set of dependencies {standard library, numpy, scipy}This is a ongoing research project with many parts currentlyunder construction- please expect bugs and sharp edges.Featuresseveral "avant-garde" kernels such as spectral kernel and neural kernel allow for exploration of new ideaseach kernel implements anisotropic variant besides isotropic one to support automatic relevance determinationa full-fledged toolkit of kernel operators enables all sorts of "kernel engineering", for example, handcrafting composite kernels based on expert knowledge or exploiting special structure of datasetscore computations such as likelihood and gradient are carefully formulated for speed and stabilitysampling inference embraces a probabilistic perspective in learning and prediction to promote robustnessBayesian optimizer offers a principled strategy to optimize expensive and black-box objectives globallyFunctionalitykernel functionswhite kernelconstant kernelradial basis function kernelrational quadratic kernelMatérn kernelOrnstein-Uhlenbeck kernelperiodic kernelspectral kernelneural kernelkernel operatorsHadamard (element-wise)sumproductexponentiationKroneckersumproductGaussian processregressionclassificationt processregressionclassificationBayesian optimizersurrogate: Gaussian process,t processacquisition: PI, EI, LCB,ES,KGsampling inferenceMarkov chain Monte CarloMetropolis-HastingsHamiltonian + no-U-turnsimulated annealingvariational inferenceNote: parts of the projectin italic fontare under construction.ExamplesGaussian process regression on Mauna Loa CO2In this example, we use Gaussian process to model the concentration of CO2at Mauna Loa as a function of time.# handcraft a composite kernel based on expert knowledge# long-term trendk1=30.0**2*RBFKernel(l=200.0)# seasonal variationsk2=3.0**2*RBFKernel(l=200.0)*PeriodicKernel(p=1.0,l=1.0)# medium-term irregularitiesk3=0.5**2*RationalQuadraticKernel(m=0.8,l=1.0)# noisek4=0.1**2*RBFKernel(l=0.1)+0.2**2*WhiteKernel()# composite kernelkernel=k1+k2+k3+k4# train GPR on datagpr=GaussianProcessRegressor(kernel=kernel)gpr.fit(X,y)In the plot, scattered dots represent historical observations, and shaded area shows the predictive interval (μ - σ, μ + σ) prophesied by a Gaussian process regressor trained on the historical data.Sampling inference for Gaussian process regressionHere we use a synthesized dataset for ease of illustration and investigate sampling inference techniques such as Markov chain Monte Carlo. As a Gaussian process defines the predictive distribution, we can get a sense of it by sampling from its prior distribution (before seeing training set) and posterior distribution (after seeing training set).# with the current hyperparameter configuration,# ... what is the prior distribution p(y_test)y_prior=gpr.prior_predictive(X,n_samples=6)# ... what is the posterior distribution p(y_test|y_train)y_posterior=gpr.posterior_predictive(X,n_samples=4)We can also sample from the posterior distribution of a hyperparameter, which characterizes its uncertainty beyond a single point estimate such as MLE or MAP.# invoke MCMC sampler to sample hyper values from its posterior distributionhyper_posterior=gpr.hyper_posterior(n_samples=10000)Bayesian optimizationWe demonstrate a simple example of Bayesian optimization. It starts by exploring the objective function globally and shifts to exploiting "promising areas" as more observations are made.# number of evaluationsn_evals=10# surrogate model (Gaussian process)surrogate=GaussianProcessRegressor(1.0*MaternKernel(d=5,l=1.0)+1.0*WhiteKernel())# bayesian optimizerbayesopt=BayesianOptimizer(fun=f,bounds=b,x0=x0,n_evals=n_evals,acquisition='lcb',surrogate=surrogate)bayesopt.minimize(callback=callback)BackendGPie makes extensive use ofde factostandard scientific computing packages in Python:numpy: linear algebra, stochastic samplingscipy: gradient-based optimization, stochastic samplingInstallationGPie requires Python 3.6 or greater. The easiest way to install GPie is from a prebuilt wheel using pip:pipinstall--upgradegpieYou can also install from source to try out the latest features (requirespep517>=0.8.0andsetuptools>=40.9.0):pipinstall--upgradegit+https://github.com/zackxzhang/gpieRoadmapimplement Hamiltonian Monte Carlo and no-U-turnadd a demo on characteristics of different kernelsadd a demo of quantified Occam's razorimplement Kronecker operators for scalable learning on grid datareplace Cholesky decomposition with Krylov subspace methods for speed...