Package: pyPDPPartitioner
Python PDP with PartitionerGitHubOriginal PaperarXivGitHubPDFInstallationYou need to either create an environment or update an existing environment.Aftercreating an environment you have to activate it:conda activate pyPDPPartitionerCreate environmentconda env create -f environment.ymlUpdate environment (if env exists)conda env update -f environment.yml --pruneInstallation via pippip install pyPDPPartitionerForHPO-Benchexamples, you further need to install HPOBench from git (e.g.pip install git+https://github.com/automl/HPOBench.git@master).UsageBlackbox functionsTo use this package you needA Blackbox function (a function that gets any input and outputs a score)A Configuration Space that matches the required input of the blackbox functionThere are some synthetic Blackbox-functions implemented that are ready to use:f=StyblinskiTang.for_n_dimensions(3)# Create 3D-StyblinskiTang functioncs=f.config_space# A config space that is suitable for this functionSamplersTo sample points for fitting a surrogate, there are multiple samplers available:RandomSamplerGridSamplerBayesianOptimizationSampler with Acquisition-Functions:LowerConfidenceBound(ExpectedImprovement)(ProbabilityOfImprovement)sampler=BayesianOptimizationSampler(f,cs)sampler.sample(80)Surrogate ModelsAll algorithms require aSurrogateModel, which can be fitted withSurrogateModel.fit(X, y)and yields means and variances withSurrogateModel.predict(X).Currently, there is only aGaussianProcessSurrogateavailable.surrogate=GaussianProcessSurrogate()surrogate.fit(sampler.X,sampler.y)AlgorithmsThere are some available algorithms:ICEPDPDecisionTreePartitionerRandomForestPartitionerEach algorithm needs:ASurrogateModelOne or many selected hyperparametersamplesnum_grid_points_per_axisSamples can be randomly generated via# Algorithm.from_random_points(...)ice=ICE.from_random_points(surrogate,selected_hyperparameter="x1")Also, all other algorithms can be built from an ICE-Instance.pdp=PDP.from_ICE(ice)dt_partitioner=DecisionTreePartitioner.from_ICE(ice)rf_partitioner=RandomForestPartitioner.from_ICE(ice)The Partitioners can split the Hyperparameterspace of not selected Hyperparameters into multiple regions.
The best region can be obtained using the incumbent of the sampler.incumbent_config=sampler.incumbent_configdt_partitioner.partition(max_depth=3)dt_region=dt_partitioner.get_incumbent_region(incumbent_config)rf_partitioner.partition(max_depth=1,num_trees=10)rf_region=rf_partitioner.get_incumbent_region(incumbent_config)Finally, a new PDP can be obtained from the region. This PDP has the properties of a single ICE-Curve since the mean
of the ICE-Curve results in a new ICE-Curve.pdp_region=region.pdp_as_ice_curvePlottingMost components can create plots. These plots can be drawn on a given axis or are drawn onplt.gca()by default.Samplerssampler.plot()# Plots all samplesSurrogatesurrogate.plot_means()# Plots mean predictions of surrogatesurrogate.plot_confidences()# Plots confidencesAcquisition Functionsurrogate.acq_func.plot()# Plot acquisition function of surrogate modelICEice.plot()# Plots all ice curves. Only possible for 1 selected hyperparameterICE Curveice_curve=ice[0]# Get first ice curveice_curve.plot_values()# Plot values of ice curveice_curve.plot_confidences()# Plot confidences of ice curveice_curve.plot_incumbent()# Plot position of smallest valuePDPpdp.plot_values()# Plot values of pdppdp.plot_confidences()# Plot confidences of pdppdp.plot_incumbent()# Plot position of smallest valuePartitionerdt_partitioner.plot()# only 1 selected hp, plots all ice curves in different color per regiondt_partitioner.plot_incumbent_cs(incumbent_config)# plot config space of best regionrf_partitioner.plot_incumbent_cs(incumbent_config)# plot incumbent config of all treesRegionsregion.plot_values()# plot pdp of regionregion.plot_confidences()# plot confidence of pdp in regionPlotting examplesSurrogateSource:tests/sampler/test_acquisition_function.py1D-Surrogate model with mean + confidenceacquisition functionSamplerSource:tests/sampler/test_mmd.pyUnderlying blackbox function (2D-Styblinski-Tang)Samples from RandomSamplerSamples from BayesianOptimizationSamplerICESource:tests/algorithms/test_ice.pyAll ICE-Curves from 2D-Styblinski-Tang with 1 selected HyperparameterPDPSource:tests/algorithms/test_pdp.py2D PDP (means)2D PDP (confidences)All Samples for surrogate modelPDPSource:examples/main_2d_pdp.py(num_grid_points_per_axis=100)2D PDP (means)Decision Tree PartitionerSource:tests/algorithms/partitioner/test_partitioner.pyAll ICE-Curves splitt into 8 different regions (3 splits) (used 2D-Styblinski-Tang with 1 selected hyperparameter)Decision Tree Config SpacesSource:tests/algorithms/partitioner/test_partitioner.pyAll Leaf-Config spaces from Decision Tree Partitioner with 3D-Styblinski-Tang Function and 1 Selected Hyperparameter (x3)2D-Styblinkski-Tang in background