I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
Optimisation using decision trees
We run forest_minimize on the function
Can you write Python code for it?

bounds = [(-1, 1.), ] * 3
n_calls = 150

result = forest_minimize(funny_func, bounds, n_calls=n_calls,
                         base_estimator="ET",
                         random_state=4)


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
Partial Dependence Plots
Sigurd Carlsen Feb 2019
Holger Nahrstaedt 2020
.. currentmodule:: skopt
Plot objective now supports optional use of partial dependence as well as
different methods of defining parameter values for dependency plots.
Can you write Python code for it?

print(__doc__)
import sys
from skopt.plots import plot_objective
from skopt import forest_minimize
import numpy as np
np.random.seed(123)
import matplotlib.pyplot as plt


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
Partial dependence plot
Here we see an example of using partial dependence. Even when setting
n_points all the way down to 10 from the default of 40, this method is
still very slow. This is because partial dependence calculates 250 extra
predictions for each point on the plots.
Can you write Python code for it?


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
Plot without partial dependence
Here we plot without partial dependence. We see that it is a lot faster.
Also the values for the other parameters are set to the default "result"
which is the parameter set of the best observed value so far. In the case
of funny_func this is close to 0 for all parameters.
Can you write Python code for it?

_ = plot_objective(result,  sample_source='result', n_points=10)


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
Set a minimum location
Lastly we can also define these parameters ourself by parsing a list
as the minimum argument:
Can you write Python code for it?

_ = plot_objective(result, n_points=10, sample_source=[1, -0.5, 0.5],
                   minimum=[1, -0.5, 0.5])


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
We can also specify how many initial samples are used for the two different
"expected_minimum" methods. We set it to a low value in the next examples
to showcase how it affects the minimum for the two methods.
Can you write Python code for it?

_ = plot_objective(result, n_points=10, sample_source='expected_minimum_random',
                   minimum='expected_minimum_random',
                   n_minimum_search=10)

_ = plot_objective(result, n_points=10, sample_source="expected_minimum",
                   minimum='expected_minimum', n_minimum_search=2)


I am working on the file "0.7/notebooks/auto_examples/plots/visualizing-results.ipynb".
The first task is:
Evaluating the objective function
Next we use an extra trees based minimizer to find one of the minima of the
:class:benchmarks.branin function. Then we visualize at which points the objective is being
evaluated using :class:plots.plot_evaluations.
Can you write Python code for it?

from functools import partial
from skopt.plots import plot_evaluations
from skopt import gp_minimize, forest_minimize, dummy_minimize

bounds = [(-5.0, 10.0), (0.0, 15.0)]
n_calls = 160

forest_res = forest_minimize(branin, bounds, n_calls=n_calls,
                             base_estimator="ET", random_state=4)

_ = plot_evaluations(forest_res, bins=10)


I am working on the file "0.7/notebooks/auto_examples/plots/visualizing-results.ipynb".
The first task is:
Going from 6 to 6+2 dimensions
To make things more interesting let's add two dimension to the problem.
As :class:benchmarks.hart6 only depends on six dimensions we know that for this problem
the new dimensions will be "flat" or uninformative. This is clearly visible
in both the placement of samples and the partial dependence plots.
Can you write Python code for it?

bounds = [(0., 1.),] * 8
n_calls = 200

forest_res = forest_minimize(hart6, bounds, n_calls=n_calls,
                             base_estimator="ET", random_state=4)

_ = plot_evaluations(forest_res)
_ = plot_objective(forest_res, n_samples=40)

# .. [Friedman (2001)] `doi:10.1214/aos/1013203451 section 8.2 <http://projecteuclid.org/euclid.aos/1013203451>`


I am working on the file "0.7/notebooks/auto_examples/plots/visualizing-results.ipynb".
The first task is:
Starting with branin
To start let's take advantage of the fact that :class:benchmarks.branin is a simple
function which can be visualised in two dimensions.
Can you write Python code for it?

from matplotlib.colors import LogNorm

def plot_branin():
    fig, ax = plt.subplots()

    x1_values = np.linspace(-5, 10, 100)
    x2_values = np.linspace(0, 15, 100)
    x_ax, y_ax = np.meshgrid(x1_values, x2_values)
    vals = np.c_[x_ax.ravel(), y_ax.ravel()]
    fx = np.reshape([branin(val) for val in vals], (100, 100))

    cm = ax.pcolormesh(x_ax, y_ax, fx,
                       norm=LogNorm(vmin=fx.min(),
                                    vmax=fx.max()))

    minima = np.array([[-np.pi, 12.275], [+np.pi, 2.275], [9.42478, 2.475]])
    ax.plot(minima[:, 0], minima[:, 1], "r.", markersize=14,
            lw=0, label="Minima")

    cb = fig.colorbar(cm)
    cb.set_label("f(x)")

    ax.legend(loc="best", numpoints=1)

    ax.set_xlabel("$X_0$")
    ax.set_xlim([-5, 10])
    ax.set_ylabel("$X_1$")
    ax.set_ylim([0, 15])

plot_branin()


I am working on the file "0.7/notebooks/auto_examples/plots/visualizing-results.ipynb".
The first task is:
The two dimensional partial dependence plot can look like the true
objective but it does not have to. As points at which the objective function
is being evaluated are concentrated around the suspected minimum the
surrogate model sometimes is not a good representation of the objective far
away from the minima.
Random sampling
Compare this to a minimizer which picks points at random. There is no
structure visible in the order in which it evaluates the objective. Because
there is no model involved in the process of picking sample points at
random, we can not plot the partial dependence of the model.
Can you write Python code for it?

dummy_res = dummy_minimize(branin, bounds, n_calls=n_calls, random_state=4)

_ = plot_evaluations(dummy_res, bins=10)


I am working on the file "0.7/notebooks/auto_examples/plots/visualizing-results.ipynb".
The first task is:
Toy models
We will use two different toy models to demonstrate how :class:plots.plot_evaluations
works.
The first model is the :class:benchmarks.branin function which has two dimensions and three
minima.
The second model is the hart6 function which has six dimension which makes
it hard to visualize. This will show off the utility of
:class:plots.plot_evaluations.
Can you write Python code for it?

from skopt.benchmarks import branin as branin
from skopt.benchmarks import hart6 as hart6_

# redefined `hart6` to allow adding arbitrary "noise" dimensions
def hart6(x):
    return hart6_(x[:6])


I am working on the file "0.7/notebooks/auto_examples/plots/visualizing-results.ipynb".
The first task is:
Visualizing optimization results
Tim Head, August 2016.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
Bayesian optimization or sequential model-based optimization uses a surrogate
model to model the expensive to evaluate objective function func. It is
this model that is used to determine at which points to evaluate the expensive
objective next.
To help understand why the optimization process is proceeding the way it is,
it is useful to plot the location and order of the points at which the
objective is evaluated. If everything is working as expected, early samples
will be spread over the whole parameter space and later samples should
cluster around the minimum.
The :class:plots.plot_evaluations function helps with visualizing the location and
order in which samples are evaluated for objectives with an arbitrary
number of dimensions.
The :class:plots.plot_objective function plots the partial dependence of the objective,
as represented by the surrogate model, for each dimension and as pairs of the
input dimensions.
All of the minimizers implemented in skopt return an OptimizeResult
instance that can be inspected. Both :class:plots.plot_evaluations and :class:plots.plot_objective
are helpers that do just that
Can you write Python code for it?

print(__doc__)
import numpy as np
np.random.seed(123)

import matplotlib.pyplot as plt


I am working on the file "0.7/notebooks/auto_examples/plots/visualizing-results.ipynb".
The first task is:
Working in six dimensions
Visualising what happens in two dimensions is easy, where
:class:plots.plot_evaluations and :class:plots.plot_objective start to be useful is when the
number of dimensions grows. They take care of many of the more mundane
things needed to make good plots of all combinations of the dimensions.
The next example uses class:benchmarks.hart6 which has six dimensions and shows both
:class:plots.plot_evaluations and :class:plots.plot_objective.
Can you write Python code for it?

bounds = [(0., 1.),] * 6

forest_res = forest_minimize(hart6, bounds, n_calls=n_calls,
                             base_estimator="ET", random_state=4)

_ = plot_evaluations(forest_res)
_ = plot_objective(forest_res, n_samples=40)


I am working on the file "0.7/notebooks/auto_examples/sklearn-gridsearchcv-replacement.ipynb".
The first task is:
Counting total iterations that will be used to explore all subspaces
Subspaces in previous examples can further increase in complexity if you add
new model subspaces or dimensions for feature extraction pipelines. For
monitoring of progress, you would like to know the total number of
iterations it will take to explore all subspaces. This can be
calculated with total_iterations property, as in the code below.
Can you write Python code for it?

from skopt import BayesSearchCV

from sklearn.datasets import load_iris
from sklearn.svm import SVC

X, y = load_iris(True)

searchcv = BayesSearchCV(
    SVC(),
    search_spaces=[
        ({'C': (0.1, 1.0)}, 19),  # 19 iterations for this subspace
        {'gamma':(0.1, 1.0)}
    ],
    n_iter=23
)

print(searchcv.total_iterations)


I am working on the file "0.7/notebooks/auto_examples/sklearn-gridsearchcv-replacement.ipynb".
The first task is:
Scikit-learn hyperparameter search wrapper
Iaroslav Shcherbatyi, Tim Head and Gilles Louppe. June 2017.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
Introduction
This example assumes basic familiarity with
scikit-learn <http://scikit-learn.org/stable/index.html>_.
Search for parameters of machine learning models that result in best
cross-validation performance is necessary in almost all practical
cases to get a model with best generalization estimate. A standard
approach in scikit-learn is using :obj:sklearn.model_selection.GridSearchCV class, which takes
a set of values for every parameter to try, and simply enumerates all
combinations of parameter values. The complexity of such search grows
exponentially with the addition of new parameters. A more scalable
approach is using :obj:sklearn.model_selection.RandomizedSearchCV, which however does not take
advantage of the structure of a search space.
Scikit-optimize provides a drop-in replacement for :obj:sklearn.model_selection.GridSearchCV,
which utilizes Bayesian Optimization where a predictive model referred
to as "surrogate" is used to model the search space and utilized to
arrive at good parameter values combination as soon as possible.
Note: for a manual hyperparameter optimization example, see
"Hyperparameter Optimization" notebook.
Can you write Python code for it?


I am working on the file "0.7/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
===========================================
Store and load skopt optimization results
===========================================
Mikhail Pak, October 2016.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
Problem statement
We often want to store optimization results in a file. This can be useful,
for example,

if you want to share your results with colleagues;
if you want to archive and/or document your work;
or if you want to postprocess your results in a different Python instance or on an another computer.

The process of converting an object into a byte stream that can be stored in
a file is called serialization.
Conversely, deserialization means loading an object from a byte stream.
Warning: Deserialization is not secure against malicious or erroneous
code. Never load serialized data from untrusted or unauthenticated sources!
Can you write Python code for it?

print(__doc__)
import numpy as np
import os
import sys

# The followings are hacks to allow sphinx-gallery to run the example.
sys.path.insert(0, os.getcwd())
main_dir = os.path.basename(sys.modules['__main__'].__file__)
IS_RUN_WITH_SPHINX_GALLERY = main_dir != os.getcwd()


I am working on the file "0.7/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
And load from file using :class:skopt.load:
Can you write Python code for it?

res_loaded = load('result.pkl')

res_loaded.fun


I am working on the file "0.7/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
As long as your Python session is active, you can access all the
optimization results via the res object.
So how can you store this data in a file? skopt conveniently provides
functions :class:skopt.dump and :class:skopt.load that handle this for you.
These functions are essentially thin wrappers around the
joblib <https://joblib.readthedocs.io/en/latest/>_ module's :obj:joblib.dump and :obj:joblib.load.
We will now show how to use :class:skopt.dump and :class:skopt.load for storing
and loading results.
Using skopt.dump() and skopt.load()
For storing optimization results into a file, call the :class:skopt.dump
function:
Can you write Python code for it?

from skopt import dump, load

dump(res, 'result.pkl')


I am working on the file "0.7/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
Notice that the entry 'func' is absent in the loaded object but is still
present in the local variable:
Can you write Python code for it?

res_loaded_without_objective = load('result_without_objective.pkl')

print('Loaded object: ', res_loaded_without_objective.specs['args'].keys())
print('Local variable:', res.specs['args'].keys())


I am working on the file "0.7/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
Possible problems

Python versions incompatibility: In general, objects serialized in
  Python 2 cannot be deserialized in Python 3 and vice versa.
Security issues: Once again, do not load any files from untrusted
  sources.
Extremely large results objects: If your optimization results object

is extremely large, calling :class:skopt.dump with store_objective=False might
cause performance issues. This is due to creation of a deep copy without the
objective function. If the objective function it is not critical to you, you
can simply delete it before calling :class:skopt.dump. In this case, no deep
copy is created:
Can you write Python code for it?

del res.specs['args']['func']

dump(res, 'result_without_objective_2.pkl')


I am working on the file "0.7/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
Simple example
We will use the same optimization problem as in the
sphx_glr_auto_examples_bayesian-optimization.py notebook:
Can you write Python code for it?

from skopt import gp_minimize
noise_level = 0.1

if IS_RUN_WITH_SPHINX_GALLERY:
    # When this example is run with sphinx gallery, it breaks the pickling
    # capacity for multiprocessing backend so we have to modify the way we
    # define our functions. This has nothing to do with the example.
    from utils import obj_fun
else:
    def obj_fun(x, noise_level=noise_level):
        return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2)) + np.random.randn() * noise_level

res = gp_minimize(obj_fun,            # the function to minimize
                  [(-2.0, 2.0)],      # the bounds on each dimension of x
                  x0=[0.],            # the starting point
                  acq_func="LCB",     # the acquisition function (optional)
                  n_calls=15,         # the number of evaluations of f including at x0
                  n_random_starts=0,  # the number of random initialization points
                  random_state=777)


I am working on the file "0.7/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
You can fine-tune the serialization and deserialization process by calling
:class:skopt.dump and :class:skopt.load with additional keyword arguments. See the
joblib <https://joblib.readthedocs.io/en/latest/>_ documentation
:obj:joblib.dump and
:obj:joblib.load for the additional parameters.
For instance, you can specify the compression algorithm and compression
level (highest in this case):
Can you write Python code for it?

dump(res, 'result.gz', compress=9)

from os.path import getsize
print('Without compression: {} bytes'.format(getsize('result.pkl')))
print('Compressed with gz:  {} bytes'.format(getsize('result.gz')))


I am working on the file "0.7/notebooks/auto_examples/strategy-comparison.ipynb".
The first task is:
Comparing surrogate models
Tim Head, July 2016.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
Bayesian optimization or sequential model-based optimization uses a surrogate
model to model the expensive to evaluate function func. There are several
choices for what kind of surrogate model to use. This notebook compares the
performance of:

gaussian processes,
extra trees, and
random forests

as surrogate models. A purely random optimization strategy is also used as
a baseline.
Can you write Python code for it?

print(__doc__)
import numpy as np
np.random.seed(123)
import matplotlib.pyplot as plt


I am working on the file "0.7/notebooks/auto_examples/strategy-comparison.ipynb".
The first task is:
This shows the value of the two-dimensional branin function and
the three minima.
Objective
The objective of this example is to find one of these minima in as
few iterations as possible. One iteration is defined as one call
to the :class:benchmarks.branin function.
We will evaluate each model several times using a different seed for the
random number generator. Then compare the average performance of these
models. This makes the comparison more robust against models that get
"lucky".
Can you write Python code for it?

from functools import partial
from skopt import gp_minimize, forest_minimize, dummy_minimize

func = partial(branin, noise_level=2.0)
bounds = [(-5.0, 10.0), (0.0, 15.0)]
n_calls = 60

def run(minimizer, n_iter=5):
    return [minimizer(func, bounds, n_calls=n_calls, random_state=n)
            for n in range(n_iter)]

# Random search
dummy_res = run(dummy_minimize)

# Gaussian processes
gp_res = run(gp_minimize)

# Random forest
rf_res = run(partial(forest_minimize, base_estimator="RF"))

# Extra trees
et_res = run(partial(forest_minimize, base_estimator="ET"))


I am working on the file "0.7/notebooks/auto_examples/strategy-comparison.ipynb".
The first task is:
Toy model
We will use the :class:benchmarks.branin function as toy model for the expensive function.
In a real world application this function would be unknown and expensive
to evaluate.
Can you write Python code for it?

from skopt.benchmarks import branin as _branin

def branin(x, noise_level=0.):
    return _branin(x) + noise_level * np.random.randn()

from matplotlib.colors import LogNorm

def plot_branin():
    fig, ax = plt.subplots()

    x1_values = np.linspace(-5, 10, 100)
    x2_values = np.linspace(0, 15, 100)
    x_ax, y_ax = np.meshgrid(x1_values, x2_values)
    vals = np.c_[x_ax.ravel(), y_ax.ravel()]
    fx = np.reshape([branin(val) for val in vals], (100, 100))

    cm = ax.pcolormesh(x_ax, y_ax, fx,
                       norm=LogNorm(vmin=fx.min(),
                                    vmax=fx.max()))

    minima = np.array([[-np.pi, 12.275], [+np.pi, 2.275], [9.42478, 2.475]])
    ax.plot(minima[:, 0], minima[:, 1], "r.", markersize=14,
            lw=0, label="Minima")

    cb = fig.colorbar(cm)
    cb.set_label("f(x)")

    ax.legend(loc="best", numpoints=1)

    ax.set_xlabel("X1")
    ax.set_xlim([-5, 10])
    ax.set_ylabel("X2")
    ax.set_ylim([0, 15])

plot_branin()


I am working on the file "0.8/notebooks/auto_examples/ask-and-tell.ipynb".
The first task is:
Async optimization Loop
Bayesian optimization is used to tune parameters for walking robots or other
experiments that are not a simple (expensive) function call.
Tim Head, February 2017.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
They often follow a pattern a bit like this:

ask for a new set of parameters
walk to the experiment and program in the new parameters
observe the outcome of running the experiment
walk back to your laptop and tell the optimizer about the outcome
go to step 1

A setup like this is difficult to implement with the _minimize()* function
interface. This is why scikit-optimize** has a ask-and-tell interface that
you can use when you want to control the execution of the optimization loop.
This notebook demonstrates how to use the ask and tell interface.
Can you write Python code for it?

print(__doc__)

import numpy as np
np.random.seed(1234)
import matplotlib.pyplot as plt
from skopt.plots import plot_gaussian_process


I am working on the file "0.8/notebooks/auto_examples/ask-and-tell.ipynb".
The first task is:
By using the :class:Optimizer class directly you get control over the
optimization loop.
You can also pickle your :class:Optimizer instance if you want to end the
process running it and resume it later. This is handy if your experiment
takes a very long time and you want to shutdown your computer in the
meantime:
Can you write Python code for it?

import pickle

with open('my-optimizer.pkl', 'wb') as f:
    pickle.dump(opt, f)

with open('my-optimizer.pkl', 'rb') as f:
    opt_restored = pickle.load(f)


I am working on the file "0.8/notebooks/auto_examples/ask-and-tell.ipynb".
The first task is:
Now we setup the :class:Optimizer class. The arguments follow the meaning and
naming of the *_minimize() functions. An important difference is that
you do not pass the objective function to the optimizer.
Can you write Python code for it?

opt = Optimizer([(-2.0, 2.0)], "GP", acq_func="EI",
                acq_optimizer="sampling",
                initial_point_generator="lhs")

# To obtain a suggestion for the point at which to evaluate the objective
# you call the ask() method of opt:

next_x = opt.ask()
print(next_x)


I am working on the file "0.8/notebooks/auto_examples/ask-and-tell.ipynb".
The first task is:
Our 1D toy problem, this is the function we are trying to
minimize
Can you write Python code for it?

def objective(x, noise_level=noise_level):
    return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2))\
           + np.random.randn() * noise_level

def objective_wo_noise(x, noise_level=0):
    return objective(x, noise_level=0)


I am working on the file "0.8/notebooks/auto_examples/bayesian-optimization.ipynb".
The first task is:
Accordingly, the approximated minimum is found to be:
Can you write Python code for it?


I am working on the file "0.8/notebooks/auto_examples/bayesian-optimization.ipynb".
The first task is:
Note. In skopt, functions $f$ are assumed to take as input a 1D
vector $x$: represented as an array-like and to return a scalar
$f(x)$:.
Can you write Python code for it?

# Plot f(x) + contours
x = np.linspace(-2, 2, 400).reshape(-1, 1)
fx = [f(x_i, noise_level=0.0) for x_i in x]
plt.plot(x, fx, "r--", label="True (unknown)")
plt.fill(np.concatenate([x, x[::-1]]),
         np.concatenate(([fx_i - 1.9600 * noise_level for fx_i in fx],
                         [fx_i + 1.9600 * noise_level for fx_i in fx[::-1]])),
         alpha=.2, fc="r", ec="None")
plt.legend()
plt.grid()
plt.show()


I am working on the file "0.8/notebooks/auto_examples/bayesian-optimization.ipynb".
The first task is:
Together these attributes can be used to visually inspect the results of the
minimization, such as the convergence trace or the acquisition function at
the last iteration:
Can you write Python code for it?

from skopt.plots import plot_convergence
plot_convergence(res);


I am working on the file "0.8/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
Note that negative values does not work with the "PI"-acquisition function
but works with "EI":
Can you write Python code for it?

acq_func_kwargs = {"xi": -1000000000000}

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
                acq_func="PI", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
                acq_func="EI", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)


I am working on the file "0.8/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
Plotting parameters
Can you write Python code for it?

plot_args = {"objective": objective_wo_noise,
             "noise_level": noise_level, "show_legend": True,
             "show_title": True, "show_next_point": False,
             "show_acq_func": True}


I am working on the file "0.8/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
We can also favor exploitaton:
Can you write Python code for it?

acq_func_kwargs = {"xi": 0.000001, "kappa": 0.001}

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
                acq_func="LCB", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
                acq_func="EI", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
                acq_func="PI", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)


I am working on the file "0.8/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
We run a an optimization loop with standard settings
Can you write Python code for it?

for i in range(30):
    next_x = opt.ask()
    f_val = objective(next_x)
    opt.tell(next_x, f_val)
# The same output could be created with opt.run(objective, n_iter=30)
_ = plot_gaussian_process(opt.get_result(), **plot_args)


I am working on the file "0.8/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
We see that some minima is found and "exploited"
Now lets try to set kappa and xi using'to other values and
pass it to the optimizer:
Can you write Python code for it?

acq_func_kwargs = {"xi": 10000, "kappa": 10000}

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
                acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)


I am working on the file "0.8/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
We see that the points are more random now.
This works both for kappa when using acq_func="LCB":
Can you write Python code for it?

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=3,
                acq_func="LCB", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
_ = plot_gaussian_process(opt.get_result(), **plot_args)


I am working on the file "0.8/notebooks/auto_examples/hyperparameter-optimization.ipynb".
The first task is:
Convergence plot
Can you write Python code for it?

from skopt.plots import plot_convergence

plot_convergence(res_gp)


I am working on the file "0.8/notebooks/auto_examples/hyperparameter-optimization.ipynb".
The first task is:
Next, we need to define the bounds of the dimensions of the search space
we want to explore and pick the objective. In this case the cross-validation
mean absolute error of a gradient boosting regressor over the Boston
dataset, as a function of its hyper-parameters.
Can you write Python code for it?

from skopt.space import Real, Integer
from skopt.utils import use_named_args

# The list of hyper-parameters we want to optimize. For each one we define the
# bounds, the corresponding scikit-learn parameter name, as well as how to
# sample values from that dimension (`'log-uniform'` for the learning rate)
space  = [Integer(1, 5, name='max_depth'),
          Real(10**-5, 10**0, "log-uniform", name='learning_rate'),
          Integer(1, n_features, name='max_features'),
          Integer(2, 100, name='min_samples_split'),
          Integer(1, 100, name='min_samples_leaf')]

# this decorator allows your objective function to receive a the parameters as
# keyword arguments. This is particularly convenient when you want to set
# scikit-learn estimator parameters
@use_named_args(space)
def objective(**params):
    reg.set_params(**params)

    return -np.mean(cross_val_score(reg, X, y, cv=5, n_jobs=-1,
                                    scoring="neg_mean_absolute_error"))


I am working on the file "0.8/notebooks/auto_examples/hyperparameter-optimization.ipynb".
The first task is:
Objective
To tune the hyper-parameters of our model we need to define a model,
decide which parameters to optimize, and define the objective function
we want to minimize.
Can you write Python code for it?

from sklearn.datasets import load_boston
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import cross_val_score

boston = load_boston()
X, y = boston.data, boston.target
n_features = X.shape[1]

# gradient boosted trees tend to do well on problems like this
reg = GradientBoostingRegressor(n_estimators=50, random_state=0)


I am working on the file "0.8/notebooks/auto_examples/hyperparameter-optimization.ipynb".
The first task is:
Tuning a scikit-learn estimator with skopt
Gilles Louppe, July 2016
Katie Malone, August 2016
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
If you are looking for a :obj:sklearn.model_selection.GridSearchCV replacement checkout
sphx_glr_auto_examples_sklearn-gridsearchcv-replacement.py instead.
Problem statement
Tuning the hyper-parameters of a machine learning model is often carried out
using an exhaustive exploration of (a subset of) the space all hyper-parameter
configurations (e.g., using :obj:sklearn.model_selection.GridSearchCV), which
often results in a very time consuming operation.
In this notebook, we illustrate how to couple :class:gp_minimize with sklearn's
estimators to tune hyper-parameters using sequential model-based optimisation,
hopefully resulting in equivalent or better solutions, but within less
evaluations.
Note: scikit-optimize provides a dedicated interface for estimator tuning via
:class:BayesSearchCV class which has a similar interface to those of
:obj:sklearn.model_selection.GridSearchCV. This class uses functions of skopt to perform hyperparameter
search efficiently. For example usage of this class, see
sphx_glr_auto_examples_sklearn-gridsearchcv-replacement.py
example notebook.
Can you write Python code for it?


I am working on the file "0.8/notebooks/auto_examples/interruptible-optimization.ipynb".
The first task is:
Interruptible optimization runs with checkpoints
Christian Schell, Mai 2018
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
Problem statement
Optimization runs can take a very long time and even run for multiple days.
If for some reason the process has to be interrupted results are irreversibly
lost, and the routine has to start over from the beginning.
With the help of the :class:callbacks.CheckpointSaver callback the optimizer's current state
can be saved after each iteration, allowing to restart from that point at any
time.
This is useful, for example,

if you don't know how long the process will take and cannot hog computational resources forever
if there might be system failures due to shaky infrastructure (or colleagues...)
if you want to adjust some parameters and continue with the already obtained results
Can you write Python code for it?

print(__doc__)
import sys
import numpy as np
np.random.seed(777)
import os


I am working on the file "0.8/notebooks/auto_examples/interruptible-optimization.ipynb".
The first task is:
Simple example
We will use pretty much the same optimization problem as in the
sphx_glr_auto_examples_bayesian-optimization.py
notebook. Additionally we will instantiate the :class:callbacks.CheckpointSaver
and pass it to the minimizer:
Can you write Python code for it?

from skopt import gp_minimize
from skopt import callbacks
from skopt.callbacks import CheckpointSaver

noise_level = 0.1

def obj_fun(x, noise_level=noise_level):
    return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2)) + np.random.randn() \
        * noise_level

checkpoint_saver = CheckpointSaver("./checkpoint.pkl", compress=9) # keyword arguments will be passed to `skopt.dump`

gp_minimize(obj_fun,            # the function to minimize
            [(-20.0, 20.0)],    # the bounds on each dimension of x
            x0=[-20.],          # the starting point
            acq_func="LCB",     # the acquisition function (optional)
            n_calls=10,         # number of evaluations of f including at x0
            n_random_starts=3,  # the number of random initial points
            callback=[checkpoint_saver],
            # a list of callbacks including the checkpoint saver
            random_state=777)


I am working on the file "0.8/notebooks/auto_examples/optimizer-with-different-base-estimator.ipynb".
The first task is:
GP kernel
Can you write Python code for it?

fig = plt.figure()
fig.suptitle("Standard GP kernel")
for i in range(10):
    next_x = opt_gp.ask()
    f_val = objective(next_x)
    res = opt_gp.tell(next_x, f_val)
    if i >= 5:
        plot_optimizer(res, n_iter=i-5, max_iters=5)
plt.tight_layout(rect=[0, 0.03, 1, 0.95])
plt.plot()


I am working on the file "0.8/notebooks/auto_examples/optimizer-with-different-base-estimator.ipynb".
The first task is:
Use different base estimators for optimization
Sigurd Carlen, September 2019.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
To use different base_estimator or create a regressor with different parameters,
we can create a regressor object and set it as kernel.
This example uses :class:plots.plot_gaussian_process which is available
since version 0.8.
Can you write Python code for it?

print(__doc__)

import numpy as np
np.random.seed(1234)
import matplotlib.pyplot as plt
from skopt.plots import plot_gaussian_process
from skopt import Optimizer


I am working on the file "0.8/notebooks/auto_examples/parallel-optimization.ipynb".
The first task is:
Example
A minimalistic example that uses joblib to parallelize evaluation of the
objective function is given below.
Can you write Python code for it?

from skopt import Optimizer
from skopt.space import Real
from joblib import Parallel, delayed
# example objective taken from skopt
from skopt.benchmarks import branin

optimizer = Optimizer(
    dimensions=[Real(-5.0, 10.0), Real(0.0, 15.0)],
    random_state=1,
    base_estimator='gp'
)

for i in range(10):
    x = optimizer.ask(n_points=4)  # x is a list of n_points points
    y = Parallel(n_jobs=4)(delayed(branin)(v) for v in x)  # evaluate points in parallel
    optimizer.tell(x, y)

# takes ~ 20 sec to get here
print(min(optimizer.yi))  # print the best objective found


I am working on the file "0.8/notebooks/auto_examples/parallel-optimization.ipynb".
The first task is:
Parallel optimization
Iaroslav Shcherbatyi, May 2017.
Reviewed by Manoj Kumar and Tim Head.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
Introduction
For many practical black box optimization problems expensive objective can be
evaluated in parallel at multiple points. This allows to get more objective
evaluations per unit of time, which reduces the time necessary to reach good
objective values when appropriate optimization algorithms are used, see for
example results in [1]_ and the references therein.
One such example task is a selection of number and activation function of a
neural network which results in highest accuracy for some machine learning
problem. For such task, multiple neural networks with different combinations
of number of neurons and activation function type can be evaluated at the same
time in parallel on different cpu cores / computational nodes.
The “ask and tell” API of scikit-optimize exposes functionality that allows to
obtain multiple points for evaluation in parallel. Intended usage of this
interface is as follows:

Initialize instance of the Optimizer class from skopt
Obtain n points for evaluation in parallel by calling the ask method of an optimizer instance with the n_points argument set to n > 0
Evaluate points
Provide points and corresponding objectives using the tell method of an optimizer instance
Continue from step 2 until eg maximum number of evaluations reached
Can you write Python code for it?


I am working on the file "0.8/notebooks/auto_examples/plots/partial-dependence-plot-with-categorical.ipynb".
The first task is:
Bayesian optimization
Can you write Python code for it?

SPACE = [
    Integer(1, 20, name='max_depth'),
    Integer(2, 100, name='min_samples_split'),
    Integer(5, 30, name='min_samples_leaf'),
    Integer(1, 30, name='max_features'),
    Categorical(list('abc'), name='dummy'),
    Categorical(['gini', 'entropy'], name='criterion'),
    Categorical(list('def'), name='dummy'),
]

result = gp_minimize(objective, SPACE, n_calls=20)


I am working on the file "0.8/notebooks/auto_examples/plots/partial-dependence-plot-with-categorical.ipynb".
The first task is:
Modify the shown minimum
Here we try with setting the other parameters to something other than
"result". When dealing with categorical dimensions we can't use
'expected_minimum'. Therefore we try with "expected_minimum_random"
which is a naive way of finding the minimum of the surrogate by only
using random sampling. n_minimum_search sets the number of random samples,
which is used to find the minimum
Can you write Python code for it?

_ = plot_objective(result, n_points=10, sample_source='expected_minimum_random',
                   minimum='expected_minimum_random', n_minimum_search=10000)


I am working on the file "0.8/notebooks/auto_examples/plots/partial-dependence-plot-with-categorical.ipynb".
The first task is:
Partial Dependence Plots  with categorical values
Sigurd Carlsen Feb 2019
Holger Nahrstaedt 2020
.. currentmodule:: skopt
Plot objective now supports optional use of partial dependence as well as
different methods of defining parameter values for dependency plots.
Can you write Python code for it?

print(__doc__)
import sys
from skopt.plots import plot_objective
from skopt import forest_minimize
import numpy as np
np.random.seed(123)
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from skopt.space import Integer, Categorical
from skopt import plots, gp_minimize
from skopt.plots import plot_objective


I am working on the file "0.8/notebooks/auto_examples/plots/partial-dependence-plot-with-categorical.ipynb".
The first task is:
Partial dependence plot
Here we see an example of using partial dependence. Even when setting
n_points all the way down to 10 from the default of 40, this method is
still very slow. This is because partial dependence calculates 250 extra
predictions for each point on the plots.
Can you write Python code for it?


I am working on the file "0.8/notebooks/auto_examples/plots/partial-dependence-plot-with-categorical.ipynb".
The first task is:
Plot without partial dependence
Here we plot without partial dependence. We see that it is a lot faster.
Also the values for the other parameters are set to the default "result"
which is the parameter set of the best observed value so far. In the case
of funny_func this is close to 0 for all parameters.
Can you write Python code for it?

_ = plot_objective(result,  sample_source='result', n_points=10)


I am working on the file "0.8/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
"expected_minimum_random" is a naive way of finding the minimum of the
surrogate by only using random sampling:
Can you write Python code for it?

_ = plot_objective(result, n_points=10, sample_source='expected_minimum_random',
                   minimum='expected_minimum_random')


I am working on the file "0.8/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
It is possible to change the location of the red dot, which normally shows
the position of the found minimum. We can set it 'expected_minimum',
which is the minimum value of the surrogate function, obtained by a
minimum search method.
Can you write Python code for it?

_ = plot_objective(result, n_points=10, minimum='expected_minimum')


I am working on the file "0.8/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
Optimisation using decision trees
We run forest_minimize on the function
Can you write Python code for it?

bounds = [(-1, 1.), ] * 3
n_calls = 150

result = forest_minimize(funny_func, bounds, n_calls=n_calls,
                         base_estimator="ET",
                         random_state=4)


I am working on the file "0.8/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
Partial dependence plot
Here we see an example of using partial dependence. Even when setting
n_points all the way down to 10 from the default of 40, this method is
still very slow. This is because partial dependence calculates 250 extra
predictions for each point on the plots.
Can you write Python code for it?


I am working on the file "0.8/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
Plot without partial dependence
Here we plot without partial dependence. We see that it is a lot faster.
Also the values for the other parameters are set to the default "result"
which is the parameter set of the best observed value so far. In the case
of funny_func this is close to 0 for all parameters.
Can you write Python code for it?


I am working on the file "0.8/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
Set a minimum location
Lastly we can also define these parameters ourself by parsing a list
as the minimum argument:
Can you write Python code for it?

_ = plot_objective(result, n_points=10, sample_source=[1, -0.5, 0.5],
                   minimum=[1, -0.5, 0.5])


I am working on the file "0.8/notebooks/auto_examples/plots/visualizing-results.ipynb".
The first task is:
Evaluating the objective function
Next we use an extra trees based minimizer to find one of the minima of the
:class:benchmarks.branin function. Then we visualize at which points the objective is being
evaluated using :class:plots.plot_evaluations.
Can you write Python code for it?

from functools import partial
from skopt.plots import plot_evaluations
from skopt import gp_minimize, forest_minimize, dummy_minimize

bounds = [(-5.0, 10.0), (0.0, 15.0)]
n_calls = 160

forest_res = forest_minimize(branin, bounds, n_calls=n_calls,
                             base_estimator="ET", random_state=4)

_ = plot_evaluations(forest_res, bins=10)


I am working on the file "0.8/notebooks/auto_examples/plots/visualizing-results.ipynb".
The first task is:
Going from 6 to 6+2 dimensions
To make things more interesting let's add two dimension to the problem.
As :class:benchmarks.hart6 only depends on six dimensions we know that for this problem
the new dimensions will be "flat" or uninformative. This is clearly visible
in both the placement of samples and the partial dependence plots.
Can you write Python code for it?

bounds = [(0., 1.),] * 8
n_calls = 200

forest_res = forest_minimize(hart6, bounds, n_calls=n_calls,
                             base_estimator="ET", random_state=4)

_ = plot_evaluations(forest_res)
_ = plot_objective(forest_res, n_samples=40)

# .. [Friedman (2001)] `doi:10.1214/aos/1013203451 section 8.2 <http://projecteuclid.org/euclid.aos/1013203451>`


I am working on the file "0.8/notebooks/auto_examples/plots/visualizing-results.ipynb".
The first task is:
Starting with branin
To start let's take advantage of the fact that :class:benchmarks.branin is a simple
function which can be visualised in two dimensions.
Can you write Python code for it?

from matplotlib.colors import LogNorm

def plot_branin():
    fig, ax = plt.subplots()

    x1_values = np.linspace(-5, 10, 100)
    x2_values = np.linspace(0, 15, 100)
    x_ax, y_ax = np.meshgrid(x1_values, x2_values)
    vals = np.c_[x_ax.ravel(), y_ax.ravel()]
    fx = np.reshape([branin(val) for val in vals], (100, 100))

    cm = ax.pcolormesh(x_ax, y_ax, fx,
                       norm=LogNorm(vmin=fx.min(),
                                    vmax=fx.max()),
                       cmap='viridis_r')

    minima = np.array([[-np.pi, 12.275], [+np.pi, 2.275], [9.42478, 2.475]])
    ax.plot(minima[:, 0], minima[:, 1], "r.", markersize=14,
            lw=0, label="Minima")

    cb = fig.colorbar(cm)
    cb.set_label("f(x)")

    ax.legend(loc="best", numpoints=1)

    ax.set_xlabel("$X_0$")
    ax.set_xlim([-5, 10])
    ax.set_ylabel("$X_1$")
    ax.set_ylim([0, 15])

plot_branin()


I am working on the file "0.8/notebooks/auto_examples/plots/visualizing-results.ipynb".
The first task is:
The two dimensional partial dependence plot can look like the true
objective but it does not have to. As points at which the objective function
is being evaluated are concentrated around the suspected minimum the
surrogate model sometimes is not a good representation of the objective far
away from the minima.
Random sampling
Compare this to a minimizer which picks points at random. There is no
structure visible in the order in which it evaluates the objective. Because
there is no model involved in the process of picking sample points at
random, we can not plot the partial dependence of the model.
Can you write Python code for it?

dummy_res = dummy_minimize(branin, bounds, n_calls=n_calls, random_state=4)

_ = plot_evaluations(dummy_res, bins=10)


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method-integer.ipynb".
The first task is:
Centered latin hypercube sampling
Can you write Python code for it?

lhs = Lhs(lhs_type="centered", criterion=None)
x = lhs.generate(space.dimensions, n_samples)
plot_searchspace(x, 'centered LHS')
print("empty fields: %d" % (36 - np.size(np.unique(x, axis=0), 0)))
pdist_data.append(pdist(x).flatten())
x_label.append("center")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method-integer.ipynb".
The first task is:
Classic latin hypercube sampling
Can you write Python code for it?

lhs = Lhs(lhs_type="classic", criterion=None)
x = lhs.generate(space.dimensions, n_samples)
plot_searchspace(x, 'classic LHS')
print("empty fields: %d" % (36 - np.size(np.unique(x, axis=0), 0)))
pdist_data.append(pdist(x).flatten())
x_label.append("lhs")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method-integer.ipynb".
The first task is:
Comparing initial sampling methods on integer space
Holger Nahrstaedt 2020 Sigurd Carlsen October 2019
.. currentmodule:: skopt
When doing baysian optimization we often want to reserve some of the
early part of the optimization to pure exploration. By default the
optimizer suggests purely random samples for the first n_initial_points
(10 by default). The downside to this is that there is no guarantee that
these samples are spread out evenly across all the dimensions.
Sampling methods as Latin hypercube, Sobol, Halton and Hammersly
take advantage of the fact that we know beforehand how many random
points we want to sample. Then these points can be "spread out" in
such a way that each dimension is explored.
See also the example on a real space
sphx_glr_auto_examples_initial_sampling_method.py
Can you write Python code for it?

print(__doc__)
import numpy as np
np.random.seed(1234)
import matplotlib.pyplot as plt
from skopt.space import Space
from skopt.sampler import Sobol
from skopt.sampler import Lhs
from skopt.sampler import Halton
from skopt.sampler import Hammersly
from skopt.sampler import Grid
from scipy.spatial.distance import pdist

def plot_searchspace(x, title):
    fig, ax = plt.subplots()
    plt.plot(np.array(x)[:, 0], np.array(x)[:, 1], 'bo', label='samples')
    plt.plot(np.array(x)[:, 0], np.array(x)[:, 1], 'bs', markersize=40, alpha=0.5)
    # ax.legend(loc="best", numpoints=1)
    ax.set_xlabel("X1")
    ax.set_xlim([0, 5])
    ax.set_ylabel("X2")
    ax.set_ylim([0, 5])
    plt.title(title)
    ax.grid(True)

n_samples = 10
space = Space([(0, 5), (0, 5)])


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method-integer.ipynb".
The first task is:
Correlation optimized hypercube sampling
Can you write Python code for it?

lhs = Lhs(criterion="correlation", iterations=10000)
x = lhs.generate(space.dimensions, n_samples)
plot_searchspace(x, 'correlation LHS')
print("empty fields: %d" % (36 - np.size(np.unique(x, axis=0), 0)))
pdist_data.append(pdist(x).flatten())
x_label.append("corr")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method-integer.ipynb".
The first task is:
Grid sampling
Can you write Python code for it?

grid = Grid(border="include", use_full_layout=False)
x = grid.generate(space.dimensions, n_samples)
plot_searchspace(x, 'Grid')
print("empty fields: %d" % (36 - np.size(np.unique(x, axis=0), 0)))
pdist_data.append(pdist(x).flatten())
x_label.append("grid")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method-integer.ipynb".
The first task is:
Halton sampling
Can you write Python code for it?

halton = Halton()
x = halton.generate(space.dimensions, n_samples)
plot_searchspace(x, 'Halton')
print("empty fields: %d" % (36 - np.size(np.unique(x, axis=0), 0)))
pdist_data.append(pdist(x).flatten())
x_label.append("halton")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method-integer.ipynb".
The first task is:
Hammersly sampling
Can you write Python code for it?

hammersly = Hammersly()
x = hammersly.generate(space.dimensions, n_samples)
plot_searchspace(x, 'Hammersly')
print("empty fields: %d" % (36 - np.size(np.unique(x, axis=0), 0)))
pdist_data.append(pdist(x).flatten())
x_label.append("hammersly")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method-integer.ipynb".
The first task is:
Maximin optimized hypercube sampling
Can you write Python code for it?

lhs = Lhs(criterion="maximin", iterations=10000)
x = lhs.generate(space.dimensions, n_samples)
plot_searchspace(x, 'maximin LHS')
print("empty fields: %d" % (36 - np.size(np.unique(x, axis=0), 0)))
pdist_data.append(pdist(x).flatten())
x_label.append("maximin")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method-integer.ipynb".
The first task is:
Pdist boxplot of all methods
This boxplot shows the distance between all generated points using
Euclidian distance. The higher the value, the better the sampling method.
It can be seen that random has the worst performance
Can you write Python code for it?

fig, ax = plt.subplots()
ax.boxplot(pdist_data)
plt.grid(True)
plt.ylabel("pdist")
_ = ax.set_ylim(0, 6)
_ = ax.set_xticklabels(x_label, rotation=45, fontsize=8)


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method-integer.ipynb".
The first task is:
Sobol
Can you write Python code for it?

sobol = Sobol()
x = sobol.generate(space.dimensions, n_samples)
plot_searchspace(x, 'Sobol')
print("empty fields: %d" % (36 - np.size(np.unique(x, axis=0), 0)))
pdist_data.append(pdist(x).flatten())
x_label.append("sobol")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method.ipynb".
The first task is:
Centered Latin hypercube sampling
Can you write Python code for it?

lhs = Lhs(lhs_type="centered", criterion=None)
x = lhs.generate(space.dimensions, n_samples)
plot_searchspace(x, 'centered LHS')
pdist_data.append(pdist(x).flatten())
x_label.append("center")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method.ipynb".
The first task is:
Correlation optimized hypercube sampling
Can you write Python code for it?

lhs = Lhs(criterion="correlation", iterations=10000)
x = lhs.generate(space.dimensions, n_samples)
plot_searchspace(x, 'correlation LHS')
pdist_data.append(pdist(x).flatten())
x_label.append("corr")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method.ipynb".
The first task is:
Grid sampling
Can you write Python code for it?

grid = Grid(border="include", use_full_layout=False)
x = grid.generate(space.dimensions, n_samples)
plot_searchspace(x, 'Grid')
pdist_data.append(pdist(x).flatten())
x_label.append("grid")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method.ipynb".
The first task is:
Halton sampling
Can you write Python code for it?

halton = Halton()
x = halton.generate(space.dimensions, n_samples)
plot_searchspace(x, 'Halton')
pdist_data.append(pdist(x).flatten())
x_label.append("halton")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method.ipynb".
The first task is:
Hammersly sampling
Can you write Python code for it?

hammersly = Hammersly()
x = hammersly.generate(space.dimensions, n_samples)
plot_searchspace(x, 'Hammersly')
pdist_data.append(pdist(x).flatten())
x_label.append("hammersly")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method.ipynb".
The first task is:
Maximin optimized hypercube sampling
Can you write Python code for it?

lhs = Lhs(criterion="maximin", iterations=10000)
x = lhs.generate(space.dimensions, n_samples)
plot_searchspace(x, 'maximin LHS')
pdist_data.append(pdist(x).flatten())
x_label.append("maximin")


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method.ipynb".
The first task is:
Pdist boxplot of all methods
This boxplot shows the distance between all generated points using
Euclidian distance. The higher the value, the better the sampling method.
It can be seen that random has the worst performance
Can you write Python code for it?

fig, ax = plt.subplots()
ax.boxplot(pdist_data)
plt.grid(True)
plt.ylabel("pdist")
_ = ax.set_ylim(0, 12)
_ = ax.set_xticklabels(x_label, rotation=45, fontsize=8)


I am working on the file "0.8/notebooks/auto_examples/sampler/initial-sampling-method.ipynb".
The first task is:
Random sampling
Can you write Python code for it?

x = space.rvs(n_samples)
plot_searchspace(x, "Random samples")
pdist_data = []
x_label = []
pdist_data.append(pdist(x).flatten())
x_label.append("random")


I am working on the file "0.8/notebooks/auto_examples/sampler/sampling_comparison.ipynb".
The first task is:
Comparing initial point generation methods
Holger Nahrstaedt 2020
.. currentmodule:: skopt
Bayesian optimization or sequential model-based optimization uses a surrogate
model to model the expensive to evaluate function func. There are several
choices for what kind of surrogate model to use. This notebook compares the
performance of:

Halton sequence,
Hammersly sequence,
Sobol sequence and
Latin hypercube sampling

as initial points. The purely random point generation is used as
a baseline.
Can you write Python code for it?

print(__doc__)
import numpy as np
np.random.seed(123)
import matplotlib.pyplot as plt


I am working on the file "0.8/notebooks/auto_examples/sampler/sampling_comparison.ipynb".
The first task is:
Toy model
We will use the :class:benchmarks.hart6 function as toy model for the expensive function.
In a real world application this function would be unknown and expensive
to evaluate.
Can you write Python code for it?

from skopt.benchmarks import hart6 as hart6_
# redefined `hart6` to allow adding arbitrary "noise" dimensions
def hart6(x, noise_level=0.):
    return hart6_(x[:6]) + noise_level * np.random.randn()

from skopt.benchmarks import branin as _branin

def branin(x, noise_level=0.):
    return _branin(x) + noise_level * np.random.randn()

from matplotlib.pyplot import cm
import time
from skopt import gp_minimize, forest_minimize, dummy_minimize

def plot_convergence(result_list, true_minimum=None, yscale=None, title="Convergence plot"):

    ax = plt.gca()
    ax.set_title(title)
    ax.set_xlabel("Number of calls $n$")
    ax.set_ylabel(r"$\min f(x)$ after $n$ calls")
    ax.grid()
    if yscale is not None:
        ax.set_yscale(yscale)
    colors = cm.hsv(np.linspace(0.25, 1.0, len(result_list)))

    for results, color in zip(result_list, colors):
        name, results = results
        n_calls = len(results[0].x_iters)
        iterations = range(1, n_calls + 1)
        mins = [[np.min(r.func_vals[:i]) for i in iterations]
                for r in results]
        ax.plot(iterations, np.mean(mins, axis=0), c=color, label=name)
        #ax.errorbar(iterations, np.mean(mins, axis=0),
        #             yerr=np.std(mins, axis=0), c=color, label=name)
    if true_minimum:
        ax.axhline(true_minimum, linestyle="--",
                   color="r", lw=1,
                   label="True minimum")
    ax.legend(loc="best")
    return ax

def run(minimizer, initial_point_generator,
        n_initial_points=10, n_repeats=1):
    return [minimizer(func, bounds, n_initial_points=n_initial_points,
                      initial_point_generator=initial_point_generator,
                      n_calls=n_calls, random_state=n)
            for n in range(n_repeats)]

def run_measure(initial_point_generator, n_initial_points=10):
    start = time.time()
    # n_repeats must set to a much higher value to obtain meaningful results.
    n_repeats = 1
    res = run(gp_minimize, initial_point_generator,
             ...


I am working on the file "0.8/notebooks/auto_examples/sklearn-gridsearchcv-replacement.ipynb".
The first task is:
Advanced example
In practice, one wants to enumerate over multiple predictive model classes,
with different search spaces and number of evaluations per class. An
example of such search over parameters of Linear SVM, Kernel SVM, and
decision trees is given below.
Can you write Python code for it?

from skopt import BayesSearchCV
from skopt.space import Real, Categorical, Integer
from skopt.plots import plot_objective, plot_histogram

from sklearn.datasets import load_digits
from sklearn.svm import LinearSVC, SVC
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split

X, y = load_digits(n_class=10, return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

# pipeline class is used as estimator to enable
# search over different model types
pipe = Pipeline([
    ('model', SVC())
])

# single categorical value of 'model' parameter is
# sets the model class
# We will get ConvergenceWarnings because the problem is not well-conditioned.
# But that's fine, this is just an example.
linsvc_search = {
    'model': [LinearSVC(max_iter=1000)],
    'model__C': (1e-6, 1e+6, 'log-uniform'),
}

# explicit dimension classes can be specified like this
svc_search = {
    'model': Categorical([SVC()]),
    'model__C': Real(1e-6, 1e+6, prior='log-uniform'),
    'model__gamma': Real(1e-6, 1e+1, prior='log-uniform'),
    'model__degree': Integer(1,8),
    'model__kernel': Categorical(['linear', 'poly', 'rbf']),
}

opt = BayesSearchCV(
    pipe,
    # (parameter space, # of evaluations)
    [(svc_search, 40), (linsvc_search, 16)],
    cv=3
)

opt.fit(X_train, y_train)

print("val. score: %s" % opt.best_score_)
print("test score: %s" % opt.score(X_test, y_test))
print("best params: %s" % str(opt.best_params_))


I am working on the file "0.8/notebooks/auto_examples/sklearn-gridsearchcv-replacement.ipynb".
The first task is:
Counting total iterations that will be used to explore all subspaces
Subspaces in previous examples can further increase in complexity if you add
new model subspaces or dimensions for feature extraction pipelines. For
monitoring of progress, you would like to know the total number of
iterations it will take to explore all subspaces. This can be
calculated with total_iterations property, as in the code below.
Can you write Python code for it?

from skopt import BayesSearchCV

from sklearn.datasets import load_iris
from sklearn.svm import SVC

X, y = load_iris(True)

searchcv = BayesSearchCV(
    SVC(),
    search_spaces=[
        ({'C': (0.1, 1.0)}, 19),  # 19 iterations for this subspace
        {'gamma':(0.1, 1.0)}
    ],
    n_iter=23
)

print(searchcv.total_iterations)


I am working on the file "0.8/notebooks/auto_examples/sklearn-gridsearchcv-replacement.ipynb".
The first task is:
Minimal example
A minimal example of optimizing hyperparameters of SVC (Support Vector machine Classifier) is given below.
Can you write Python code for it?

from skopt import BayesSearchCV
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split

X, y = load_digits(n_class=10, return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.75, test_size=.25, random_state=0)

# log-uniform: understand as search over p = exp(x) by varying x
opt = BayesSearchCV(
    SVC(),
    {
        'C': (1e-6, 1e+6, 'log-uniform'),
        'gamma': (1e-6, 1e+1, 'log-uniform'),
        'degree': (1, 8),  # integer valued parameter
        'kernel': ['linear', 'poly', 'rbf'],  # categorical parameter
    },
    n_iter=32,
    cv=3
)

opt.fit(X_train, y_train)

print("val. score: %s" % opt.best_score_)
print("test score: %s" % opt.score(X_test, y_test))


I am working on the file "0.8/notebooks/auto_examples/sklearn-gridsearchcv-replacement.ipynb".
The first task is:
Partial Dependence plot of the objective function for SVC
Can you write Python code for it?

_ = plot_objective(opt.optimizer_results_[0],
                   dimensions=["C", "degree", "gamma", "kernel"],
                   n_minimum_search=int(1e8))
plt.show()


I am working on the file "0.8/notebooks/auto_examples/sklearn-gridsearchcv-replacement.ipynb".
The first task is:
Progress monitoring and control using callback argument of fit method
It is possible to monitor the progress of :class:BayesSearchCV with an event
handler that is called on every step of subspace exploration. For single job
mode, this is called on every evaluation of model configuration, and for
parallel mode, this is called when n_jobs model configurations are evaluated
in parallel.
Additionally, exploration can be stopped if the callback returns True.
This can be used to stop the exploration early, for instance when the
accuracy that you get is sufficiently high.
An example usage is shown below.
Can you write Python code for it?

from skopt import BayesSearchCV

from sklearn.datasets import load_iris
from sklearn.svm import SVC

X, y = load_iris(return_X_y=True)

searchcv = BayesSearchCV(
    SVC(gamma='scale'),
    search_spaces={'C': (0.01, 100.0, 'log-uniform')},
    n_iter=10,
    cv=3
)

# callback handler
def on_step(optim_result):
    score = searchcv.best_score_
    print("best score: %s" % score)
    if score >= 0.98:
        print('Interrupting!')
        return True

searchcv.fit(X, y, callback=on_step)


I am working on the file "0.8/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
And load from file using :class:skopt.load:
Can you write Python code for it?

res_loaded = load('result.pkl')

res_loaded.fun


I am working on the file "0.8/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
As long as your Python session is active, you can access all the
optimization results via the res object.
So how can you store this data in a file? skopt conveniently provides
functions :class:skopt.dump and :class:skopt.load that handle this for you.
These functions are essentially thin wrappers around the
joblib <https://joblib.readthedocs.io/en/latest/>_ module's :obj:joblib.dump and :obj:joblib.load.
We will now show how to use :class:skopt.dump and :class:skopt.load for storing
and loading results.
Using skopt.dump() and skopt.load()
For storing optimization results into a file, call the :class:skopt.dump
function:
Can you write Python code for it?

from skopt import dump, load

dump(res, 'result.pkl')


I am working on the file "0.8/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
Notice that the entry 'func' is absent in the loaded object but is still
present in the local variable:
Can you write Python code for it?

res_loaded_without_objective = load('result_without_objective.pkl')

print('Loaded object: ', res_loaded_without_objective.specs['args'].keys())
print('Local variable:', res.specs['args'].keys())


I am working on the file "0.8/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
Possible problems

Python versions incompatibility: In general, objects serialized in
  Python 2 cannot be deserialized in Python 3 and vice versa.
Security issues: Once again, do not load any files from untrusted
  sources.
Extremely large results objects: If your optimization results object

is extremely large, calling :class:skopt.dump with store_objective=False might
cause performance issues. This is due to creation of a deep copy without the
objective function. If the objective function it is not critical to you, you
can simply delete it before calling :class:skopt.dump. In this case, no deep
copy is created:
Can you write Python code for it?

del res.specs['args']['func']

dump(res, 'result_without_objective_2.pkl')


I am working on the file "0.8/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
Simple example
We will use the same optimization problem as in the
sphx_glr_auto_examples_bayesian-optimization.py notebook:
Can you write Python code for it?

from skopt import gp_minimize
noise_level = 0.1

def obj_fun(x, noise_level=noise_level):
    return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2)) + np.random.randn() \
        * noise_level

res = gp_minimize(obj_fun,            # the function to minimize
                  [(-2.0, 2.0)],      # the bounds on each dimension of x
                  x0=[0.],            # the starting point
                  acq_func="LCB",     # the acquisition function (optional)
                  n_calls=15,         # the number of evaluations of f including at x0
                  n_random_starts=3,  # the number of random initial points
                  random_state=777)


I am working on the file "0.8/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
Unserializable objective functions
Notice that if your objective function is non-trivial (e.g. it calls MATLAB
engine from Python), it might be not serializable and :class:skopt.dump will
raise an exception when you try to store the optimization results.
In this case you should disable storing the objective function by calling
:class:skopt.dump with the keyword argument store_objective=False:
Can you write Python code for it?


I am working on the file "0.8/notebooks/auto_examples/store-and-load-results.ipynb".
The first task is:
You can fine-tune the serialization and deserialization process by calling
:class:skopt.dump and :class:skopt.load with additional keyword arguments. See the
joblib <https://joblib.readthedocs.io/en/latest/>_ documentation
:obj:joblib.dump and
:obj:joblib.load for the additional parameters.
For instance, you can specify the compression algorithm and compression
level (highest in this case):
Can you write Python code for it?

dump(res, 'result.gz', compress=9)

from os.path import getsize
print('Without compression: {} bytes'.format(getsize('result.pkl')))
print('Compressed with gz:  {} bytes'.format(getsize('result.gz')))


I am working on the file "0.8/notebooks/auto_examples/strategy-comparison.ipynb".
The first task is:
Comparing surrogate models
Tim Head, July 2016.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
Bayesian optimization or sequential model-based optimization uses a surrogate
model to model the expensive to evaluate function func. There are several
choices for what kind of surrogate model to use. This notebook compares the
performance of:

gaussian processes,
extra trees, and
random forests

as surrogate models. A purely random optimization strategy is also used as
a baseline.
Can you write Python code for it?


I am working on the file "0.8/notebooks/auto_examples/strategy-comparison.ipynb".
The first task is:
Note that this can take a few minutes.
Can you write Python code for it?

from skopt.plots import plot_convergence

plot = plot_convergence(("dummy_minimize", dummy_res),
                        ("gp_minimize", gp_res),
                        ("forest_minimize('rf')", rf_res),
                        ("forest_minimize('et)", et_res),
                        true_minimum=0.397887, yscale="log")

plot.legend(loc="best", prop={'size': 6}, numpoints=1)


I am working on the file "0.8/notebooks/auto_examples/strategy-comparison.ipynb".
The first task is:
This shows the value of the two-dimensional branin function and
the three minima.
Objective
The objective of this example is to find one of these minima in as
few iterations as possible. One iteration is defined as one call
to the :class:benchmarks.branin function.
We will evaluate each model several times using a different seed for the
random number generator. Then compare the average performance of these
models. This makes the comparison more robust against models that get
"lucky".
Can you write Python code for it?

from functools import partial
from skopt import gp_minimize, forest_minimize, dummy_minimize

func = partial(branin, noise_level=2.0)
bounds = [(-5.0, 10.0), (0.0, 15.0)]
n_calls = 60

def run(minimizer, n_iter=5):
    return [minimizer(func, bounds, n_calls=n_calls, random_state=n)
            for n in range(n_iter)]

# Random search
dummy_res = run(dummy_minimize)

# Gaussian processes
gp_res = run(gp_minimize)

# Random forest
rf_res = run(partial(forest_minimize, base_estimator="RF"))

# Extra trees
et_res = run(partial(forest_minimize, base_estimator="ET"))


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
(If you're typing this into an IPython notebook, or otherwise using notebook file, you hit shift-Enter to evaluate a cell.)
There are some gotchas compared to using a normal calculator.
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
(Quick quiz: why did the snippet print "Go to work" here? What is the variable "day" set to?)
Let's take the snippet apart to see what happened. First, note the statement
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
But not both at the same time, unless you want one of the symbols to be part of the string.
Can you write Python code for it?

"He's a Rebel"

'She asked, "How are you today?"'


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Dictionaries are an object called "mappings" or "associative arrays" in other languages. Whereas a list associates an integer index with a set of objects:
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Don't forget the space between the strings, if you want one there.
Can you write Python code for it?

statement = "Hello, " + "World!"
print statement


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Here we did two things with tuples you haven't seen before. First, we unpacked an object into a set of named variables using tuple assignment:
>>> name,x,y = obj

We also returned multiple values (minx,miny), which were then assigned to two other variables (x,y), again by tuple assignment. This makes what would have been complicated code in C++ rather simple.
Tuple assignment is also a convenient way to swap variables:
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
However, if we did want to write a function ourselves, we could do recursively by noting that
$$ n! = n(n-1)!$$
The program then looks something like:
Can you write Python code for it?

def fact(n):
    if n <= 0:
        return 1
    return n*fact(n-1)

fact(20)


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
If statements can have elif parts ("else if"), in addition to if/else parts. For example:
Can you write Python code for it?

if day == "Sunday":
    print "Sleep in"
elif day == "Saturday":
    print "Do chores"
else:
    print "Go to work"


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
If we evaluate it by itself, as we just did, we see that it returns a boolean value, False. The "==" operator performs equality testing. If the two items are equal, it returns True, otherwise it returns False. In this case, it is comparing two variables, the string "Sunday", and whatever is stored in the variable "day", which, in this case, is the other string "Saturday". Since the two strings are not equal to each other, the truth test has the false value.
The if statement that contains the truth test is followed by a code block (a colon followed by an indented block of code). If the boolean is true, it executes the code in that block. Since it is false in the above example, we don't see that code executed.
The first block of code is followed by an else statement, which is executed if nothing else in the above if statement is true. Since the value was false, this code is executed, which is why we see "Go to work".
You can compare any data types in Python:
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
If we want the list containing the first two elements of a list, we can do this via
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
If you have a lot of words to concatenate together, there are other, more efficient ways to do this. But this is fine for linking a few strings together.
Lists
Very often in a programming language, one wants to keep a group of similar items together. Python does this using a data type called lists.
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
In the last few lines, we have sped by a lot of things that we should stop for a moment and explore a little more fully. We've seen, however briefly, two different data types: integers, also known as whole numbers to the non-programming world, and floating point numbers, also known (incorrectly) as decimal numbers to the rest of the world.
We've also seen the first instance of an import statement. Python has a huge number of libraries included with the distribution. To keep things simple, most of these variables and functions are not accessible from a normal Python interactive session. Instead, you have to import the name. For example, there is a math module containing many useful functions. To access, say, the square root function, you can either first
from math import sqrt

and then call sqrt(81)
or you can simply import the math library itself
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Iteration, Indentation, and Blocks
One of the most useful things you can do with lists is to iterate through them, i.e. to go through each element one at a time. To do this in Python, we use the for statement:
Can you write Python code for it?

for day in days_of_the_week:
    print day


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Lists do not have to hold the same data type. For example,
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Note that in this example I was even able to omit the second argument, so that the slice started at 2, went to the end of the list, and took every second element, to generate the list of even numbers less that 40.
Booleans and Truth Testing
We have now learned a few data types. We have integers and floating point numbers, strings, and lists to contain them. We have also learned about lists, a container that can hold any data type. We have learned to print things out, and to iterate over items in lists. We will now learn about boolean variables that can be either True or False.
We invariably need some concept of conditions in programming to control branching behavior, to allow a program to react differently to different situations. If it's Monday, I'll go to work, but if it's Sunday, I'll sleep in. To do this in Python, we use a combination of boolean variables, which evaluate to either True or False, and if statements, that control branching based on boolean values.
For example:
Can you write Python code for it?

if day == "Sunday":
    print "Sleep in"
else:
    print "Go to work"


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Of course we can combine if statements with for loops, to make a snippet that is almost interesting:
Can you write Python code for it?

for day in days_of_the_week:
    statement = "Today is " + day
    print statement
    if day == "Sunday":
        print "   Sleep in"
    elif day == "Saturday":
        print "   Do chores"
    else:
        print "   Go to work"


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Plotting with Matplotlib
We can generally understand trends in data by using a plotting program to chart it. Python has a wonderful plotting library called Matplotlib. The IPython notebook interface we are using for these notes has that functionality built in.
As an example, we have looked at two different functions, the Fibonacci function, and the factorial function, both of which grow faster than polynomially. Which one grows the fastest? Let's plot them. First, let's generate the Fibonacci sequence of length 20:
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Python lists, like C, but unlike Fortran, use 0 as the index of the first element of a list. Thus, in this example, the 0 element is "Sunday", 1 is "Monday", and so on. If you need to access the nth element from the end of the list, you can use a negative index. For example, the -1 element of a list is the last element:
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Recursion can be very elegant, and can lead to very simple programs.
Two More Data Structures: Tuples and Dictionaries
Before we end the Python overview, I wanted to touch on two more data structures that are very useful (and thus very common) in Python programs.
A tuple is a sequence object like a list or a string. It's constructed by grouping a sequence of objects together with commas, either without brackets, or with parentheses:
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Slicing
Lists and strings have something in common that you might not suspect: they can both be treated as sequences. You already know that you can iterate through the elements of a list. You can also iterate through the letters in a string:
Can you write Python code for it?

for letter in "Sunday":
    print letter


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
The factorial function grows much faster. In fact, you can't even see the Fibonacci sequence. It's not entirely surprising: a function where we multiply by n each iteration is bound to grow faster than one where we add (roughly) n each iteration.
Let's plot these on a semilog plot so we can see them both a little more clearly:
Can you write Python code for it?

semilogy(facts,label="factorial")
semilogy(fibs,label="Fibonacci")
xlabel("n")
legend()


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
The lists created above with range have a step of 1 between elements. You can also give a fixed step size via a third command:
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
This code snippet goes through each element of the list called days_of_the_week and assigns it to the variable day. It then executes everything in the indented block (in this case only one line of code, the print statement) using those variable assignments. When the program has gone through every element of the list, it exists the block.
(Almost) every programming language defines blocks of code in some way. In Fortran, one uses END statements (ENDDO, ENDIF, etc.) to define code blocks. In C, C++, and Perl, one uses curly braces {} to define these blocks.
Python uses a colon (":"), followed by indentation level to define code blocks. Everything at a higher level of indentation is taken to be in the same block. In the above example the block was only a single line, but we could have had longer blocks as well:
Can you write Python code for it?

for day in days_of_the_week:
    statement = "Today is " + day
    print statement


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
This is clearly what we want.
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
This is something of an advanced topic, but ordinary data types have boolean values associated with them, and, indeed, in early versions of Python there was not a separate boolean object. Essentially, anything that was a 0 value (the integer or floating point 0, an empty string "", or an empty list []) was False, and everything else was true. You can see the boolean value of any data object using the bool() function.
Can you write Python code for it?

bool(1)

bool(0)

bool(["This "," is "," a "," list"])


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Tuples are useful anytime you want to group different pieces of data together in an object, but don't want to create a full-fledged class (see below) for them. For example, let's say you want the Cartesian coordinates of some objects in your program. Tuples are a good way to do this:
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
Tuples can be used when functions return more than one value. Say we wanted to compute the smallest x- and y-coordinates of the above list of objects. We could write:
Can you write Python code for it?

def minmax(objects):
    minx = 1e20 # These are set to really big numbers
    miny = 1e20
    for obj in objects:
        name,x,y = obj
        if x < minx:
            minx = x
        if y < miny:
            miny = y
    return minx,miny

x,y = minmax(positions)
print x,y


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
We can do boolean tests on lists as well:
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
We see a few other boolean operators here, all of which which should be self-explanatory. Less than, equality, non-equality, and so on.
Particularly interesting is the 1 == 1.0 test, which is true, since even though the two objects are different data types (integer and floating point number), they have the same value. There is another boolean operator is, that tests whether two objects are the same object:
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
We've introduced a several new features here. First, note that the function itself is defined as a code block (a colon followed by an indented block). This is the standard way that Python delimits things. Next, note that the first line of the function is a single string. This is called a docstring, and is a special kind of comment that is often available to people using the function through the python command line:
Can you write Python code for it?


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
You can add additional items to the list using the .append() command:
Can you write Python code for it?

languages = ["Fortran","C","C++"]
languages.append("Python")
print languages


I am working on the file "0/Introduction to Python.ipynb".
The first task is:
You can define variables using the equals (=) sign:
Can you write Python code for it?

width = 20
length = 30
area = length*width
area


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
As well as Jacobi, Laguerre, Hermite polynomials, Hypergeometric functions, and many others. There's a full listing at the Scipy Special Functions Page.
Least squares fitting
Very often we deal with some data that we want to fit to some sort of expected behavior. Say we have the following:
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
Example: Finite Differences
Now that we have these tools in our toolbox, we can start to do some cool stuff with it. Many of the equations we want to solve in Physics involve differential equations. We want to be able to compute the derivative of functions:
$$ y' = \frac{y(x+h)-y(x)}{h} $$
by discretizing the function $y(x)$ on an evenly spaced set of points $x_0, x_1, \dots, x_n$, yielding $y_0, y_1, \dots, y_n$. Using the discretization, we can approximate the derivative by
$$ y_i' \approx \frac{y_{i+1}-y_{i-1}}{x_{i+1}-x_{i-1}} $$
We can write a derivative function in Python via
Can you write Python code for it?

def nderiv(y,x):
    "Finite difference derivative of the function f"
    n = len(y)
    d = zeros(n,'d') # assume double
    # Use centered differences for the interior points, one-sided differences for the ends
    for i in range(1,n-1):
        d[i] = (y[i+1]-y[i-1])/(x[i+1]-x[i-1])
    d[0] = (y[1]-y[0])/(x[1]-x[0])
    d[n-1] = (y[n-1]-y[n-2])/(x[n-1]-x[n-2])
    return d


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
For a pure exponential decay like this, we can fit the log of the data to a straight line. The above plot suggests this is a good approximation. Given a function
$$ y = Ae^{-ax} $$
$$ \log(y) = \log(A) - ax$$
Thus, if we fit the log of the data versus x, we should get a straight line with slope $a$, and an intercept that gives the constant $A$.
There's a numpy function called polyfit that will fit data to a polynomial form. We'll use this to fit to a straight line (a polynomial of order 1)
Can you write Python code for it?

params = polyfit(data[:,0],log(data[:,1]),1)
a = params[0]
A = exp(params[1])


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
If we have more complicated functions, we may not be able to get away with fitting to a simple polynomial. Consider the following data:
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
If you provide a third argument, it takes that as the number of points in the space. If you don't provide the argument, it gives a length 50 linear space.
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
It is generally more efficient to generate a list of random numbers all at once, particularly if you're drawing from a non-uniform distribution. Numpy has functions to generate vectors and matrices of particular types of random distributions.
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
Let's compare the first function to our solution.
Can you write Python code for it?

plot(x,ho_evec(x,0,1,1),label="Analytic")
plot(x,-U[:,0]/sqrt(h),label="Numeric")
xlabel('x (bohr)')
ylabel(r'$\psi(x)$')
title("Comparison of numeric and analytic solutions to the Harmonic Oscillator")
legend()


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
Let's see whether this curve fits the data.
Can you write Python code for it?

x = linspace(1,45)
title("Raw Data")
xlabel("Distance")
semilogy(data[:,0],data[:,1],'bo')
semilogy(x,A*exp(a*x),'b-')


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
One of the first programs I ever wrote was a program to compute $\pi$ by taking random numbers as x and y coordinates, and counting how many of them were in the unit circle. For example:
Can you write Python code for it?

npts = 5000
xs = 2*rand(npts)-1
ys = 2*rand(npts)-1
r = xs**2+ys**2
ninside = (r<1).sum()
figsize(6,6) # make the figure square
title("Approximation to pi = %f" % (4*ninside/float(npts)))
plot(xs[r<1],ys[r<1],'b.')
plot(xs[r>1],ys[r>1],'r.')
figsize(8,6) # change the figsize back to 4x3 for the rest of the notebook


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
Pretty close!
One-Dimensional Harmonic Oscillator using Finite Difference
Now that we've convinced ourselves that finite differences aren't a terrible approximation, let's see if we can use this to solve the one-dimensional harmonic oscillator.
We want to solve the time-independent Schrodinger equation
$$ -\frac{\hbar^2}{2m}\frac{\partial^2\psi(x)}{\partial x^2} + V(x)\psi(x) = E\psi(x)$$
for $\psi(x)$ when $V(x)=\frac{1}{2}m\omega^2x^2$ is the harmonic oscillator potential. We're going to use the standard trick to transform the differential equation into a matrix equation by multiplying both sides by $\psi^*(x)$ and integrating over $x$. This yields
$$ -\frac{\hbar}{2m}\int\psi(x)\frac{\partial^2}{\partial x^2}\psi(x)dx + \int\psi(x)V(x)\psi(x)dx = E$$
We will again use the finite difference approximation. The finite difference formula for the second derivative is
$$ y'' = \frac{y_{i+1}-2y_i+y_{i-1}}{x_{i+1}-x_{i-1}} $$
We can think of the first term in the Schrodinger equation as the overlap of the wave function $\psi(x)$ with the second derivative of the wave function $\frac{\partial^2}{\partial x^2}\psi(x)$. Given the above expression for the second derivative, we can see if we take the overlap of the states $y_1,\dots,y_n$ with the second derivative, we will only have three points where the overlap is nonzero, at $y_{i-1}$, $y_i$, and $y_{i+1}$. In matrix form, this leads to the tridiagonal Laplacian matrix, which has -2's along the diagonals, and 1's along the diagonals above and below the main diagonal.
The second term turns leads to a diagonal matrix with $V(x_i)$ on the diagonal elements. Putting all of these pieces together, we get:
Can you write Python code for it?

def Laplacian(x):
    h = x[1]-x[0] # assume uniformly spaced points
    n = len(x)
    M = -2*identity(n,'d')
    for i in range(1,n):
        M[i,i-1] = M[i-1,i] = 1
    return M/h**2

x = linspace(-3,3)
m = 1.0
ohm = 1.0
T = (-0.5/m)*Laplacian(x)
V = 0.5*(ohm**2)*(x**2)
H =  T + diag(V)
E,U = eigh(H)
h = x[1]-x[0]

# Plot the Harmonic potential
plot(x,V,color='k')

for i in range(4):
    # For each of the first few solutions, plot the energy level:
    axhline(y=E[i],color='k',ls=":")
    # as well as the eigenfunction, displaced by the energy level so they don't
    # all pile up on each other:
    plot(x,-U[:,i]/sqrt(h)+E[i])
title("Eigenfunctions of the Quantum Harmonic Oscillator")
xlabel("Displacement (bohr)")
ylabel("Energy (hartree)")


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
Scipy has a numerical integration routine quad (since sometimes numerical integration is called quadrature), that we can use for this:
Can you write Python code for it?

from scipy.integrate import quad
quad(f,0,inf)


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
Something that confuses Matlab users is that the times (*) operator give element-wise multiplication rather than matrix multiplication:
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
The agreement is almost exact.
We can use the subplot command to put multiple comparisons in different panes on a single plot:
Can you write Python code for it?

phase_correction = [-1,1,1,-1,-1,1]
for i in range(6):
    subplot(2,3,i+1)
    plot(x,ho_evec(x,i,1,1),label="Analytic")
    plot(x,phase_correction[i]*U[:,i]/sqrt(h),label="Numeric")


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
The curve_fit routine we just used is built on top of a very good general minimization capability in Scipy. You can learn more at the scipy documentation pages.
Monte Carlo, random numbers, and computing $\pi$
Many methods in scientific computing rely on Monte Carlo integration, where a sequence of (pseudo) random numbers are used to approximate the integral of a function. Python has good random number generators in the standard library. The random() function gives pseudorandom numbers uniformly distributed between 0 and 1:
Can you write Python code for it?

from random import random
rands = []
for i in range(100):
    rands.append(random())
plot(rands)


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
There are also 2d and 3d numerical integrators in Scipy. See the docs for more information.
Fast Fourier Transform and Signal Processing
Very often we want to use FFT techniques to help obtain the signal from noisy data. Scipy has several different options for this.
Can you write Python code for it?

from scipy.fftpack import fft,fftfreq

npts = 4000
nplot = npts/10
t = linspace(0,120,npts)
def acc(t): return 10*sin(2*pi*2.0*t) + 5*sin(2*pi*8.0*t) + 2*rand(npts)

signal = acc(t)

FFT = abs(fft(signal))
freqs = fftfreq(npts, t[1]-t[0])

subplot(211)
plot(t[:nplot], signal[:nplot])
subplot(212)
plot(freqs,20*log10(FFT),',')
show()


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
There's also a diag() function that takes a list or a vector and puts it along the diagonal of a square matrix.
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
There's also an identity command that behaves as you'd expect:
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
To build matrices, you can either use the array command with lists of lists:
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
We've made a couple of hacks here to get the orbitals the way we want them. First, I inserted a -1 factor before the wave functions, to fix the phase of the lowest state. The phase (sign) of a quantum wave function doesn't hold any information, only the square of the wave function does, so this doesn't really change anything.
But the eigenfunctions as we generate them aren't properly normalized. The reason is that finite difference isn't a real basis in the quantum mechanical sense. It's a basis of Dirac δ functions at each point; we interpret the space betwen the points as being "filled" by the wave function, but the finite difference basis only has the solution being at the points themselves. We can fix this by dividing the eigenfunctions of our finite difference Hamiltonian by the square root of the spacing, and this gives properly normalized functions.
Special Functions
The solutions to the Harmonic Oscillator are supposed to be Hermite polynomials. The Wikipedia page has the HO states given by
$$\psi_n(x) = \frac{1}{\sqrt{2^n n!}}
\left(\frac{m\omega}{\pi\hbar}\right)^{1/4}
\exp\left(-\frac{m\omega x^2}{2\hbar}\right)
H_n\left(\sqrt{\frac{m\omega}{\hbar}}x\right)$$
Let's see whether they look like those. There are some special functions in the Numpy library, and some more in Scipy. Hermite Polynomials are in Numpy:
Can you write Python code for it?

from numpy.polynomial.hermite import Hermite
def ho_evec(x,n,m,ohm):
    vec = [0]*9
    vec[n] = 1
    Hn = Hermite(vec)
    return (1/sqrt(2**n*math.factorial(n)))*pow(m*ohm/pi,0.25)*exp(-0.5*m*ohm*x**2)*Hn(x*sqrt(m*ohm))


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
You can also form empty (zero) matrices of arbitrary shape (including vectors, which Numpy treats as vectors with one row), using the zeros command:
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
You can pass in a second argument to array that gives the numeric type. There are a number of types listed here that your matrix can be. Some of these are aliased to single character codes. The most common ones are 'd' (double precision floating point number), 'D' (double precision complex number), and 'i' (int32). Thus,
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
as well as matrix-vector products.
There are determinant, inverse, and transpose functions that act as you would suppose. Transpose can be abbreviated with ".T" at the end of a matrix object:
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
as well as when you add two matrices together. (However, the matrices have to be the same shape.)
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
dot can also do dot products (duh!):
Can you write Python code for it?


I am working on the file "0/More on Python Tools.ipynb".
The first task is:
linspace is an easy way to make coordinates for plotting. Functions in the numpy library (all of which are imported into IPython notebook) can act on an entire vector (or even a matrix) of points at once. Thus,
Can you write Python code for it?


I am working on the file "0/Pandas Intro.ipynb".
The first task is:
As you can see the Births column is of type int64, thus no floats (decimal numbers) or alpha numeric characters will be present in this column.
Analyze Data
To find the most popular name or the baby name with the higest birth rate, we can do one of the following.

Sort the dataframe and select the top row
Use the max() attribute to find the maximum value
Can you write Python code for it?

# Method 1:
Sorted = df.sort_values(['Births'], ascending=False)
Sorted.head(1)

# Method 2:
df['Births'].max()