I am working on the file "2.3/tutorials/plotting.ipynb".
The first task is:
And run a forward model.  See Computing Observables for more details.
Can you write Python code for it?


I am working on the file "2.3/tutorials/plotting.ipynb".
The first task is:
Any call to plot returns two objects - the autofig and matplotlib figure instances.  Generally we won't need to do anything with these, but having them returned could come in handy if you want to manually edit either before drawing/saving the image.
In this example with so many different models and datasets, it is quite simple to build a single plot by filtering the bundle and calling the plot method on the resulting ParameterSet.
Can you write Python code for it?

afig, mplfig = b.filter(dataset='lc01').plot(show=True)

afig, mplfig = b.plot(dataset='lc01', show=True)


I am working on the file "2.3/tutorials/plotting.ipynb".
The first task is:
First we're going to create some fake observations so that we can show how to plot observational data.  In real life, we would use something like np.loadtxt to get arrays from a data file instead.
Can you write Python code for it?

b = phoebe.default_binary()
b.add_dataset('lc', compute_phases=phoebe.linspace(0,1,101))
b.run_compute(irrad_method='none')

times = b.get_value('times', context='model')
fluxes = b.get_value('fluxes', context='model') + np.random.normal(size=times.shape) * 0.01
sigmas = np.ones_like(times) * 0.05


I am working on the file "2.3/tutorials/plotting.ipynb".
The first task is:
For more information on each of the available arrays, see the relevant tutorial on that dataset method:

ORB dataset
MESH dataset
LC dataset
RV dataset
LP dataset

Selecting Phase
And to plot in phase we just send x='phases' or x='phases:binary'.
Setting x='phases' will use the ephemeris from the top-level of the hierarchy
(as if you called b.get_ephemeris()), whereas passing a string after the colon,
will use the ephemeris of that component.
Can you write Python code for it?

afig, mplfig = b.plot(dataset='lc01', x='phases', z=0, show=True)


I am working on the file "2.3/tutorials/plotting.ipynb".
The first task is:
Selecting Arrays
So far, each plotting call automatically chose default arrays from that dataset to plot along each axis.  To override these defaults, simply point to the qualifier of the array that you'd like plotted along a given axis.
Can you write Python code for it?

afig, mplfig = b.filter(dataset='orb01').plot(x='times', y='vus', show=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
Adding a Colorbar
To add a colorbar (or sizebar, etc), send draw_sidebars=True to the plot call.
Can you write Python code for it?

afig, mplfig = b['orb01@primary@run_with_incl_80'].plot(x='times', c='vws', draw_sidebars=True, show=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
And run several forward models.  See Computing Observables for more details.
Can you write Python code for it?

b.set_value(qualifier='incl', kind='orbit', value=90)
b.run_compute(model='run_with_incl_90')

b.set_value(qualifier='incl', kind='orbit', value=85)
b.run_compute(model='run_with_incl_85')

b.set_value(qualifier='incl', kind='orbit', value=80)
b.run_compute(model='run_with_incl_80')


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
Axes Limits
Axes limits are determined by the data automatically.  To set custom axes limits, either use matplotlib methods on the returned axes objects, or pass limits as a list or tuple.
Can you write Python code for it?

afig, mplfig = b['orb01@primary@run_with_incl_80'].plot(xlim=(-2,2), show=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
Choosing colors works slightly differently for meshes (ie you can set fc for facecolor and ec for edgecolor).  For more details, see the tutorial on the MESH dataset.
Colormaps
The colormaps is determined automatically based on the parameter used for coloring (ie RVs will be a red-blue colormap).  To override this, pass a matplotlib recognized colormap to the cmap keyword.
Can you write Python code for it?

afig, mplfig = b['orb01@primary@run_with_incl_80'].plot(x='times', c='vws', cmap='spring', show=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
Colors
Colors of points and lines, by default, cycle according to matplotlib's color policy.  To manually set the color, simply pass a matplotlib recognized color to the 'c' keyword.
Can you write Python code for it?

afig, mplfig = b['orb01@primary@run_with_incl_80'].plot(c='r', show=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
Labels and Legends
To add a legend, include legend=True.
For details on placement and formatting of the legend see matplotlib's documentation.
Can you write Python code for it?

afig, mplfig = b['orb@run_with_incl_80'].plot(show=True, legend=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
Now we'll create a new Bundle and attach an orbit dataset (without observations) and a light curve dataset (with our "fake" observations - see Datasets for more details):
Can you write Python code for it?

b = phoebe.default_binary()
b.set_value('q', 0.8)
b.set_value('ecc', 0.1)
b.set_value('irrad_method', 'none')

b.add_dataset('orb', compute_times=np.linspace(0,4,1000), dataset='orb01', component=['primary', 'secondary'])
b.add_dataset('lc', times=times, fluxes=fluxes, sigmas=sigmas, dataset='lc01')


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
Other Plotting Options
Valid plotting options that are directly passed to matplotlib include:
- linestyle
- marker
Note that sizes (markersize, linewidth) should be handled by passing the size to 's' and attempting to set markersize or linewidth directly will raise an error.  See also the autofig documention on size scales.
Can you write Python code for it?

afig, mplfig = b['orb01@primary@run_with_incl_80'].plot(linestyle=':', s=0.1, show=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
This first line is only necessary for ipython noteboooks - it allows the plots to be shown on this page instead of in interactive mode.  Depending on your version of Jupyter, Python, and matplotlib - you may or may not need this line in order to see plots in the notebook.
Can you write Python code for it?

%matplotlib inline

import phoebe
from phoebe import u # units
import numpy as np

logger = phoebe.logger()


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
Time (highlight and uncover)
The built-in plot method also provides convenience options to either highlight the interpolated point for a given time, or only show the dataset up to a given time.
Highlight
The higlight option is enabled by default so long as a time (or times) is passed to plot.  It simply adds an extra marker at the sent time - interpolating in the synthetic model if necessary.
Can you write Python code for it?

afig, mplfig = b['orb@run_with_incl_80'].plot(time=1.0, show=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
To change the style of the "highlighted" points, you can pass matplotlib recognized markers, colors, and markersizes to the highlight_marker, highlight_color, and highlight_ms keywords, respectively.
Can you write Python code for it?

afig, mplfig = b['orb@run_with_incl_80'].plot(time=1.0,
                                              highlight_marker='s',
                                              highlight_color='g',
                                              highlight_ms=20,
                                              show=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
To disable highlighting, simply send highlight=False
Can you write Python code for it?

afig, mplfig = b['orb@run_with_incl_80'].plot(time=1.0,
                                              highlight=False,
                                              show=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
To disable the errorbars, simply set yerror=None.
Can you write Python code for it?

afig, mplfig = b['lc01@dataset'].plot(yerror=None, show=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
To override the position or styling of the legend, you can pass valid options to legend_kwargs which will be passed on to plt.legend
Can you write Python code for it?

afig, mplfig = b['orb@run_with_incl_80'].plot(show=True, legend=True, legend_kwargs={'loc': 'center', 'facecolor': 'r'})


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
Uncover
Uncover shows the observations or synthetic model up to the provided time and is disabled by default, even when a time is provided, but is enabled simply by providing uncover=True.  There are no additional options available for uncover.
Can you write Python code for it?

afig, mplfig = b['orb@run_with_incl_80'].plot(time=0.5,
                                              uncover=True,
                                              show=True)


I am working on the file "2.3/tutorials/plotting_advanced.ipynb".
The first task is:
Units
Likewise, each array that is plotted is automatically plotted in its default units. To override these defaults, simply provide the unit (as a string or as a astropy units object) for a given axis.
Can you write Python code for it?

afig, mplfig = b['orb01@primary@run_with_incl_80'].plot(xunit='AU', yunit='AU', show=True)


I am working on the file "2.3/tutorials/reflection_heating.ipynb".
The first task is:
As always, let's do imports and initialize a logger and a new bundle.
Can you write Python code for it?

import phoebe
from phoebe import u # units
import numpy as np
import matplotlib.pyplot as plt

#logger = phoebe.logger('error')

b = phoebe.default_binary()


I am working on the file "2.3/tutorials/reflection_heating.ipynb".
The first task is:
In order to see the effect of reflection, let's set "irrad_frac_refl_bol" of both of our stars to 0.9 - that is 90% of the incident flux will go towards reflection and 10% will be ignored.
Can you write Python code for it?


I am working on the file "2.3/tutorials/reflection_heating.ipynb".
The first task is:
Influence on Meshes (Intensities)
Can you write Python code for it?

b.add_dataset('mesh', times=[0.2], columns=['teffs', 'intensities@lc01'])

b.disable_dataset('lc01')

b.run_compute(irrad_method='none', ntriangles=700, model='refl_false', overwrite=True)

b.run_compute(irrad_method='wilson', ntriangles=700, model='refl_true', overwrite=True)

#phoebe.logger('debug')

afig, mplfig = b.plot(component='secondary', kind='mesh', model='refl_false',
                      fc='intensities', ec='face',
                      draw_sidebars=True, show=True)

afig, mplfig = b.plot(component='secondary', kind='mesh', model='refl_true',
                      fc='intensities', ec='face',
                      draw_sidebars=True, show=True)

afig, mplfig = b.plot(component='secondary', kind='mesh', model='refl_false',
                      fc='teffs', ec='face',
                      draw_sidebars=True, show=True)

afig, mplfig = b.plot(component='secondary', kind='mesh', model='refl_true',
                      fc='teffs', ec='face',
                      draw_sidebars=True, show=True)


I am working on the file "2.3/tutorials/reflection_heating.ipynb".
The first task is:
Let's run models with the reflection switch both turned on and off so that we can compare the two results.  We'll also override delta to be a larger number since the computation time required by delta depends largely on the number of surface elements.
Can you write Python code for it?

b.run_compute(irrad_method='none', ntriangles=700, model='refl_false')

b.run_compute(irrad_method='wilson', ntriangles=700, model='refl_true')

afig, mplfig = b.plot(show=True, legend=True)

artists = plt.plot(b['value@times@refl_false'], b['value@fluxes@refl_true']-b['value@fluxes@refl_false'], 'r-')


I am working on the file "2.3/tutorials/reflection_heating.ipynb".
The first task is:
Reflection has the most noticeable effect when the two stars are close to each other and have a large temperature ratio.
Can you write Python code for it?

b['sma@orbit'] = 4.0

b['teff@primary'] = 10000

b['teff@secondary'] = 5000


I am working on the file "2.3/tutorials/reflection_heating.ipynb".
The first task is:
Since reflection can be a computationally expensive process and in most cases is a low-order effect, there is a switch in the compute options that needs to be enabled in order for reflection to be taken into account.  If this switch is False (which it is by default), the albedos are completely ignored and will be treated as if all incident light is lost/ignored.
Can you write Python code for it?


I am working on the file "2.3/tutorials/requiv.ipynb".
The first task is:
Critical Potentials and System Checks
Additionally, for each detached component, there is an requiv_max Parameter which shows the critical value at which the Roche surface will overflow.  Setting requiv to a larger value will fail system checks and raise a warning.
Can you write Python code for it?

print(b['requiv_max@primary@component'])

print(b['requiv_max@primary@constraint'])

b.set_value('requiv@primary@component', 3)


I am working on the file "2.3/tutorials/requiv.ipynb".
The first task is:
Now let's add a mesh dataset at a few different times so that we can see how the potentials affect the surfaces of the stars.
Can you write Python code for it?


I am working on the file "2.3/tutorials/requiv.ipynb".
The first task is:
Relevant Parameters
The 'requiv' parameter defines the stellar surface to have a constant volume of 4./3 pi requiv^3.
Can you write Python code for it?


I am working on the file "2.3/tutorials/requiv_crit_contact.ipynb".
The first task is:
Changing Parameterization
It is possible to change this parameterization to allow any one of the four parameters (requiv@primary, requiv@secondary, pot@contact_envelope, fillout_factor@contact_envelope) to be adjustable and the other two to be constrained.  Doing so requires flipping one or two constraints via b.flip_constraint().
Let's first flip the  constraints so that we can provide the potential of the envelope.
Can you write Python code for it?

b.flip_constraint('pot', solve_for='requiv@primary')

print(b.filter(qualifier=['requiv', 'pot', 'fillout_factor']))


I am working on the file "2.3/tutorials/requiv_crit_contact.ipynb".
The first task is:
Contact Systems
Contact systems are created by passing contact_binary=True to phoebe.default_binary() or by manually adding an envelope and setting the hierarchy correctly.
By default, requiv@primary is the free Parameter, with requiv@secondary, pot@contact_envelope, and fillout_factor@contact_envelope constrained such that there is one single surface defining the envelope.
Can you write Python code for it?


I am working on the file "2.3/tutorials/requiv_crit_contact.ipynb".
The first task is:
In order to pass the system checks, these values must be between their minimum and maximum value ensuring the system is not underflowing (in which case it should be detached or semi-detached) or overflowing and losing mass.
These limiting values are constrained parameters, allowing us to see the allowed range for any parameterization.
Can you write Python code for it?

print(b.filter(qualifier=['requiv_max', 'requiv_min', 'pot_max', 'pot_min']))


I am working on the file "2.3/tutorials/requiv_crit_contact.ipynb".
The first task is:
Or instead we could allow providing the equivalent radius of the secondary star.
Can you write Python code for it?

b = phoebe.default_binary(contact_binary=True)
b.flip_constraint('pot', solve_for='requiv@primary')
b.flip_constraint('requiv@secondary', solve_for='pot')

print(b.filter(qualifier=['requiv', 'pot', 'fillout_factor']))


I am working on the file "2.3/tutorials/requiv_crit_detached.ipynb".
The first task is:
If we increase 'requiv' past the critical point, we'll receive a warning from the logger and would get an error if attempting to call b.run_compute().
Can you write Python code for it?

b['requiv@primary'] = 2.2

print(b.run_checks())


I am working on the file "2.3/tutorials/requiv_crit_detached.ipynb".
The first task is:
We can see that the default system is well within this critical value by printing all radii and critical radii.
Can you write Python code for it?


I am working on the file "2.3/tutorials/requiv_crit_semidetached.ipynb".
The first task is:
Note that the point region in semi-detached systems often falls outside of the atmosphere tables.  Since this requires the full mesh, this cannot be checked in advance with b.run_checks but will raise an error during b.run_compute.
Can you write Python code for it?

b.add_dataset('lc', compute_phases=phoebe.linspace(0,1,101))

try:
    b.run_compute()
except Exception as err:
    print(err)


I am working on the file "2.3/tutorials/requiv_crit_semidetached.ipynb".
The first task is:
Now whenever any of the relevant parameters (q, ecc, syncpar, sma) are changed, the value of requiv will change to match the critical value as defined by requiv_max.
Can you write Python code for it?


I am working on the file "2.3/tutorials/requiv_crit_semidetached.ipynb".
The first task is:
Semi-Detached Systems
Semi-detached systems are implemented by constraining the value of requiv to be the same as requiv_max by appyling the 'semidetached' constraint on the 'primary' component.
Can you write Python code for it?


I am working on the file "2.3/tutorials/requiv_crit_semidetached.ipynb".
The first task is:
The default values of the ld_func and ld_coeffs should be changed to appropriate values for a given system, but should still compute without errors.
Can you write Python code for it?

print(b.filter(qualifier=['ld_func', 'ld_coeffs']))

b.run_compute()


I am working on the file "2.3/tutorials/rv_offset.ipynb".
The first task is:
As always, let's do imports and initialize a logger and a new bundle.
Can you write Python code for it?

import phoebe
from phoebe import u # units
import numpy as np
import matplotlib.pyplot as plt

logger = phoebe.logger()

b = phoebe.default_binary()

b.add_dataset('rv', times=np.linspace(0,1,101), dataset='rv01')

b.set_value_all('ld_mode', 'manual')
b.set_value_all('ld_func', 'logarithmic')
b.set_value_all('ld_coeffs', [0.0, 0.0])
b.set_value_all('atm', 'blackbody')


I am working on the file "2.3/tutorials/rv_offset.ipynb".
The first task is:
Influence on Radial Velocities
Can you write Python code for it?

afig, mplfig = b.plot(legend=True, show=True)


I am working on the file "2.3/tutorials/rv_offset.ipynb".
The first task is:
Now let's run another model, with the offset applied to the primary component.
Can you write Python code for it?


I am working on the file "2.3/tutorials/rv_offset.ipynb".
The first task is:
Relevant Parameters
Radial velocity offsets allow for a per-component and per-dataset offset applied to the synthetic RVs.
First let's run a model without any offsets applied.
Can you write Python code for it?


I am working on the file "2.3/tutorials/saving_and_loading.ipynb".
The first task is:
Exporting to a PHOEBE Legacy file is also possible (although note that some parameters don't translate exactly or are not supported in PHOEBE Legacy), via b.export_legacy.
Can you write Python code for it?


I am working on the file "2.3/tutorials/saving_and_loading.ipynb".
The first task is:
For the parameters that could not be directly translated, you should see a warning message (if you have warning messages enabled in your logger).
We can now look at the beginning of the saved file and see that it matches the PHOEBE Legacy file-format.
Can you write Python code for it?


I am working on the file "2.3/tutorials/saving_and_loading.ipynb".
The first task is:
Loading a Bundle
To open an existing Bundle from the file we just created, call Bundle.open and pass the filename.
Can you write Python code for it?


I am working on the file "2.3/tutorials/saving_and_loading.ipynb".
The first task is:
Saving a Bundle
Can you write Python code for it?


I am working on the file "2.3/tutorials/saving_and_loading.ipynb".
The first task is:
Support for Other Codes
Legacy
Importing from a PHOEBE Legacy file is as simple as passing the filename to from_legacy:
Can you write Python code for it?


I am working on the file "2.3/tutorials/saving_and_loading.ipynb".
The first task is:
To save the Bundle to a file, we can call the save method of the Bundle and pass a filename.
Can you write Python code for it?


I am working on the file "2.3/tutorials/saving_and_loading.ipynb".
The first task is:
We can now inspect the contents of the created file.
This file is in the JSON-format and is simply a list of dictionaries - where each dictionary represents the attributes of a single Parameter.
You could edit this file in a text-editor - but do be careful if changing any of the tags.  For example: if you want to change the component tag of one of your stars, make sure to change ALL instances of the component tag to match (as well as the hierarchy Parameter).
Can you write Python code for it?


I am working on the file "2.3/tutorials/settings.ipynb".
The first task is:
If you want dictionary access to use set_value_all instead of set_value, you can enable this parameter
Can you write Python code for it?

b['dict_set_all@setting'] = True
b['teff@component'] = 8000
print(b['value@teff@primary@component'], b['value@teff@secondary@component'])


I am working on the file "2.3/tutorials/settings.ipynb".
The first task is:
Now let's reset this option... keeping in mind that we no longer have access to the 'setting' context through twig access, we'll have to use methods to clear the dict_filter
Can you write Python code for it?


I am working on the file "2.3/tutorials/settings.ipynb".
The first task is:
Now we no longer see the constraint parameters.
All parameters are always accessible with method access:
Can you write Python code for it?


I am working on the file "2.3/tutorials/settings.ipynb".
The first task is:
and can be set as any other Parameter in the Bundle
Available Settings
Now let's look at each of the available settings and what they do
phoebe_version
phoebe_version is a read-only parameter in the settings to store the version of PHOEBE used.
dict_set_all
dict_set_all is a BooleanParameter (defaults to False) that controls whether attempting to set a value to a ParameterSet via dictionary access will set all the values in that ParameterSet (if True) or raise an error (if False)
Can you write Python code for it?


I am working on the file "2.3/tutorials/settings.ipynb".
The first task is:
dict_filter
dict_filter is a Parameter that accepts a dictionary.  This dictionary will then always be sent to the filter call which is done under-the-hood during dictionary access.
Can you write Python code for it?


I am working on the file "2.3/tutorials/settings.ipynb".
The first task is:
or via filtering/twig access
Can you write Python code for it?


I am working on the file "2.3/tutorials/settings.ipynb".
The first task is:
run_checks_compute (/figure/solver/solution)
The run_checks_compute option allows setting the default compute option(s) sent to b.run_checks, including warnings in the logger raised by interactive checks (see phoebe.interactive_checks_on).
Similar options also exist for checks at the figure, solver, and solution level.
Can you write Python code for it?

b['run_checks_compute@setting']

b.add_dataset('lc')
b.add_compute('legacy')
print(b.run_checks())

b['run_checks_compute@setting'] = ['phoebe01']

print(b.run_checks())


I am working on the file "2.3/tutorials/settings.ipynb".
The first task is:
web_client, web_client_url
The web_client and web_client_url settings determine whether the client is opened in a web-browser or with the installed desktop client whenever calling b.ui or b.ui_figures.  For more information, see the UI from Jupyter tutorial.
Can you write Python code for it?

b['web_client']

b['web_client'].description

b['web_client_url']

b['web_client_url'].description


I am working on the file "2.3/tutorials/solver.ipynb".
The first task is:
Adding Solver Options
As there are quite a few different solvers implemented in PHOEBE and each have their own available options, we won't get in to the details here.  See LC esimators, RV estimators, Nelder-Mead Optimizer, and emcee sampler for details on some of the most commonly-used solver.  The solver API docs or solver example scripts may also help.
As you may expect, to use a solver you must first call b.add_solver, set the desired options, and then call b.run_solver (or b.export_solver and b.import_solution).
Can you write Python code for it?

b.add_solver('estimator.lc_geometry', solver='my_lcgeom_solver')

print(b.get_solver(solver='my_lcgeom_solver'))


I am working on the file "2.3/tutorials/solver.ipynb".
The first task is:
In addition to the solver API docs, remember that each parameter has a description and possibly a set of available choices (if its a ChoiceParameter or SelectParameter).
Can you write Python code for it?

print(b.get_parameter('expose_model').description)

print(b.get_parameter('lc_datasets').description)

print(b.get_parameter('lc_datasets').choices)


I am working on the file "2.3/tutorials/solver.ipynb".
The first task is:
In many cases, the solution itself is plottable - showing some sort of diagnostic figures.  In some cases, such as sampler.emcee or sampler.dynesty, there are several different diagnostic figures available which can be chosen by passing the available options to style.
Can you write Python code for it?

_ = b.plot(solution='my_lcgeom_solution', show=True)


I am working on the file "2.3/tutorials/solver.ipynb".
The first task is:
The Merit Function
Both optimizers and samplers require running a forward model and use a merit function to compare the synthetic model to the observational data.  This merit function is described in detail in the 2.3 release paper (Conroy+ 2020).
Several bundle methods allow for accessing the values used in the merit function:

b.calculate_residuals
b.calculate_chi2
b.calculate_lnlikelihood
b.calculate_lnp

The log-probability used as the merit function within optimizers and samplers is defined as calculate_lnp(priors, combine=priors_combine) + calculate_lnlikelihood.
To see the affect of priors_combine, we can pass the solver tag directly to b.get_distribution_collection, b.plot_distribution_collection, or b.calculate_lnp.
Can you write Python code for it?

b.add_distribution('teff@primary', phoebe.gaussian(6000,100), distribution='mydist01')
b.add_distribution('teff@secondary', phoebe.gaussian(5500,600), distribution='mydist01')

b.add_distribution('teff@primary', phoebe.uniform(5800,6200), distribution='mydist02')

b.add_solver('sampler.emcee', priors=['mydist01', 'mydist02'], solver='myemceesolver')

print(b.filter(qualifier='prior*'))


I am working on the file "2.3/tutorials/solver.ipynb".
The first task is:
run_solver
b.run_solver (or b.export_solver and b.import_solution) allows optionally setting a solution tag (if not provided, one will be created automatically), just as b.run_compute allows setting a model tag.
Can you write Python code for it?


I am working on the file "2.3/tutorials/solver_times.ipynb".
The first task is:
And then we'll build a synthetic "dataset" and initialize a new bundle with those data
Can you write Python code for it?

b = phoebe.default_binary()
b.add_dataset('lc', times=phoebe.linspace(0,5,1001))
b.add_compute('ellc', compute='ellc01')
b.set_value_all('ld_mode', 'lookup')
b.run_compute('ellc01')

times = b.get_value('times@model')
fluxes = b.get_value('fluxes@model')
sigmas = np.ones_like(times) * 0.01

b = phoebe.default_binary()
b.add_dataset('lc', compute_phases=phoebe.linspace(0,1,101),
              times=times, fluxes=fluxes, sigmas=sigmas)
b.add_compute('ellc', compute='ellc01')
b.set_value_all('ld_mode', 'lookup')
b.add_solver('optimizer.nelder_mead', compute='ellc01', fit_parameters=['teff'], solver='nm_solver')


I am working on the file "2.3/tutorials/solver_times.ipynb".
The first task is:
Now we'll just flip the constraint back for the remaining examples
Can you write Python code for it?

_ = b.flip_constraint('compute_phases', solve_for='compute_times')


I am working on the file "2.3/tutorials/solver_times.ipynb".
The first task is:
The logic for solver times is generally only used internally within b.run_solver (for optimizers and samplers which require a forward-model to be computed).  However, it is useful (in order to diagnose any issues, for example) to be able to see how the combination of solver_times, times, compute_times/compute_phases, mask_enabled, and mask_phases will be interpretted within PHOEBE during b.run_solver.
See also:
* Advanced: mask_phases
* Advanced: Compute Times & Phases
To access the underlying times that would be used, we can call b.parse_solver_times.  Let's first look at the docstring (also available from the link above):
Can you write Python code for it?


I am working on the file "2.3/tutorials/solver_times.ipynb".
The first task is:
This will always be the case when providing compute_phases when the dataset times cover more than a single cycle.  Here we'll follow the advice from the error and provide compute_times instead.
Can you write Python code for it?

b.flip_constraint('compute_times', solve_for='compute_phases')
b.set_value('compute_times', phoebe.linspace(0,5,501))

print(b.run_checks_solver())

dataset_times = b.get_value('times', context='dataset')
_ = plt.plot(times, np.ones_like(times)*1, 'k.')

compute_times = b.get_value('compute_times', context='dataset')
_ = plt.plot(compute_times, np.ones_like(compute_times)*2, 'b.')

solver_times = b.parse_solver_times()
_ = plt.plot(solver_times['lc01'], np.ones_like(solver_times['lc01'])*3, 'g.')

b.run_compute(solver='nm_solver')
_ = b.plot(show=True)


I am working on the file "2.3/tutorials/solver_times.ipynb".
The first task is:
solver_times = 'times'
without phase_mask enabled
Can you write Python code for it?

b.set_value('solver_times', 'times')
b.set_value('compute_phases', phoebe.linspace(0,1,101))
b.set_value('mask_enabled', False)
b.set_value('dperdt', 0.0)

dataset_times = b.get_value('times', context='dataset')
_ = plt.plot(times, np.ones_like(times)*1, 'k.')

compute_times = b.get_value('compute_times', context='dataset')
_ = plt.plot(compute_times, np.ones_like(compute_times)*2, 'b.')

solver_times = b.parse_solver_times()
print(solver_times)
_ = plt.plot(solver_times['lc01'], np.ones_like(solver_times['lc01'])*3, 'g.')

b.run_compute(solver='nm_solver')
_ = b.plot(show=True)


I am working on the file "2.3/tutorials/solver_times.ipynb".
The first task is:
times shorter
Can you write Python code for it?

b.set_value('solver_times', 'auto')
b.set_value('compute_phases', phoebe.linspace(0,1,2001))
b.set_value('mask_enabled', False)
b.set_value('dperdt', 0.0)

dataset_times = b.get_value('times', context='dataset')
_ = plt.plot(times, np.ones_like(times)*1, 'k.')

compute_times = b.get_value('compute_times', context='dataset')
_ = plt.plot(compute_times, np.ones_like(compute_times)*2, 'b.')

solver_times = b.parse_solver_times()
_ = plt.plot(solver_times['lc01'], np.ones_like(solver_times['lc01'])*3, 'g.')

b.run_compute(solver='nm_solver')
_ = b.plot(show=True)


I am working on the file "2.3/tutorials/solver_times.ipynb".
The first task is:
with phase_mask enabled
Can you write Python code for it?

b.set_value('solver_times', 'times')
b.set_value('compute_phases', phoebe.linspace(0,1,101))
b.set_value('mask_enabled', True)
b.set_value('mask_phases', [(-0.1, 0.1), (0.45,0.55)])
b.set_value('dperdt', 0.0)

dataset_times = b.get_value('times', context='dataset')
_ = plt.plot(times, np.ones_like(times)*1, 'k.')

compute_times = b.get_value('compute_times', context='dataset')
_ = plt.plot(compute_times, np.ones_like(compute_times)*2, 'b.')

solver_times = b.parse_solver_times()
_ = plt.plot(solver_times['lc01'], np.ones_like(solver_times['lc01'])*3, 'g.')

b.run_compute(solver='nm_solver')
_ = b.plot(show=True)


I am working on the file "2.3/tutorials/solver_times.ipynb".
The first task is:
with phase_mask enabled and time-independent hierarchy
Can you write Python code for it?

b.set_value('solver_times', 'compute_times')
b.set_value('compute_phases', phoebe.linspace(0,1,101))
b.set_value('mask_enabled', True)
b.set_value('mask_phases', [(-0.1, 0.1), (0.45,0.55)])
b.set_value('dperdt', 0.0)

dataset_times = b.get_value('times', context='dataset')
_ = plt.plot(times, np.ones_like(times)*1, 'k.')

compute_times = b.get_value('compute_times', context='dataset')
_ = plt.plot(compute_times, np.ones_like(compute_times)*2, 'b.')

solver_times = b.parse_solver_times()
_ = plt.plot(solver_times['lc01'], np.ones_like(solver_times['lc01'])*3, 'g.')

b.run_compute(solver='nm_solver')
_ = b.plot(show=True)


I am working on the file "2.3/tutorials/spots.ipynb".
The first task is:
As a shortcut, we can also call add_spot directly.
Can you write Python code for it?


I am working on the file "2.3/tutorials/spots.ipynb".
The first task is:
As always, let's do imports and initialize a logger and a new bundle.
Can you write Python code for it?

import phoebe
from phoebe import u # units
import numpy as np
import matplotlib.pyplot as plt

logger = phoebe.logger()

b = phoebe.default_binary()


I am working on the file "2.3/tutorials/spots.ipynb".
The first task is:
At a later time, the spot is still technically at the same coordinates, but longitude of 0 no longer corresponds to pointing to the companion star.  The coordinate system has rotated along with the asyncronous rotation of the star.
Can you write Python code for it?

afig, mplfig = b.plot(time=0.25, y='ws', fc='teffs', facecmap='YlOrRd', ec='None', show=True)

afig, mplfig = b.plot(time=0.5, y='ws', fc='teffs', facecmap='YlOrRd', ec='None', show=True)

ax, artists = b.plot(time=0.75, y='ws', fc='teffs', facecmap='YlOrRd', ec='None', show=True)


I am working on the file "2.3/tutorials/spots.ipynb".
The first task is:
Since the syncpar was set to 1.5, one full orbit later the star (and the spot) has made an extra half-rotation.
Can you write Python code for it?

ax, artists = b.plot(time=1.0, y='ws', fc='teffs', facecmap='YlOrRd', ec='None', show=True)


I am working on the file "2.3/tutorials/spots.ipynb".
The first task is:
Spot Corotation
The positions (colat, long) of a spot are defined at t0 (note: t0@system, not necessarily t0_perpass or t0_supconj).  If the stars are not synchronous, then the spots will corotate with the star.  To illustrate this, let's set the syncpar > 1 and plot the mesh at three different phases from above.
Can you write Python code for it?

b.set_value(qualifier='syncpar', component='primary', value=1.5)

b.run_compute(irrad_method='none')


I am working on the file "2.3/tutorials/spots.ipynb".
The first task is:
To see the spot, add a mesh dataset and plot it.
Can you write Python code for it?

b.add_dataset('mesh', times=[0,0.25,0.5,0.75,1.0], columns=['teffs'])

b.run_compute()

afig, mplfig = b.filter(component='primary', time=0.75).plot(fc='teffs', show=True)


I am working on the file "2.3/tutorials/t0s.ipynb".
The first task is:
And let's make our system a little more interesting so that we can discriminate between the various t0s
Can you write Python code for it?

b.set_value('sma@binary', 20)
b.set_value('q', 0.8)
b.set_value('ecc', 0.8)
b.set_value('per0', 45)


I am working on the file "2.3/tutorials/t0s.ipynb".
The first task is:
In addition, there is a single t0 parameter that is system-wide.  This parameter simply defines the time at which all parameters are defined and therefore at which all computations start.  The value of this parameter begins to play an important role if any parameter is given a time-derivative (see apsidal motion and dpdt, for example) or when using N-body instead of Keplerian dynamics (coming in a future release).
Can you write Python code for it?


I am working on the file "2.3/tutorials/t0s.ipynb".
The first task is:
Influence on Oribits (positions)
Can you write Python code for it?

b.add_dataset('orb', compute_times=np.linspace(-1,1,1001))

b.run_compute(ltte=False)


I am working on the file "2.3/tutorials/t0s.ipynb".
The first task is:
The t0_ref defines the time at which the primary component in our orbit passes an arbitrary reference point.  This t0_ref is defined in the same way as PHOEBE legacy's 'HJD0' parameter, so is included for convenience translating between the two.
Can you write Python code for it?

b.get_parameter('t0_ref', context='component')

b.get_parameter('t0_ref', context='constraint')


I am working on the file "2.3/tutorials/t0s.ipynb".
The first task is:
To visualize where these times are with respect to the orbits, we can plot the model orbit and highlight the positions of each star at the times defined by these parameters.  Note here that the observer is in the positive w-direction.
NOTE: sending z=0 will override the default of ordering in z by vs (the unused coordinate in the same system), which can be expensive to draw.
Can you write Python code for it?

afig, mplfig = b.plot(x='us', y='ws', z=0, time='t0_supconj', show=True)

afig, mplfig = b.plot(x='us', y='ws', z=0, time='t0_perpass', show=True)

afig, mplfig = b.plot(x='us', y='ws', z=0, time='t0_ref', show=True)


I am working on the file "2.3/tutorials/t0s.ipynb".
The first task is:
t0 Parameters
There are three t0 parameters that are available to define an orbit (but only one of which is editable at any given time), as well as a t0 parameter for the entire system.  Let's first access the three t0 parameters for our binary orbit.
t0_supconj defines the time at which the primary component in our orbit is at superior conjunction.  For a binary system in which there are eclipses, this is defined as the primary eclipse.  By default this parameter is editable.
Can you write Python code for it?


I am working on the file "2.3/tutorials/t0s.ipynb".
The first task is:
t0_perpass defines the time at which both components in our orbit is at periastron passage.  By default this parameter is constrained by t0_supconj.  For more details or information on how to change which parameter is editable, see the Constraints Tutorial.
Can you write Python code for it?

b.get_parameter('t0_perpass', context='component')

b.get_parameter('t0_perpass', context='constraint')


I am working on the file "2.3/tutorials/ui_jupyter.ipynb".
The first task is:
Showing the Jupyter in-line UI
Whether you have the UI Client installed locally or not, calling b.ui or PS.ui will launch an inline-representation of that ParameterSet directly below the cell by default.  This allows you to interactively view and change parameter values without having to write code.
This can be super useful to create "interactive" scripts where you (or a student, etc) can change specific values at certain points throughout the script.  But do keep in mind that any values you change in the UI will not be reproduced when re-running the notebook (unless you manually do the exact same thing).
By default this is "blocking", which means that Jupyter will not continue to the next line until you click the blue "continue" button.  Once clicked, the UI will become "frozen" (and will show a "disconnected" note in the corner in place of the "continue" button) and Jupyter will continue to allow further entry and execution.
NOTE: The following lines are commented-out to avoid the error messages when viewing the notebook in read-only mode.  Below each commented line are a screenshot of the resulting UI.  To see the interactive versions, run this notebook locally and uncomment the calls to ui.
Can you write Python code for it?


I am working on the file "2.3/tutorials/units.ipynb".
The first task is:
Both get_value and get_quantity also accept a unit argument which will return the value or quantity in the requested units (if able to convert).  This unit argument takes either a unit object (we imported a forked version of astropy units from within PHOEBE) or a string representation that can be parsed.
Can you write Python code for it?

b.get_parameter(qualifier='sma', component='binary', context='component').get_value(unit=u.km)

b.get_parameter(qualifier='sma', component='binary', context='component').get_quantity(unit='km')


I am working on the file "2.3/tutorials/units.ipynb".
The first task is:
From the representation above, we can already see that the units are in solar radii.  We can access the units directly via get_default_unit.
Can you write Python code for it?


I am working on the file "2.3/tutorials/units.ipynb".
The first task is:
If for some reason you want to change the default units, you can, but just be careful that this could cause some float-point precision issues.
Can you write Python code for it?

b.get_parameter(qualifier='sma', component='binary', context='component').set_default_unit('mm')

b.get_parameter(qualifier='sma', component='binary', context='component').get_quantity()

b.get_parameter(qualifier='sma', component='binary', context='component').get_quantity(unit='solRad')


I am working on the file "2.3/tutorials/units.ipynb".
The first task is:
Similarly when setting the value, you can provide either a Quantity object or a value and unit.  These will still be stored within PHOEBE according to the default_unit of the Parameter object.
Can you write Python code for it?

b.get_parameter(qualifier='sma', component='binary', context='component').set_value(3800000*u.km)

b.get_parameter(qualifier='sma', component='binary', context='component').get_quantity()

b.get_parameter(qualifier='sma', component='binary', context='component').set_value(3900000, unit='km')

b.get_parameter(qualifier='sma', component='binary', context='component').get_quantity()


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
As you can see, since the center of mass of the system was at 0,0,0 at t0 - including systemic velocity actually shows the system spiraling towards or away from the observer (who is in the positive w direction).  In other words - the positions of the meshes are affected in the same way as the orbits (note the offset on the ylimit scales).
In addition, the actual values of vw and rv in the meshes are adjusted to include the systemic velocity.
Can you write Python code for it?

b['primary@mesh1@0_false'].get_value('vws', time=0.0)[:5]

b['primary@mesh1@100_false'].get_value('vws', time=0.0)[:5]

b['primary@mesh1@100_true'].get_value('vws', time=0.0)[:5]


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
For our second model, we'll set a somewhat ridiculous value for the systemic velocity (so that the affects are exagerated and clearly visible over one orbit), but leave ltte off.
Can you write Python code for it?

b['vgamma@system'] = 100

b.run_compute(irrad_method='none', model='100_false')


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
In each of the figures below, the first panel will be the first cycle (days 0-3) and the second panel will be 100 cycles later (days 900-903).
Without light-time effects, the light curve remains unchanged by the introduction of a systemic velocity (blue and red overlap each other).   However, once ltte is enabled, the time between two eclipses (ie the observed period of the system) changes.  This occurs because the path between the system and observer has changed.  This is an important effect to note - the period parameter sets the TRUE period of the system, not necessarily the observed period between two successive eclipses.
Can you write Python code for it?

afig, mplfig = b['lc'].plot(c=colors, linestyle='solid',
                            axorder={'lc1': 0, 'lc2': 1},
                            subplot_grid=(1,2), tight_layout=True, show=True)


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
Influence on Light Curves (fluxes)
Now let's compare the various models across all our different datasets.
Let's set the colors so that all figures will have systemic velocity shown in blue, systemic velocity with ltte=False in red, and systemic velocity with ltte=True in green.
Can you write Python code for it?


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
Influence on Meshes
Can you write Python code for it?

afig, mplfig = b.filter(kind='mesh', model=['0_false', '100_false']).plot(x='us', y='ws',
                                                                          axorder={'mesh1': 0, 'mesh2': 1},
                                                                          subplot_grid=(1,2), tight_layout=True, show=True)

afig, mplfig = b.filter(kind='mesh', model=['100_false', '100_true']).plot(x='us', y='ws',
                                                                          axorder={'mesh1': 0, 'mesh2': 1},
                                                                          subplot_grid=(1,2), tight_layout=True, show=True)


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
Influence on Orbits (positions, velocities)
In the orbit, the addition of a systemic velocity affects both the positions and velocities.  So if we plot the orbits from above (u-w plane) we can see see orbit spiral in the w-direction.  Note that this actually shows the barycenter of the orbit moving - and it was only at 0,0,0 at t0.  This also stresses the importance of using a reasonable t0 - here 900 days later, the barycenter has moved significantly from the center of the coordinate system.
Can you write Python code for it?

afig, mplfig = b.filter(kind='orb', model=['0_false', '100_false']).plot(x='us', y='ws',
                             c=colors, linestyle='solid',
                             axorder={'orb1': 0, 'orb2': 1},
                             subplot_grid=(1,2), tight_layout=True, show=True)


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
Influence on Radial Velocities
Radial velocities are perhaps the most logical observable in the case of systemic velocities.  Introducing a non-zero value for vgamma simply offsets the observed values.
Light-time will have a similar affect on RVs as it does on LCs - it simply changes the observed period.
Can you write Python code for it?

afig, mplfig = b['rv'].plot(c=colors, linestyle='solid',
                            axorder={'rv1': 0, 'rv2': 1},
                            subplot_grid=(1,2), tight_layout=True, show=True)


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
Lastly, let's leave this value of vgamma, but enable light-time effects.
Can you write Python code for it?


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
Now let's look at the effect that enabling ltte has on these same plots.
Can you write Python code for it?

afig, mplfig = b.filter(kind='orb', model=['100_false', '100_true']).plot(x='us', y='ws',
                             c=colors, linestyle='solid',
                             axorder={'orb1': 0, 'orb2': 1},
                             subplot_grid=(1,2), tight_layout=True, show=True)

afig, mplfig = b.filter(kind='orb', model=['100_false', '100_true']).plot(x='times', y='vws',
                             c=colors, linestyle='solid',
                             axorder={'orb1': 0, 'orb2': 1},
                             subplot_grid=(1,2), tight_layout=True, show=True)


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
Now we'll create empty lc, rv, orb, and mesh datasets.  We'll then look to see how the systemic velocity (vgamma) affects the observables in each of these datasets, and how those are also affected by light-time effects (ltte).
To see the effects over long times, we'll compute one cycle starting at t=0, and another in the distant future.
Can you write Python code for it?

times1 = np.linspace(0,1,201)
times2 = np.linspace(90,91,201)

b.add_dataset('lc', times=times1, dataset='lc1')
b.add_dataset('lc', times=times2, dataset='lc2')

b.add_dataset('rv', times=times1, dataset='rv1')
b.add_dataset('rv', times=times2, dataset='rv2')

b.add_dataset('orb', times=times1, dataset='orb1')
b.add_dataset('orb', times=times2, dataset='orb2')

b.add_dataset('mesh', times=[0], dataset='mesh1', columns=['vws'])
b.add_dataset('mesh', times=[90], dataset='mesh2', columns=['vws'])


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
Plotting the w-velocities with respect to time would show the same as the RVs, except without any Rossiter-McLaughlin like effects.  Note however the flip in w-convention between vw and radial velocities (+w is defined as towards the observer to make a right-handed system, but by convention +rv is a red shift).
Can you write Python code for it?

afig, mplfig = b.filter(kind='orb', model=['0_false', '100_false']).plot(x='times', y='vws',
                             c=colors, linestyle='solid',
                             axorder={'orb1': 0, 'orb2': 1},
                             subplot_grid=(1,2), tight_layout=True, show=True)


I am working on the file "2.3/tutorials/vgamma.ipynb".
The first task is:
We'll leave it set at 0.0 for now, and then change vgamma to see how that affects the observables.
The other relevant parameter here is t0 - that is the time at which all quantities are provided, the time at which nbody integration would start (if applicable), and the time at which the center-of-mass of the system is defined to be at (0,0,0).  Unless you have a reason to do otherwise, it makes sense to have this value near the start of your time data... so if we don't have any other changing quantities defined in our system and are using BJDs, we would want to set this to be non-zero.  In this case, our times all start at 0, so we'll leave t0 at 0 as well.
Can you write Python code for it?


I am working on the file "2.4 Transfer Learning & Fine-Tuning.ipynb".
The first task is:
Dataset Preparation
Can you write Python code for it?

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()

# create two datasets one with digits below 5 and one with 5 and above
X_train_lt5 = X_train[y_train < 5]
y_train_lt5 = y_train[y_train < 5]
X_test_lt5 = X_test[y_test < 5]
y_test_lt5 = y_test[y_test < 5]

X_train_gte5 = X_train[y_train >= 5]
y_train_gte5 = y_train[y_train >= 5] - 5  # make classes start at 0 for
X_test_gte5 = X_test[y_test >= 5]         # np_utils.to_categorical
y_test_gte5 = y_test[y_test >= 5] - 5

# define two groups of layers: feature (convolutions) and classification (dense)
feature_layers = [
    Convolution2D(nb_filters, kernel_size, kernel_size,
                  border_mode='valid',
                  input_shape=input_shape),
    Activation('relu'),
    Convolution2D(nb_filters, kernel_size, kernel_size),
    Activation('relu'),
    MaxPooling2D(pool_size=(pool_size, pool_size)),
    Dropout(0.25),
    Flatten(),
]
classification_layers = [
    Dense(128),
    Activation('relu'),
    Dropout(0.5),
    Dense(nb_classes),
    Activation('softmax')
]

# create complete model
model = Sequential(feature_layers + classification_layers)

# train model for 5-digit classification [0..4]
train_model(model,
            (X_train_lt5, y_train_lt5),
            (X_test_lt5, y_test_lt5), nb_classes)

# freeze feature layers and rebuild model
for l in feature_layers:
    l.trainable = False

# transfer: train dense layers for new classification task [5..9]
train_model(model,
            (X_train_gte5, y_train_gte5),
            (X_test_gte5, y_test_gte5), nb_classes)


I am working on the file "2.4 Transfer Learning & Fine-Tuning.ipynb".
The first task is:
Settings
Can you write Python code for it?

now = datetime.datetime.now

batch_size = 128
nb_classes = 5
nb_epoch = 5

# input image dimensions
img_rows, img_cols = 28, 28
# number of convolutional filters to use
nb_filters = 32
# size of pooling area for max pooling
pool_size = 2
# convolution kernel size
kernel_size = 3

if K.image_data_format() == 'channels_first':
    input_shape = (1, img_rows, img_cols)
else:
    input_shape = (img_rows, img_cols, 1)

def train_model(model, train, test, nb_classes):

    X_train = train[0].reshape((train[0].shape[0],) + input_shape)
    X_test = test[0].reshape((test[0].shape[0],) + input_shape)
    X_train = X_train.astype('float32')
    X_test = X_test.astype('float32')
    X_train /= 255
    X_test /= 255

    print('X_train shape:', X_train.shape)
    print(X_train.shape[0], 'train samples')
    print(X_test.shape[0], 'test samples')

    # convert class vectors to binary class matrices
    Y_train = np_utils.to_categorical(train[1], nb_classes)
    Y_test = np_utils.to_categorical(test[1], nb_classes)

    model.compile(loss='categorical_crossentropy',
                  optimizer='adadelta',
                  metrics=['accuracy'])

    t = now()
    model.fit(X_train, Y_train,
              batch_size=batch_size, nb_epoch=nb_epoch,
              verbose=1,
              validation_data=(X_test, Y_test))
    print('Training time: %s' % (now() - t))
    score = model.evaluate(X_test, Y_test, verbose=0)
    print('Test score:', score[0])
    print('Test accuracy:', score[1])


I am working on the file "2.4 Transfer Learning & Fine-Tuning.ipynb".
The first task is:
Your Turn
Try to Fine Tune a VGG16 Network
Can you write Python code for it?

from keras.applications import VGG16
from keras.applications.vgg16 import VGG16
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input
from keras.layers import Input, Flatten, Dense
from keras.models import Model
import numpy as np

#Get back the convolutional part of a VGG network trained on ImageNet
model_vgg16_conv = VGG16(weights='imagenet', include_top=False)
model_vgg16_conv.summary()

#Create your own input format (here 3x200x200)
inp = Input(shape=(48,48,3),name = 'image_input')

#Use the generated model
output_vgg16_conv = model_vgg16_conv(inp)

#Add the fully-connected layers
x = Flatten(name='flatten')(output_vgg16_conv)
x = Dense(4096, activation='relu', name='fc1')(x)
x = Dense(4096, activation='relu', name='fc2')(x)
x = Dense(5, activation='softmax', name='predictions')(x)

#Create your own model
my_model = Model(input=inp, output=x)

#In the summary, weights and layers from VGG part will be hidden, but they will be fit during the training
my_model.summary()


I am working on the file "2.5 - String exercises.ipynb".
The first task is:
B. both_ends
Given a string s, return a string made of the first 2
and the last 2 chars of the original string,
so 'spring' yields 'spng'. However, if the string length
is less than 2, return instead the empty string.
Can you write Python code for it?

def both_ends(s):
    # +++your code here+++
    return

test(both_ends('spring'), 'spng')
test(both_ends('Hello'), 'Helo')
test(both_ends('a'), '')
test(both_ends('xyz'), 'xyyz')


I am working on the file "2.5 - String exercises.ipynb".
The first task is:
D. mix_up
Given strings a and b, return a single string with a and b separated
by a space ' ', except swap the first 2 chars of each string.
Assume a and b are length 2 or more.
Can you write Python code for it?

def mix_up(a, b):
    # +++your code here+++
    return

test(mix_up('mix', 'pod'), 'pox mid')
test(mix_up('dog', 'dinner'), 'dig donner')
test(mix_up('gnash', 'sport'), 'spash gnort')
test(mix_up('pezzy', 'firm'), 'fizzy perm')


I am working on the file "2.5 - String exercises.ipynb".
The first task is:
E. verbing
Given a string, if its length is at least 3,
add 'ing' to its end.
Unless it already ends in 'ing', in which case
add 'ly' instead.
If the string length is less than 3, leave it unchanged.
Return the resulting string.
Can you write Python code for it?

def verbing(s):
    # +++your code here+++
    return

test(verbing('hail'), 'hailing')
test(verbing('swimming'), 'swimmingly')
test(verbing('do'), 'do')


I am working on the file "2.5 - String exercises.ipynb".
The first task is:
F. not_bad
Given a string, find the first appearance of the
substring 'not' and 'bad'. If the 'bad' follows
the 'not', replace the whole 'not'...'bad' substring
with 'good'.
Return the resulting string.
So 'This dinner is not that bad!' yields:
This dinner is good!
Can you write Python code for it?

def not_bad(s):
    # +++your code here+++
    return

test(not_bad('This movie is not so bad'), 'This movie is good')
test(not_bad('This dinner is not that bad!'), 'This dinner is good!')
test(not_bad('This tea is not hot'), 'This tea is not hot')
test(not_bad("It's bad yet not"), "It's bad yet not")


I am working on the file "2.5 - String exercises.ipynb".
The first task is:
Fill in the code for the functions below. main() is already set up
to call the functions with a few different inputs,
printing 'OK' when each function is correct.
The starter code for each function includes a 'return'
which is just a placeholder for your code.
A. doughnuts
Given an int count of a number of doughnuts, return a string
of the form 'Number of doughnuts: <count>', where <count> is the number
passed in. However, if the count is 10 or more, then use the word 'many'
instead of the actual count.
So doughnuts(5) returns 'Number of doughnuts: 5'
and doughnuts(23) returns 'Number of doughnuts: many'
Can you write Python code for it?

def doughnuts(count):
    # +++your code here+++
    return

test(doughnuts(4), 'Number of doughnuts: 4')
test(doughnuts(9), 'Number of doughnuts: 9')
test(doughnuts(10), 'Number of doughnuts: many')
test(doughnuts(99), 'Number of doughnuts: many')


I am working on the file "2.5 - String exercises.ipynb".
The first task is:
G. front_back
Consider dividing a string into two halves.
If the length is even, the front and back halves are the same length.
If the length is odd, we'll say that the extra char goes in the front half.
e.g. 'abcde', the front half is 'abc', the back half 'de'.
Given 2 strings, a and b, return a string of the form
a-front + b-front + a-back + b-back
Can you write Python code for it?

def front_back(a, b):
    # +++your code here+++
    return

test(front_back('abcd', 'xy'), 'abxcdy')
test(front_back('abcde', 'xyz'), 'abcxydez')
test(front_back('Kitten', 'Donut'), 'KitDontenut')


I am working on the file "2.Backpropagation.ipynb".
The first task is:
All together now
Can you write Python code for it?

class loss:
    def __init__(self):
        self.y = 0
    def evaluate(self,model,x,t):
        #Evaluates the loss function L(y,t)
        self.y = model.forward(x)
        return (t-self.y)*(t-self.y)
    def gradient(self,model,x,t):
        self.y = model.forward(x)
        return -2.*(t-self.y)

class reluLayer(layer): #this is a linear layer with relu activation
    def __init__(self,input_dim,n_neurons):
        self.x = np.zeros((1,input_dim))
        self.z = np.zeros((1,n_neurons))
        self.w = np.random.randn(input_dim,n_neurons) #Normal Random Initialization
        self.y = np.zeros((1,n_neurons))
    def forward(self,x): # evaluate the layer
        self.z = np.dot(x, self.w)
        self.y = np.maximum(0,self.z)
        return self.y
    def backward(self): # gradient with respect to the inputs
        return np.select([self.z>0],[self.w],default=0).T
    def gradient(self): # gradient with respect to the parameters
        dydz = np.where(self.z>0,1.,0.)
        return np.dot(dydz[:,np.newaxis],self.x).T

class model:
    def __init__(self):
        self.architecture = []
        self.y_ = []
    def addLayer(self,layer):
        self.architecture.append(layer)
    def forward(self,x):
        #Takes a data point and evaluates f(x,w)
        self.y_= x
        for layer in self.architecture:
            self.y_=layer.forward(self.y_)
        return self.y_

class optimize:
    def __init__(self):
        pass

    def run(self, data, target, model, loss, num_iter, eta):
        #Takes a loss function and a model and find the optimal parameters for an specific data set
        N_samples = data.shape[0]
        for t in xrange(num_iter):
            #Step 1.-take a sample x at random from the training set
            idx = np.random.randint(N_samples)
            xi = data[idx,:]
            yi = target[idx,:]

            #Step 2.- update the parameters to optimise
            dLdx= loss.gradient(model,xi,yi)[:,np.newaxis]
            for layer in...


I am working on the file "2.Backpropagation.ipynb".
The first task is:
All together now ... with bias
Can you write Python code for it?

import numpy as np

#Example dataset
N_samples_per_class = 1000
d_dimensions = 2
x = np.vstack((np.random.randn(N_samples_per_class, d_dimensions),np.random.randn(N_samples_per_class, d_dimensions)+np.array([3,3])))
y = np.vstack((-1.*np.ones((N_samples_per_class,1)),1.*np.ones((N_samples_per_class,1))))

%matplotlib inline
import matplotlib.pyplot as plt

idx = y==1
plt.scatter(x[idx.ravel(),0],x[idx.ravel(),1],alpha=0.1)
idx = y==-1
plt.scatter(x[idx.ravel(),0],x[idx.ravel(),1],alpha=0.1,color='pink')

import numpy as np
import tqdm

class loss:
    def __init__(self):
        self.y = 0
    def evaluate(self,model,x,t):
        #Evaluates the loss function L(y,t)
        self.y = model.forward(x)
        return (t-self.y)*(t-self.y)
    def gradient(self,model,x,t):
        self.y = model.forward(x)
        return -2.*(t-self.y)

class reluLayer(layer): #this is a linear layer with relu activation
    def __init__(self,input_dim,n_neurons):
        self.z = np.zeros((1,n_neurons))
        self.w = np.random.randn(input_dim+1,n_neurons)
        #self.w = self.w/np.dot(np.ones((input_dim+1,1)),np.sum(self.w*self.w,axis=0)[np.newaxis,:])
        self.y = np.zeros((1,n_neurons))
    def forward(self,x): # evaluate the layer
        self.xext = np.concatenate((x,np.ones((x.shape[0],1))),axis=1)
        self.z = np.dot(self.xext, self.w)
        self.y = np.maximum(0,self.z)
        return self.y
    def backward(self): # gradient with respect to the inputs
        dydz = np.tile(np.where(self.z>0.,1.,0.),(self.w.shape[0],1))
        tw = dydz*self.w
        return tw[:-1,:]
    def gradient(self): # gradient with respect to the parameters
        dydz = np.where(self.z>0.,1.,0.)
        return np.dot(dydz.T,self.xext)

class linearLayer(layer): #this is a linear layer with relu activation
    def __init__(self,input_dim,n_neurons):
        self.w = np.random.randn(input_dim+1,n_neurons)
        self.y = np.zeros((1,n_neurons))
    def forward(self,x): # evaluate the layer
        self.xext =...


I am working on the file "2.Backpropagation.ipynb".
The first task is:
Just for fun.
Let us create a new loss function more adequate for classification purposes. The hinge loss
$\mathcal{L}(y,t) = \max(0, 1-t\cdot y)$
Can you write Python code for it?

#display
class loss_hinge:
    def __init__(self):
        self.y = 0

    def evaluate(self,model,x,t):
        #Evaluates the loss function L(y,t)
        self.y = model.forward(x)
        return np.maximum(0,1-t*self.y)

    def gradient(self,model,x,t):
        self.y = model.forward(x)
        if (1-t*self.y)>0:
            return -t
        else:
            return 0

num_iter = 10000
eta = 0.01 #optimization step/learning rate

f = model(d_dimensions)
L = loss_hinge()
opt = optimize()
opt.run(x,y,f,L,num_iter,eta)

opt.plot()

delta = 0.05
xx = np.arange(-5.0, 8.0, delta)
yy = np.arange(-5.0, 8.0, delta)
XX, YY = np.meshgrid(xx, yy)
Xf = XX.flatten()
Yf = YY.flatten()
sz=XX.shape
test_data = np.concatenate([Xf[:,np.newaxis],Yf[:,np.newaxis]],axis=1);

idx = y==1
plt.scatter(x[idx.ravel(),0],x[idx.ravel(),1],alpha=0.1)
idx = y==-1
plt.scatter(x[idx.ravel(),0],x[idx.ravel(),1],alpha=0.1,color='pink')

Z = f.forward(test_data)
Z.shape=sz

plt.imshow(Z, interpolation='bilinear', origin='lower', extent=(-6,8,-6,8),alpha=0.3, vmin=-15, vmax=15)
plt.contour(XX,YY,Z,[0])
fig = plt.gcf()
fig.set_size_inches(9,9)


I am working on the file "2.Backpropagation.ipynb".
The first task is:
Let us add some control to check how it works. We will compute the loss every 1000 iterations, just for  checking purposes and add a plot method for showing convergence.
Can you write Python code for it?

import matplotlib.pyplot as plt
class optimize:
    def __init__(self):
        pass

    def run(self, data, target, model, loss, num_iter, eta):
        #Takes a loss function and a model and find the optimal parameters for an specific data set
        self.l = np.zeros((np.ceil(num_iter/1000),1))
        N_samples = data.shape[0]
        print N_samples
        i=0
        for t in xrange(num_iter):
            #Step 1.-take a sample x at random from the training set
            idx = np.random.randint(N_samples)
            xi = data[idx,:]
            yi = target[idx]
            # <-- Start new code
            if t%1000==0:
                self.l[i] = np.sum(loss.evaluate(model,data[:1000,:],target[:1000,0]))
                i=i+1
            ## End new code -->
            #Step 2.- update the parameters to optimise
            model.w = model.w - eta * loss.gradient(model,xi,yi)*model.gradient(xi)
    def plot(self):
        plt.plot(self.l)


I am working on the file "2.Backpropagation.ipynb".
The first task is:
Let us put all the pieces together and solve a toy problem
Can you write Python code for it?

import numpy as np

#Example dataset
N_samples_per_class = 1000
d_dimensions = 2
x = np.vstack((np.random.randn(N_samples_per_class, d_dimensions),np.random.randn(N_samples_per_class, d_dimensions)+np.array([3,3])))
y = np.vstack((-1.*np.ones((N_samples_per_class,1)),1.*np.ones((N_samples_per_class,1))))

%matplotlib inline
import matplotlib.pyplot as plt

idx = y==1
plt.scatter(x[idx.ravel(),0],x[idx.ravel(),1],alpha=0.1)
idx = y==-1
plt.scatter(x[idx.ravel(),0],x[idx.ravel(),1],alpha=0.1,color='pink')

delta = 0.05
xx = np.arange(-5.0, 8.0, delta)
yy = np.arange(-5.0, 8.0, delta)
XX, YY = np.meshgrid(xx, yy)
Xf = XX.flatten()
Yf = YY.flatten()
sz=XX.shape
test_data = np.concatenate([Xf[:,np.newaxis],Yf[:,np.newaxis]],axis=1);

idx = y==1
plt.scatter(x[idx.ravel(),0],x[idx.ravel(),1],alpha=0.1)
idx = y==-1
plt.scatter(x[idx.ravel(),0],x[idx.ravel(),1],alpha=0.1,color='pink')

f=model(d_dimensions)
Z = f.forward(test_data)
Z.shape=sz

plt.imshow(Z, interpolation='bilinear', origin='lower', extent=(-6,8,-6,8),alpha=0.3, vmin=-15, vmax=15)
plt.contour(XX,YY,Z,[0])
fig = plt.gcf()
fig.set_size_inches(9,9)


I am working on the file "2.Backpropagation.ipynb".
The first task is:
Let us train a classifier first.
Can you write Python code for it?

num_iter = 100000
eta = 0.001 #optimization step/learning rate

f = model(d_dimensions)
L = loss_hinge()
opt = optimize()
opt.run(x,y,f,L,num_iter,eta)


I am working on the file "2.Backpropagation.ipynb".
The first task is:
The gradient with respect to the parameter to optimise is $$\nabla_\omega\mathcal{L(f(x,\omega),t)} = (\frac{\partial \mathcal{L}}{\partial \omega_1},\frac{\partial \mathcal{L}}{\partial \omega_2}, \dots, \frac{\partial \mathcal{L}}{\partial \omega_d}) = \frac{\partial \mathcal{L}}{\partial \bar{\omega}}$$.
Recall the graph representation
<img src = "./images/pipeline1.png"  width = "200">
and consider the chain rule
$$\frac{\partial \mathcal{L}}{\partial \bar{\omega}} = \frac{\partial \mathcal{L}}{\partial y}\frac{\partial y}{\partial \bar{\omega}}$$
Observe that the complete differentiation is easier this way.
Given that $\mathcal{L}(y,t) = (t-y)^2$, then
$$\frac{\partial \mathcal{L}}{\partial y} = -2(t-y)$$
Given that $y = \omega^Tx$, then
$$ \frac{\partial y}{\partial \bar{\omega}} = x$$
The flow of the chain rule
Let us try to understand visually what the chain rule is about. This will constitute the basis of the backpropagation algorithm, that will be next used to train deep learning techniques and the basis of automatic differentiation, a programatically optimum way for computing differentials.
Can you write Python code for it?

class optimize:
    def __init__(self):
        pass

    def run(self, data, target, model, loss, num_iter, eta):
        #Takes a loss function and a model and find the optimal parameters for an specific data set
        N_samples = data.shape[0]
        for t in xrange(num_iter):
            #Step 1.-take a sample x at random from the training set
            idx = np.random.randint(N_samples)
            xi = data[idx,:]
            yi = target[idx]

            #Step 2.- update the parameters to optimise
            model.w = model.w - eta * loss.gradient(model,xi,yi)*model.gradient(xi)


I am working on the file "2.Backpropagation.ipynb".
The first task is:
Thus we need to add the gradient terms to the model and the loss.
Can you write Python code for it?

import numpy as np
class model:
    def __init__(self,d):
        self.w = np.zeros((1,d+1))
    def forward(self,x):
        #Takes a data point and evaluates f(x,w)
        return np.dot(self.w[0,:-1],x.T)+self.w[0,-1]
    def gradient(self,x):
        return np.concatenate((x,np.array([1])))
    def gradient_x(self,x):
        return self.w[0,:-1]

class loss:
    def __init__(self):
        self.y = 0

    def evaluate(self,model,x,t):
        #Evaluates the loss function L(y,t)
        self.y = model.forward(x)
        return (t-self.y)*(t-self.y)

    def gradient(self,model,x,t):
        self.y = model.forward(x)
        return -2.*(t-self.y)


I am working on the file "2.Backpropagation.ipynb".
The first task is:
We know one way of optimising algorithms in front of large scale data sets, i.e. stochastic subgradient methods. In short this is an iterative algorithm that updates the unknown variables proportional to minus the gradient magnitude at each iteration.
Let us fill in the gaps for a classical linear regression where the model is $f(x,\omega) = \omega^Tx$,
Can you write Python code for it?

import numpy as np
class model:
    def __init__(self,w):
        self.w = w
    def forward(self,x):
        #Takes a data point and evaluates f(x,w)
        return np.dot(self.w,x.T)


I am working on the file "2.Backpropagation.ipynb".
The first task is:
using a least squares loss function $\mathcal{L}(y,t) = (t-y)^2$,
Can you write Python code for it?

class loss:
    def __init__(self):
        pass
    def evaluate(self,model,x,t):
        #Evaluates the loss function L(y,t)
        y = model.forward(x)
        return (t-y)*(t-y)


I am working on the file "2.Backpropagation.ipynb".
The first task is:
using stochastic gradient descend,
Can you write Python code for it?

class optimize:
    def __init__(self,t):
        self.num_iter = t

    def run(self, data, target, model, loss):
        #Takes a loss function and a model and find the optimal parameters for an specific data set
        N_samples = data.shape[0]
        for t in xrange(self.num_iter):
            #Step 1.-take a sample x at random from the training set
            idx = np.random.randint(N_samples)
            xi = data[idx,:]
            yi = target[idx]

            #Step 2.- update the parameters to optimise
            model.w = model.w - eta * "gradient_L_omega(xi,yi)"


I am working on the file "2.DataPreprocessing.ipynb".
The first task is:
Exercise

Load the drinks1960 and drinks1980 csv files and fix the column header
Can you write Python code for it?

df2 = pd.read_csv('data/drinks1980.csv')
years2 = list(range(1999, 1979, -1))
header2 = ['description']
header2.extend(years2)
df2.columns = header2
df2.head()

df3 = pd.read_csv('data/drinks1960.csv')
years3 = list(range(1979, 1959, -1))
header3 = ['description']
header3.extend(years3)
df3.columns = header3
df3.head()


I am working on the file "2.DataPreprocessing.ipynb".
The first task is:
Lets check in the value whether we have numeric or not
Can you write Python code for it?

df.value.unique()

df[df.value.str.isnumeric() == False].shape


I am working on the file "2.DataPreprocessing.ipynb".
The first task is:
Model the Data

"All models are wrong, Some of them are useful"

Statistical testing
The power and limits of models
Tradeoff between Prediction Accuracy and Model Interpretability
Assessing Model Accuracy
Regression models (Simple, Multiple)
Classification model

We want to test whether the alcohol consumption has really declined in recent times. To do that we will do a t-test, in which we will check whether the consumption before and after 1990 are really different.
https://en.wikipedia.org/wiki/Student%27s_t-test#Independent_two-sample_t-test
https://en.wikipedia.org/wiki/Welch%27s_t-test
Can you write Python code for it?

dfSing.head()

dfSingAll = dfSing[dfSing.beverage == 'all'].copy()

dfSingAll.head()

# Create a new column
dfSingAll['split'] = dfSingAll.year < 1990

dfSingAll.head()

# Let us plot the two samples
dfSingAll.hist(column = "serving", by = "split", sharex = True, sharey= True)

from scipy import stats
np.random.seed(12345678)

sampleA = dfSingAll[dfSingAll.split == True].serving
sampleB = dfSingAll[dfSingAll.split == False].serving

sampleA.shape

sampleB.shape

stats.ttest_ind(sampleA, sampleB, equal_var = False, nan_policy = 'omit')


I am working on the file "2.DataPreprocessing.ipynb".
The first task is:
Principle: append one dataframe to another
Can you write Python code for it?

df1.shape

df2.shape

df = df1.append(df2)

df.shape

df = df.append(df3)

df.shape


I am working on the file "2.DataPreprocessing.ipynb".
The first task is:
Principle: mutate to create new variables
It is hard to think of alcohol in terms of 'litres of pure alcohol content'. It is easy to understand in terms of number of typical serving of drinks.
http://rethinkingdrinking.niaaa.nih.gov/How-much-is-too-much/what-counts-as-a-drink/whats-A-Standard-drink.aspx

For one standard serving -
- One glasses of wine: 12% alcohol in 5fl oz of standard serving of wine glass (0.6 fl oz)
- One can of beer: 5% alcohol in 12fl oz of a standard serving of beer can (0.6 fl oz)
- One shot of spirits: 40% alcohol in 1.5fl oz of standard shot of spirit (0.6 fl oz)
1 US fluid ounce (fl oz) = 0.0295735 litres (l) ~ 30ml
So for:
- 1 Standard serving of Wine = 12% * 5 * 0.03= 0.018 litres of pure alcohol
- 1 Standard serving of Beer = 5% * 12 * 0.03= 0.018 litres of pure alcohol
- 1 Standard serving of Spirit = 40% * 1.5 * 0.03 = 0.018 litres of pure alcohol
Can you write Python code for it?

df.dtypes

df['serving'] = round(df['value']/0.018, 0)

df.head()


I am working on the file "2.DataPreprocessing.ipynb".
The first task is:
Principle: str to extract text from a strings
String manipulation is very common and we often need to extract a substring from a long string. In this case, we want to get the country and type of beverage from the description
Can you write Python code for it?

df.head()

df['country'] = df.description.str.split(';').str[0]

df.head()

df['beverage'] = df.description.str.split(";").str[-1]

df.tail()


I am working on the file "2.DataPreprocessing.ipynb".
The first task is:
We can now drop the description column from our dataframe
Can you write Python code for it?

df.drop('description', axis = 1, inplace= True)

df.head()


I am working on the file "2.DataPreprocessing.ipynb".
The first task is:
We will use pd.to_numeric which will coerce to NaN everything that cannot be converted to a numeric value, so strings that represent numeric values will not be removed. For example '1.25' will be recognized as the numeric value 1.25
Can you write Python code for it?

df.value = pd.to_numeric(df.value, errors='coerce')

df.value.unique()

df.dtypes

df.country.unique()

df.beverage.unique()

# Convert from an np array to a list
beverage_old = df.beverage.unique().tolist()

beverage_old

# Create a new list with white space removed and shorter names
beverage_new = ['all', 'beer', 'wine', 'spirits', 'others']

beverage_new

df.beverage = df.beverage.replace(beverage_old, beverage_new)


I am working on the file "2.LINEAR_MODELS_FOR_REGRESSION/Old_Faithful_Geyser_Experiment.ipynb".
The first task is:
上面对余量residues进行分析，发现可视化residues后，residues依然存在某种模式(pattern)，resudue不能很好的模拟随机的噪声，所以现在得到模型还不能很好地来描述数据，于是我们需要进一步来优化修正。
从residues图形我们可以看出其是分段线性的，在waiting=70这里点上将数据分成两部分，我们可以通过人为观察来构造几个非线性函数将原数据转化成新的特征，比如这里我们新加入两个特征，分别是$max(0, \textbf{$X_i$}-68)$和$max(0, \textbf{$X_i$}-72)$。
下面是加入非线性转换得到的线性模型的代码：
Can you write Python code for it?

xi = waiting.values
feature2 = np.maximum(0, xi-68)
feature3 = np.maximum(0, xi-72)
train_data = {'f1': xi, 'f2': feature2, 'f3': feature3}
trainXFrame = pd.DataFrame(train_data)

linreg2 = LinearRegression()
linreg2.fit(trainXFrame.values, trainy)

x = np.linspace(40, 100, 200)
f2 = np.maximum(0, x-68)
f3 = np.maximum(0, x-72)
tdata = {'f1': x, 'f2': f2, 'f3': f3}
tX = pd.DataFrame(tdata)
y = linreg2.predict(tX.values)

fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot()
plt.scatter(waiting.values, eruptions.values)
plt.plot(x, y, 'r')
plt.xlim(40,100)
plt.xlabel("waiting")
plt.ylabel("duration")
plt.title("Old Faithful Geyser Data")
plt.show()


I am working on the file "2.LINEAR_MODELS_FOR_REGRESSION/Old_Faithful_Geyser_Experiment.ipynb".
The first task is:
通过上面的可视化过程，我们可以认为其数据的分布是近似线性的，所以我们可以用简单的线性回归模型来去拟合数据。
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
linreg = LinearRegression()
trainx = waiting.values.reshape(272, 1)
trainy = eruptions.values
linreg.fit(trainx, trainy)

print "intercept b0 = %f" % linreg.intercept_
print "coefficiet b1 = %f" % linreg.coef_[0]

x = np.linspace(40, 100, 200)
x = x.reshape(200,1)
y = linreg.predict(x)

fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot()
plt.scatter(waiting.values, eruptions.values)
plt.plot(x, y, 'r')
plt.xlim(40,100)
plt.ylim(1,6)
plt.xlabel("waiting")
plt.ylabel("duration")
plt.title("Old Faithful Geyser Data")
plt.show()


I am working on the file "2.ProcessText Data.ipynb".
The first task is:
請修改以下的程式碼，並計算"好吃"所有出現次數，注意!!不是頁面數
提示:修改 word_count 函式
Can you write Python code for it?

def word_count(text):

    return text.count(u"好吃")

print "好吃出現了",word_count(u"老師好吃好吃好吃好吃！！！！"),"次"

pixnet.mapValues(word_count).collect()

total_count = pixnet.mapValues(word_count).map(lambda x : x[1]).reduce(lambda x,y: x+y)
if total_count == 23 : print "你答對了"
else : print "答錯了！你的答案是 %d, 正確答案是59" % (total_count)


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
1.4 Python's print() function
The print function is used to print variables and expressions to screen. print() offers a lot of functionality which we'll encounter during the workshop. For now, note that:<br/>

You can pass anything to the print() function and it will attempt to print its arguments.
Can you write Python code for it?

print(1) # Print a constant

x = 2014
print(x) # Print an integer variable

xstr = "Hello World." # Print a string
print(xstr)

print(x,xstr) # Print multiple objects

print("String 1" + " " + "String2") # Concatenate multiple strings and print them


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
2. Numeric Variable Types
2.1 Integers
Can you write Python code for it?

year = 2014
print(year)

print("The year is %d." % year)

print(type(year))

help(year)

help(int)


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
3. Basic Operators
3.1 Arithmetic Operators
Standard arithmetic operators for addition (+), subtraction (-), multiplication (*) and division (/) are supported in Python. We have already seen use of the addition (+) and division (/) operators. Some more operators that are commonly encountered are demonstrated below.
Can you write Python code for it?

x = 2**3  # ** is the exponentiation operator
print(x)

x = 9 % 4 # % is the modulus operator
print(x)

x = 9 // 4 # // is the operator for floor division
print(x)


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
3.2. Assignment Operators
In addition to using the = (simple assignment operator) for assigning values to variables, one can use a composite assignment operator(+=, -=, etc.) that combines the simple assignment operator with all of these arithmetic expressions. For example:
Can you write Python code for it?

x = 2.0
y = 5.0
y += x # y = y + x
print(y)

y %= x # y = y%x
print(y)


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
5.2. Tuples

Tuples are another sequence data type consisting of arbitrary items separated by commas. In contrast to lists, tuples are immutable, i.e., they cannot be modified. See below for a declaration of a tuple. Note the parentheses in the declaration.
Can you write Python code for it?

# pyTuple contains an integer, string and floating point number
pyTuple = (2014,"02 June", 74.5)

# Print all the elements of pyTuple
print("pyTuple is: ")
print(pyTuple)
print("\n")

# Print the length of pyTuple obtained using the len() function
print("Length of pyTuple is: {0}.\n".format(len(pyTuple)))


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
5.3. Dictionaries

A Python dictionary is an unordered set of key:value pairs that acts as an associate arrays. The keys are immutable and unique within one dictionary. In contrast to lists and tuples, dictionaries are indexed by keys. Note the use of curly braces in the declaration of the dictionary below.
Can you write Python code for it?

pyDict = {"Canada":"CAN","Argentina":"ARG","Austria":"AUT"}
print("pyDict: {0}.".format(pyDict))

print("pyDict['Argentina']: " + pyDict['Argentina'] + ".") # Print the value corresponding to key 'afghanistan'

print(pyDict.keys())

print(pyDict.values()) # Return all the values in the dictionary as a list.

print(pyDict.items()) # Return key, value pairs from the dictionary as a list of tuples.


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
6.5 <tt>break</tt> and <tt>continue</tt> Statements

Now, if some condition is evaluated within the for/while loop and based on that, we wish to exit the loop, we can use the break statement. Note that the break statement exits the innermost loop which contains it.
Can you write Python code for it?

countryList = ["Canada", "United States of America", "Mexico"]
for country in countryList:
    if(country == "United States of America"):
        # if the country name matches, then break out of the for loop
        break
    else:
        # do some processing
        print(country)


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
7. Python File I/O

This is a quick intro to reading and writing plain text files in Python. As we proceed through the workshop, we'll look at more sophisticated ways of reading/writing files, in non-English languages and using specialized Python modules to handle files in formats such as CSV, JSON.

7.1 Writing to a File

In order to write to a file, the syntax is very similar. Open the file using the "w" mode instead of the "r" mode. Use the write() method of the file object as shown below. The syntax for the write() method is very similar to print(). Although, it does not automatically insert a newline at the end of the statement as does print().
Can you write Python code for it?

filename = "tmp.txt"
fout = open(filename,"w") # The 'r' option indicates that the file is being opened to be read

for i in range(0,5): # Read in each line from the file
    # Do some processing
    fout.write("i = {0}.\n".format(i))

fout.close() # Once the file has been read, close the file


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
Alternative syntax for writing to file using 'with open' is shown below.
Can you write Python code for it?

filename = "tmp.txt"
with open(filename,"w") as fout:
    for i in range(0,5):
        fout.write("i = {0}.\n".format(i))

fout.close()


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
An input file can also be read in as one string by using the read() method.

7.3. The <tt>csv</tt> module

Python's <tt>csv</tt> module provides convenient functionality for reading and writing csv files similar to that available in R. The csv files can then be imported in other statistical packages such as R and Excel.

Here is a short example of using the csv module to write consecutive rows in to a comma-separated file. The delimiter can be chosen to be an arbitrary string.
Can you write Python code for it?

import csv

with open("game.csv","wb") as csvfile:
    csvwriter = csv.writer(csvfile,delimiter=',')
    csvwriter.writerow(["rock","paper","scissor"])
    csvwriter.writerow(["ji","gu","pa"])
    csvwriter.writerow(["rock","paper","scissor","lizard","Spock"])

cat game.csv


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
And this is an example of reading the games.csv file. Each row of the csv file is read in as a list.
Can you write Python code for it?

import csv

with open("game.csv","r") as csvfile:
    csvreader = csv.reader(csvfile,delimiter=",")
    for row in csvreader:
        print(row)


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
For multi-line comments, use triple-quoted strings.
Can you write Python code for it?

"""This is a multi-line comment.
Assign value 1 to variable x."""
x = 1


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
For web-scraping and text-processing type tasks, we'd like better control over how things get printed out, such as the number of decimal places when printing out floating point numbers. Use the format() method on the string to be printed out to control the output format.
Can you write Python code for it?

x = 1
print("Formatted integer is {0:06d}".format(x)) # Note the format specification, 06d, for the integer.

y = 12.66666666667
print("Formatted floating point number is {0:2.3f}".format(y)) # Note the format specification, 2.3f, for the floating point number.

iStr = "Hello World"
fStr = "Goodbye World"
print("Initial string: {0:s} . Final string: {1:s}.".format(iStr,fStr)) # Note the format specification, s, for the string.

print("Initial string: {0} . Final string: {1}.".format(iStr,fStr)) # In this case, omitting the s format specified works too.

x = 1
print("Formatted integer is {0:06d}".format(x))

y = 12.66666666667
print("Formatted floating point number is {0:2.3f}".format(y))


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
If, instead of exiting the loop, one merely wishes to skip that iteration, then use the continue statement as shown here.
Can you write Python code for it?

countryList = ["Canada", "United States of America", "Mexico"]
for country in countryList:
    if(country == "United States of America"):
        # if the country name matches, then break out of the for loop
        continue
    else:
        # do some processing
        print(country)


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
Lastly, we can have lists of dictionaries
Can you write Python code for it?

pyNested = [{"year":2011,"countries":["Canada","Argentina"]},
            {"year":2008,"countries":["Canada","Austria"]},
            {"year":2006,"countries":["Canada","Austria"]},
            {"year":2013,"countries":["Argentina","Austria"]}]
print("pyNested[0] = {0}".format(pyNested[0]))

print("pyNested[0]['year'] = {0}, pyNested[0]['countries'] = {1}.".format(pyNested[0]['year'],pyNested[0]['countries']))


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
List elements can be individually referenced using their index in the list. Python indexing starts with 0 and runs up to the length of the sequence - 1. The square bracket is used to specify the index in to the list. This notation can also be used to assign values to the elements of the list. In contrast to strings, lists are mutable.
Can you write Python code for it?

print(pyList)
print("\n")

# Print the first element of pyList. Remember, indexing starts with 0.
print("First element of pyList: {0}.\n".format(pyList[0]))

# Print the last element of pyList. Last element can be conveniently indexed using -1.
print("Last element of pyList: {0}.\n".format(pyList[-1]))

# Also the last element has index = (length of list - 1)
check = (pyList[2] == pyList[-1])
print("Is pyList[2] equal to pyList[-1]?\n{0}.\n".format(check))

# Assign a new value to the third element of the list
pyList[2] = -99.0
print("Modified element of pyList[2]: {0}.\n".format(pyList[2]))


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
One can declare a tuple of lists.
Can you write Python code for it?

pyNested = (["rock","paper","scissors"],["ji","gu","pa"])
pyNested[0][2] = "lizard" # OK, list within the tuple is mutable
print(pyNested[0]) # Print first list element of the tuple


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
One can declare tuples of tuples.
Can you write Python code for it?

pyTuples = (("rock","paper","scissors"),("ji","gu","pa"))
print("pyTuples[0] = {0}.".format(pyTuples[0])) # Print the first sub-tuple in pyTuples.
print("pyTuples[1] = {0}.".format(pyTuples[1])) # Print the second sub-tuple in pyTuples.


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
Parsing hierarchical data structures involving Python dictionaries will be very useful when working with the JSON data format and APIs such as the Twitter API.

Values in a dictionary can be any object including other dictionaries.
Can you write Python code for it?

pyDicts = {"Canada":{"Alpha-2":"CA","Alpha-3":"CAN","Numeric":"124"},
               "Argentina":{"Alpha-2":"AR","Alpha-3":"ARG","Numeric":"032"},
                                         "Austria":{"Alpha-2":"AT","Alpha-3":"AUT","Numeric":"040"}}

print("pyDicts['Canada'] = {0}.".format(pyDicts['Canada']))

print("pyDicts['Canada']['Alpha-2'] = {0}.".format(pyDicts['Canada']['Alpha-2']))


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
Python lists can be nested - list within a list within a list and so on. An index needs to be specified for each level of nesting.
Can you write Python code for it?

pyLists = [["rock","paper","scissors"], ["ji","gu","pa"]]

# Print the first element (0-th index) of pyLists which is itself a list
print("pyLists[0] = ")
print(pyLists[0])
print("\n")

# Print the 0-th index element of the first list element in pyLists
print("pylists[0][0] = " + pyLists[0][0] + ".")
print("\n")

# Print the second element of pyLists which is itself a list
print("pyLists[1] = ")
print(pyLists[1])
print("\n")

# Print the 0-th index element of the second list element in pyLists
print("pyLists[1][0] = " + pyLists[1][0] + ".")
print("\n")

pyList = [1,3,4,2]
pyList.sort(reverse=True)
sum(pyList)
2*(pyList)
#2**(pyList)


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
Scripting languages, such as Python, make it easy to automate repetitive tasks. In this workshop, we'll use two of Python's syntactic constructs for iteration - the for loop and the while loop.

6.2 <tt>for</tt> Statements

Given an iterable, such as a list, the for loop construct can iterate over each of its values as shown below.
Can you write Python code for it?

countryList = ["Canada", "United States of America", "Mexico"]
for country in countryList: # Loop over countryList, set country to next element in list.
    print(country)

countryDict = {"Canada":"124","United States":"840","Mexico":"484"}
print("Country\t\tISO 3166-1 Numeric Code")
for country,code in countryDict.items(): # Loop over all the key and value pairs in the dictionary
    print("{0:12s}\t\t{1:12s}".format(country,code))


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
Since, the Python string is a sequence of characters, individual characters in the string can be indexed. Note that, unlike R, in Python sequences indexing starts at 0 and goes up to one less than the length of the sequence.
Can you write Python code for it?

str1 = "Python"
print(str1[0]) # Print the first character element of the string.

print(str1[len(str1)-1]) # Print the last character element of the string.

print(str1[2:4]) # Print a 2-element slice of the string, starting from the 2-nd element up to but not including the 4-th element.


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
Strings are immutable. That is, an existing instance of a string cannot be modified. Instead, a new string that contains the modification should be created.
Can you write Python code for it?

str1 = "Python"
str1[1] = "3" # Error, strings can't be modified.


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
Strings can also be repeated with the multiplication (*) operator.
Can you write Python code for it?

str1 = "Rock,Paper,Scissors\n"
repeatStr = str1*5
print(repeatStr)


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
The code below demonstrates another way to open a file and read each line. With this syntax, the file is automatically closed after the <tt >with</tt> block.
Can you write Python code for it?

filename = "tmp.txt"
with open(filename,"r") as fin:
    for line in fin:
        # Do some processing
        print(line)


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
The len() function returns the length of a string.
Can you write Python code for it?

str1 = "Python"
lenStr1 = len(str1)
print("The length of str: is " + str(lenStr1) + ".")


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
Tuples are immutable. Attempting to change elements of a tuple will result in errors.
Can you write Python code for it?

pyTuple[1] = "31 December" # Error as pyTuple is a tuple and hence, immutable


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
Tuples can be packed from and unpacked into individual elements.
Can you write Python code for it?

pyTuple = "rock", "paper", "scissors" # pack the strings into a tuple named pyTuple
print(pyTuple)

str0,str1,str2 = pyTuple # unpack the tuple into strings named str0, str1, str2
print("str0 = " + str0 + ".")
print("str1 = " + str1 + ".")
print("str2 = " + str2 + ".")


I am working on the file "2.Python_Quickstart.ipynb".
The first task is:
Values in a dictionary can also be lists.
Can you write Python code for it?


I am working on the file "2.Topics.ipynb".
The first task is:
For the stop words we use the one given by nltk. This set seems small so we include also other common English stop words found online or in the titles
Can you write Python code for it?

english_stop_words = ["a", "about", "above", "above", "across", "after", "afterwards", "again", "against", "all", "almost", "alone", "along", "already", "also","although","always","am","among", "amongst", "amoungst", "amount",  "an", "and", "another", "any","anyhow","anyone","anything","anyway", "anywhere", "are", "around", "as",  "at", "back","be","became", "because","become","becomes", "becoming", "been", "before", "beforehand", "behind", "being", "below", "beside", "besides", "between", "beyond", "bill", "both", "bottom","but", "by", "call", "can", "cannot", "cant", "co", "con", "could", "couldnt", "cry", "de", "describe", "detail", "do", "done", "down", "due", "during", "each", "eg", "eight", "either", "eleven","else", "elsewhere", "empty", "enough", "etc", "even", "ever", "every", "everyone", "everything", "everywhere", "except", "few", "fifteen", "fify", "fill", "find", "fire", "first", "five", "for", "former", "formerly", "forty", "found", "four", "from", "front", "full", "further", "get", "give", "go", "had", "has", "hasnt", "have", "he", "hence", "her", "here", "hereafter", "hereby", "herein", "hereupon", "hers", "herself", "him", "himself", "his", "how", "however", "hundred", "ie", "if", "in", "inc", "indeed", "interest", "into", "is", "it", "its", "itself", "keep", "last", "latter", "latterly", "least", "less", "ltd", "made", "many", "may", "me", "meanwhile", "might", "mill", "mine", "more", "moreover", "most", "mostly", "move", "much", "must", "my", "myself", "name", "namely", "neither", "never", "nevertheless", "next", "nine", "no", "nobody", "none", "noone", "nor", "not", "nothing", "now", "nowhere", "of", "off", "often", "on", "once", "one", "only", "onto", "or", "other", "others", "otherwise", "our", "ours", "ourselves", "out", "over", "own","part", "per", "perhaps", "please", "put", "rather", "re", "same", "see", "seem", "seemed", "seeming", "seems", "serious", "several", "she", "should", "show", "side", "since", "sincere", "six", "sixty", "so", "some", "somehow", "someone", "something",...


I am working on the file "2.Topics.ipynb".
The first task is:
Indeed the lemmatizer keep 2 different words. Let's use the stemmer
Can you write Python code for it?

def pre_processing(titles):
    list_of_tokens = []
    for title in titles:
        tokens = utils.simple_preprocess(title)
        tokens = [stemmer.stem(x) for x in tokens]
        tokens = list(filter(lambda t: t not in sw, tokens))
        list_of_tokens.append(tokens)
    return list_of_tokens
authorID_to_titles_stem = {id_: pre_processing(titles) for id_, titles in tqdm(authorID_to_titles.items())}

utl.pickle_data(authorID_to_titles_stem, "../pmi_data/authorID_to_titles_stem.p")


I am working on the file "2.Topics.ipynb".
The first task is:
Now we have a list of author->document. We can build the dictionaray and transform each document to a list of (token, token_count)
Can you write Python code for it?

dictionary = corpora.Dictionary([doc for doc in tqdm(authorID_to_document.values())])
corpus = [dictionary.doc2bow(doc) for doc in tqdm(authorID_to_document.values())]


I am working on the file "2.Topics.ipynb".
The first task is:
Set up the number of parameter, we select 20 topics.
Can you write Python code for it?

#parameters
num_topics = 20 # number of topics LDA has to select
passes = 1 # number of passe in the lda training
num_words = 5 # number of most important word in one topic to be printed

tmp = corpus

corpus = tmp
corpus = np.random.choice(corpus, int(len(corpus)/1000))

len(corpus)

c = [c for c in tqdm(tmp) if len(c)> 100]
len(c)

start = time()
pp = pprint.PrettyPrinter(depth=2)
lda = models.LdaModel(c, num_topics=num_topics, id2word = dictionary, passes=passes)
print("Training time:", round((time()-start)/60,2),"[min]")
pp.pprint(lda.print_topics(lda.num_topics, num_words=num_words))
lda.save('lda.model')
utl.pickle_data(lda, "../pmi_data/lda_model__20_100.p")

def compute_score(titles):
    total_score = np.zeros(num_topics)
    for title in titles:
        #lda output : [(id1, score1), (id2, score2),... if id != 0]
        for id_, value in lda[dictionary.doc2bow(title)]:
            total_score[id_] += value
    return total_score

score_by_author_by_document = [compute_score([doc]) for _, doc in tqdm(authorID_to_document.items())]
utl.pickle_data(score_by_author_by_document, "../pmi_data/score_by_author_by_document.p")

score_by_author_by_titles = [compute_score(titles) for _, titles in tqdm(authorID_to_titles_stem.items())]
utl.pickle_data(score_by_author_by_titles,"../pmi_data/score_by_author_by_titles.p")