I am working on the file "GSOC/notebooks/ipython/examples/Notebook/Running Code.ipynb".
The first task is:
Cell menu
The "Cell" menu has a number of menu items for running code in different ways. These includes:

Run and Select Below
Run and Insert Below
Run All
Run All Above
Run All Below

Restarting the kernels
The kernel maintains the state of a notebook's computations. You can reset this state by restarting the kernel. This is done by clicking on the <button class='btn btn-default btn-xs'><i class='fa fa-repeat icon-repeat'></i></button> in the toolbar above.
sys.stdout and sys.stderr
The stdout and stderr streams are displayed as text in the output area.
Can you write Python code for it?

print("hi, stdout")

from __future__ import print_function
print('hi, stderr', file=sys.stderr)


I am working on the file "GSOC/notebooks/ipython/examples/Notebook/Running Code.ipynb".
The first task is:
If the Kernel dies you will be prompted to restart it. Here we call the low-level system libc.time routine with the wrong argument via
ctypes to segfault the Python interpreter:
Can you write Python code for it?

import sys
from ctypes import CDLL
# This will crash a Linux or Mac system
# equivalent calls can be made on Windows
dll = 'dylib' if sys.platform == 'darwin' else 'so.6'
libc = CDLL("libc.%s" % dll)
libc.time(-1)  # BOOM!!


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Condensed.ipynb".
The first task is:
Defining your own magic
As we have seen already, IPython has cell and line magics. You can define your own magics using any Python function and the register_magic_function method:
Can you write Python code for it?

from IPython.core.magic import (register_line_magic, register_cell_magic,
                                register_line_cell_magic)

@register_line_magic
def sleep(line):
    """A simple function for sleeping"""
    import time
    t = float(line)
    time.sleep(t)

%sleep 2

%sleep?


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Condensed.ipynb".
The first task is:
Let's peek at our config file
Can you write Python code for it?


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Condensed.ipynb".
The first task is:
Restart the kernel and then run the following cells immediately to verify these scripts have been executed:
Can you write Python code for it?


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Condensed.ipynb".
The first task is:
Startup Files
Startup files are simple Python or IPython scripts
that are run whenever you start IPython.
These are a useful way to do super common imports,
or for building database connections to load on startup of a non-default profile.
We can use a startup file to ensure that our %tic/toc magics are always defined,
every time we start IPython.
Can you write Python code for it?

!ls $profile/startup

!cat $profile/startup/README


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Condensed.ipynb".
The first task is:
This creates a profile in your IPYTHONDIR (ipython locate is a quick way to see where your IPYTHONDIR is),
and populates it with automatically generated default config files.
Can you write Python code for it?

!ipython locate profile default
!ipython locate profile newprofile


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Condensed.ipynb".
The first task is:
This is an important trick for finding out configuration info:
$> ipython [subcommand] --help-all | grep [-C context] PATTERN

--help-all exposes everything configurable in IPython,
there is a good chance you will find what you are looking for.
A common configuration question is:

how do I disable the "Do you really want to exit" message when quitting with Ctrl-d?

Well, logically this has to do with exit, so let's look for it:
Can you write Python code for it?

!ipython --help-all | GREP_COLOR='1;31;46' grep --color exit


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Condensed.ipynb".
The first task is:
Which shows me that I can disable the confirmation for a single IPython session with
$> ipython --no-confirm-exit

or I can set the TerminalInteractiveShell.confirm_exit=False in a config file,
to have it be the default behavior.
Configuration principles
Here are the design principles of the IPython configuration system:

Configuration is always done using class attributes
Classes that have configurable attributes are subclasses of Configurable
Attributes that are configurable are typed traitlets objects (Bool, Unicode, etc.) that have config=True
In config files, configurable attributes can be set using the format Class.attr_name=the_value
At the command line, configurable attributes can be set using the syntax --Class.attr_name=the_value
At the command line, some attributes have shorthands of the form --attr-name=value
Values set at the command line have higher priority than those set in config files

The IPython Profile
IPython has a notion of 'profiles' - these are directories that live in your IPYTHONDIR,
which contain configuration and runtime information.
Let's create the default profile
Can you write Python code for it?


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Extension.ipynb".
The first task is:
%install_ext is a function that can take any file path or URL, and puts the target into IPYTHONDIR/extensions
Can you write Python code for it?

%install_ext soln/mymagics.py

%load_ext mymagics

import time
%tic
time.sleep(0.1)
%toc

%nbrun _Sample


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Magics.ipynb".
The first task is:
A notebook is just a dictionary with attribute access for convenience.
Can you write Python code for it?


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Magics.ipynb".
The first task is:
Excercise
Can you write and register a cell magic that automates the outer iteration,
timing a block for various values of a particular variable:
Can you write Python code for it?

%load soln/scalemagic.py

%%scale N [ int(10**p) for p in range(1,4) ]

A = np.random.random((N,N))
np.linalg.eigvals(A)

%%scale N [ int(2**p) for p in np.linspace(6, 11, 11) ]

A = np.random.random((N,N))
np.linalg.eigvals(A)


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Magics.ipynb".
The first task is:
Executing Notebooks
We can load a notebook into memory using IPython.nbformat.
Can you write Python code for it?

import io
import os

import IPython.nbformat as nbf

def load_notebook(filename):
    """load a notebook object from a filename"""
    if not os.path.exists(filename) and not filename.endswith(".ipynb"):
        filename = filename + ".ipynb"
    with io.open(filename) as f:
        return nbf.read(f, as_version=4)

nb = load_notebook("_Sample")


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Magics.ipynb".
The first task is:
Exercise
Can you write and register an %nbrun line magic to run a notebook?
python
%nbrun Sample
Can you write Python code for it?


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Magics.ipynb".
The first task is:
Now I can run all of the code cells with get_ipython().run_cell
Can you write Python code for it?

for cell in cells:
    ip = get_ipython()
    if cell.cell_type == 'code':
        ip.run_cell(cell.source, silent=True)


I am working on the file "GSOC/notebooks/ipython/exercises/Customization/Magics.ipynb".
The first task is:
We can see all the cells and their type
Can you write Python code for it?

for cell in cells:
    print()
    print('----- %s -----' % cell.cell_type)
    print(cell.source)


I am working on the file "GS_Wintrich_4params.ipynb".
The first task is:
Get the MLE (i.e., the parameter values at the maximum likelihood), in the same order than the definition of the parameters
Can you write Python code for it?


I am working on the file "GS_Wintrich_4params.ipynb".
The first task is:
Grid search setup summary
Can you write Python code for it?


I am working on the file "GS_Wintrich_4params.ipynb".
The first task is:
Plot the measured concentrations and the predicted profile corresponding to the best fitted data model
Can you write Python code for it?

sns.set_context('notebook')

depths = np.linspace(profile_data['depth'].min(),
                     profile_data['depth'].max(),
                     100)
Cm_fitted = C_10Be_wintrich(depths, *gstest.mle)

plt.figure()
plt.plot(Cm_fitted, -depths, label='best-fitted model')
plt.errorbar(profile_data['C'],
             -profile_data['depth'],
             xerr=profile_data['std'],
             fmt='o', markersize=4,
             label='data')
plt.setp(plt.gca(),
         xlabel='10Be concentration [atoms g-1]',
         ylabel='-1 * depth [cm]',
         xlim=[0, None], ylim=[None, 0])

plt.legend(loc='lower right')


I am working on the file "GS_Wintrich_4params.ipynb".
The first task is:
Plot the profile log-likelihood for each parameter. The blue lines represent the difference between the profile log-likelihood and the maximum log-likelihood, The intersections between the blue line and the black lines define the confidence intervals at the given confidence levels (based on the likelihood ratio test). The red lines indicate the true values.
Can you write Python code for it?

%matplotlib inline

def plot_proflike1d(cobj, pname, clevels=[0.68, 0.95, 0.997],
                    true_val=None, ax=None):

    p = cobj.parameters[pname]
    pindex = cobj.parameters.keys().index(pname)

    x = cobj.grid[pindex].flatten()
    proflike = cobj.proflike1d[pindex]

    if ax is None:
        ax = plt.subplot(111)

    difflike = proflike - cobj.maxlike

    ax.plot(x, difflike, label='profile loglike')

    ccrit = gridsearch.profile_likelihood_crit(
        cobj.proflike1d[pindex],
        cobj.maxlike,
        clevels=clevels
    )
    ccrit -= cobj.maxlike

    for lev, cc in zip(clevels, ccrit):
        l = ax.axhline(cc, color='k')
        hpos = x.min() + (x.max() + x.min()) * 0.05
        ax.text(hpos, cc, str(lev * 100),
                size=9, color = l.get_color(),
                ha="center", va="center",
                bbox=dict(ec='1',fc='1'))

    if true_val is not None:
        ax.axvline(true_val, color='r')

    plt.setp(ax, xlabel=pname,
             ylabel='profile log-like - max log-like',
             xlim=p['range'][0:2],
             ylim=[ccrit[-1], 0.])

def plot_proflike1d_all(cobj, n_subplot_cols=2, **kwargs):

    n_subplots = len(cobj.parameters)
    n_subplot_rows = int(math.ceil(1. *
                                   n_subplots /
                                   n_subplot_cols))

    fig, aax = plt.subplots(nrows=n_subplot_rows,
                            ncols=n_subplot_cols,
                            **kwargs)
    axes = aax.flatten()
    fig.text(0.5, 0.975,
             "Profile log-like: " + cobj.description,
             horizontalalignment='center',
             verticalalignment='top')

    for i, pname in enumerate(cobj.parameters.keys()):
        ax = axes[i]
        plot_proflike1d(cobj, pname,
                        ax=ax)

    plt.tight_layout()
    plt.subplots_adjust(top=0.93)

plot_proflike1d_all(gstest, figsize=(12, 6))


I am working on the file "GS_Wintrich_4params.ipynb".
The first task is:
Set the data
Can you write Python code for it?

gstest.set_profile_measured(
    profile_data['depth'].values,
    profile_data['C'].values,
    profile_data['std'].values,
    None,
)


I am working on the file "GS_Wintrich_4params.ipynb".
The first task is:
Show the profile log-likelihood for couples of parameters. Confidence regions are also shown (also based on the likelihood ratio test).
Can you write Python code for it?

def plot_proflike2d(cobj, p1p2, ax=None,
                    cmap='Blues', show_colorbar=True):

    pname1, pname2 = p1p2
    idim = cobj.parameters.keys().index(pname2)
    jdim = cobj.parameters.keys().index(pname1)

    if ax is None:
        ax = plt.subplot(111)

    X, Y = np.meshgrid(cobj.grid[idim].flatten(),
                       cobj.grid[jdim].flatten())

    difflike = cobj.proflike2d[idim][jdim] - cobj.maxlike

    ccrit = gridsearch.profile_likelihood_crit(
        cobj.proflike2d[idim][jdim],
        cobj.maxlike,
        clevels=[0.68, 0.95]
    )
    ccrit -= cobj.maxlike

    contours = np.linspace(np.median(difflike),
                           0,
                           10)

    P2D = ax.contourf(Y, X, difflike,
                      contours,
                      cmap=plt.get_cmap(cmap))

    ci68 = ax.contour(Y, X, difflike,
                      [ccrit[0]], colors='w',
                      linestyles='solid')
    plt.clabel(ci68, fontsize=8, inline=True,
               fmt='68')
    ci95 = ax.contour(Y, X, difflike,
                      [ccrit[1]], colors=['k'],
                      linestyles='solid')
    plt.clabel(ci95, fontsize=8, inline=True,
               fmt='95')

    plt.setp(ax, xlabel=pname1, ylabel=pname2)

    if show_colorbar:
        plt.colorbar(P2D, ax=ax)

    #ax.axhline(true_exposure, color='r')
    #ax.axvline(true_erosion, color='r')

fig = plt.figure(figsize=(11, 9))
ax = plt.subplot(321)
plot_proflike2d(gstest, ('erosion_rate', 'exposure_time'), ax=ax)
ax2 = plt.subplot(322)
plot_proflike2d(gstest, ('exposure_time', 'inheritance'), ax=ax2)
ax3 = plt.subplot(323)
plot_proflike2d(gstest, ('erosion_rate', 'inheritance'), ax=ax3)
ax4 = plt.subplot(324)
plot_proflike2d(gstest, ('exposure_time', 'soil_density'), ax=ax4)
ax5 = plt.subplot(325)
plot_proflike2d(gstest, ('erosion_rate', 'soil_density'), ax=ax5)
ax6 = plt.subplot(326)
plot_proflike2d(gstest, ('soil_density', 'inheritance'), ax=ax6)
plt.tight_layout()


I am working on the file "GS_Wintrich_4params.ipynb".
The first task is:
The Data
Can you write Python code for it?

profile_data = pd.read_csv('profiles_data/wintrich_10Be_profile_data.csv',
                           index_col='sample',
                           delim_whitespace=True,
                           comment='#',
                           quoting=csv.QUOTE_NONNUMERIC, quotechar='\"',
                           na_values=[-9999],
                           dtype={'depth': 'f', 'depth_g-cm-2': 'f',
                                  'C': 'f', 'std': 'f'})

profile_data

with open('profiles_data/wintrich_10Be_settings.yaml') as f:
    wintrich_settings = yaml.load(f)

wintrich_settings


I am working on the file "GS_Wintrich_4params.ipynb".
The first task is:
The mathematical (Predictive) Model
The mathematical model is available in the models Python module (see the notebook Models)
Can you write Python code for it?


I am working on the file "GTSDB/GTSDB_RPN_Train.ipynb".
The first task is:
Base Layers
Build base layers (a.k.a. shared layers in the frcnn paper) on the given input layer. The vgg16 network pre-trained on ImageNet was one of the base networks used in the frcnn paper. However, most traffic signs in our datasets are 32x32 in size. Vgg16 has 32x downsample, making it unsuitable for use. The base network is a simple network used in our training.
Can you write Python code for it?

def vgg(input_layer, trainable=False):
    vgg = keras.applications.vgg16.VGG16(include_top=False)
    vgg.trainable = trainable
    return vgg(input_layer)

def base(input_layer):
    return Sequential([
        Conv2D(filters=32, kernel_size=(3,3), padding="same", activation='relu', input_shape=(800,1360,3)),
        MaxPooling2D(pool_size=(2, 2)),
        Conv2D(filters=64, kernel_size=(5,5), padding="same", activation='relu'),
        MaxPooling2D(pool_size=(2, 2)),
    ])(input_layer)


I am working on the file "GTSDB/GTSDB_RPN_Train.ipynb".
The first task is:
Build model
Search for models in the checkpoint directory
Can you write Python code for it?

import os, re
temp = [re.compile('gtsdb_rpn-(\d+)\.hdf5').match(fn) for fn in os.listdir('models/gtsdb_rpn/')]
temp = [int(m.group(1)) for m in temp if m is not None]


I am working on the file "GTSDB/GTSDB_RPN_Train.ipynb".
The first task is:
Data
Data generator. Reads from the GTSDB training set. Generate a mini-batch (one image) each iteration.
Can you write Python code for it?

def datagen(start, stop, ancs=None, shuffle=True):
    """Generator for GTSDB dataset
        Args:
            start, stop = range of images to use
            ancs = anchor boxes to use. Use get_anchors() to generate these.
                Dimensions should be (imageheight/basenet_stride, imagewidht/basenet_stride, k, basenet_stride)
                Defaults to be get_anchors(200, 340, [16,24,32], 4)
            shuffle = whether to shuffle the data
    """
    if ancs is None:
        ancs = get_anchors(200, 340, [16,24,32], 4)
    csv = np.loadtxt('../dataset/PNG_train/gt.txt', delimiter=',', converters = {0: lambda x:x[:-4]}, dtype=np.int32)

    idx = np.arange(start, stop)

    for i in idx:
        temp = csv[csv[:,0] == i]
        temp = temp[:,[2,1,4,3]]
        # temp.shape = (gtb_num, 4)

        ## print("fname = ../dataset/PNG_train/{:05d}.png".format(i))
        pos, neg, gtbs = anchors_vs_gt(ancs, temp)
        gtbs = gtbs.reshape((gtbs.shape[0], gtbs.shape[1], -1))
        # pos.shape = neg.shape = (rows, cols, anc_num)
        # gtbs.shape = (row, cols, anc_num * 4)
        pos, neg = filter_boxes(pos, neg)

        # x_img.shape  = (1, imgh, imgw, imgchannels)
        # y_cls.shape  = (row, cols, anc_num * 2)
        # y_regr.shape = (row, cols, anc_num * 8)
        x_img  = np.expand_dims(imread('../dataset/PNG_train/{:05d}.png'.format(i)), 0)
        y_cls  = np.expand_dims(np.concatenate((pos, neg), axis=-1).astype(np.int32), 0)
        y_regr = np.expand_dims(np.concatenate((pos.repeat(4, axis=-1), gtbs), axis=-1), 0)

        yield (x_img, [y_cls, y_regr])


I am working on the file "GTSDB/GTSDB_RPN_Train.ipynb".
The first task is:
If models are found, load the latest model.
Otherwise build new model
Can you write Python code for it?

if len(temp) == 0:
    inp = Input(shape=(800,1360,3))
    # Can use vgg(inp) instead of base(inp)
    model = Model(inputs=inp, outputs=rpn(base(inp), 3))
else:
    model = keras.models.load_model('models/gtsdb_rpn/gtsdb_rpn-{:d}.hdf5'.format(max(temp)))

model.compile(optimizer='sgd', loss={'rpn_cls': rpn_cls_loss(3), 'rpn_regr': rpn_regr_loss(3)})

model.summary()


I am working on the file "GTSDB/GTSDB_RPN_Train.ipynb".
The first task is:
Losses
Loss functions as defined in the frcnn paper:
L({pi}, {ti}) = (1/Ncls) Σ Lcls(pi, pi*) + λ(1/Nreg) Σ pi* Lreg(ti, ti*)
In this implementation we ignore the balancing factor λ and the normalization, which the paper say is okay:
L({pi}, {ti}) = Σ Lcls(pi, pi*) + Σ pi* Lreg(ti, ti*)
where Lcls is binary log loss
and Lreg is robust loss
Here, rpn_cls_loss is Σ Lcls(pi, pi*)
and rpn_regr_loss is Σ pi* Lreg(ti, ti*)
Can you write Python code for it?

def rpn_regr_loss(num_ancs):
    def loss(ytrue, ypred, ptrue):
        # ytrue.shape = (rows, cols, num_ancs * 4)
        # ypred.shape = (rows, cols, num_ancs * 4)
        # ptrue.shape = (rows, cols, num_ancs * 4)
        # ancs.shape  = (rows, cols, num_ancs * 4)

        dy = ytrue - ypred
        sw = K.cast(K.less(K.abs(dy), 1), dtype=K.floatx())
        r1 = sw*dy*dy*.5
        r2 = (1-sw)*(K.abs(dy)-.5)
        return K.sum((r1+r2) * ptrue)
    return lambda ytrue, ypred: \
        loss(ytrue[...,4*num_ancs:],
             ypred,
             ytrue[...,:4*num_ancs])

def rpn_cls_loss(num_ancs):
    def loss(postrue, negtrue, ppred):
        # Add epsilon = 1e-4 to prevent log(0)
        return K.sum(- postrue * K.log(1e-4 + ppred)
                     - negtrue * K.log(1e-4 + 1-ppred))
    return lambda ptrue, ppred: \
        loss(ptrue[...,:num_ancs], ptrue[...,num_ancs:], ppred)


I am working on the file "GW150914/AdjustCoM.ipynb".
The first task is:
Indeed it's pretty short, so the system doesn't get very far.  I start to worry when we need to be careful with higher modes, or when the displacements are a few times larger than this.
Can you write Python code for it?


I am working on the file "GW150914/HybridizeNR.ipynb".
The first task is:
Now read the NR waveform and offset so that the "relaxed" measurement time is $0$.
Can you write Python code for it?

nr = GWFrames.ReadFromNRAR(data_dir + 'rhOverM_Asymptotic_GeometricUnits_CoM.h5/Extrapolated_N4.dir')
nr.SetT(nr.T()-metadata.relaxed_measurement_time);

approximant = 'TaylorT4'  # 'TaylorT1'|'TaylorT4'|'TaylorT5'
delta = (m1 - m2) / (m1 + m2)  # Normalized BH mass difference (M1-M2)/(M1+M2)
chi1_i = chi1  # Initial dimensionless spin vector of BH1
chi2_i = chi2  # Initial dimensionless spin vector of BH2
Omega_orb_i = Omega_orb_i  # Initial orbital angular frequency
Omega_orb_0 = Omega_orb_i/3.25  # Earliest orbital angular frequency to compute (default: Omega_orb_i)
# R_frame_i: Initial rotation of the binary (default: No rotation)
# MinStepsPerOrbit =   # Minimum number of time steps at which to evaluate (default: 32)
# PNWaveformModeOrder: PN order at which to compute waveform modes (default: 3.5)
# PNOrbitalEvolutionOrder: PN order at which to compute orbital evolution (default: 4.0)

pn = GWFrames.PNWaveform(approximant, delta, chi1_i, chi2_i, Omega_orb_i, Omega_orb_0)

plt.close()
plt.semilogy(pn.T(), np.abs(pn.Data(0)))
plt.semilogy(nr.T(), np.abs(nr.Data(0)))

! /Users/boyle/.continuum/anaconda/envs/gwframes/bin/python ~/Research/Code/misc/GWFrames/Code/Scripts/HybridizeOneWaveform.py {data_dir} \
  --Waveform=rhOverM_Asymptotic_GeometricUnits_CoM.h5/Extrapolated_N4.dir --t1={metadata.relaxed_measurement_time} --t2=2000.0 \
  --InitialOmega_orb={Omega_orb_0} --Approximant=TaylorT4

--DirectAlignmentEvaluations 100

h = hybrid.EvaluateAtPoint(0.0, 0.0)[:-1]

hybrid = scri.SpEC.read_from_h5(data_dir + 'rhOverM_Inertial_Hybrid.h5')
hybrid = hybrid[:-1]

h = hybrid.SI_units(current_unit_mass_in_solar_masses=36.+29., distance_from_source_in_megaparsecs=410)

t_merger = 16.429
h.max_norm_time()
h.t = h.t - h.max_norm_time() + t_merger

plt.close()
plt.semilogy(h.t, np.abs(h.data[:, 0]))

sampling_rate = 4096.  # Hz
dt = 1 / sampling_rate  # sec
t = np.linspace(0, 32, num=int(32*sampling_rate))
h_discrete = h.interpolate(t)
h_discrete.data[np.argmax(t>16.4739):, :] = 1e-40j

from utilities import...


I am working on the file "GW150914/HybridizeNR_GW151226.ipynb".
The first task is:
Now read the NR waveform and offset so that the "relaxed" measurement time is $0$.
Can you write Python code for it?

nr = GWFrames.ReadFromNRAR(data_dir + 'rhOverM_Asymptotic_GeometricUnits_CoM.h5/Extrapolated_N4.dir')
nr.SetT(nr.T()-metadata.relaxed_measurement_time);

approximant = 'TaylorT4'  # 'TaylorT1'|'TaylorT4'|'TaylorT5'
delta = (m1 - m2) / (m1 + m2)  # Normalized BH mass difference (M1-M2)/(M1+M2)
chi1_i = chi1  # Initial dimensionless spin vector of BH1
chi2_i = chi2  # Initial dimensionless spin vector of BH2
Omega_orb_i = Omega_orb_i  # Initial orbital angular frequency
Omega_orb_0 = Omega_orb_i/3  # Earliest orbital angular frequency to compute (default: Omega_orb_i)
# R_frame_i: Initial rotation of the binary (default: No rotation)
# MinStepsPerOrbit =   # Minimum number of time steps at which to evaluate (default: 32)
# PNWaveformModeOrder: PN order at which to compute waveform modes (default: 3.5)
# PNOrbitalEvolutionOrder: PN order at which to compute orbital evolution (default: 4.0)

pn = GWFrames.PNWaveform(approximant, delta, chi1_i, chi2_i, Omega_orb_i, Omega_orb_0)

plt.close()
plt.semilogy(pn.T(), np.abs(pn.Data(0)))
plt.semilogy(nr.T(), np.abs(nr.Data(0)))

! /Users/boyle/.continuum/anaconda/envs/gwframes/bin/python ~/Research/Code/misc/GWFrames/Code/Scripts/HybridizeOneWaveform.py {data_dir} \
  --Waveform=rhOverM_Asymptotic_GeometricUnits_CoM.h5/Extrapolated_N4.dir --t1={metadata.relaxed_measurement_time} --t2=10000.0 \
  --InitialOmega_orb={Omega_orb_0} --Approximant=TaylorT4 --DirectAlignmentEvaluations=400

hybrid = scri.SpEC.read_from_h5(data_dir + 'rhOverM_Inertial_Hybrid.h5')
hybrid = hybrid[:-1]

h = hybrid.SI_units(current_unit_mass_in_solar_masses=m_tot, distance_from_source_in_megaparsecs=440)

t_merger = 16.65
h.max_norm_time()
h.t = h.t - h.max_norm_time() + t_merger

plt.close()
plt.semilogy(h.t, np.abs(h.data[:, 0]))

plt.close()
plt.plot(h.t, np.unwrap(np.angle(h.data[:, 0])))

import quaternion

plt.close()
plt.semilogy(h.t, np.abs(quaternion.derivative(np.unwrap(np.angle(h.data[:, 0])), h.t)))

sampling_rate = 4*4096.  # Hz
dt = 1 / sampling_rate  # sec
t = np.linspace(0, 32,...


I am working on the file "GW150914/HybridizeNR_GW151226.ipynb".
The first task is:
Now, the mass ratio is fixed by the simulation, and the chirp mass is the most accurately determined quantity — not the total mass.  So we deduce the best total mass given this mass ratio and the observed chirp mass of $8.9^{+0.3}{-0.3}\, M\odot$:
Can you write Python code for it?

m_tot = total_mass_from_chirp_mass_and_mass_ratio(8.9, m1/m2)
m_tot


I am working on the file "GW150914/HybridizeNR_GW151226.ipynb".
The first task is:
We need about 16.65 seconds of data, after we scale the system to $25.1\, M_{\odot}$.  In terms of $M$ as we know it, that's about...
Can you write Python code for it?


I am working on the file "Gabarito.ipynb".
The first task is:
Exercício 01: Calcule a distância média, o diâmetro e o coeficiente de agrupamento das redes abaixo.
Can you write Python code for it?

G1 = nx.erdos_renyi_graph(10,0.4)
nx.draw_shell(G1)

print "Dist. media: ", nx.average_shortest_path_length(G1)
print "Diametro: ", nx.diameter(G1)
print "Coef. Agrupamento médio: ", nx.average_clustering(G1)

G2 = nx.barabasi_albert_graph(10,3)
nx.draw_shell(G2)

print "Dist. media: ", nx.average_shortest_path_length(G2)
print "Diametro: ", nx.diameter(G2)
print "Coef. Agrupamento médio: ", nx.average_clustering(G2)

G3 = nx.barabasi_albert_graph(10,4)
nx.draw_shell(G3)

print "Dist. media: ", nx.average_shortest_path_length(G3)
print "Diametro: ", nx.diameter(G3)
print "Coef. Agrupamento médio: ", nx.average_clustering(G3)


I am working on the file "Gabarito/Exercicio01_Titanic_Gabarito.ipynb".
The first task is:
Por fim, geramos nossa base de treino e teste no formato para o ScikitLearn
Can you write Python code for it?

dataset = {
    'data': df.values[0:,2:],
    'target': df.values[0:,0]
}

X = dataset['data']
y = dataset['target']


I am working on the file "Gabarito/Exercicio01_Titanic_Gabarito.ipynb".
The first task is:
Resposta do item 2
Can you write Python code for it?

from sklearn import tree

clf_tree = tree.DecisionTreeClassifier(criterion='entropy')
clf_tree = clf_tree.fit(X_train, Y_train)

pred = clf_tree.predict(X_test)

from sklearn import metrics

acc = metrics.accuracy_score(Y_test, pred)

print("Acurácia da árvore de decisão", acc)


I am working on the file "Gabarito/Exercicio01_Titanic_Gabarito.ipynb".
The first task is:
Resposta do item 3
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

k_range = list(range(1, 30))

maior_k = 0.0
maior_acc_test = 0.0

print("k\t\tAcc Test\t\tAcc Train")

for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    knn = knn.fit(X_train, Y_train)
    acc_test = knn.score(X_test, Y_test)
    acc_train = knn.score(X_train, Y_train)

    if acc_test > maior_acc_test:
        maior_k = k
        maior_acc_test = acc_test

    print(str(k)+"\t\t"+str(acc_test)+"\t\t"+str(acc_train))

print()
print("Maior valor de K é", maior_k, "com acurácia em teste igual a", maior_acc_test)


I am working on the file "Gabarito/Exercicio01_Titanic_Gabarito.ipynb".
The first task is:
Tarefas
Como o propósito desse exercício não é submeter para o Kaggle, vamos ter que criar nossa própria base de testes. A base de teste disponibilizada pelo kaggle não vem com o gabarito, logo não poderíamos utilizar o aprendizado supervisionado.
Sendo assim, as tarefas deste exercício são:

Separe sua base em treino e teste. Para o teste use 10% da base original.
Aplique o algoritmo de Árvore de Decisão visto no HelloWorld e calcule a acurácia na base de teste
Pesquise sobre o KNN e aplique-o sobre a mesma base de dados e calcule a acurácia

Resposta do item 1
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_test, Y_train, Y_test = train_test_split(X, y, test_size=0.1, random_state=4)

print(X.shape)
print(X_train.shape)
print(X_test.shape)


I am working on the file "GabaritoFacil.ipynb".
The first task is:
Exercício 01: Calcule a distância média, o diâmetro e o coeficiente de agrupamento das redes abaixo.
Can you write Python code for it?

G1 = nx.erdos_renyi_graph(5,0.4)
nx.draw_shell(G1)

print "Dist. media: ", nx.average_shortest_path_length(G1)
print "Diametro: ", nx.diameter(G1)
print "Coef. Agrupamento médio: ", nx.average_clustering(G1)

G2 = nx.barabasi_albert_graph(6,3)
nx.draw_shell(G2)

print "Dist. media: ", nx.average_shortest_path_length(G2)
print "Diametro: ", nx.diameter(G2)
print "Coef. Agrupamento médio: ", nx.average_clustering(G2)

G3 = nx.barabasi_albert_graph(6,4)
nx.draw_shell(G3)

print "Dist. media: ", nx.average_shortest_path_length(G3)
print "Diametro: ", nx.diameter(G3)
print "Coef. Agrupamento médio: ", nx.average_clustering(G3)


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
424x424 pixels, with three color channels? Yeah, we don't need all of that. Let's make it 50x50 and black & white instead.
Can you write Python code for it?

from skimage import transform, color
smallgalaxy = transform.resize(galaxy, (50, 50))
smallbwgalaxy = color.rgb2gray(smallgalaxy)
io.imshow(smallbwgalaxy)
io.show()


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
How fast is our pipeline? It might be too slow.
Can you write Python code for it?

n = 300
pix = 10
t1 = time.time()
images = [downsample_and_grayscale(fn, pix) for fn in small_training_filenames[:n]]
t2 = time.time()
print("Processed", n, "images at", pix, "x", pix, "resolution in", t2-t1, "seconds.")


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
Let's try altering the kernel.
Can you write Python code for it?

kernel = Amplitude(1.0, (1E-3, 1E3)) + SquaredExponential(10, (1e-2, 1e2))

gp = GaussianProcessRegressor(kernel=kernel,
                              n_restarts_optimizer=10)

# Fit to data using Maximum Likelihood Estimation of the parameters
gp.fit(X, Y)

predictedY = gp.predict(newX)
print(rmse(newY, predictedY))
len(newX)

from sklearn.gaussian_process.kernels import Matern

kernel = Amplitude(1.0, (1E-3, 1E3)) * Matern(1.0, (1e-3, 1e3), 2.5)

gp = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=10)

gp.fit(X, Y)

predictedY = gp.predict(newX)
print(rmse(newY, predictedY))
len(newX)

kernel = Amplitude(1.0, (1E-4, 1E4)) * Matern(1.0, (1e-4, 1e4), 0.5)

gp = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=10)

gp.fit(X, Y)

predictedY = gp.predict(newX)
print(rmse(newY, predictedY))
len(newX)

from sklearn.gaussian_process.kernels import RationalQuadratic

kernel = Amplitude(1.0, (1E-4, 1E4)) * RationalQuadratic()

gp = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=10)

gp.fit(X, Y)

predictedY = gp.predict(newX)
print(rmse(newY, predictedY))
len(newX)

kernel = Amplitude(1.0, (1E-3, 1E3))

gp = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=10)

gp.fit(X, Y)

predictedY = gp.predict(newX)
print(rmse(newY, predictedY))
len(newX)


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
Much better. But that's an elliptical galaxy — can we still resolve the arms of a spiral galaxy, or the bars, or the "odd features" that are part of the Galaxy Zoo decision tree?
Can you write Python code for it?

fn = np.random.choice(training_filenames)
randgalaxy = io.imread("images_training_rev1/" + fn)
randsmallgalaxy = color.rgb2gray(transform.resize(randgalaxy, (50, 50)))
io.imshow(randgalaxy)
io.show()
io.imshow(randsmallgalaxy)
io.show()


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
Next, let's have a look at the beginning of that CSV file of galaxy classifications.
Can you write Python code for it?

import pandas as pd
training_solutions = pd.read_csv("training_solutions_rev1.csv",index_col=0)
training_solutions.head()


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
Nope. I think we've reached the limits of what random forest is capable of when acting on raw pixels. We'd need to run something with a huge forest and large images on the whole dataset to be sure, but I doubt we'd get much better. Time to try something else.
V. Messing around with a few other ideas
Gaussian processes
Can you write Python code for it?

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF as SquaredExponential
from sklearn.gaussian_process.kernels import ConstantKernel as Amplitude

kernel = Amplitude(1.0, (1E-3, 1E3)) * SquaredExponential(10, (1e-2, 1e2))

gp = GaussianProcessRegressor(kernel=kernel,
                              n_restarts_optimizer=10)

# Fit to data using Maximum Likelihood Estimation of the parameters
gp.fit(X, Y)

predictedY = gp.predict(newX)
print(rmse(newY, predictedY))
len(newX)


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
Not bad -- we're about a third of the way up the leaderboard on Kaggle. Let's make the images bigger now and see if that helps.
Can you write Python code for it?

n_images = 1000
size = 50
longerX = [pipeline(fn, size) for fn in small_training_filenames[:n_images]]
longerX = np.array(longerX)
longerY = np.array(small_training_answers[:n_images])
randforest = RandomForestRegressor(n_estimators=100,
                            n_jobs=4, verbose=1, warm_start=False)
randforest.fit(longerX, longerY)
testlongerX = [pipeline(fn, size) for fn in small_training_filenames[n_images:2*n_images]]
testlongerX = np.array(testlongerX)
testlongerY = np.array(small_training_answers[n_images:2*n_images])
predictedY = randforest.predict(testlongerX)
print("RMSE with a larger training set and larger images:", rmse(predictedY, testlongerY))


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
Now we have our baseline. Just to be sure, let's generate a bunch of those random arrays and get the average RMSE across all of them. (Though, given the sheer number of elements in our arrays, it's likely that we'll get something very close to what we already have.)
Can you write Python code for it?

n = 50
print("Generating", n, "random arrays...")
t1 = time.time()
random_errs = [rmse(randarray(n_files), training_solutions_array) for i in range(n)]
t2 = time.time()
print("Finished in", t2 - t1, "seconds.")
random_baseline_error = sum(random_errs)/n
print("Baseline RMSE is", baseline)


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
OK, looks like we might want to go up to 100x100. Let's try 50x50 first, though, and then go back up to 100x100 later.
Splitting the training set and creating a pipeline
Unfortunately, the only way to check our guesses for the test images is to submit the guesses to Kaggle. We don't want to do that every time — so let's train our machine-learning algorithms on half of our training set (randomly selected), and use the other half as our "test" set. If we end up with an algorithm that performs well on that test set, we'll retrain the algoritm on the entire training set, then use the actual test set to create a submission to Kaggle.
Can you write Python code for it?

import random
shuffled_training_set = list(zip(training_filenames, training_solutions_array))
random.shuffle(shuffled_training_set)
small_training_set = shuffled_training_set[:n_files//2]
small_test_set = shuffled_training_set[n_files//2:]
print(len(small_training_set), len(small_test_set))
small_training_filenames = [i[0] for i in small_training_set]
small_training_answers = [i[1] for i in small_training_set]
small_test_filenames = [i[0] for i in small_test_set]
small_test_answers = [i[1] for i in small_test_set]


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
OK, that gained us a few leaderboard spots. Let's try making the images bigger again, and increasing the number of trees to compensate for the larger input vectors. Increasing those individually didn't help before, but doing it in tandem might.
Can you write Python code for it?

n_images = 1000
size = 100
longerX = [pipeline(fn, size) for fn in small_training_filenames[:n_images]]
longerX = np.array(longerX)
longerY = np.array(small_training_answers[:n_images])
randforest = RandomForestRegressor(n_estimators=400,
                            n_jobs=4, verbose=1, warm_start=False)
randforest.fit(longerX, longerY)
testlongerX = [pipeline(fn, size) for fn in small_training_filenames[n_images:2*n_images]]
testlongerX = np.array(testlongerX)
testlongerY = np.array(small_training_answers[n_images:2*n_images])
predictedY = randforest.predict(testlongerX)
print("RMSE with a larger training set, a bigger forest, and much larger images:", rmse(predictedY, testlongerY))


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
OK, that's not terrible. Let's try a larger training set and see how we do.
Can you write Python code for it?

n_images = 1000
size = 20
longerX = [pipeline(fn, size) for fn in small_training_filenames[:n_images]]
longerX = np.array(longerX)
longerY = np.array(small_training_answers[:n_images])
randforest = RandomForestRegressor(n_estimators=100,
                            n_jobs=4, verbose=1, warm_start=False)
randforest.fit(longerX, longerY)
testlongerX = [pipeline(fn, size) for fn in small_training_filenames[n_images:2*n_images]]
testlongerX = np.array(testlongerX)
testlongerY = np.array(small_training_answers[n_images:2*n_images])
predictedY = randforest.predict(testlongerX)
print("RMSE with a larger training set:", rmse(predictedY, testlongerY))


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
Ok, let's try 50x50 images with more trees, since we probably didn't fully sample the space there.
Can you write Python code for it?

n_images = 1000
size = 50
longerX = [pipeline(fn, size) for fn in small_training_filenames[:n_images]]
longerX = np.array(longerX)
longerY = np.array(small_training_answers[:n_images])
randforest = RandomForestRegressor(n_estimators=500,
                            n_jobs=4, verbose=1, warm_start=False)
randforest.fit(longerX, longerY)
testlongerX = [pipeline(fn, size) for fn in small_training_filenames[n_images:2*n_images]]
testlongerX = np.array(testlongerX)
testlongerY = np.array(small_training_answers[n_images:2*n_images])
predictedY = randforest.predict(testlongerX)
print("RMSE with a larger training set and larger images:", rmse(predictedY, testlongerY))


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
That did very little! The answer is comparable to what we got before. We're clearly limited by the size of the training set.
Rather than making the forest bigger and shorter, let's make the forest smarter instead: we'll try boosting the performance of the trees with AdaBoost. AdaBoost forces the random forest to re-examine the parts of the training set that it initially doesn't handle well. More on this algorithm in the scikit-learn documentation.
Can you write Python code for it?

size = 20
n_images = 200
X = [pipeline(fn, size) for fn in small_training_filenames[:n_images]]
X = np.array(X)
Y = np.array(small_training_answers[:n_images])
newX = [pipeline(fn, size) for fn in small_training_filenames[n_images:2*n_images]]
newX = np.array(newX)

from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import AdaBoostRegressor

#AdaBoost in sklearn can't handle multi-dimensional outputs! So let's roll our own.
# This should be an object, but I'm going to make functions instead because that's how I roll.
def ABRegressors_init(X, Y, n_estimators):
    '''
    Create a group of AdaBoostRegressors suitable for the given X and Y input.
    '''
    abregressors = [AdaBoostRegressor(DecisionTreeRegressor(), n_estimators = n_estimators) for i in range(Y.shape[1])]
    return abregressors

def ABRegressors_fit(ABregressors, X, Y):
    '''
    Use a group of AdaBoostRegressors to fit to the training set X and Y.
    '''
    return [abr.fit(X, Y[:, i]) for i, abr in enumerate(ABregressors)]

def ABRegressors_predict(ABregressors, X):
    '''
    Use a group of AdaBoostRegressors to predict for the test set X.
    '''
    predictions = [abr.predict(X) for abr in ABregressors]
    predictions = np.array(predictions)
    predictions = predictions.transpose()
    return predictions

boostedforest = ABRegressors_init(X, Y, 100)
ABRegressors_fit(boostedforest, X, Y)
predictedY = ABRegressors_predict(boostedforest, newX)

print(rmse(newY, predictedY))


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
That is surprisingly un-horrible! Let's see if we can do better with the same training set by altering the hyper-parameters (e.g. number of trees, number of pixels fed in, size of the training set, etc.) We'll change them one at a time, and see what gets us the biggest improvement.
Can you write Python code for it?

randforest2 = RandomForestRegressor(n_estimators=400,
                            n_jobs=4, verbose=1, warm_start=False)
randforest2.fit(X, Y)
predictedY = randforest2.predict(newX)
print("RMSE with more trees:", rmse(newY, predictedY))

n_images = 400
size = 20
longerX = [pipeline(fn, size) for fn in small_training_filenames[:n_images]]
longerX = np.array(longerX)
longerY = np.array(small_training_answers[:n_images])
randforest = RandomForestRegressor(n_estimators=100,
                            n_jobs=4, verbose=1, warm_start=False)
randforest.fit(longerX, longerY)
testlongerX = [pipeline(fn, size) for fn in small_training_filenames[n_images:2*n_images]]
testlongerX = np.array(testlongerX)
testlongerY = np.array(small_training_answers[n_images:2*n_images])
predictedY = randforest.predict(testlongerX)
print("RMSE with a larger training set:", rmse(predictedY, testlongerY))

n_images = 200
size = 40
biggerX = [pipeline(fn, size) for fn in small_training_filenames[:n_images]]
biggerX = np.array(biggerX)
biggerY = np.array(small_training_answers[:n_images])
randforest = RandomForestRegressor(n_estimators=100,
                            n_jobs=4, verbose=1, warm_start=False)
randforest.fit(biggerX, biggerY)
testbiggerX = [pipeline(fn, size) for fn in small_training_filenames[n_images:2*n_images]]
testbiggerX = np.array(testbiggerX)
testbiggerY = np.array(small_training_answers[n_images:2*n_images])
predictedY = randforest.predict(testbiggerX)
print("RMSE with larger images:", rmse(predictedY, testbiggerY))


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
Unsurprisingly, a larger training set wins. We'll have to find a way to get that to work before we can go any further with this, I think.
V. Increasing the size of the training set
Let's start by profiling our existing file I/O.
Can you write Python code for it?

import cProfile
N = 800
offset = 400
cProfile.run("list(io.imread('images_training_rev1/' + fn) for fn in small_training_filenames[offset:N+offset])")


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
We'll also need a pipeline function for these images:
Can you write Python code for it?

def downsample_and_grayscale(filename, size):
    '''
    Grabs the file, scales it down to the given size (in pixels), and grayscales it.
    Returns a numpy array with the downscaled and grayscaled image data.
    '''
    raw_image = io.imread("images_training_rev1/" + filename)
    downscaled_gray_image = color.rgb2gray(transform.resize(raw_image, (size, size)))
    return downscaled_gray_image

io.imshow(downsample_and_grayscale(small_training_filenames[2], 50))
io.show()


I am working on the file "Galaxy-Zoo-Machine-Learning.ipynb".
The first task is:
Wow, that's not a lot better! So that means the way to crank down that RMSE on a pixel-based approach is really just to crank up the number of images. Let's do that.
Can you write Python code for it?

n_images = 3000
size = 20
longerX = [pipeline(fn, size) for fn in small_training_filenames[:n_images]]
longerX = np.array(longerX)
longerY = np.array(small_training_answers[:n_images])
randforest = RandomForestRegressor(n_estimators=100,
                            n_jobs=4, verbose=1, warm_start=False)
randforest.fit(longerX, longerY)
testlongerX = [pipeline(fn, size) for fn in small_training_filenames[n_images:2*n_images]]
testlongerX = np.array(testlongerX)
testlongerY = np.array(small_training_answers[n_images:2*n_images])
predictedY = randforest.predict(testlongerX)
print("RMSE:", rmse(predictedY, testlongerY))


I am working on the file "Galclean Example.ipynb".
The first task is:
It is possible to change the standard background threshold to change the extraction results. As an example, we can extract only very bright objects (siglevel = 6)
Can you write Python code for it?

segmented_galaxy_5std = galclean(galaxy_data, 6)
plot_result(galaxy_data, segmented_galaxy_5std)


I am working on the file "Galclean Example.ipynb".
The first task is:
OBS: Warnings are thrown by PhotUtils due to a recent update in Astropy. Galclean does not use iters explictly, so this warning should disappear when photutils get an update.
Load EFIGI galaxy image from fits (EFIGI dataset available at https://www.astromatic.net/projects/efigi)
Can you write Python code for it?


I am working on the file "Galclean Example.ipynb".
The first task is:
Remove bright sources from the galaxy image using galclean. Default threshold is 4 std above the median of the sk background.
Can you write Python code for it?

segmented_galaxy = galclean(galaxy_data)
plot_result(galaxy_data, segmented_galaxy)


I am working on the file "Galclean Example.ipynb".
The first task is:
Show galaxy image with top 0.5% clip
Can you write Python code for it?


I am working on the file "Gambit.ipynb".
The first task is:
mixed_behavior_profile()
それぞれの情報集合(information set)における行動の確率分布を表す。
行動プロファイルの初期値はそれぞれの情報集合において一様分布で導かれる。
Can you write Python code for it?


I am working on the file "Gambit.ipynb".
The first task is:
mixed_strategy_profile()のときと扱い方はほとんど変わらないが、infosetsが増えたところを注意して扱う必要がある。
Can you write Python code for it?

p[g.players[0]]

p[g.players[0].infosets[0]]

p[g.players[0].infosets[0].actions[0]]


I am working on the file "Gambit.ipynb".
The first task is:
nash.ExternalEnumPureSolver()
純戦略における解を返す。
Can you write Python code for it?

solver = gambit.nash.ExternalEnumPureSolver()
solver.solve(g)


I am working on the file "Gambit.ipynb".
The first task is:
payoff()
payoff()によってプレイヤーに対する期待利得を得ることが出来る。
Can you write Python code for it?


I am working on the file "Gambit.ipynb".
The first task is:
players
players属性によりプレイヤーの設定が出来ます。len()を使って人数を確認したり、add()でプレイヤーを増やすことが出来ます。
Can you write Python code for it?


I am working on the file "Gambit.ipynb".
The first task is:
strategy_value()
strategy_value()によってプレイヤーが与えられた戦略を行うときの期待利得を得る。
Can you write Python code for it?


I am working on the file "Gambit.ipynb".
The first task is:
title
title属性によりゲームタイトルを入力出来る。
Can you write Python code for it?

str(g)

g.title = "A simple poker example"
g.title

str(g)


I am working on the file "Gambit.ipynb".
The first task is:
ゲームの読み込み(Reading a game from a file)
Game.read_game()
Gambitにもとから入っているサンプルファイルをGame.read_game()を使って読み込むことが出来る。
ためしにmixed_strategy.nfgを読み込んでみる。
Can you write Python code for it?

g = gambit.Game.read_game("test_games/mixed_strategy.nfg")
g


I am working on the file "Gambit.ipynb".
The first task is:
ナッシュ均衡
戦略型ゲーム
mixed_strategy.nfgのゲームを例に扱う。
Player1は戦略0を確率1で行うが、Player2は戦略0を確率0.5以上で混合戦略をとる。
Can you write Python code for it?


I am working on the file "Gambit.ipynb".
The first task is:
展開型ゲーム
sample_extensive_game.efgを例として扱う。
Can you write Python code for it?

g = gambit.Game.read_game("test_games/sample_extensive_game.efg")
g

solver = gambit.nash.ExternalLCPSolver()
solver.solve(g)

solver.solve(g, use_strategic=True)
solver.solve(g)


I am working on the file "Gambit.ipynb".
The first task is:
戦略型ゲーム(strategic game)
ここでは2x2の囚人のジレンマを例として取り扱う。
Game.new_table()
新しいゲームを生成する。それぞれのプレイヤーにおける戦略の数の組を整数で構成されるリストとして引く。
Can you write Python code for it?

g = gambit.Game.new_table([2,2])
g.title = "A prisoner's dilemma game"
g.players[0].label = "Alphonse"
g.players[1].label = "Gaston"
g


I am working on the file "Gambit.ipynb".
The first task is:
混合戦略と行動戦略(Mixed strategy and behavior profiles)
混合戦略(Mixed strategy)
mixed_strategy_profile()
mixed_strategy_profile()によって各プレイヤーの純戦略における確率分布を表すことが出来る。
Can you write Python code for it?

p = g.mixed_strategy_profile()

list(p)

p[g.players[0]]

p[g.players[1].strategies[0]]


I am working on the file "Game of Life.ipynb".
The first task is:
Initial layout parameters
width: The width of the board in pixels (squares).
height: The height of the board in pixels (squares).
q: $ \frac{1}{q} $ probability that a square is initially alive.
Can you write Python code for it?


I am working on the file "Game of Life.ipynb".
The first task is:
Produce the plot
Can you write Python code for it?

size = np.array(Z.shape)
figsize= size[1]/float(dpi),size[0]/float(dpi)
fig = plt.figure(figsize=figsize, dpi=dpi, facecolor="white")
fig.add_axes([0.0, 0.0, 1.0, 1.0], frameon=False)
plt.xticks([]), plt.yticks([])

im = plt.imshow(Z,interpolation='nearest', cmap=plt.cm.gray_r)
ani = animation.FuncAnimation(fig, iterate_plot, interval=50, frames=frames)
ani.save(output_file, codec="libx264")
plt.show()


I am working on the file "Game of Life.ipynb".
The first task is:
Randomization Parameters
k_1: $ \frac{1}{k_1} $ probability that a square with only one neighbour comes alive.
k_2: $ \frac{1}{k_2} $ probability that a square with two neighbours dies.
k_3: $ \frac{1}{k_3} $ probability that a square with three neighbour dies.
k_4: $ \frac{1}{k_4} $ probability that a square with four neighbours comes alive.
Can you write Python code for it?


I am working on the file "Game of Life.ipynb".
The first task is:
The initial state of the board
Can you write Python code for it?

Z = (np.random.randint(0,q,(height,width)) == 0).astype(int)


I am working on the file "Game of Life.ipynb".
The first task is:
The iteration step
Z: Should be a matrix of ones and zeros which represents the state of the board. The board will then be iterated and returned as a matrix of ones and zeros.

This function will add in the randomness according to the parameters listed above.
Can you write Python code for it?


I am working on the file "Gamma-iterated-factorial.ipynb".
The first task is:
A new zoom:
Can you write Python code for it?


I am working on the file "Gamma-iterated-factorial.ipynb".
The first task is:
Choosing randomly points in the complex plane, their orbits either approach 1 or after a number of iterations
the function gamma returns a nan (not a number), except for fixed points of the function  $f(z)=\Gamma(z+1)$, i.e. points $z$, sucha that $f(z)=z$.
$z=1$ and $z=2$ are fixed points:
Let us check if $z_0=1$ and $z_0=2$ are  attracting, repelling or neutral fixed points, i.e. if $|f'(z_0)|$ is less than 1, greater than 1 or equal to 1. For we import mpmath library:
Can you write Python code for it?


I am working on the file "Gamma-iterated-factorial.ipynb".
The first task is:
Enlarging the above rectangle we get:
Can you write Python code for it?

plotGammaiterated((-5, 5),(-5,5), 'cubehelix' )


I am working on the file "Gamma-iterated-factorial.ipynb".
The first task is:
Let us  compute a segment of orbit of a point $z_0=z$, i.e. a few terms of the associated sequence $z_n=\Gamma(z_{n-1}+1)$:
Can you write Python code for it?

z=1+2*1j
for k in range(25):
    z=gamma(z+1)
    print  z


I am working on the file "Gamma-iterated-factorial.ipynb".
The first task is:
Let us zoom the central protuberance:
Can you write Python code for it?

plt.rcParams['figure.figsize']=10, 6

plotGammaiter3cols(re=(1.75,6.5), im=(-1,1))


I am working on the file "Gamma-iterated-factorial.ipynb".
The first task is:
The colored representation of this protuberance is the following:
Can you write Python code for it?

plotGammaiterated((1.75,6.5), (-1,1), 'cubehelix', N=300, Miter=100)


I am working on the file "Gamma-iterated-factorial.ipynb".
The first task is:
The next Python function generates the visual representation  of  the patterns  associated to the function $f(z)=\Gamma(z+1)$, through the above  algorithm. More precisely, to a narrow grid of a rectangular region in the complex plane, one associates an array w of integer numbers, n, returned by the iteratedGamma function. This array is transformed into an image, mapping the n values to a colormap.
After many experiments we decided to use the cubehelix colormap for most of our plots in this IPython Notebook.  The list of all colormaps provided by matplotlib can be found here.
Can you write Python code for it?

def plotGammaiterated(re, im,  cmap, filename=False, N=100, Miter=50):

    # re is a tuple giving the interval on the real axis for the rectangular region
    # im is a tuple giving the interval on the imaginary axis
    # cmap is the name of the colormap used to plot the fractal set
    # filename is a string such as 'imagefile.png'
    # N is the number of points in an unit interval

    Nx=int((re[1]-re[0])*N)# horizontal resolution
    Ny=int((im[1]-im[0])*N)# vertical resolution
    x=np.linspace(re[0], re[1], Nx)
    y=np.linspace(im[0], im[1], Ny)
    w=np.zeros((Ny,Nx), dtype=int)

    for n in xrange(Ny):
        for m in xrange(Nx):
            z =x[m]+1j*y[n]
            w[n][m] = iterGamma(z, Miter)

    fig=plt.figure()
    ax = fig.add_subplot(111)

    im=plt.imshow(w, cmap=cmap, extent=(re[0], re[1], im[0], im[1]), interpolation='nearest', origin='lower')
    divider = make_axes_locatable(ax)
    cax = divider.append_axes("right", size="5%", pad=0.05)
    plt.colorbar(im, cax=cax)

    if (filename):
        plt.savefig(filename)

plt.rcParams['figure.figsize']=11,  12

plotGammaiterated((-5.5, 0),(-2,2), 'cubehelix' )


I am working on the file "Gamma-iterated-factorial.ipynb".
The first task is:
To be continued :)
This IPython Notebook is included in the folder Math:
https://github.com/empet/Math/blob/master/README.md
Can you write Python code for it?

from IPython.core.display import HTML
def  css_styling():
    styles = open("./custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Gamma-iterated-factorial.ipynb".
The first task is:
We change the colormap:
Can you write Python code for it?

plotGammaiterated((-5, 5),(-5,5), 'gist_earth' )


I am working on the file "Gamma-iterated-factorial.ipynb".
The first task is:
and its colored version illustrates the self similarity:
Can you write Python code for it?


I am working on the file "Gamma-iterated-factorial.ipynb".
The first task is:
mpmath  is a Python library for real and complex floating-point arithmetic with arbitrary precision.  It provides  functions to evaluate special mathematical functions such as gamma, zeta, hurwitz, and more.
The function of interest for gamma fractals is mpmath.factorial(z).
The derivative of mpm.factorial at 1 is:
Can you write Python code for it?

deriv = mpm.diffs(mpm.factorial, 1) # deriv is a generator
print next(deriv)

deriv = mpm.diffs(mpm.factorial, 2)
print next(deriv)


I am working on the file "Gaussian Process.ipynb".
The first task is:
:: NOTE ::
Constructing squared exponential covariance kernel over a large number of points will produce an ill-conditioned covariance matrix
Can you write Python code for it?

s = 2
b = 1
C_eigvals = np.linalg.eigvals(C(x_, s, b))
plt.plot(range(len(C_eigvals)), C_eigvals, 'go');
plt.xlabel("$\lambda$")
plt.ylabel("$\sigma(C)$")
plt.yscale('log')
print "Condition Number of C(x,x'): %.2f" % (max(C_eigvals)/min(C_eigvals))


I am working on the file "Gaussian Process.ipynb".
The first task is:
Drawing samples from prior over functions
Can you write Python code for it?

# Using squared exponential covariance function
def C(Xs, s, b):
    diff = squareform(pdist(Xs.reshape(len(Xs),1), 'euclidean'))
    return np.around(b*np.exp(-np.power(diff/s, 2)), 16)

s = 2
b = 1

N = 25
ys = []
x_ = np.linspace(0, 10, N)

for f in range(3):
    y = np.random.multivariate_normal(np.zeros(len(x_)), C(x_,s,b))
    ys.append(y)

for y in ys:
    plt.plot(x_, y)


I am working on the file "GaussianProcesses/GPC.ipynb".
The first task is:
Predictive gaussian parameter finding (with gaussian cumulative likelihood)
based on algo 3.1 page 65 GPML book
Can you write Python code for it?

sigmaf = 1; l = 1; sigman = 1;# non optimum values

K = get_kernel(x,x,sigmaf,l,sigman);
# K = K + np.finfo(float).eps*np.eye(x.shape[0]);#for stability
K = nearestSPD(K);
f = np.zeros_like(y);

old_log_marg = np.inf; new_log_marg = 0;
it = 0; maxit=10000;
for it in range(maxit): #newton iterations
    npf = norm.pdf(f); cpyf = norm.cdf(y*f);

    dlp = y*npf/cpyf #3.16
    d2lp = -npf**2/cpyf**2 - y*f*npf/cpyf # 3.16

    W = -1.*d2lp; sW = np.sqrt(W)
    B = np.eye(x.shape[0])+sW*K*sW #B=I+sqrt(W)*K*sqrt(W)

    #B = make_PD(B)
    B = nearestSPD(B)

    L = cholesky(B, lower=True)
    b = W*f + dlp; # 3.18 part 1
    a = b - sW * cho_solve((L, True),np.dot(sW*K,b)) # b - sW.*(L'\(L\(sW.*K*b)))
    f = K.dot(a)

    new_log_marg = -0.5 * a.T.dot(f) \
                + np.log(norm.cdf(y*f)).sum() \
                - np.log(np.diag(L)).sum()
    if it%10==0:
        print '%e'%abs(new_log_marg - old_log_marg)
    if abs(new_log_marg - old_log_marg) < 1e-3: # need to adapt step sizes
        print('FOUND! %f at %d iteration!'%(abs(new_log_marg - old_log_marg),it))
        break
    else:
        old_log_marg = new_log_marg;

fhat = f

npf = norm.pdf(fhat); cpyf = norm.cdf(y*fhat);

lp = np.log(cpyf);
dlp = y*npf/cpyf #3.16
d2lp = -npf**2/cpyf**2 - y*f*npf/cpyf # 3.16

W = -1.*d2lp; sW = np.sqrt(W)
B = np.eye(x.shape[0])+sW*K*sW #B=I+sqrt(W)*K*sqrt(W)
#B = make_PD(B)
B = nearestSPD(B)
L = cholesky(B, lower=True)

K_s = get_kernel(x_test, x, sigmaf, l, sigman);
K_ss = get_kernel(x_test, x_test, sigmaf, l, sigman);

fs = np.dot(K_s.T,dlp);
v = np.dot(inv(L),(sW*K_s));

y_test_var = np.diag(K_ss - np.dot(v.T,v));# uncertainty
y_test_mean = norm.cdf(np.real(np.divide(fs[:,0],np.sqrt(1+y_test_var))));# best prediction

plt.plot(x[y[:,0]==1,0], x[y[:,0]==1,1], 'r+', x[y[:,0]==-1,0], x[y[:,0]==-1,1], 'b+')
plt.plot(x_test[y_test[:,0]==1,0], x_test[y_test[:,0]==1,1], 'r+', x_test[y_test[:,0]==-1,0], x_test[y_test[:,0]==-1,1], 'b+')

plt.plot(x_test[y_test_mean>=0.5,0],...


I am working on the file "GaussianProcesses/GPC.ipynb".
The first task is:
We will define kernel function here:
Can you write Python code for it?


I am working on the file "GaussianProcesses/GRP.ipynb".
The first task is:
Gassuain Processes Regression (GPR):
We have noisy sensor readings (indicated by errorbars) and we want to predict  readings for desired new points. In GPs every new point will be a dimension to infinite dimensional multivariate gaussian distribution (usualy mean zero) in which covariance between points will be defined by the kernel function.
In other words, in GPs we can have infinte number of random variables, that any finite subset of them will be jointly gaussian as well. GP can then be called a distribution over functions where value of each function at a certain point will be a gussian RV.
By fiting a GP to our (n) trainig points (observations) we can get then nearly back with a single sample from a n-dimentional GP.
Train set points:
$$\mathbf{x} = \begin{bmatrix}
x_1 & x_2 & \cdots & x_n
\end{bmatrix}$$
Test set points:
$$\mathbf{x} = \begin{bmatrix}
x_{1} & x{2} & \cdots & x_{m}
\end{bmatrix}$$
Kernel function with integrated reading noise:
$$ k(x,x') = \sigma_f^2 e^{\frac{-(x-x')^2}{2l^2}} + \sigma_n^2\delta(x,x')$$
and then our GP kernel will read:
\begin{equation}
\begin{bmatrix}
\mathbf{y}\
\mathbf{y}_
\end{bmatrix}
\sim N\Bigl(
0,\begin{bmatrix}
\mathbf{K} & \mathbf{K}^T \
\mathbf{K}_ & \mathbf{K}{*} \
\end{bmatrix}
)
\end{equation}
where
$$\mathbf{K} = \begin{bmatrix}
k(x_1,x_1) & k(x_1,x_2) & \cdots & k(x_1,x_n) \
k(x_2,x_1) & k(x_2,x_2) & \cdots & k(x_2,x_n) \
\vdots & \vdots & \ddots & \vdots \
k(x_n,x_1) & k(x_n,x_2) & \cdots & k(x_n,x_n) \\end{bmatrix}$$
$$\mathbf{K} = \begin{bmatrix}
k(x_{1},x_1) & k(x{1},x_2) & \cdots & k(x_{1},x_n) \
k(x_{2},x_1) & k(x_{2},x_2) & \cdots & k(x_{2},x_n) \
\vdots & \vdots & \ddots & \vdots \
k(x_{m},x_1) & k(x_{m},x_2) & \cdots & k(x_{m},x_n) \\end{bmatrix}$$
$$\mathbf{K}{} = \begin{bmatrix}
k(x_{1},x_{1}) & k(x{1},x_{2}) & \cdots & k(x_{1},x_{m}) \
k(x_{2},x_{1}) & k(x_{2},x_{2}) & \cdots & k(x_{2},x_{m}) \
\vdots & \vdots & \ddots & \vdots \
k(x_{m},x_{1}) & k(x_{m},x_{2}) & \cdots & k(x_{m},x_{m}) \ \end{bmatrix}$$
Next, for prediction we are...
Can you write Python code for it?

def get_kernel(X1,X2,sigmaf,l,sigman):
    k = lambda x1,x2,sigmaf,l,sigman:(sigmaf**2)*np.exp(-(1/float(2*(l**2)))*np.dot((x1-x2),(x1-x2).T)) + (sigman**2);
    K = np.zeros((X1.shape[0],X2.shape[0]))
    for i in range(0,X1.shape[0]):
        for j in range(0,X2.shape[0]):
            if i==j:
                K[i,j] = k(X1[i,:],X2[j,:],sigmaf,l,sigman);
            else:
                K[i,j] = k(X1[i,:],X2[j,:],sigmaf,l,0);
    return K


I am working on the file "GaussianProcesses/GRP.ipynb".
The first task is:
Single point prediction with GPR
We wont use optimized kernel hyperparameters here and only guess some values and predict the target for a single point.
Can you write Python code for it?

n_pts = 1

x = np.array([-1.2, -1., -0.8, -0.6, -.4, -0.2, 0.0, 0.2, 0.4, 0.6],ndmin=2).T
y = np.array([-2, -1, -0.5, -0.25, 0.5, 0.4, 0.0, 1.2, 1.7, 1.4],ndmin=2).T
x_predict = np.array([0.8,]).reshape(n_pts,1)

sigman = 0.1; # noise of the reading
sigmaf = 1.1; # parameters of the GP - next to be computed by optimization
l = 0.2; #lenght-scale of our GP with squared exponential kernel

K = get_kernel(x, x, sigmaf, l, sigman) #+ np.finfo(float).eps*np.identity(x.size) # numerically stable
K_s = get_kernel(x_predict, x, sigmaf, l, 0)
K_ss = get_kernel(x_predict, x_predict, sigmaf, l, sigman)

y_predict_mean = np.dot(np.dot(K_s,np.linalg.inv(K)),y).reshape(n_pts,1)
y_predict_var = np.diag(K_ss - np.dot(K_s,(np.dot(np.linalg.inv(K),K_s.T)))).reshape(n_pts,1)

plt.errorbar(x[:,0], y[:,0], sigman*np.ones_like(y),linestyle='None',marker = '.')

plt.errorbar(x_predict[:,0], y_predict_mean[:,0], y_predict_var[:,0], linestyle='None',marker = '.')
plt.xlabel('x');plt.ylabel('y');plt.title('single point prediction')
plt.show()

y_predict_var.shape


I am working on the file "GaussianProcesses/GRP.ipynb".
The first task is:
With current optimized parameters we will check our GP again:
Can you write Python code for it?

K = get_kernel(x, x, sigmaf, l, sigman) #+ np.finfo(float).eps*np.identity(x.size) # numerically stable
K_s = get_kernel(x_predict, x, sigmaf, l, 0)
K_ss = get_kernel(x_predict, x_predict, sigmaf, l, sigman)

y_predict_mean = np.dot(np.dot(K_s,np.linalg.inv(K)),y).reshape(n_pts,1)
y_predict_var = np.diag(K_ss - np.dot(K_s,(np.dot(np.linalg.inv(K),K_s.T)))).reshape(n_pts,1)

plt.errorbar(x[:,0], y[:,0], sigman*np.ones_like(y),linestyle='None',marker = '.')

plt.errorbar(x_predict[:,0], y_predict_mean[:,0], y_predict_var[:,0], linestyle='None',marker = '.')
plt.xlabel('x');plt.ylabel('y');plt.title('multiple prediction with MAP hyperparamters')
plt.show()


I am working on the file "Gender Prediction.ipynb".
The first task is:
Akurasi prediksi menggunakan data test yang didapat cukup lumayan berada pada tingkat 93.6%
Can you write Python code for it?

# dataset training
print(clf.score(X_train, y_train))

# dataset test
print(clf.score(X_test, y_test))


I am working on the file "Gender Prediction.ipynb".
The first task is:
Cleansing dataset
Can you write Python code for it?

# mengecek apakah ada data yang berisi null
df.isnull().values.any()

# mengecek jumlah baris data yang berisi null
len(df[pd.isnull(df).any(axis=1)])

# menghapus baris null dan recheck kembali
df = df.dropna(how='all')
len(df[pd.isnull(df).any(axis=1)])

# mengecek dimensi dataset
df.shape

# mengubah isi kolom jenis kelamin dari text menjadi integer (Laki-laki = 1; Perempuan= 0)
jk_map = {"Laki-Laki" : 1, "Perempuan" : 0}
df["jenis_kelamin"] = df["jenis_kelamin"].map(jk_map)

# cek kembali data apakah telah berubah
df.head(5)

# Mengecek distribusi jenis kelamin pada dataset

num_obs = len(df)
num_true = len(df.loc[df['jenis_kelamin'] == 1])
num_false = len(df.loc[df['jenis_kelamin'] == 0])
print("Jumlah Pria:  {0} ({1:2.2f}%)".format(num_true, (num_true/num_obs) * 100))
print("Jumlah Wanita: {0} ({1:2.2f}%)".format(num_false, (num_false/num_obs) * 100))


I am working on the file "Gender Prediction.ipynb".
The first task is:
Dengan algoritman Naive Bayes, tingkat akurasi yang didapatkan sedikit saja lebih rendah dari Logistic Regression yaitu 93.3%. Mari kita lakukan kembali testing prediksi jenis kelamin
Can you write Python code for it?

result = clf_nb.predict(["Alifah Rahmah"])
print(jk_label[result[0]])


I am working on the file "Gender Prediction.ipynb".
The first task is:
Dengan algoritman Random Forest, tingkat akurasi yang didapatkan lebih rendah dari dua algoritma sebelumnya, itu sebesar 93.12%.
Algoritma ini juga mempunyai kekurangan, yaitu performance yang yang lebih lambat.
Ok, Mari kita lakukan kembali testing prediksi jenis kelamin
Can you write Python code for it?

result = clf_rf.predict(["Yuni ahmad"])
print(jk_label[result[0]])


I am working on the file "Gender Prediction.ipynb".
The first task is:
Logistic Regression
Percobaan pertama menggunakan algoritma Logistic Regression. Data hasil feature extraction akan diinput sebagai data training.
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression()
clf.fit(X_train, y_train.ravel())


I am working on the file "Gender Prediction.ipynb".
The first task is:
Menggunakan Pipeline
Scikit memiliki fitur untuk memudahkan proses diatas dengan mengguanakan Pipeline. Penulisan kode jadi lebih simple dan rapih, berikut konversi kode diatas jika menggunakan Pipeline
Can you write Python code for it?

from sklearn.pipeline import Pipeline

clf_lg = Pipeline([('vect', CountVectorizer(analyzer = 'char_wb', ngram_range=(2,6))),
                     ('clf', LogisticRegression()),
])
_ = clf_lg.fit(text_train.ravel(), y_train.ravel())
predicted = clf_lg.predict(text_test.ravel())
np.mean(predicted == y_test.ravel())


I am working on the file "Gender Prediction.ipynb".
The first task is:
Naive Bayes
Algoritma berikutnya yang akan digunakan adalah Naive Bayes. Lansung saja kita coba
Can you write Python code for it?

from sklearn.naive_bayes import MultinomialNB

clf_nb = Pipeline([('vect', CountVectorizer(analyzer = 'char_wb', ngram_range=(2,6))),
                     ('clf', MultinomialNB()),
])

clf_nb = clf_nb.fit(text_train.ravel(), y_train.ravel())
predicted = clf_nb.predict(text_test.ravel())
np.mean(predicted == y_test.ravel())


I am working on the file "Gender Prediction.ipynb".
The first task is:
Split Dataset
Dataset yang adalah akan dipecah menjadi dua bagian, 70% data akan digunakan sebagai data training untuk melatih mesin. Kemudian 30% sisanya akan digunakan sebagai data testing untuk mengevaluasi akurasi predisksi machine learning.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

feature_col_names = ["nama"]
predicted_class_names = ["jenis_kelamin"]

X = df[feature_col_names].values
y = df[predicted_class_names].values
split_test_size = 0.30

text_train, text_test, y_train, y_test = train_test_split(X, y, test_size=split_test_size, stratify=y, random_state=42)


I am working on the file "Gender Prediction.ipynb".
The first task is:
Tingkat akurasi persis sama, dan lebih mudah dalam penulisan kode nya.
Mari kita lakukan kembali testing prediksi jenis kelamin
Can you write Python code for it?

result = clf_lg.predict(["muhammad irfani sahnur"])
print(jk_label[result[0]])


I am working on the file "General_Analysis.ipynb".
The first task is:
Clustering Attention Heads (Section 6, Figure 6)
Can you write Python code for it?

ENTROPY_THRESHOLD = 3.8  # When to say a head "attends broadly"
POSITION_THRESHOLD = 0.5  # When to say a head "attends to next/prev"
SPECIAL_TOKEN_THRESHOLD = 0.6  # When to say a heads attends to [CLS]/[SEP]"
# Heads that were found to have linguistic behaviors
LINGUISTIC_HEADS = {
    (4, 3): "Coreference",
    (7, 10): "Determiner",
    (7, 9): "Direct object",
    (8, 5): "Object of prep.",
    (3, 9): "Passive auxiliary",
    (6, 5): "Possesive",
}

# Use multi-dimensional scaling to compute 2-dimensional embeddings that
# reflect Jenson-Shannon divergences between attention heads.
mds = sklearn.manifold.MDS(metric=True, n_init=5, n_jobs=4, eps=1e-10,
                           max_iter=1000, dissimilarity="precomputed")
pts = mds.fit_transform(js_divergences)
pts = pts.reshape((12, 12, 2))
pts_flat = pts.reshape([144, 2])

colormap = cm.seismic(np.linspace(0, 1.0, 12))
plt.figure(figsize=(4.8, 9.6))
plt.title("BERT Attention Heads")

for color_by_layer in [False, True]:
  ax = plt.subplot(2, 1, int(color_by_layer) + 1)
  seen_labels = set()
  for layer in range(12):
    for head in range(12):
      label = ""
      color = GREY
      marker = "o"
      markersize = 4
      x, y = pts[layer, head]

      if avg_attns["right"][layer, head] > POSITION_THRESHOLD:
        color = RED
        marker = ">"
        label = "attend to next"

      if avg_attns["left"][layer, head] > POSITION_THRESHOLD:
        color = BLUE
        label = "attend to prev."
        marker = "<"

      if entropies[layer, head] > ENTROPY_THRESHOLD:
        color = ORANGE
        label = "attend broadly"
        marker = "^"

      if avg_attns["cls"][layer, head] > SPECIAL_TOKEN_THRESHOLD:
        color = PURPLE
        label = "attend to [CLS]"
        marker = "$C$"
        markersize = 5

      if avg_attns["sep"][layer, head] > SPECIAL_TOKEN_THRESHOLD:
        color = GREEN
        marker = "$S$"
        markersize = 5
        label = "attend to [SEP]"

      if avg_attns["punct"][layer, head] >...


I am working on the file "General_Analysis.ipynb".
The first task is:
Computing Attention Head Entropies (Section 3.3)
Can you write Python code for it?

uniform_attn_entropy = 0  # entropy of uniform attention
entropies = np.zeros((12, 12))  # entropy of attention heads
entropies_cls = np.zeros((12, 12))  # entropy of attention from [CLS]

print("Computing entropy stats")
for tokens, attns in data_iterator():
  attns = 0.9999 * attns + (0.0001 / attns.shape[-1])  # smooth to avoid NaNs
  uniform_attn_entropy -= np.log(1.0 / attns.shape[-1])
  entropies -= (attns * np.log(attns)).sum(-1).mean(-1)
  entropies_cls -= (attns * np.log(attns))[:, :, 0].sum(-1)

uniform_attn_entropy /= n_docs
entropies /= n_docs
entropies_cls /= n_docs


I am working on the file "General_Analysis.ipynb".
The first task is:
Computing Average Attention to Particular Tokens/Positions (Sections 3.1 and 3.2 of the Paper)
Can you write Python code for it?

avg_attns = {
    k: np.zeros((12, 12)) for k in [
      "self", "right", "left", "sep", "sep_sep", "rest_sep",
      "cls", "punct"]
}

print("Computing token stats")
for tokens, attns in data_iterator():
  n_tokens = attns.shape[-1]

  # create masks indicating where particular tokens are
  seps, clss, puncts = (np.zeros(n_tokens) for _ in range(3))
  for position, token in enumerate(tokens):
    if token == "[SEP]":
      seps[position] = 1
    if token == "[CLS]":
      clss[position] = 1
    if token == "." or token == ",":
      puncts[position] = 1

  # create masks indicating which positions are relevant for each key
  sep_seps = np.ones((n_tokens, n_tokens))
  sep_seps *= seps[np.newaxis]
  sep_seps *= seps[:, np.newaxis]

  rest_seps = np.ones((n_tokens, n_tokens))
  rest_seps *= (np.ones(n_tokens) - seps)[:, np.newaxis]
  rest_seps *= seps[np.newaxis]

  selectors = {
      "self": np.eye(n_tokens, n_tokens),
      "right": np.eye(n_tokens, n_tokens, 1),
      "left": np.eye(n_tokens, n_tokens, -1),
      "sep": np.tile(seps[np.newaxis], [n_tokens, 1]),
      "sep_sep": sep_seps,
      "rest_sep": rest_seps,
      "cls": np.tile(clss[np.newaxis], [n_tokens, 1]),
      "punct": np.tile(puncts[np.newaxis], [n_tokens, 1]),
  }

  # get the average attention for each token type
  for key, selector in selectors.items():
    if key == "sep_sep":
      denom = 2
    elif key == "rest_sep":
      denom = n_tokens - 2
    else:
      denom = n_tokens
    avg_attns[key] += (
        (attns * selector[np.newaxis, np.newaxis]).sum(-1).sum(-1) /
        (n_docs * denom))


I am working on the file "General_Analysis.ipynb".
The first task is:
Entropy Plots (Section 3.3, Figure 4)
Can you write Python code for it?

xs, es, avg_es = get_data_points(entropies)
xs, es_cls, avg_es_cls = get_data_points(entropies_cls)

plt.figure(figsize=(5, 5))

def plot_entropies(ax, data, avgs, label, c):
  ax.scatter(xs, data, c=c, s=5, label=label)
  ax.plot(1 + np.arange(12), avgs, c=c)
  ax.plot([1, 12], [uniform_attn_entropy, uniform_attn_entropy],
          c="k", linestyle="--")
  ax.text(7, uniform_attn_entropy - 0.45, "uniform attention",
          ha="center")
  ax.legend(loc="lower right")
  ax.set_ylabel("Avg. Attention Entropy (nats)")
  ax.set_xlabel("Layer")

plot_entropies(plt.subplot(2, 1, 1), es, avg_es, "BERT Heads",
               c=BLUE)
plot_entropies(plt.subplot(2, 1, 2), es_cls, avg_es_cls,
               "BERT Heads from [CLS]", c=RED)

plt.show()


I am working on the file "General_Machine_Learning/Kaggle_Titanic/titanic_survival_exploration/titanic_survival_exploration.ipynb".
The first task is:
After exploring the survival statistics visualization, fill in the missing code below so that the function will make your prediction.
Make sure to keep track of the various features and conditions you tried before arriving at your final prediction model.
Hint: You can start your implementation of this function using the prediction code you wrote earlier from predictions_2.
Can you write Python code for it?

def predictions_3(data):
    """ Model with multiple features. Makes a prediction with an accuracy of at least 80%. """

    predictions = []
    for _, passenger in data.iterrows():

        if (data['Sex'][_]== 'female' and data['Pclass'][_]==3 and data['Age'][_] > 38 and data['Age'][_] < 50):
            predictions.append(0)
        elif data['Sex'][_] == 'female':
            predictions.append(1)
        elif (data['Sex'][_]== 'male' and data['Age'][_]<10):
            predictions.append(1)
        else:
            predictions.append(0)

    # Return our predictions
    return pd.Series(predictions)

# Make the predictions
predictions = predictions_3(data)


I am working on the file "General_Machine_Learning/Kaggle_Titanic/titanic_survival_exploration/titanic_survival_exploration.ipynb".
The first task is:
Answer: Predictions have an accuracy of 78.68%

Using just the Sex feature for each passenger, we are able to increase the accuracy of our predictions by a significant margin. Now, let's consider using an additional feature to see if we can further improve our predictions. For example, consider all of the male passengers aboard the RMS Titanic: Can we find a subset of those passengers that had a higher rate of survival? Let's start by looking at the Age of each male, by again using the survival_stats function. This time, we'll use a fourth parameter to filter out the data so that only passengers with the Sex 'male' will be included.
Run the code cell below to plot the survival outcomes of male passengers based on their age.
Can you write Python code for it?

vs.survival_stats(data, outcomes, 'Age', ["Sex == 'male'"])


I am working on the file "General_Machine_Learning/Kaggle_Titanic/titanic_survival_exploration/titanic_survival_exploration.ipynb".
The first task is:
Examining the survival statistics, a large majority of males did not survive the ship sinking. However, a majority of females did survive the ship sinking. Let's build on our previous prediction: If a passenger was female, then we will predict that they survived. Otherwise, we will predict the passenger did not survive.
Fill in the missing code below so that the function will make this prediction.
Hint: You can access the values of each feature for a passenger like a dictionary. For example, passenger['Sex'] is the sex of the passenger.
Can you write Python code for it?

def predictions_1(data):
    """ Model with one feature:
            - Predict a passenger survived if they are female. """

    predictions = []
    for _, passenger in data.iterrows():

        if data['Sex'][_] == 'female':
            predictions.append(1)
        else:
            predictions.append(0)

    # Return our predictions
    return pd.Series(predictions)

# Make the predictions
predictions = predictions_1(data)


I am working on the file "General_Machine_Learning/Kaggle_Titanic/titanic_survival_exploration/titanic_survival_exploration.ipynb".
The first task is:
Examining the survival statistics, the majority of males younger than 10 survived the ship sinking, whereas most males age 10 or older did not survive the ship sinking. Let's continue to build on our previous prediction: If a passenger was female, then we will predict they survive. If a passenger was male and younger than 10, then we will also predict they survive. Otherwise, we will predict they do not survive.
Fill in the missing code below so that the function will make this prediction.
Hint: You can start your implementation of this function using the prediction code you wrote earlier from predictions_1.
Can you write Python code for it?

def predictions_2(data):
    """ Model with two features:
            - Predict a passenger survived if they are female.
            - Predict a passenger survived if they are male and younger than 10. """

    predictions = []
    for _, passenger in data.iterrows():

        if data['Sex'][_] == 'female':
            predictions.append(1)
        elif data['Sex'][_]== 'male' and data['Age'][_]<10:
            predictions.append(1)
        else:
            predictions.append(0)

    # Return our predictions
    return pd.Series(predictions)

# Make the predictions
predictions = predictions_2(data)


I am working on the file "General_Machine_Learning/Kaggle_Titanic/titanic_survival_exploration/titanic_survival_exploration.ipynb".
The first task is:
Question 1
Using the RMS Titanic data, how accurate would a prediction be that none of the passengers survived?
Hint: Run the code cell below to see the accuracy of this prediction.
Can you write Python code for it?


I am working on the file "General_Machine_Learning/Kaggle_Titanic/titanic_survival_exploration/titanic_survival_exploration.ipynb".
The first task is:
Question 3
How accurate would a prediction be that all female passengers and all male passengers younger than 10 survived?
Hint: Run the code cell below to see the accuracy of this prediction.
Can you write Python code for it?


I am working on the file "General_Machine_Learning/Kaggle_Titanic/titanic_survival_exploration/titanic_survival_exploration.ipynb".
The first task is:
Question 4
Describe the steps you took to implement the final prediction model so that it got an accuracy of at least 80%. What features did you look at? Were certain features more informative than others? Which conditions did you use to split the survival outcomes in the data? How accurate are your predictions?
Hint: Run the code cell below to see the accuracy of your predictions.
To quickly find an improvement to our existing prediction of 79.35% from question 3 I looked first at the survial rate of Embarked and then passenger class with respect to Men & Women. I decided after reviewing the graphs that the quickest way to improve our score was to improve the prediction of woment who did not survive. Currently we were predicting that all women lived, so any improvement to correctly predicting a few women who did not survive should impact our score.
In the graphs above it was obviouse that 3rd class women did no survive nearly as well as 2nd or 1st class women. I then optimised based on age to pull a group who did not survive at all.
I made this prediction the root of the decisiong tree after which all remaining woment were able to be classified as survived.
This along with the previouse predictions turned out to be enough to surpass the 80% requirement.
If I was to take this project further I would have next looked into the impact of the S embarked category and then SibSy / Parch.
I have also done a fair bit of analysis on this problem on Kaggle.
https://github.com/Z0m6ie/Zombie_Code/tree/Testaddfile/General_Machine_Learning/Kaggle_Titanic
Can you write Python code for it?


I am working on the file "General_Machine_Learning/Kaggle_Titanic/titanic_survival_exploration/titanic_survival_exploration.ipynb".
The first task is:
Tip: If you save an iPython Notebook, the output from running code blocks will also be saved. However, the state of your workspace will be reset once a new session is started. Make sure that you run all of the code blocks from your previous session to reestablish variables and functions before picking up where you last left off.

Making Predictions
If we were asked to make a prediction about any passenger aboard the RMS Titanic whom we knew nothing about, then the best prediction we could make would be that they did not survive. This is because we can assume that a majority of the passengers (more than 50%) did not survive the ship sinking.
The predictions_0 function below will always predict that a passenger did not survive.
Can you write Python code for it?

def predictions_0(data):
    """ Model with no features. Always predicts a passenger did not survive. """

    predictions = []
    for _, passenger in data.iterrows():

        # Predict the survival of 'passenger'
        predictions.append(0)

    # Return our predictions
    return pd.Series(predictions)

# Make the predictions
predictions = predictions_0(data)


I am working on the file "Generate Netlist from Nengo.ipynb".
The first task is:
But, we can compute the stats on it.
Can you write Python code for it?


I am working on the file "Generate Netlist from Nengo.ipynb".
The first task is:
This is a much smaller memory requirement than we'd quoted before! (down to 30M from 400M).  What happened here?
One possibility is that there's a bug in my code for extracting the netlist, and I'm looking into that.  But there have also been a lot of changes to Spaun since we last measured this number.  In particular, it looks like it's using a lot more smaller ensembles.  This is specific to this particular version of Spaun (and this particular model).  I'm looking into grabbing a few more largish models to see if I can get a wider variety of components.
However, in the meantime, we can look at the sizes of the neuron groups in Spaun:
Can you write Python code for it?

ens = {}
for ee in e.values():
    key = (ee['size_in'], ee['n_neurons'], ee['size_out'])
    ens[key] = ens.get(key, 0) + 1
pprint.pprint(ens)


I am working on the file "Generate Netlist from Nengo.ipynb".
The first task is:
This sort of reorganizing can be thought of as a sort of compilier optimization.  Making changes like this won't affect the behaviour of the model, ubt may make it more (or less) efficient in terms of the hardware.   Some optimizations are very obvious: for example, in the above graph, after the first group of neurons in the top left, we have three boxes, each of which perform matrix multiplications.  All of these can get merged together into one matrix, just by multiplying those matrices together.
In fact, they could even get rolled all the way back into the decoder for the neural group itself!  That is, instead of doing x->A->y->p->q->r as it is now, we coule instead multiply all those matrices together and get x->A->r directly.  (Note that even though there's a filter on the first box, we can move that filter to the end without changing anything, because low-pass filters are linear).
So, this sort of optimization can be interesting, and so far we've seen it to be very hardware-specific.
The above graph is for a semantic memory of size 1 (i.e. it stores associated 1-dimensional vectors).  For human-level cognitive tasks, we need somewhere around 512-dimensional vectors.  Let's see how the memory requirements scale with size:
Can you write Python code for it?

import ctn_benchmark.spa.memory

stats = []
Ds = [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
for D in Ds:
    model = ctn_benchmark.spa.memory.SemanticMemory().make_model(D=D)

    e, n, c = netlist.compute_graph(model)
    netlist.simplify_conns(e, n, c)
    s = netlist.calc_stats(e, n, c)
    s['D'] = D
    stats.append(s)

df = pandas.DataFrame(stats)
df

pylab.semilogy(df['memory'], label='memory')
pylab.semilogy(df['neurons'], label='neurons')
pylab.semilogy(df['messages'], label='messages')
pylab.semilogy(df['values'], label='values')
pylab.xticks(np.arange(len(Ds)), Ds)
pylab.legend(loc='best')
pylab.xlabel('size of semantic memory')
pylab.show()


I am working on the file "Generate Netlist from Nengo.ipynb".
The first task is:
We can also take a look at the distribution of the ensembles in the network.  For this network, we have a very regular structure.
Can you write Python code for it?

model = ctn_benchmark.spa.memory.SemanticMemory().make_model(D=512)
e, n, c = netlist.compute_graph(model)

ens = {}
for ee in e.values():
    key = (ee['size_in'], ee['n_neurons'], ee['size_out'])
    ens[key] = ens.get(key, 0) + 1
pprint.pprint(ens)


I am working on the file "GenerateTrainingSet.ipynb".
The first task is:
存储
Can you write Python code for it?

np.save('test_X', X)
np.save('test_Y', Y)

np.save('train_X', X)
np.save('train_Y', Y)


I am working on the file "GenerateTrainingSet.ipynb".
The first task is:
标准化是依照特征矩阵的列处理数据(to do)
Can you write Python code for it?

X[0]

"""
mean = (X.sum(axis=0) /sample_size)
Image.fromarray(mean)
np.sum(mean)/mean.size
X = (X.astype("float64") - mean)/256
"""
""""""

X = np.expand_dims(X, axis=3)
X.shape


I am working on the file "Generating the FCC.ipynb".
The first task is:
Now comes the interesting part, where we stack the nuclear row, and the 12-around-nucleus layer vertically, as numpy arrays, then pour those total of 13 balls into our multi-indexed DataFrame object.
Can you write Python code for it?

layer0 = np.zeros((1,7), dtype=np.float)
layer0[0, :] = tuple(nucleus[0].coords) + tuple(nucleus[0].xyz().xyz)

layer1 = np.zeros((12,7), dtype=np.float)
for idx in range(12):
    layer1[idx,:] = tuple(nl[idx].coords) + tuple(nl[idx].xyz().xyz)

layer2 = np.zeros((42,7), dtype=np.float)
for idx in range(42):
    layer2[idx,:] = tuple(nnl[idx].coords) + tuple(nnl[idx].xyz().xyz)

allvectors = np.vstack([layer0, layer1, layer2])  # vertical stack two numpy.ndarray objects

pd.options.display.float_format = '{:,.3f}'.format
ve = pd.DataFrame(allvectors, index = rowindex, columns = colindex)
ve.loc[:,['ivm']] = ve.loc[:,['ivm']].astype(np.int)
ve


I am working on the file "Generating the FCC.ipynb".
The first task is:
See the Online Encyclopedia of Integer Sequences for more information on this sequence (and maybe scroll down to find my name in the links section, Urner, K.).  As we learn from observing the Jitterbug Transformation, the cuboctahedral shells may transform into Icosahedral shells, which is why the number sequences are the same.
The final step involves generating scene description language and interpolating the resulting instructions into a POV-Ray script.
Can you write Python code for it?

sph = """sphere { %s 0.5 texture { pigment { color rgb <1,0,0> } } }"""

def get_xyz(qvectors):
    xyz_vectors = []
    for qv in qvectors:
        xyz_vectors.append(qv.xyz())
    return xyz_vectors

def make_spheres(xyz_vectors, fname="fragment.pov", append=True):
    if append:
        pov = open(fname, "a")
    else:
        pov = open(fname, "w")
    print("#declare cubocta = union{", file=pov)
    for xyzv in xyz_vectors:
        s = sph % "<{0.x}, {0.y}, {0.z}>".format(xyzv)
        print(s, file=pov)
    print("}\n", file=pov)
    print("object{cubocta scale<0.5, 0.5, 0.5>}", file=pov)

xyz_vs = get_xyz(nnl)
make_spheres(xyz_vs, fname="fragment.pov", append=False)

xyz_vs = get_xyz(nnnl)
make_spheres(xyz_vs, fname="fragment.pov")


I am working on the file "Generating the FCC.ipynb".
The first task is:
Three frequency:

Four frequency:

C6XTY at Saturday Academy

Many thanks to our sponsor.  Related reading on edu-sig.
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo("rGMrTWPxPZg")

YouTubeVideo("ilGCHYgZITY")

YouTubeVideo("9L5x-C1EglY")


I am working on the file "Generating the exponent and log tables.ipynb".
The first task is:
I used 3 as the generator for this field. For a field defined with the polynomial x^8 + x^4 + x^3 + x + 1, there may be other generators (I can't remember)
Can you write Python code for it?


I am working on the file "Generating the exponent and log tables.ipynb".
The first task is:
Multiplying the generator by itself is the same as raising it to a power. I show up to the 3rd power here
Can you write Python code for it?

generator*generator

generator*generator*generator

generator**1

generator**2

generator**3


I am working on the file "Generating the exponent and log tables.ipynb".
The first task is:
That's now our exponent table. We can look up the nth element in this list to get generator^n
Can you write Python code for it?

exptable[5] == generator**5

all(exptable[n] == generator**n for n in range(256))

[int(x) for x in exptable] == [int(x) for x in ff.GF2int_exptable]


I am working on the file "Generating the exponent and log tables.ipynb".
The first task is:
The log table is the inverse function of the exponent table
Can you write Python code for it?

logtable = [None for _ in range(256)]
# Ignore the last element of the field because fields wrap back around.
# The log of 1 could be 0 (g^0=1) or it could be 255 (g^255=1)
for i, x in enumerate(exptable[:-1]):
    logtable[x] = i
print([int(x) if x is not None else None for x in logtable])

tuple(int(x) if x is not None else None for x in logtable) == ff.GF256int.logtable


logtable = [None for _ in range(256)]
# Ignore the last element of the field because fields wrap back around.
# The log of 1 could be 0 (g^0=1) or it could be 255 (g^255=1)
for i, x in enumerate(exptable[:-1]):
    logtable[x] = i
print([int(x) if x is not None else None for x in logtable])

[int(x) if x is not None else None for x in logtable] == [int(x) if x >= 0 else None for x in ff.GF2int_logtable]


I am working on the file "Generating the exponent and log tables.ipynb".
The first task is:
The slow multiply method implemented without the lookup table has the same results
Can you write Python code for it?


I am working on the file "Generating the exponent and log tables.ipynb".
The first task is:
We can enumerate the entire field by repeatedly multiplying by the generator. (The first element is 1 because generator^0 is 1). This becomes our exponent table.
Can you write Python code for it?

exptable = [ff.GF256int(1), generator]
for _ in range(254): # minus 2 because the first 2 elements are hardcoded
    exptable.append(exptable[-1].multiply(generator))

# Turn back to ints for a more compact print representation
print([int(x) for x in exptable])


exptable = [ff.GF2int(1), generator]
for _ in range(254): # minus 2 because the first 2 elements are hardcoded
    exptable.append(exptable[-1].multiply(generator))

# Turn back to ints for a more compact print representation
print([int(x) for x in exptable])


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
$\implies$ the Johnson-Trotter algorithm is, as expected, quicker than the previous naive implementations, but it's still pretty slow compared to the reference implementation itertools.permutation.

4. Comparing our Johnson-Trotter implementation and itertools.permutation
To compare them fairly, we need to run them several times:
Can you write Python code for it?

%timeit len(list(itertools_permutations([1, 2, 3])))
%timeit len(list(third_permutations([1, 2, 3])))

%timeit len(list(itertools_permutations([1, 2, 3, 4, 5])))
%timeit len(list(third_permutations([1, 2, 3, 4, 5])))


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
$\implies$ this second algorithm is more efficient, as it requires only $0.6 s$ to generate the $8! = 40320$ different permutations of the list $[0, 1, 2, 3, 4, 5, 6, 7]$.
Can you write Python code for it?


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
5. Testing our $3$ implementations
Additionnally to comparing the speed efficiency, we would also like to simply check that all the functions we wrote are working as expected!
Can you write Python code for it?

def test(list_of_f, iterable):
    """ Test that all functions in list_of_f give the same list of permutation on this iterable."""
    print("Testing for the list of functions {} ...".format([f.__name__ for f in list_of_f]))  # DEBUG
    result = True
    print("Testing for the iterable {} ...".format(iterable))  # DEBUG
    i = iterable
    allperms = []
    for f in list_of_f:
        allperms.append(sorted([list(p) for p in f(iterable)]))
    for i, pi in enumerate(allperms):
        for j in range(i + 1, len(allperms)):
            pj = allperms[j]
            if pi != pj:
                print(" - Function #{} ({.__name__}) gave a different list of permutations as function #{} ({.__name__}) ...".format(i, list_of_f[i], j, list_of_f[j]))  # DEBUG
                result = False
            else:
                print(" - Function #{} ({.__name__}) gave the same list of permutations as function #{} ({.__name__}) ...".format(i, list_of_f[i], j, list_of_f[j]))  # DEBUG
    return result


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
And finally, the fixed-head algorithm is easy to implement, as a recursive function.
- The case of en empty list or a list with only one element are easy,
- The recursion case uses, again, a call to reduce(fun acc, x: acc + f(x), list, []) to permforms the concatenation of all lists f(x) for x in l.
Can you write Python code for it?

def second_permutations(iterable):
    """Second algorithm, fixed-head solution."""
    if len(iterable) == 0:
        return []
    # we must specify this edge case
    elif len(iterable) == 1:
        return [[iterable[0]]]
    else:
        return reduce(lambda acc, x: acc + head_of_all(x, second_permutations(rm(x, iterable))), iterable, [])


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
And let's measure its efficiency on small lists of size $4,5,6,7,8$:
Can you write Python code for it?

%time len(list(first_permutations(list(range(4)))))
%time len(list(first_permutations(list(range(5)))))
%time len(list(first_permutations(list(range(6)))))
%time len(list(first_permutations(list(range(7)))))
%time len(list(first_permutations(list(range(8)))))


%time len(list(second_permutations(list(range(4)))))
%time len(list(second_permutations(list(range(5)))))
%time len(list(second_permutations(list(range(6)))))
%time len(list(second_permutations(list(range(7)))))
%time len(list(second_permutations(list(range(8)))))
%time len(list(second_permutations(list(range(9)))))


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
Directions will be flipped, alternating left and right, with flip(d):
Can you write Python code for it?

def flip(d):
    """Flip direction d : left -> right, right -> left"""
    return not d


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
Let's try it out:
Can you write Python code for it?


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
Note that with comparisons on indexes instead of comparisons on values, we could treat the general case not much harder.

Then, we need, as before, a function to add $x$ as the head of all lists $p$ in a certain list of lists $l$.
Can you write Python code for it?

def head_of_all(x, l):
    """List of lists from l where x is the head of all the lists."""
    return [[x] + p for p in l]


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
So, what's the advantage of returning an iterator and not a list of lists? Memory and time efficiency !
The $n!$ permutations (if the list is of size $n$) take both a lot of time to compute and a lot of memory to store, so it's very clever if we can generate one after another, on demand, instead of having to compute all of them and storing them.
The first two algorithms presented below are not iterators, but the last one will be.

Permutations of the list are given as tuples, but there is no difference.

We can check how quick is this first function:
Can you write Python code for it?

%time len(list(itertools_permutations(list(range(4)))))
%time len(list(itertools_permutations(list(range(8)))))
%time len(list(itertools_permutations(list(range(9)))))

%timeit len(list(itertools_permutations(list(range(10)))))


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
Then the function move(a, i) simply swaps a[i] to the left or right, if it is possible.
It raises a ValueError exception if it cannot swap, to be general, but of course the algorithm will never be in such a undesirable state.
Can you write Python code for it?

def move(a, i):
    """Move it if possible."""
    x, d = a[i]
    if is_movable(a, i):
        if d == left:
            swap(a, i, i - 1)
        elif d == right:
            swap(a, i, i + 1)
        else:
            raise ValueError("unknown direction d = {}".format(d))
    else:
        raise ValueError("not movable")


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
Then the list will need to be scanned, and flip the directions of all elements larger than some x:
Can you write Python code for it?

def scan_flip_larger(x, a):
    """Scan to flip larger."""
    for i, (y, d) in enumerate(a):
        if y > x:
            a[i] = y, flip(d)