I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
Then we need a function to scan the array a, from its beginning, to find the largest movable element.
This can cost upto a time of $O(n)$ (if $n = #a$), but it could hardly be improved.
Can you write Python code for it?

def scan_largest_movable(a):
    """Find the largest movable element."""
    def aux(acc, i):
        if i >= len(a):
            return acc
        else:
            if not is_movable(a, i):
                return aux(acc, i + 1)
            else:
                x, _ = a[i]
                if acc is None:
                    return aux(i, i + 1)
                else:
                    j = acc if x < a[acc][0] else i
                    return aux(j, i + 1)
    return aux(None, 0)


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
Then we write a recursive function, following the description of the algorithm:
Can you write Python code for it?

from functools import reduce
# reduce(lambda acc, p: acc + f(p), l, []) is the same as the concatenation of list f(p) for p in l

# Now the main permutations generator.
def first_permutations(iterable):
    """Second algorithm, insert-into-all-positions solution."""
    if len(iterable) == 0:
        return []
    # we must specify this edge case
    elif len(iterable) == 1:
        return [[iterable[0]]]
    else:
        x, xs = iterable[0], iterable[1:]
        # reduce is needed instead of a simple sum(...) as sum() only works for numerical values
        return reduce(lambda acc, p: acc + ins_all_positions(x, p), first_permutations(xs), [])


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
There is $n!$ permutations to generate, so obviously any algorithm is running in $\Omega(n!)$ time to generate all of them, and that is approximately the behavior observed above.

This claim should need better measurements to be really empirically supported!

2. First algorithm : The insert-into-all-positions solution
The basic idea is to separate the first element $x$ of the list, and the rest $xs$.

For instance, for $l = [1, 2, 3]$, $x = 1$ and $xs = [2, 3]$.
Then the permutations of $l$ are obtained by inserting $x$ in every possible positions of every permutations of $xs$.
Here, the permutations of $xs$ are $[2, 3]$ and $[3, 2]$. Inserting $x = 1$ in the first one give $[1, 2, 3]$ (first position), $[2, 1, 3]$ and $[2, 3, 1]$. Similarly, we obtain the last permutations: $[1, 3, 2]$, $[3, 1, 2]$ and $[3, 2, 1]$.

So we first need a function that insert an element $x$ in every possible index of a list $l$.
Can you write Python code for it?

def ins_all_positions(x, l):
    """Return a list of lists obtained from l by inserting x at every possible index."""
    res = []
    for i in range(0, len(l) + 1):
        res.append(l[:i] + [x] + l[i:])
    return res


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
This classical function swap(t, i, j) exchange the position of the elements t[i] and t[j]:
Can you write Python code for it?

def swap(t, i, j):
    """Swap t[i] and t[j] in array t."""
    t[i], t[j] = t[j], t[i]


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
This will obviously be the quickest implementation, and there is no hope of beating it with pure Python (in terms of computational speed), as it is written in C and not in Python.
Let's check that it works as wanted:
Can you write Python code for it?


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
We first need to know if the element a[i] can be moved, according to its attached direction, to the left or right.
The rule is that an element can only be swapped to a small element.
Can you write Python code for it?

def is_movable(a, i):
    """Can a[i] be moved?"""
    x, d = a[i]
    if d == left:
        return i > 0 and x > a[i - 1][0]
    elif d == right:
        return i < len(a) - 1 and x > a[i + 1][0]
    else:
        raise ValueError("unknown direction d = {}".format(d))


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
We will need a first function to attach a direction to every element of an array t, and then to remove them.
Can you write Python code for it?

def attach_direction(t, d=left):
    """Attach the direction d to all elements of array t."""
    return [(x, d) for x in t]

def remove_direction(t):
    """Remove the attached direction d to all elements of array t."""
    return [y for y, _ in t]


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
We will test and compare the reference implementation, itertools.permutation, with the three other implementations given above.
Can you write Python code for it?

list_of_f = [itertools_permutations, first_permutations, second_permutations, third_permutations]

iterable = [1, 2, 3]
test(list_of_f, iterable)

iterable = [1, 2, 3, 4, 5]
test(list_of_f, iterable)

iterable = [1, 2, 3, 4, 5, 6]
test(list_of_f, iterable)


I am working on the file "Generating_permutations_with_Python.ipynb".
The first task is:
What's that weird result? In fact, itertools.permutations() does not return the list of all permutations, but rather an iterator.
It can be looped on in a similar way, and can be converted to a list easily:
Can you write Python code for it?

for p in itertools_permutations([1, 2]):
    print(p)

list(itertools_permutations([1, 2, 3]))


I am working on the file "Generating_reports_CEC2015.ipynb".
The first task is:
By Category
Can you write Python code for it?

def results_by_accuracy(acc):
    styles = ['blue', 'orange', 'yellow', 'green', 'brown']
    funs = range(15)
    results_cat = pd.DataFrame(columns=funs_group_names)

    for id, fun in enumerate(funs_group_names):
        fig_name = 'fe{}'.format(acc)
        results = get_scores(df, algs, funs_categories[fun], [acc])
        results_cat[fun] = pd.Series(results)

    title = 'Results after {} Fitness Evaluations'.format(acc)
    results_cat.plot(kind='bar', title=title, stacked=True)
    results_cat.to_excel(excel, acc)

for acc in accuracies:
    plt.figure()
    results_by_accuracy(acc)
    fname = '{}.png'.format(acc.replace('.',''))
    lgd = plt.legend(loc='center left', bbox_to_anchor=(1.0, 0.5))
    locs, labels = plt.xticks()
    plt.setp(labels, rotation=90)
    plt.savefig(fname, bbox_extra_artists=(lgd,), bbox_inches='tight')

def results_by_all():
    styles = ['blue', 'orange', 'yellow', 'green', 'brown']
    funs = range(15)
    results_cat = pd.DataFrame(columns=funs_group_names)

    for id, fun in enumerate(funs_group_names):
        fig_name = 'fall'
        results = get_scores(df, algs, funs_categories[fun], accuracies)
        results_cat[fun] = pd.Series(results)

    title = 'Overall score'
    results_cat.plot(kind='bar', title=title, stacked=True)
    results_cat.to_excel(excel, 'all')

plt.figure()
results_by_all()
fname = 'all.png'
lgd = plt.legend(loc='center left', bbox_to_anchor=(1.0, 0.5))
locs, labels = plt.xticks()
plt.setp(labels, rotation=90)
plt.savefig(fname, bbox_extra_artists=(lgd,), bbox_inches='tight')

excel.save()


I am working on the file "Generating_reports_CEC2015.ipynb".
The first task is:
By Groups
Can you write Python code for it?

fig_names = ['cat1', 'cat2', 'cat3', 'cat4', 'cat5']
styles = ['blue', 'orange', 'yellow', 'green', 'brown']

for id, title in enumerate(funs_group_names):
    fig_name = fig_names[id]
    print_results(df, algs, title, funs_categories[title], accuracies, styles[id])
    plt.savefig(fig_name, bbox_inches='tight')


I am working on the file "Generating_reports_CEC2015.ipynb".
The first task is:
Finally, a function get_scores combine the previous two functions.
Can you write Python code for it?

def get_scores(df, algs, funs, accuracies):
    """
    This function returns the scores for the algs 'algs', functions 'funs' and accuracies 'accuracies'.

    Keyword parameters:
    df -- dataframe with the data.
    algs -- algorithms to compare (must be included in df).
    funs -- functions to compare.
    accuracies -- accuracy levels to compare (in string).
    """
    size = len(algs)
    f1 = get_f1_score(size)
    result = np.zeros(size)

    for acc in accuracies:
        for i in funs:
            result += f1[calculate_points(i, df, algs, acc)-1]

    results_alg = {alg: res for alg, res in zip(algs, result)}
    return results_alg


I am working on the file "Generating_reports_CEC2015.ipynb".
The first task is:
First, Comparing by group of variables
It is very simple
Can you write Python code for it?

for fid, funs in enumerate(funs_group):
    title = funs_group_names[fid]
    results = get_scores(df, algs, funs, accuracies)
    results_df = pd.Series(results)
    plt.figure()
    results_df.plot(kind='bar', title=title)
    locs, labels = plt.xticks()
    plt.setp(labels, rotation=90)


I am working on the file "Generating_reports_CEC2015.ipynb".
The first task is:
Get all the data for comparisons
Can you write Python code for it?

df = {}

for alg in table_alg.sheet_names:
    df[alg]=table_alg.parse(alg)

#    if "_orig" in alg:
#        df[alg] = parse_table_orig(df[alg])


I am working on the file "Generating_reports_CEC2015.ipynb".
The first task is:
Get function that calculate the points for function
Can you write Python code for it?

def calculate_points(fun, dfs, algs, acc=0):
    """
    Returns the ranking in positions for the function and algorithm desired.

    Keyword parameters:
    - fun -- function to compare.
    - dfs -- hash with the dataframes for algorithm.
    - algs -- algorithms to compare (must be into df).
    - acc -- accuracy level (from 0 to 2).
    """
    values = pd.DataFrame(columns=algs)

    for alg in algs:
        df = dfs[alg]
        values[alg] = [df[acc][fun-1]]

    ranks = values.rank(1, method='min')
    return np.array(ranks, dtype=np.int).reshape(len(algs))


I am working on the file "Generating_reports_CEC2015.ipynb".
The first task is:
The following function return the scores for each position following the Formula 1 criterion.
Can you write Python code for it?

def get_f1_score(num_algs):
    """
    Return a np.array with the scoring criterio by position from the Formula 1, in which
    the first 10 items have scores.
    The array have num_algs positions.

    - If num_algs is lower than 10, it is shorten.
    - If num_algs is greater than 10, it is increased with 0s.
    """
    f1 = np.array([25, 18, 15, 12, 10, 8, 6, 4, 2, 1])

    if len(algs) < len(f1):
        f1 = f1[:len(algs)]
    else:
        f1 += np.zeros(len(algs))

    return f1


I am working on the file "Generative-Adversarial-Networks/Intro_to_GANs_Exercises.ipynb".
The first task is:
Build network
Now we're building the network from the functions defined above.
First is to get our inputs, input_real, input_z from model_inputs using the sizes of the input and z.
Then, we'll create the generator, generator(input_z, input_size). This builds the generator with the appropriate input and output sizes.
Then the discriminators. We'll build two of them, one for real data and one for fake data. Since we want the weights to be the same for both real and fake data, we need to reuse the variables. For the fake data, we're getting it from the generator as g_model. So the real data discriminator is discriminator(input_real) while the fake discriminator is discriminator(g_model, reuse=True).

Exercise: Build the network from the functions you defined earlier.
Can you write Python code for it?

tf.reset_default_graph()
# Create our input placeholders
input_real, input_z = model_inputs(input_size, z_size)

# Generator network here
g_model = generator(input_z, input_size)
# g_model is the generator output

# Disriminator network here
d_model_real, d_logits_real = discriminator(input_real)
d_model_fake, d_logits_fake = discriminator(g_model, reuse=True)


I am working on the file "Generative-Adversarial-Networks/Intro_to_GANs_Exercises.ipynb".
The first task is:
Discriminator
The discriminator network is almost exactly the same as the generator network, except that we're using a sigmoid output layer.

Exercise: Implement the discriminator network in the function below. Same as above, you'll need to return both the logits and the sigmoid output. Make sure to wrap your code in a variable scope, with 'discriminator' as the scope name, and pass the reuse keyword argument from the function arguments to tf.variable_scope.
Can you write Python code for it?

def discriminator(x, n_units=128, reuse=False, alpha=0.01):
    ''' Build the discriminator network.

        Arguments
        ---------
        x : Input tensor for the discriminator
        n_units: Number of units in hidden layer
        reuse : Reuse the variables with tf.variable_scope
        alpha : leak parameter for leaky ReLU

        Returns
        -------
        out, logits:
    '''
    with tf.variable_scope('discriminator', reuse=reuse): # finish this
        # Hidden layer
        h1 = tf.layers.dense(x, n_units, activation=None)
        # Leaky ReLU
        h1 = tf.maximum(alpha * h1, h1)

        logits = tf.layers.dense(h1, 1, activation=None)
        out = tf.sigmoid(logits)

        return out, logits


I am working on the file "Generative-Adversarial-Networks/Intro_to_GANs_Exercises.ipynb".
The first task is:
Generator network

Here we'll build the generator network. To make this network a universal function approximator, we'll need at least one hidden layer. We should use a leaky ReLU to allow gradients to flow backwards through the layer unimpeded. A leaky ReLU is like a normal ReLU, except that there is a small non-zero output for negative input values.
Variable Scope
Here we need to use tf.variable_scope for two reasons. Firstly, we're going to make sure all the variable names start with generator. Similarly, we'll prepend discriminator to the discriminator variables. This will help out later when we're training the separate networks.
We could just use tf.name_scope to set the names, but we also want to reuse these networks with different inputs. For the generator, we're going to train it, but also sample from it as we're training and after training. The discriminator will need to share variables between the fake and real input images. So, we can use the reuse keyword for tf.variable_scope to tell TensorFlow to reuse the variables instead of creating new ones if we build the graph again.
To use tf.variable_scope, you use a with statement:
python
with tf.variable_scope('scope_name', reuse=False):
    # code here
Here's more from the TensorFlow documentation to get another look at using tf.variable_scope.
Leaky ReLU
TensorFlow doesn't provide an operation for leaky ReLUs, so we'll need to make one . For this you can just take the outputs from a linear fully connected layer and pass them to tf.maximum. Typically, a parameter alpha sets the magnitude of the output for negative values. So, the output for negative input (x) values is alpha*x, and the output for positive x is x:
$$
f(x) = max(\alpha * x, x)
$$
Tanh Output
The generator has been found to perform the best with $tanh$ for the generator output. This means that we'll have to rescale the MNIST images to be between -1 and 1, instead of 0 and 1.

Exercise: Implement the generator network in the function below. You'll need to return the tanh output. Make sure...
Can you write Python code for it?

def generator(z, out_dim, n_units=128, reuse=False,  alpha=0.01):
    ''' Build the generator network.

        Arguments
        ---------
        z : Input tensor for the generator
        out_dim : Shape of the generator output
        n_units : Number of units in hidden layer
        reuse : Reuse the variables with tf.variable_scope
        alpha : leak parameter for leaky ReLU

        Returns
        -------
        out, logits:
    '''
    with tf.variable_scope('generator', reuse=reuse):   # finish this
        # Hidden layer
        h1 = tf.layers.dense(z, n_units, activation=None)
        # Leaky ReLU
        h1 = tf.maximum(alpha * h1, h1)

        # Logits and tanh output
        logits = tf.layers.dense(h1, out_dim, activation=None)
        out = tf.tanh(logits)

        return out


I am working on the file "Generative-Adversarial-Networks/Intro_to_GANs_Exercises.ipynb".
The first task is:
Generator samples from training
Here we can view samples of images from the generator. First we'll look at images taken while training.
Can you write Python code for it?

def view_samples(epoch, samples):
    fig, axes = plt.subplots(figsize=(7,7), nrows=4, ncols=4, sharey=True, sharex=True)
    for ax, img in zip(axes.flatten(), samples[epoch]):
        ax.xaxis.set_visible(False)
        ax.yaxis.set_visible(False)
        im = ax.imshow(img.reshape((28,28)), cmap='Greys_r')

    return fig, axes

# Load samples from generator taken while training
with open('train_samples.pkl', 'rb') as f:
    samples = pkl.load(f)


I am working on the file "Generative-Adversarial-Networks/Intro_to_GANs_Exercises.ipynb".
The first task is:
Hyperparameters
Can you write Python code for it?

# Size of input image to discriminator
input_size = 784 # 28x28 MNIST images flattened
# Size of latent vector to generator
z_size = 100
# Sizes of hidden layers in generator and discriminator
g_hidden_size = 128
d_hidden_size = 128
# Leak factor for leaky ReLU
alpha = 0.01
# Label smoothing
smooth = 0.1


I am working on the file "Generative-Adversarial-Networks/Intro_to_GANs_Exercises.ipynb".
The first task is:
It starts out as all noise. Then it learns to make only the center white and the rest black. You can start to see some number like structures appear out of the noise. Looks like 1, 9, and 8 show up first. Then, it learns 5 and 3.
Sampling from the generator
We can also get completely new images from the generator by using the checkpoint we saved after training. We just need to pass in a new latent vector $z$ and we'll get new samples!
Can you write Python code for it?

saver = tf.train.Saver(var_list=g_vars)
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))
    sample_z = np.random.uniform(-1, 1, size=(16, z_size))
    gen_samples = sess.run(
                   generator(input_z, input_size, reuse=True),
                   feed_dict={input_z: sample_z})
view_samples(0, [gen_samples])


I am working on the file "Generative-Adversarial-Networks/Intro_to_GANs_Exercises.ipynb".
The first task is:
Optimizers
We want to update the generator and discriminator variables separately. So we need to get the variables for each part and build optimizers for the two parts. To get all the trainable variables, we use tf.trainable_variables(). This creates a list of all the variables we've defined in our graph.
For the generator optimizer, we only want to generator variables. Our past selves were nice and used a variable scope to start all of our generator variable names with generator. So, we just need to iterate through the list from tf.trainable_variables() and keep variables that start with generator. Each variable object has an attribute name which holds the name of the variable as a string (var.name == 'weights_0' for instance).
We can do something similar with the discriminator. All the variables in the discriminator start with discriminator.
Then, in the optimizer we pass the variable lists to the var_list keyword argument of the minimize method. This tells the optimizer to only update the listed variables. Something like tf.train.AdamOptimizer().minimize(loss, var_list=var_list) will only train the variables in var_list.

Exercise:  Below, implement the optimizers for the generator and discriminator. First you'll need to get a list of trainable variables, then split that list into two lists, one for the generator variables and another for the discriminator variables. Finally, using AdamOptimizer, create an optimizer for each network that update the network variables separately.
Can you write Python code for it?

# Optimizers
learning_rate = 0.002

# Get the trainable_variables, split into G and D parts
t_vars = tf.trainable_variables()
g_vars = [var for var in t_vars if var.name.startswith('generator')]
d_vars = [var for var in t_vars if var.name.startswith('discriminator')]

d_train_opt = tf.train.AdamOptimizer(learning_rate).minimize(d_loss, var_list=d_vars)
g_train_opt = tf.train.AdamOptimizer(learning_rate).minimize(g_loss, var_list=g_vars)


I am working on the file "Generative-Adversarial-Networks/Intro_to_GANs_Exercises.ipynb".
The first task is:
These are samples from the final training epoch. You can see the generator is able to reproduce numbers like 5, 7, 3, 0, 9. Since this is just a sample, it isn't representative of the full range of images this generator can make.
Can you write Python code for it?


I am working on the file "Generative-Adversarial-Networks/Intro_to_GANs_Exercises.ipynb".
The first task is:
Training
Can you write Python code for it?

batch_size = 100
epochs = 100
samples = []
losses = []
saver = tf.train.Saver(var_list = g_vars)
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for e in range(epochs):
        for ii in range(mnist.train.num_examples//batch_size):
            batch = mnist.train.next_batch(batch_size)

            # Get images, reshape and rescale to pass to D
            batch_images = batch[0].reshape((batch_size, 784))
            batch_images = batch_images*2 - 1

            # Sample random noise for G
            batch_z = np.random.uniform(-1, 1, size=(batch_size, z_size))

            # Run optimizers
            _ = sess.run(d_train_opt, feed_dict={input_real: batch_images, input_z: batch_z})
            _ = sess.run(g_train_opt, feed_dict={input_z: batch_z})

        # At the end of each epoch, get the losses and print them out
        train_loss_d = sess.run(d_loss, {input_z: batch_z, input_real: batch_images})
        train_loss_g = g_loss.eval({input_z: batch_z})

        print("Epoch {}/{}...".format(e+1, epochs),
              "Discriminator Loss: {:.4f}...".format(train_loss_d),
              "Generator Loss: {:.4f}".format(train_loss_g))
        # Save losses to view after training
        losses.append((train_loss_d, train_loss_g))

        # Sample from generator as we're training for viewing afterwards
        sample_z = np.random.uniform(-1, 1, size=(16, z_size))
        gen_samples = sess.run(
                       generator(input_z, input_size, reuse=True),
                       feed_dict={input_z: sample_z})
        samples.append(gen_samples)
        saver.save(sess, './checkpoints/generator.ckpt')

# Save training generator samples
with open('train_samples.pkl', 'wb') as f:
    pkl.dump(samples, f)


I am working on the file "Generative-Adversarial-Networks/Intro_to_GANs_Exercises.ipynb".
The first task is:
Training loss
Here we'll check out the training losses for the generator and discriminator.
Can you write Python code for it?

%matplotlib inline

import matplotlib.pyplot as plt

fig, ax = plt.subplots()
losses = np.array(losses)
plt.plot(losses.T[0], label='Discriminator')
plt.plot(losses.T[1], label='Generator')
plt.title("Training Losses")
plt.legend()


I am working on the file "Generator.ipynb".
The first task is:
1. GIS data selection
First, run the cell below to browse to the directory your input CSV file is located at and select the input file. Sample file shall be located at .\ gep-onsset\test_data.
Can you write Python code for it?

import tkinter as tk
from tkinter import filedialog, messagebox
from openpyxl import load_workbook
root = tk.Tk()
root.withdraw()
root.attributes("-topmost", True)
messagebox.showinfo('OnSSET', 'Open the input file with extracted GIS data')
input_file = filedialog.askopenfilename()

onsseter = SettlementProcessor(input_file)
onsseter.df['IsUrban'] = 0
onsseter.df['Conflict'] = 0
onsseter.df['PerCapitaDemand'] = 0


I am working on the file "Generator.ipynb".
The first task is:
2. Modelling period and target electrification rate
Next, define the modelling period and the electrification rate to be achieved by the end of the analysis. Further down you will also define an intermediate year and target (in the Levers section).
Can you write Python code for it?

start_year = 2018
end_year = 2030
electrification_rate_target = 1 # E.g. 1 for 100% electrification rate or 0.80 for 80% electrification rate


I am working on the file "Generator.ipynb".
The first task is:
5. GIS data import and processing
OnSSET is a GIS based tool and its proper function depends heavily on the diligent preparation and calibration of the necessary geospatial data. Documentation on GIS processing in regards to OnSSET can be found here. The following cell reads the CSV-file containing the extracted GIS data for the country chosen in the previous section, and displays a snap-shot of some of the data.
Can you write Python code for it?

yearsofanalysis = [intermediate_year, end_year]

onsseter.condition_df()
onsseter.grid_penalties()
onsseter.calc_wind_cfs()
onsseter.calibrate_pop_and_urban(pop_start_year, end_year_pop, end_year_pop, urban_ratio_start_year,
                                 urban_ratio_end_year, start_year, end_year, intermediate_year)

eleclimits = {intermediate_year: intermediate_electrification_target, end_year: electrification_rate_target}
time_steps = {intermediate_year: intermediate_year-start_year, end_year: end_year-intermediate_year}

display(Markdown('#### The csv file has been imported correctly. Here is a preview:'))
display(onsseter.df[['Country','Pop','NightLights','TravelHours','GHI','WindVel','Hydropower','HydropowerDist']].sample(7))


I am working on the file "Generator.ipynb".
The first task is:
6. Define the demand
This piece of code defines the target electricity demand in the region/country. Residential electricity demand is defined as kWh/household/year, while all other demands are defined as kWh/capita/year. Note that at the moment, all productive uses demands are set to 0 by default.
Can you write Python code for it?

# Define the annual household electricity targets to choose from
tier_1 = 38.7  # 38.7 refers to kWh/household/year.
tier_2 = 219
tier_3 = 803
tier_4 = 2117
tier_5 = 2993

onsseter.prepare_wtf_tier_columns(num_people_per_hh_rural, num_people_per_hh_urban, tier_1, tier_2, tier_3, tier_4, tier_5)

onsseter.df[SET_EDU_DEMAND] = 0           # Demand for educational facilities (kWh/capita/year)
onsseter.df[SET_HEALTH_DEMAND] = 0        # Demand for health facilities (kWh/capita/year)
onsseter.df[SET_COMMERCIAL_DEMAND] = 0    # Demand for commercial activities (kWh/capita/year)
onsseter.df[SET_AGRI_DEMAND] = 0          # Demand for agricultural activities (kWh/capita/year)
productive_demand = 0 # 1 if productive demand is defined and should be included, else 0


I am working on the file "Generator.ipynb".
The first task is:
8. Results, Summaries and Visualization
With all the calculations and grid-extensions complete, this block gets the final results on which technology was chosen for each point, how much capacity needs to be installed and what it will cost. Then the summaries, plots and maps are generated.
Can you write Python code for it?

elements = []
for year in yearsofanalysis:
    elements.append("Population{}".format(year))
    elements.append("NewConnections{}".format(year))
    elements.append("Capacity{}".format(year))
    elements.append("Investment{}".format(year))

techs = ["Grid", "SA_Diesel", "SA_PV", "MG_Diesel", "MG_PV", "MG_Wind", "MG_Hydro"]

sumtechs = []
for year in yearsofanalysis:
    sumtechs.extend(["Population{}".format(year) + t for t in techs])
    sumtechs.extend(["NewConnections{}".format(year) + t for t in techs])
    sumtechs.extend(["Capacity{}".format(year) + t for t in techs])
    sumtechs.extend(["Investment{}".format(year) + t for t in techs])

summary = pd.Series(index=sumtechs, name='country')

for year in yearsofanalysis:
    for t in techs:
        summary.loc["Population{}".format(year) + t] = onsseter.df.loc[(onsseter.df[SET_MIN_OVERALL + '{}'.format(year)] == t + '{}'.format(year)), SET_POP + '{}'.format(year)].sum()
        summary.loc["NewConnections{}".format(year) + t] = onsseter.df.loc[(onsseter.df[SET_MIN_OVERALL + '{}'.format(year)] == t + '{}'.format(year)) & (onsseter.df[SET_ELEC_FINAL_CODE + '{}'.format(year)] < 99), SET_NEW_CONNECTIONS + '{}'.format(year)].sum()
        summary.loc["Capacity{}".format(year) + t] = onsseter.df.loc[(onsseter.df[SET_MIN_OVERALL + '{}'.format(year)] == t + '{}'.format(year)) & (onsseter.df[SET_ELEC_FINAL_CODE + '{}'.format(year)] < 99), SET_NEW_CAPACITY + '{}'.format(year)].sum()/1000
        summary.loc["Investment{}".format(year) + t] = onsseter.df.loc[(onsseter.df[SET_MIN_OVERALL + '{}'.format(year)] == t + '{}'.format(year)) & (onsseter.df[SET_ELEC_FINAL_CODE + '{}'.format(year)] < 99), SET_INVESTMENT_COST + '{}'.format(year)].sum()

index = techs + ['Total']
columns = []
for year in yearsofanalysis:
    columns.append("Population{}".format(year))
    columns.append("NewConnections{}".format(year))
    columns.append("Capacity{} (MW)".format(year))
    columns.append("Investment{} (million USD)".format(year))
                                         ...


I am working on the file "Generator.ipynb".
The first task is:
9. Exporting results
This code generates three csv files:
 - one containing all the results for the scenario created
 - one containing the summary for the scenario created
 - one containing some if the key input variables of the scenario
Before we proceed, please write the scenario_name in the first cell below. then move on to the next cell and run it to browse to the directory where you want to save your results. Sample file shall be located at .\ gep-onsset\sample_output.
**Note that if you do not change the scenario name, the previous output files will be overwritten
Can you write Python code for it?

scenario_name = "scenario_5"

list1 = [('Start_year',start_year,'','',''),
         ('End_year',end_year,'','',''),
         ('End year electrification rate target',electrification_rate_target,'','',''),
         ('Intermediate target year', intermediate_year,'','',''),
         ('Intermediate electrification rate target', intermediate_electrification_target,'','',''),
         ('PV cost adjustment factor', pv_adjustment_factor, '', '', ''),
         ('Urban target tier', urban_target_tier, '', '', ''),
         ('Rural target tier', rural_target_tier, '', '', ''),
         ('Prioritization', prioritization,'','','1 = baseline, 2 = intensification'),
         ('Auto intensification distance', auto_intensification, '', '', 'Buffer distance (km) for automatic intensification if choosing prioritization 1'),
         ('coordinate_units',coordinate_units,'','','1000 if coordinates are in m, 1 if coordinates are in km'),
         ('discount_rate',discount_rate,'','',''),
         ('pop_threshold',pop_threshold,'','',''),
         ('pop_start_year',pop_start_year,'','','the population in the base year (e.g. 2016)'),
         ('pop_end_year',end_year_pop,'','','the projected population in the end year (e.g. 2030)'),
         ('urban_ratio_start_year',urban_ratio_start_year,'','','the urban population population ratio in the base year (e.g. 2016)'),
         ('urban_ratio_end_year',urban_ratio_end_year,'','','the urban population population ratio in the end year (e.g. 2030)'),
         ('num_people_per_hh_urban',num_people_per_hh_urban,'','','the number of people per household expected in the end year (e.g. 2030)'),
         ('num_people_per_hh_rural',num_people_per_hh_rural,'','','the number of people per household expected in the end year (e.g. 2030)'),
         ('elec_ratio_start_year',elec_ratio_start_year,'','','the electrification rate in the base year (e.g. 2016)'),
         ('urban_elec_ratio',urban_elec_ratio,'','','urban electrification rate in the base year (e.g. 2016)'),
        ...


I am working on the file "Generator.ipynb".
The first task is:
Calibration of currently electrified settlements
The model calibrates which settlements are likely to be electrified in the start year, to match the national statistical values defined above. A settlement is considered to be electrified if it meets all of the following conditions:
- Has more night-time lights than the defined threshold (this is set to 0 by default)
- Is closer to the existing grid network than the distance limit
- Has more population than the threshold
First, define the threshold limits. Then run the calibration and check if the results seem okay. Else, redefine these thresholds and run again.
Can you write Python code for it?

min_night_lights = 0    ### 0 Indicates no night light, while any number above refers to the night-lights intensity
min_pop = 0      ### Settlement population above which we can assume that it could be electrified

max_service_transformer_distance = 2    ### Distance  in km from the existing grid network below which we can assume a settlement could be electrified
max_mv_line_distance = 2
max_hv_line_distance = 25

Technology.set_default_values(base_year=start_year, start_year=start_year, end_year=end_year, discount_rate=discount_rate,
                             HV_line_type=hv_line_capacity, HV_line_cost=hv_line_cost, MV_line_type=mv_line_capacity,
                             MV_line_amperage_limit=MV_line_amperage_limit, MV_line_cost=mv_line_cost, LV_line_type=lv_line_capacity,
                             LV_line_cost=lv_line_cost, LV_line_max_length=lv_line_max_length,
                             service_Transf_type=service_Transf_type, service_Transf_cost = service_Transf_cost,
                             max_nodes_per_serv_trans=max_nodes_per_serv_trans, MV_LV_sub_station_cost=mv_lv_transformer_cost,
                             MV_MV_sub_station_cost=mv_mv_transformer_cost, HV_LV_sub_station_cost=hv_lv_transformer_cost,
                             HV_MV_sub_station_cost=hv_mv_transformer_cost)

elec_modelled, urban_internal_elec_ratio, rural_internal_elec_ratio = onsseter.elec_current_and_future(elec_ratio_start_year,
                                                                                urban_elec_ratio,
                                                                                rural_elec_ratio,
                                                                                pop_start_year,
                                                                                start_year,
                                                                                ...


I am working on the file "Generator.ipynb".
The first task is:
In some cases it can be of interest to filter out clusters with very low populations, e.g. to increase computational speed or to remove false positives in the data. Setting the pop_threshold variable below larger than 0 will filter out all settlements below that threshold form the analysis.
Can you write Python code for it?

pop_threshold = 0  # If you wish to remove low density population cells, enter a threshold above 0
onsseter.df = onsseter.df.loc[onsseter.df[SET_POP] > pop_threshold]


I am working on the file "Generator.ipynb".
The first task is:
Lever 2: Electricity demand target
For the second lever, enter the target tier (level of electricity access) for urban and rural households respectively. This can take a value between 1 (lowest level of electricity access) and 5 (highest level of electricity access) as in ESMAPs Multi-Tier Framework for Measuring Electricity Access (found here). Alternatively, enter 6 to use a distribution of the tiers across the country based on poverty levels and GDP according to the methodology found here link.
Can you write Python code for it?

urban_target_tier = 5
rural_target_tier = 3


I am working on the file "Generator.ipynb".
The first task is:
Lever 3: Intermediate electrification rate target
For the third lever, enter the intermediate target year and target electrification rate for that year.
Can you write Python code for it?

intermediate_year = 2025
intermediate_electrification_target = 0.63 # E.g. for a target electrification rate of 75%, enter 0.75


I am working on the file "Generator.ipynb".
The first task is:
Lever 5: PV system cost adjustment
This lever reflects the role of PV system costs on electrification results. All PV based systems will be adjusted by a factor to simulate a higher or lower cost of PV systems (compared to the baseline values entered below). A value lower than 1 means lower investment costs for PV systems compared to baseline, and a value larger than 1 means higher investment cost for PV systems compared to baseline. E.g. 0.75 would mean a cost that is 25% lower compared to baseline costs.
Can you write Python code for it?


I am working on the file "Generator.ipynb".
The first task is:
Lever 6: Prioritization algorithm
This lever reflects the prioritization approach in order to achieve the electrification rate specified (in Lever 3) in intermediate target year of the analysis. There are currently two options available:
Baseline: Prioritizes grid densification first (ramp up in already electrified clusters) then selection based on lowest invetsment cost per capita. Grid densification is limited by a grid capacity cap per year and a number of sensible grid connections per year.
Intensification: Same as above, plus automatic grid intensification to all clusters within a predefined buffer of X km. (Default value for X is 2km)
Can you write Python code for it?

prioritization = 1                        # Select 1 or 2. 1 = baseline, 2 = intensification, 3 = easily accessible settlements

auto_intensification = 2                  # Buffer distance (km) for automatic intensification if choosing prioritization 1
annual_new_grid_connections_limit = 109 # This is the maximum amount of new households that can be connected to the grid in one year (thousands)
annual_grid_cap_gen_limit = 100       # This is the maximum generation capacity that can be added to the grid in one year (MW)


I am working on the file "Generator.ipynb".
The first task is:
The figure below show the results of the calibration. Settlements in blue are considered to be (at least partly) electrified already in the start year of the analysis, while settlements in yellow are yet to be electrified. Re-running the calibration step with different intial values may change the map below.
Can you write Python code for it?

from matplotlib import pyplot as plt
colors = ['#73B2FF','#EDD100','#EDA800','#1F6600','#98E600','#70A800','#1FA800']
plt.figure(figsize=(9,9))
plt.plot(onsseter.df.loc[onsseter.df[SET_ELEC_CURRENT]==0, SET_X_DEG], onsseter.df.loc[onsseter.df[SET_ELEC_CURRENT]==0, SET_Y_DEG], 'y,')
plt.plot(onsseter.df.loc[onsseter.df[SET_ELEC_CURRENT]==1, SET_X_DEG], onsseter.df.loc[onsseter.df[SET_ELEC_CURRENT]==1, SET_Y_DEG], 'b,')
if onsseter.df[SET_X_DEG].max() - onsseter.df[SET_X_DEG].min() > onsseter.df[SET_Y_DEG].max() - onsseter.df[SET_Y_DEG].min():
    plt.xlim(onsseter.df[SET_X_DEG].min() - 1, onsseter.df[SET_X_DEG].max() + 1)
    plt.ylim((onsseter.df[SET_Y_DEG].min()+onsseter.df[SET_Y_DEG].max())/2 - 0.5*abs(onsseter.df[SET_X_DEG].max() - onsseter.df[SET_X_DEG].min()) - 1, (onsseter.df[SET_Y_DEG].min()+onsseter.df[SET_Y_DEG].max())/2 + 0.5*abs(onsseter.df[SET_X_DEG].max() - onsseter.df[SET_X_DEG].min()) + 1)
else:
    plt.xlim((onsseter.df[SET_X_DEG].min()+onsseter.df[SET_X_DEG].max())/2 - 0.5*abs(onsseter.df[SET_Y_DEG].max() - onsseter.df[SET_Y_DEG].min()) - 1, (onsseter.df[SET_X_DEG].min()+onsseter.df[SET_X_DEG].max())/2 + 0.5*abs(onsseter.df[SET_Y_DEG].max() - onsseter.df[SET_Y_DEG].min()) + 1)
    plt.ylim(onsseter.df[SET_Y_DEG].min() -1, onsseter.df[SET_Y_DEG].max() +1)
plt.figure(figsize=(30,30))


I am working on the file "Generator.ipynb".
The first task is:
b. Technology specifications & costs
The cell below contains all the information that is used to calculate the levelized costs for all the technologies, including grid. These default values should be updated to reflect the most accurate values in the country. There are currently 7 potential technologies to include in the model:
* Grid
* PV Mini-grid
* Wind Mini-grid
* Hydro Mini-grid
* Diesel Mini-grid
* PV Stand-alone systems
* Diesel Stand-alone systems
First, decide whether to include diesel technologies or not:
Can you write Python code for it?

diesel_techs = 0                      ### 0 = diesel NOT included, 1 = diesel included

grid_power_plants_capital_cost = 2000 ### The cost in USD/kW to for capacity upgrades of the grid
grid_losses = 0.1                     ### The fraction of electricity lost in transmission and distribution (percentage)
base_to_peak = 0.8                    ### The ratio of base grid demand to peak demand (percentage)
existing_grid_cost_ratio = 0.1        ### The additional cost per round of electrification (percentage)

diesel_price = 0.5                   ### This is the diesel price in USD/liter as expected in the end year of the analysis.

sa_diesel_capital_cost = 938          ### Stand-alone Diesel capital cost (USD/kW) as expected in the years of the analysis
mg_diesel_capital_cost = 721          ### Mini-grid Diesel capital cost (USD/kW) as expected in the years of the analysis
mg_pv_capital_cost = 2950             ### Mini-grid PV capital cost (USD/kW) as expected in the years of the analysis
mg_wind_capital_cost = 3750           ### Mini-grid Wind capital cost (USD/kW) as expected in the years of the analysis
mg_hydro_capital_cost = 3000          ### Mini-grid Hydro capital cost (USD/kW) as expected in the years of the analysis

sa_pv_capital_cost_1 = 9620          ### Stand-alone PV capital cost (USD/kW) for household systems under 20 W
sa_pv_capital_cost_2 = 8780          ### Stand-alone PV capital cost (USD/kW) for household systems between 21-50 W
sa_pv_capital_cost_3 = 6380           ### Stand-alone PV capital cost (USD/kW) for household systems between 51-100 W
sa_pv_capital_cost_4 = 4470           ### Stand-alone PV capital cost (USD/kW) for household systems between 101-1000 W
sa_pv_capital_cost_5 = 6950           ### Stand-alone PV capital cost (USD/kW) for household systems over 1 kW


I am working on the file "Generators_and_Coroutines.ipynb".
The first task is:
Any object expecting to be the target of a for loop, if not already an iterator, needs to either:

dedicate an __iter__ method to showing what to return when iter() gets applied, or
have a __getitem__ ready to go, as iter( ) is smart enough to make up an object wherein consecutive integers, starting from 0, go to any __getitem__ method.
Can you write Python code for it?

class Any:

    def __init__(self):
        self.__dict__ = {0:'scissors', 1:'paper', 2:'rock'}

    def __getitem__(self, n):  # enough for iter() to go on
        if n == len(self.__dict__):
            raise StopIteration  # tells for loop when to stop
        return self.__dict__[n]

for thing in Any():
    print(thing)


I am working on the file "Generators_and_Coroutines.ipynb".
The first task is:
Feeding an open-ended iterator to a for loop runs the risk of a "forever loop".  The itertools module is filled with tools designed to rein in the infinite loopers.  Just use islice(obj, start, stop) to keep the for loop finite:
Can you write Python code for it?

from itertools import islice
p = Primes()
for n in islice(p, 0, 20):
    print(n, end=", ")


I am working on the file "Generators_and_Coroutines.ipynb".
The first task is:
In the code below, we see (yield) turned around and used not to return objects, but to receive them.  The parentheses are by convention and suggest a "mouth" bringing something in from outside.
A generator function's .send() method resumes its execution inside its body with an intake of some passed-in object from the caller, the argument to send.  In the above example, two callers are nested, the inner one writing a prime to a file, the outer one feeding it next primes for said catalog.
The coroutine decorator may seem a bit mysterious at first.  A generator function does not run any of its code upon being instanced.  No yield statement has yet been encountered, so use of .send(obj) would raise an exception were obj any object but None.
The decorator has already fed a .send(None) to the generator in question, equivalent to feeding it to next() a first time.  The decorator applies a first action somewhat like cocking a pistol, putting the generator or coroutine in the firing position, positioned at a first yield.
Can you write Python code for it?

# -*- coding: utf-8 -*-
"""
Created on Thu Oct 13 13:48:52 2016

@author: Kirby Urner

David Beazley:
https://youtu.be/Z_OAlIhXziw?t=23m42s

Trial by division, but this time the primes coroutine acts
more as a filter, passing qualified candidates through to
print_me, which writes to a file.
"""
import pprint

def coroutine(func):
    """
    Advances decorated generator function to the first yield
    """
    def start(*args, **kwargs):
        cr = func(*args, **kwargs)
        cr.send(None)  # or next(cr) or cr.__next__()
        return cr
    return start

@coroutine
def print_me(file_name):
    with open(file_name, 'w') as file_obj:
        while True:
            to_print = (yield)
            file_obj.write(str(to_print)+"\n")

@coroutine
def primes(target):
    _primes_so_far = [2]
    target.send(2)
    while True:
        candidate = (yield)
        for prev in _primes_so_far:
            if not divmod(candidate, prev)[1]:
                break
            if prev**2 > candidate:
                _primes_so_far.append(candidate)
                target.send(candidate)
                break

output = print_me("primes.txt")
p = primes(output)

for x in range(3, 200, 2):  # test odds 3-199
    p.send(x)

with open("primes.txt", 'r') as file_obj:
    print(", ".join(file_obj.read().split("\n"))[:-2])


I am working on the file "Generators_and_Coroutines.ipynb".
The first task is:
The generator function below shows what most Pythonistas consider the base case for the keyword yield: using it much like return, to provide an object to the generator function's caller, in this case a next prime number, going in sequence.
The trial-by-division algorithm requires keeping a growing sequence of successive primes, and using them to test new candidates.  After taking care of the only even prime, 2, it makes sense to jump forward through the odds, screening out all the composites.
Can you write Python code for it?

import pprint

def primes():
    """generate successive prime numbers (trial by division)"""
    candidate = 1
    _primes_so_far = [2]     # first prime, only even prime
    yield _primes_so_far[0]  # share it!
    while True:
        candidate += 2    # check odds only from now on
        for prev in _primes_so_far:
            if prev**2 > candidate:
                yield candidate  # new prime!
                _primes_so_far.append(candidate)
                break
            if not divmod(candidate, prev)[1]: # no remainder!
                break                          # done looping

p = primes()  # generator function based iterator
pp = pprint.PrettyPrinter(width=40, compact=True)
pp.pprint([next(p) for _ in range(50)])  # next 30 primes please!


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
Ensuite il faut un peu de nettoyage pour enlever les lignes qui ont été incorrectement ajoutées dans le fichier (j'ai fait ça à la main).
Can you write Python code for it?

!head data/latin.txt

!ls -larth data/latin.txt
!wc data/latin.txt


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
Et on peut générer 3 phrases aléatoires :
Can you write Python code for it?

for _ in range(3):
    words = markov(corpus, start, length)
    print(' '.join(words))


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
Exemples
Can you write Python code for it?

for _ in range(10):
    print(">", citation_aleatoire(italic=True))


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
Fausses locutions latines
On va extraire le corpus, la liste des premiers mots, et la probabilité qu'un mot en début de citation commence par une majuscule.
Can you write Python code for it?

WORD_LIST = "data/latin.txt"
corpus = open(WORD_LIST).readlines()

print("Exemple d'une citation :", corpus[0])
print("Il y a", len(corpus), "citations.")

starts = [c.split()[0] for c in corpus]
start = random.choice(starts)
print("Exemple d'un mot de début de citation :", start)
print("Il y a", len(starts), "mots de débuts de citations.")

proba_title = len([1 for s in starts if s.istitle()]) / len(starts)
print("Il y a {:.3%} chance de commencer une citation par une majuscule.".format(proba_title))


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
Générer aléatoirement les explications foireuses du Roi Loth
C'est moins facile... Mais sans chercher à être parfait, on va juste prendre une explication parmi celles qui existent :
Can you write Python code for it?

explications = [
    ". Ah non, parce que là, j’en ai marre !",
    ". Voilà. Eh bien ça, par exemple, ça veut absolument rien dire, mais l’effet reste le même, et pourtant j’ai jamais foutu les pieds dans une salle de classe attention !",
    ". Bon, ça ne veut absolument rien dire, mais je trouve que c’est assez dans le ton.",
    ", ça veut rien dire, mais je suis très en colère contre moi-même.",
    " : seul les dieux décident.",
    """, ça n'a aucun sens mais on pourrait très bien imaginer une traduction du type : "Le roseau plie, mais ne cède... qu'en cas de pépin", ce qui ne veut rien dire non plus.""",
]


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
Le module lea sera très pratique pour manipuler les probabilités pour les chaînes de Markov.
Can you write Python code for it?


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
On a 1571 citations latines, c'est déjà un corpus conséquent !
Exploration de chaînes de Markov pour la génération aléatoire
J'utilise cette fonction markov écrite par Jill-Jênn Vie.
Can you write Python code for it?

def markov(corpus, start, length):
    # Counting occurrences
    next_one = defaultdict(Counter)
    for sentence in corpus:
        words = sentence.split()
        nb_words = len(words)
        for i in range(nb_words - 1):
            next_one[words[i]][words[i + 1]] += 1

    # Initializing states
    states = {}
    for word in next_one:
        states[word] = Lea.fromValFreqsDict(next_one[word])

    # Outputting visited states
    word = start
    words = [word]
    for _ in range(length - 1):
        word = states[word].random()
        words.append(word)
    return(words)


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
On a bientôt ce qu'il faut pour générer une locution latine aléatoire.
Il arrive que la chaîne de Markov se bloque, donc on va juste essayer plusieurs fois avec des débuts différents.
Can you write Python code for it?

def markov_try_while_failing(corpus, starts, length_min, length_max, proba_title, nb_max_trial=100):
    # Try 100 times to generate a sentence
    start = random.choice(starts)
    length = random.randint(length_min, length_max)
    for trial in range(nb_max_trial):
        try:
            words = markov(corpus, start, length)
            if random.random() <= proba_title:
                words[0] = words[0].title()
            return words  # comment to debug
            print(' '.join(words))
            break
        except KeyError:
            start = random.choice(starts)
            length = random.randint(length_min, length_max)
            continue
    raise ValueError("Echec")


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
On peut essayer :
Can you write Python code for it?

for _ in range(10):
    words = markov_try_while_failing(corpus, starts, length_min, length_max, proba_title)
    print(' '.join(words))


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
On va générer des locutions de 3 à 6 mots :
Can you write Python code for it?


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
Par exemple :
Can you write Python code for it?

corpus = [
    'je mange des cerises',
    'je mange des bananes',
    'je conduis des camions',
]
start = 'je'
length = 4


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
Récupérer et nettoyer les données
J'ai utilisé cette page Wikipédia et deux lignes de Bash :
Can you write Python code for it?

%%bash
wget --no-verbose "https://en.wikipedia.org/wiki/List_of_Latin_phrases_(full)" -O /tmp/latin.html
grep -o '<b>[^<]*</b>' /tmp/latin.html | sed s_'</\?b>'_''_g | sort | uniq | sort | uniq > /tmp/data_latin.txt

!head data/latin.txt


I am working on the file "Generer_des_fausses_citations_latines_du_Roi_Loth.ipynb".
The first task is:
Ça a déjà l'air pas mal latin !
Fausses citations du Roi Loth
Pour générer une citation du Roi Loth, il ne suffit pas d'avoir des locutions latines.
Il faut le contexte, l'explication, une fausse citation d'un épisode de Kaamelott etc...
Premier exemple
Ecouter celle là : Misa brevis, et spiritus maxima.
<audio src="data/tres_en_colere.mp3" controls="controls">Your browser does not support the audio element.</audio>
Exemples

Ave Cesar, rosae rosam, et spiritus rex ! Ah non, parce que là, j’en ai marre !
-- François Rollin, Kaamelott, Livre III, L’Assemblée des rois 2e partie, écrit par Alexandre Astier.
Tempora mori, tempora mundis recorda. Voilà. Eh bien ça, par exemple, ça veut absolument rien dire, mais l’effet reste le même, et pourtant j’ai jamais foutu les pieds dans une salle de classe attention !
-- François Rollin, Kaamelott, Livre III, L’Assemblée des rois 2e partie, écrit par Alexandre Astier.
Victoriae mundis et mundis lacrima. Bon, ça ne veut absolument rien dire, mais je trouve que c’est assez dans le ton.
-- François Rollin, Kaamelott, Livre IV, Le désordre et la nuit, écrit par Alexandre Astier.
Misa brevis et spiritus maxima, ça veut rien dire, mais je suis très en colère contre moi-même.
-- François Rollin, Kaamelott, Livre V, Misère noire, écrit par Alexandre Astier.
Deus minimi placet : seul les dieux décident.
-- François Rollin, Kaamelott, Livre VI, Arturus Rex, écrit par Alexandre Astier.
"Mundi placet et spiritus minima", ça n'a aucun sens mais on pourrait très bien imaginer une traduction du type : "Le roseau plie, mais ne cède... qu'en cas de pépin" ce qui ne veut rien dire non plus.
-- François Rollin, Kaamelott, Livre VI, Lacrimosa, écrit par Alexandre Astier.

Générer aléatoirement les métadonnées de l'épisode
C'est facile.
Can you write Python code for it?

episodes = [
    "Livre III, L’Assemblée des rois 2e partie, écrit par Alexandre Astier.",
    "Livre III, L’Assemblée des rois 2e partie, écrit par Alexandre Astier.",  # présent deux fois
    "Livre IV, Le désordre et la nuit, écrit par Alexandre Astier.",
    "Livre V, Misère noire, écrit par Alexandre Astier.",
    "Livre VI, Arturus Rex, écrit par Alexandre Astier.",
    "Livre VI, Lacrimosa, écrit par Alexandre Astier."
]

def metadonnee_aleatoire(episodes=episodes):
    episode = random.choice(episodes)
    return "D'après François Rollin, inspiré par Kaamelott, " + episode


I am working on the file "Generic-GitHub.ipynb".
The first task is:
Basic API request
Can you write Python code for it?

url = 'https://api.github.com/orgs/jupyterhub/repos'

response = requests.get(url)
if response.status_code != 200:
    # This means something went wrong.
    raise ApiError('GET /orgs/ {}'.format(resp.status_code))

repos = response.json()
pprint.pprint(repos)

print('The total number of repos in the organization is {}'.format(len(repos)))

# print repos
print('{0:30s}      {1:20s}\n'.format('Repository name', 'open_issues_count'))

for num in range(0, len(repos)):
    print('{0:30s}      {1:4d}\n'.format(repos[num]['name'],  repos[num]['open_issues_count']))

for num in range(0, len(repos)):
    print('{0:30s}      {1:50s}\n'.format(repos[num]['name'], repos[num]['description']))

print('{0:30s}      {1:20s}\n'.format('Repository name', 'open_issues_count'))

for num in range(0, len(repos)):
    print('{0:30s}      {1:4d}  {2:20s}\n'.format(repos[num]['name'], repos[num]['open_issues_count'], repos[num]['description']))


I am working on the file "Generic-GitHub.ipynb".
The first task is:
GitHub Authorization
Note: Be careful. Don't check in to version control your username and password.
Can you write Python code for it?

TOKEN=''
gh = github3.login(token=TOKEN)
type(gh)


I am working on the file "Generiranje kolokvija.ipynb".
The first task is:
Kreiranje datoteke
Can you write Python code for it?

datoteka = "ms_kol1.tex"

with open(datoteka,'w') as f:
    f.write(header1+'\n')
    for i in range(broj_studenata):
        random.seed()
        r=random.randint(1,broj_zadataka,5)
        f.write(header2)
        f.write("\\begin{center}{\large \\textbf{Student: "+studenti[i][:-1]+"}}\end{center}\n\n")
        f.write(header3)
        for j in range(5):
            z = str(j+1)+str(r[j]).zfill(2)
            f.write('\\input zadaci-1/z'+z+'\n')
        f.write(footer1)
    f.write(footer2)


I am working on the file "Generiranje kolokvija.ipynb".
The first task is:
Učitavanje potrebnih paketa & podataka
Can you write Python code for it?

from numpy import random
with open('studenti.txt','r') as f:
    studenti = list(f)
broj_studenata = len(studenti)
broj_zadataka = 30


I am working on the file "GeneticAlgorithm.ipynb".
The first task is:
Definition of the problem
We are given an compound oscillatory signal $W(t)$ at given times and we would like to find its components. We are too primitive to use FFT or non-linear fitting. We are dumb and strong, so we brute-force it!
Definition of element
Each $i$-th element of the genetic population represents a possible candidate signal $W_i(t) = \sum_{k=1}^n A_k \cos\left( 2\pi k t\right)$. Thus, the element can be represented by the list of amplitudes for each wave component: $\mathbf{e} = { A_1, A_2, ... A_n}$.
The training set is a collection of $\left( t,W(t) \right)$ points corresponding to the signal we are trying to fit.
Can you write Python code for it?

# set a decent population size to work with
populationSize = 100

# set the amount of plane waves
nWaves = 5

# limit amplitudes to [-2, 2]
scale = 2.0

# create a GA engine
ga = GAEngine(popSize=populationSize, dnaSize=nWaves, scale=scale)

ga.tau = 0.1

# print the first 3 elements to see what we are dealing with!
print(ga.population[0:3])

# load a training set
trainT = numpy.load("./data/ga-train-T.npy") # time values
trainY = numpy.load("./data/ga-train-Y.npy") # corresponding signal values

# plot the training set
plt.plot(trainT, trainY, 'o')
plt.show()


I am working on the file "GeneticAlgorithm.ipynb".
The first task is:
Evaluating fitness
This function takes in the element descriptor, performs the calculations necessary to estimate how well it performs, and returns its fitness.<br>
It is actually easy to calculate its badness (mean square error) and flip the sign. This way, the perfect element will have 0 fitness, while any mismatch with the training will give a negative value.
Can you write Python code for it?

def EvaluateFitness(element):

    # prepare a 0 array
    y = numpy.zeros(len(trainT))
    freq = 1 # starting frequency

    for amp in element: # loop through the amplitudes
        y += amp * numpy.cos(2*numpy.pi*freq*trainT) # add the wave
        freq += 1 # increment the frequency for the next wave

    # compute square error
    y = (y-trainY)*(y-trainY)

    return -numpy.mean(y)

# tell the engine to use this evaluation function
ga.Evaluate = EvaluateFitness


I am working on the file "GeneticAlgorithm.ipynb".
The first task is:
Evolving...
Let the population evolve!
Can you write Python code for it?

# number of generations to compute
nGens = 30

stats = numpy.zeros((nGens,3))

# start the evolution loop!
for g in range(nGens):
    stats[g] = ga.Evolve()

# evolution plot
plt.plot(stats[:,0],'o-', label='best')
plt.plot(stats[:,1],'o-', label='average')
plt.plot(stats[:,2],'o-', label='worst')
plt.legend()
plt.show()

# best element info
print("best model amplitudes:", ga.best)
print("best model fitness:", ga.bestFit)
t = numpy.arange(0,1,0.01)
y = t * 0
freq = 1 # starting frequency

for amp in ga.best: # loop through the amplitudes
    y += amp * numpy.cos(2*numpy.pi*freq*t) # add the wave
    freq += 1 # increment the frequency for the next wave

plt.plot(trainT, trainY, 'o', label='training points')
plt.plot(t, y, '-', label='best model')
plt.legend()
plt.show()


I am working on the file "GeneticAlgorithm.ipynb".
The first task is:
Final remarks
Do not be fooled by the speed of this example... in real applications, training an AI with GA can take up to several weeks on supercomputers.

You can see one in action here! Try to beat it!
Beast Exercise
Train neural networks to output the energy of a molecule using ACSFs as input.
There should be one network for each atom type and ACSF contribution. For example, all H atoms will have 5 NNs that process the ACSFs calculated for that atom with respect to C, N, O, or F atoms.
Additionally there will be NNs for the 3-body parts: H-HH, H-HC, H-HN, ... total 15 NNs
In total, each atom type should have 5 + 15 = 20 NNs, giving a total of 100 NNs and lots of parameters to train!
Some serious python hacking is required!
Can you write Python code for it?

# load structures and energies from the database
# the file is data/structures.xyz and it contains lots of molecules in the following xyz format:
#
# natoms_mol1
# energy_mol1
# type1 x y z
# type2 x y z
# ...
# natoms_mol2
# energy_mol2
# type1 x y z
# ...
#

# use DScribe to compute ACSFs
# ...

# the ACSFs should be divided into their logical contributions (x-H, x-C, ... x-HH, x-HC, ...)
# the length of each contribution

from sklearn.neural_network import MLPRegressor

# setup model networks for all possible atoms
# there are only H, C, N, O, F atoms in the database
#
# H -> H (2-body)
# H -> C (2-body)
# ...
# H -> H-H (3-body)
# H -> H-C (3-body) (no need for H -> C-H, it is symmetric!)
# ...

# sklearn network weights and neuron biases are stored in
# model.coefs_ and model.intercepts_
# these are assigned only after calling model.fit(x, y), does not matter if the training succeeded
# as long as x,y have the correct shape
#
# the GA DNA is the concatenation of all these numbers
#
# calculate the total length of the DNA
# ...

# write a function that converts the DNA (list of numbers) into a set of model.coefs_ and
# model.intercepts_ for a sklearn model
# ...

# write an evaluation function for the fitness of a GA element
# ...

# evolve the population
# ...


I am working on the file "Genetic_Chords.ipynb".
The first task is:
Also between two chords, you have to avoid the consecutive fifths and octaves, and a normal person tends to make the intervals in a voice more "natural" making jumps not bigger than a fifth that often.
Can you write Python code for it?

def check_2_chords(ch1, ch2):
    """Return the number of mistakes in the intervals between 2 chords."""
    res = 0

    # Check for 5° and 8°
    ite1 = map(lambda x,y: y-x, ch1[:-1], ch1[1:])
    ite2 = map(lambda x,y: y-x, ch2[:-1], ch2[1:])
    for inter1, inter2 in zip(ite1, ite2):
        if inter1 == 7 and inter2 == 7:
            res += 15
        elif inter1 == 0 and inter2 == 0:
            res += 15
        elif inter1 == 12 and inter2 == 12:
            res += 15

    # Check for big intervals, just to make it more "human"
    for note1, note2 in zip(ch1[1:], ch2[1:]):
        if abs(note1-note2) >= 7: # 7 equals 5° interval
            res += .7

    return res


I am working on the file "Genetic_Chords.ipynb".
The first task is:
And for the evaluation of an individual I used this generator to access an element and the two neighbors:
Can you write Python code for it?

def neighborhood(iterable):
    """Generator gives the prev actual and next."""
    iterator = iter(iterable)
    prev = None
    item = next(iterator)  # throws StopIteration if empty.
    for nex in iterator:
        yield (prev,item,nex)
        prev = item
        item = nex
    yield (prev,item,None)


I am working on the file "Genetic_Chords.ipynb".
The first task is:
And just to show the result I made a little script to trim the pdf that lilypond generate.
Can you write Python code for it?

import os

os.system('python auto_trim.py {} {}'.format('lily/'+ton+'.ly', 'temp.jpg'))
display.Image('img/temp.jpg', width=600)


I am working on the file "Genetic_Chords.ipynb".
The first task is:
In the main function is where the actual algorithm's running, it's a simple evolutionary algorithm with a hall of fame where the best individuals will be saved. The little while loop is just if you want to run multiple times the algorithm until an individual gets an evaluation lower than, in this case, 15.
Can you write Python code for it?

def main(ton):
    pop = toolbox.population(n=400)
    hof = tools.HallOfFame(3)
    stats = tools.Statistics(lambda ind: ind.fitness.values)
    stats.register('avg', numpy.mean)
    stats.register('std', numpy.std)
    stats.register('min', numpy.min)
    stats.register('max', numpy.max)

    pop, log = algorithms.eaSimple(pop, toolbox, cxpb=0.5, mutpb=0.3, ngen=70, stats=stats, halloffame=hof, verbose=True)
    while min(log.select('min')) > 15:
        pop = toolbox.population(n=400)
        pop, log = algorithms.eaSimple(pop, toolbox, cxpb=0.5, mutpb=0.3, ngen=70, stats=stats, halloffame=hof, verbose=True)

    for best in hof:
        print([x[0] for x in best], end='\n============\n')
    transform_lilypond(ton, [x[0] for x in hof[0]], make_file=True)


I am working on the file "Genetic_Chords.ipynb".
The first task is:
Initialization
In the initialization part, you have to create the initial population with all the individuals. In this example an individual will be a chord progression, with all the chords beeing of four notes each. I represented each individual as a list of chords, and each chord a list of notes, for example:
Can you write Python code for it?


I am working on the file "Genetic_Chords.ipynb".
The first task is:
Now, DEAP requires a generator to create the individuals, so I just create a simple generator that yields each chord of the progression.
Can you write Python code for it?

def newChordProg(ton, notes):
    """Create a new individual given the tonality and the base notes"""
    chords = selChord(ton, notes)
    for c in chords:
        yield c


I am working on the file "Genetic_Chords.ipynb".
The first task is:
The actual evaluation function:
Can you write Python code for it?

def evalNumErr(ton, individual):
    """Evaluation function."""
    res = 0
    for prev, item, nex in neighborhood(individual):
        res += check_interval(item[0])
        if prev == None:
            if item[1] != 0:
                res += 6
            continue
        else:
            if prev[1] in [4, 6] and item[1] in [3, 1]:
                res += 20
            res += check_2_chords(prev[0], item[0])
        if nex == None:
            if item[1] in [1, 2, 3, 4, 5, 6]:
                res += 6
    return (res,)


I am working on the file "Genetic_Chords.ipynb".
The first task is:
The function returns a tuple with the tonality represented as a number from 0 to 11 (C to B) and a 'M' if the tonality is major or 'm' if it's minor.
Then I wrote a function to create a single chord given the name of the chord and the fundamental note. The interesting part of this is that a chord of four notes can have multiple permutations, different set of notes and it can have a close or open harmony.
For example all these chords are D major and have the same bass:
Can you write Python code for it?

display.Image('img/all_same.jpg', width=400)

def creatChord(nameC, noteF):
    """Create one chord given the name of the chord and the fundamental note"""
    num_funda = int(nameC[:-1])
    if nameC[-1] == 'M':
        val_notes = [num_funda, (num_funda+4)%12, (num_funda+7)%12]
    elif nameC[-1] == 'm':
        val_notes = [num_funda, (num_funda+3)%12, (num_funda+7)%12]
    elif nameC[-1] == 'd':
        val_notes = [num_funda, (num_funda+3)%12, (num_funda+6)%12]

    # Tessitura of each voice
    tenorR = list(range(48, 69))
    contR = list(range(52, 77))
    sopR = list(range(60, 86))

    # Depending in the bass note this are the options for the others voices
    if noteF%12 == val_notes[0]:
        opc = [[1,1,1], [2,1,0], [0,1,2]]
    elif noteF%12 == val_notes[1]:
        opc = [[1,0,2], [3,0,0], [2,0,1]]
    elif noteF%12 == val_notes[2]:
        opc = [[1,1,1], [2,1,0]]

    opc = random.choice(opc)
    chordN = list()
    for num, val in zip(opc, val_notes):
        chordN += [val]*num

    random.shuffle(chordN)

    chord = [noteF,]
    for nte, voce in zip(chordN, [tenorR, contR, sopR]):
        posible_n = [x for x in voce if x%12 == nte]
        chord.append(random.choice(posible_n))

    return chord


I am working on the file "Genetic_Chords.ipynb".
The first task is:
The next function is just to create a lilypond file using the chords in the individual to see a nice sheet.
Can you write Python code for it?

def transform_lilypond(ton, indiv, make_file=False):
    """Take one list of chords and print the it in lilypond notation."""
    note_map = dict()
    if ton[-1] == 'M':
        note_map = {0: 'c',
                    1: 'cis',
                    2: 'd',
                    3: 'dis',
                    4: 'e',
                    5: 'f',
                    6: 'fis',
                    7: 'g',
                    8: 'gis',
                    9: 'a',
                    10:'ais',
                    11:'b'
                    }
        gra = 'major'
    else:
        note_map = {0: 'c',
                    1: 'des',
                    2: 'd',
                    3: 'ees',
                    4: 'e',
                    5: 'f',
                    6: 'ges',
                    7: 'g',
                    8: 'aes',
                    9: 'a',
                    10:'bes',
                    11:'b'
                    }
        gra = 'minor'
    voces = [[], [], [], []]

    for chord in indiv:
        for note, voce in zip(chord, voces):

            octave = (note // 12)-4
            name_lily = note_map[note % 12]
            if octave < 0:
                name_lily += ',' * (octave * -1)
            elif octave > 0:
                name_lily += "'" * octave
            voce.append(name_lily)

    if make_file:
        with open('lily/'+ton+'.ly', 'w') as f:
            key_map = {'0': 'c',
                    '1': 'des',
                    '2': 'd',
                    '3': 'ees',
                    '4': 'e',
                    '5': 'f',
                    '6': 'ges',
                    '7': 'g',
                    '8': 'aes',
                    '9': 'a',
                    '10':'bes',
                    '11':'b'
                    }
            print(ton)
            f.write(Template(TEMPLATE).substitute(key=key_map[ton[:-1]], grade=gra, notes='{}|\n{}|\n{}|\n{}|\n'.format(*(' '.join(voce) for voce in reversed(voces)))))

    print('{}|\n{}|\n{}|\n{}|\n'.format(*('...


I am working on the file "Genetic_Chords.ipynb".
The first task is:
This individual would be represented as
[[48, 64, 67, 79], [53, 59, 65, 74], [55, 59, 62, 74], [48, 55, 67, 76]]
To do anything I need the tonality of the exercise, so I defined a function to find it given the bass voice and the key signature, it just looks at the key signature and it looks if the voice has any sight of being a minor key:
Can you write Python code for it?

def setTon(line):
    """Return the tonality of the exercise and the bass notes of it"""
    ton = line[:2]
    notes = list(map(int, line[3:].split(' ')))
    if ton[1] == '#':
        ton = (int(ton[0])*7)%12
    else:
        ton = (int(ton[0])*5)%12
    for note in notes:
        if (ton+6)%12 == note%12:
            ton = str((ton-3)%12)+'m'
            break
    else:
        if ton-3 == notes[-1]%12:
            ton = str((ton-3)%12)+'m'
        else:
            ton = str(ton)+'M'
    return ton, notes


I am working on the file "Genetic_Chords.ipynb".
The first task is:
Will be rempresented as: 0# 57 59 60 62
Lilypond
Going from music notation to this MIDI number representation with just one voice is quite easy (although I might work a hack for that also), but once I generate all the notes that form the chords the process ends up being a pain in the ass. Therefore, at the end I take the output and pass it through a little script that transform the numbers to Lilypond, generate a .ly file and an .jpg of the sheet.
The genetic algorithm
A genetic algorithm works pretty much like a species evolution in real life. You take a group of individuals, called population, in this population there are a few individuals that have the best attributes, those are the ones that will survive and carry on the genes to the next generation. This process continues generation over generation until there is an individual with the perfect attributes, or at least with the best so far. You can read more in Wikipedia.
The process can be structured in this steps:
- Initialization
- Selection
- Crossover
- Mutation
- Termination
Let's tackle them one by one. But first let me explain the framework that I used.
Working enviroment
I used a library called DEAP, Distributed Evolutionary Algorithms in Python, it is a novel evolutionary computation framework for rapid prototyping and testing of ideas. It has all the basic tools to work with genetic algorithms you only have to create the functions to create, select, mate and mutate the individuals.
Can you write Python code for it?

import random
import math
import numpy
from string import Template

from deap import base
from deap import creator
from deap import tools
from deap import algorithms

from lily_template import TEMPLATE


I am working on the file "Gensim - Basics.ipynb".
The first task is:
結論
HM的方法最慢，用np.dot及np.linalg.norm組合的最快
日期時間的格式
Can you write Python code for it?

from datetime import datetime as dt

start = dt.now()
start.date(), start.time(), start

dt.now() - start


I am working on the file "Gensim - Corpora and Vector.ipynb".
The first task is:
Corpus Formats

Market Matrix format
儲存: corpora.MmCorpus.serialize('corpus.mm', myCorpus)
載入: myCorpus = corpora.MmCorpus('corpus.mm')

Joachim's SVMlight format
儲存: corpora.SvmLightCorpus.serialize('corpus.svml', myCorpus)
載入: myCorpus = corpora.SvmLightCorpus('corpus.svml')

Blei's LDA-C format
儲存: corpora.BleiCorpus.serialize('corpus.ldac', myCorpus)
載入: myCorpus = corpora.BleiCorpus('corpus.ldac')

GibbsLDA++ format
儲存: corpora.LowCorpus.serialize('corpus.low', myCorpus)
載入: myCorpus = corpora.LowCorpus('corpus.low')

corpus在使用時，也可以用streaming的方式:
for doc in corpus:
    print doc
Compatibility with NumPy and SciPy
Can you write Python code for it?

import gensim
# term-document matrix
numpy_matrix = np.array([[1,2,3,4],[2,4,5,6]])
corpus = gensim.matutils.Dense2Corpus(numpy_matrix)

numpy_matrix = gensim.matutils.corpus2dense(corpus, num_terms=3)
numpy_matrix

from scipy import sparse

corpus = gensim.matutils.Sparse2Corpus(sparse.rand(5, 3))
scipy_matrix = gensim.matutils.corpus2csc(corpus, 5)
scipy_matrix


I am working on the file "Gensim - Corpora and Vector.ipynb".
The first task is:
Corpus Streaming
在上面的作法中，整個corpus都放在記憶體裡。如果corpus太大，一種可行的作法是一次讀一部分，這時可以自定義corpus class
Can you write Python code for it?

class MyCorpus(object):
    def __iter__(self):
        for line in open('mycorpus.txt'):
            # 假設每一行代表一個document，而且tokens以空白字元分隔
            yield dict.doc2bow(line.lower().split())
            # 這裡用yield會變成一個generator，因此可以將物件放在for裡面
            # 例如 for line in myCorpus:

myCorpus = MyCorpus()
myCorpus

for vec in myCorpus:
    print vec


I am working on the file "Gensim - Corpora and Vector.ipynb".
The first task is:
Introduction
在gensim中，最基本的representation是document-term的組合，例如corpus[0]代表第一個document內所包含的term及出現次數。
Can you write Python code for it?


I am working on the file "Gensim - Corpora and Vector.ipynb".
The first task is:
同樣的，建立dictionary時，也可以用這樣的技巧避免載入檔案到記憶體。
Can you write Python code for it?

dict2 = corpora.Dictionary(line.lower().split() for line in open("mycorpus.txt"))
print dict2

stopid = [dict2.token2id[s] for s in stoplist]
onceid = [id for id, freq in dict2.dfs.iteritems() if freq == 1]
dict2.filter_tokens(stopid + onceid)
dict2.compactify()
print dict2


I am working on the file "Gensim - Corpora and Vector.ipynb".
The first task is:
如果要取得原本corpus的vector，需要用tfidf model轉換一次。
下面將轉換的結果產生一個similarity matrix:
Can you write Python code for it?

index = similarities.SparseMatrixSimilarity(tfidf[corpus], num_features=12)

sims = index[tfidf[(0, 2.0), (2, 3.0)]]

sims


I am working on the file "Gensim - Corpora and Vector.ipynb".
The first task is:
這個結果表示(0,2)-(2,3)的document，與文件[0]具有80%的相似度，和文件[1]具有36%、文件[3]具有27%的相似度。
Corpora and Vector Spaces
Can you write Python code for it?

documents = ["Human machine interface for lab abc computer applications",
             "A survey of user opinion of computer system response time",
             "The EPS user interface management system",
             "System and human system engineering testing of EPS",
             "Relation of user perceived response time to error measurement",
             "The generation of random binary unordered trees",
             "The intersection graph of paths in trees",
             "Graph minors IV Widths of trees and well quasi ordering",
             "Graph minors A survey"]

stoplist = set('for a of the and to in'.split())
texts = [[word for word in document.lower().split() if word not in stoplist] for document in documents]

import itertools
cnt = nltk.FreqDist(list(itertools.chain.from_iterable(texts)))

ntexts = [[term for term in doc if cnt[term] > 1] for doc in texts]
ntexts

dict = corpora.Dictionary(ntexts)
# dict.save('corpus.dict')  # 存到檔案中

dict.token2id

dict.doc2bow("Human computer interaction".lower().split())

corpus = [dict.doc2bow(text) for text in texts]
# corpora.MmCorpus.serialize('corpus.mm', corpus)  # 存到檔案中
corpus


I am working on the file "Gensim - Topics and Transformations.ipynb".
The first task is:
Save and Load
Can you write Python code for it?

tfidf.save('model.tfidf')
tfidf = models.TfidfModel.load('model.tfidf')

lsi.save('model.lsi')
lsi = models.LsiModel.load('model.lsi')


I am working on the file "Gensim - Topics and Transformations.ipynb".
The first task is:
Transformation
一般有兩個步驟

第一步，利用traning corpus產生一個model
第二步，使用model轉換corpus或未來的test data
Can you write Python code for it?

tfidf = models.TfidfModel(corpus)  # 第一步，產生model

tfidf[corpus[0]]  # 第二步，轉換vector


I am working on the file "Gensim - Topics and Transformations.ipynb".
The first task is:
注意: 呼叫model[corpus]是即時運算，結果不會保存，所以每次所需的時間都相同。如果轉換很花時間，你需要將轉換的結果存到硬碟再用streaming讀取。
不同轉換之間可以串接，例如先作TF-IDF再作LSI。
Can you write Python code for it?

tfidf = models.TfidfModel(corpus)
corpus_tfidf = tfidf[corpus]

lsi = models.LsiModel(corpus_tfidf, num_topics=200, id2word=dictb)
corpus_lsi = lsi[corpus_tfidf]

lsi.print_topics(5, 8)  # 列出前5名的topics


I am working on the file "Gensim - Word2Vec.ipynb".
The first task is:
Build Doc2Vec Model
Doc2Vec的參數

documents=None: 用來訓練的document，可以是list of TaggedDocument，或TaggedDocument generator
size=300: vector的維度
alpha=0.025: 初始的學習速度
window=8: context window的大小
min_count=5: 出現次數小於min_count的單字直接忽略
max_vocab_size=None: 限制vocabulary的大小，如果單字太多，就忽略最少見的單字，預設為無限制
sample=0: subsampling，隨機刪除機率小於sample的單字，兼具擴大context windows與減少stopword的功能
seed=1: 隨機產生器的random seed
workers=1: 在多核心的系統上，要用幾個核心來train
min_alpha=0.0001: 學習速度最後收斂的最小值
hs=1: 1表示用hierarchical soft-max，0表示用negative sampling
negative=0: 表示使用幾組negative sample來訓練
dbow_words=0: 1表示同時訓練出word-vector(用skip-gram)及doc-vector(用DBOW)，0表示只訓練doc-vector
dm=1: 1表示用distributed memory(PV-DM)來訓練，0表示用distributed bag-of-word(PV-DBOW)來訓練
dm_concat=0: 1表示不要sum/average而用concatenation of context vectors，0表示用sum/average。使用concatenation會產生較大的model，而且輸入的vector長度會變長
dm_mean=0: 在使用DBOW而且dm_concat=0的前提下，0表示使用sum作為hidden layer，1表示使用mean作為hidden layer
dm_tag_count=1: 當dm_concat=1時，預期每個document有幾個document tags
trim_rule=None: None表示小於min_count的單字會被忽略，也可以指定一個function(word, count, min_count)，這個function的傳回值有三種，util.RULE_DISCARD、util.RULE_KEEP、util.RULE_DEFAULT。這個參數會影響dictionary的生成
Can you write Python code for it?

from gensim.models.doc2vec import Doc2Vec, TaggedDocument

class PatentDocGenerator(object):
    def __init__(self, filename):
        self.filename = filename

    def __iter__(self):
        f = codecs.open(self.filename, 'r', 'UTF-8')
        for line in f:
            text, appnum = docs_out(line)
            yield TaggedDocument(text.split(), appnum.split())

doc = PatentDocGenerator('/share/USPatentData/tokenized_appDate_2013/2013USPTOPatents_by_skip_1.txt.tokenized')
%timeit model_d2v = Doc2Vec(doc, size=200, window=8, sample=1e-5, hs=0, negative=5)

doc = PatentDocGenerator('/share/USPatentData/tokenized_appDate_2013/2013USPTOPatents_by_skip_1.txt.tokenized')
model_d2v = Doc2Vec(doc, size=200, window=8, sample=1e-5, hs=0, negative=5)

model_d2v.docvecs.most_similar(['20140187118'])

m = Doc2Vec(size=200, window=8, sample=1e-5, hs=0, negative=5)

m.build_vocab(doc)

m.train(doc)

m.docvecs.most_similar(['20140187118'])


I am working on the file "Gensim - Word2Vec.ipynb".
The first task is:
Build Doc2Vec Model from 2013 USPTO Patents
Can you write Python code for it?

from gensim.models.doc2vec import Doc2Vec, TaggedDocument

class PatentDocGenerator(object):
    def __init__(self, filename):
        self.filename = filename

    def __iter__(self):
        f = codecs.open(self.filename, 'r', 'UTF-8')
        for line in f:
            text, appnum = docs_out(line)
            yield TaggedDocument(text.split(), appnum.split())

model_d2v = Doc2Vec(size=200, window=8, sample=1e-5, hs=0, negative=5)
root = '/share/USPatentData/tokenized_appDate_2013/'

for fn in sorted(listdir(root)):
    doc = PatentDocGenerator(os.path.join(root, fn))
    start = dt.now()
    model_d2v.build_vocab(doc)
    model_d2v.train(doc)
    logging.info('{} training time: {}'.format(fn, str(dt.now() - start)))

model_d2v.save("doc2vec_uspto_2013.model")


I am working on the file "Gensim - Word2Vec.ipynb".
The first task is:
Build LSI Model
Can you write Python code for it?

model_tfidf = models.TfidfModel(corpus)
corpus_tfidf = model_tfidf[corpus]


I am working on the file "Gensim - Word2Vec.ipynb".
The first task is:
Build Word2Vec Model
Word2Vec的參數

sentences: 用來訓練的list of list of words，但不是必要的，因為可以先建好model，再慢慢丟資料訓練
size=100: vector的維度
alpha=0.025: 初始的學習速度
window=5: context window的大小
min_count=5: 出現次數小於min_count的單字直接忽略
max_vocab_size: 限制vocabulary的大小，如果單字太多，就忽略最少見的單字，預設為無限制
sample=0.001: subsampling，隨機刪除機率小於0.001的單字，兼具擴大context windows與減少stopword的功能
seed=1: 隨機產生器的random seed
workers=3: 在多核心的系統上，要用幾個核心來train
min_alpha=0.0001: 學習速度最後收斂的最小值
sg=0: 0表示用CBOW，1表示用skip-gram
hs=0: 1表示用hierarchical soft-max，0表示用negative sampling
negative=5: 表示使用幾組negative sample來訓練
cbow_mean=1: 在使用CBOW的前提下，0表示使用sum作為hidden layer，1表示使用mean作為hidden layer
hashfxn=<build-in hash function>: 隨機初始化weights使用的hash function
iter=5: 整個corpus要訓練幾次
trim_rule: None表示小於min_count的單字會被忽略，也可以指定一個function(word, count, min_count)，這個function的傳回值有三種，util.RULE_DISCARD、util.RULE_KEEP、util.RULE_DEFAULT。這個參數會影響dictionary的生成
sorted_vocab=1: 1表示在指定word index前，先按照頻率將單字排序
batch_words=10000: 要傳給worker的單字長度

訓練方法
先產生一個空的model
model_w2v = models.Word2Vec(size=200, sg=1)
傳入一個list of words更新vocabulary
sent = [['first','sent'], ['second','sent']]
model_w2v.build_vocab(sent)
傳入一個list of words更新model
model_w2v.train(sent)
Can you write Python code for it?

all_text = [doc.split() for doc in documents]

model_w2v = models.Word2Vec(size=200, sg=1)

%timeit model_w2v.build_vocab(all_text)

%timeit model_w2v.train(all_text)

model_w2v.most_similar_cosmul(['deep','learning'])


I am working on the file "GeoInfoProcessing.ipynb".
The first task is:
Metadata
Can you write Python code for it?

# def of record fields
sf.fields

# number of shape records
len(shape_records)


I am working on the file "GeoInfoProcessing.ipynb".
The first task is:
OSGEO Unions - Doesn't work
Python GDAL/OGR Cookbook!
From lib osgeo doc: Calculate union between two Geometries
Can you write Python code for it?

# from osgeo import ogr

# dataSource = ogr.Open("cb_2015_us_zcta510_500k/cb_2015_us_zcta510_500k.shp")
# layer = dataSource.GetLayer()
# print(layer.GetGeomType()) # 3 means polygons

# print(layer.GetFeatureCount())

# geometries = []
# for feature in layer:
#     geom = feature.GetGeometryRef()
#     geometries.append(geom)

# geo0 = geometries[0]
# geo1 = geometries[1]
# union_poly = geo0.Union(geo1)


I am working on the file "GeoInfoProcessing.ipynb".
The first task is:
Reading shape & records
Can you write Python code for it?

shape_records = {}
for shape_record in sf.shapeRecords():
    zip_code = shape_record.record[0]
    shape_records[zip_code] = shape_record


I am working on the file "Geolocalisation.ipynb".
The first task is:
Localisation d' adresse quelconque.
Créer un programme qui renvoie la géolocalisation d'un point dont l'adresse est saisie par l'utilisateur.
Can you write Python code for it?

import geopy
from geopy.geocoders import Nominatim
geocoder = Nominatim()
adresse = input("saissez l'adresse du lieu, la ville, le pays ")
location = geocoder.geocode(adresse, True, 30)
print("Les coordonnées du point situé au ",adresse,"sont : ")
print("longitude = ",location.longitude,"°","latitude = ",location.latitude,"°")


I am working on the file "Geolocalisation.ipynb".
The first task is:
Saisie des résultats dans un fichier csv : Etape 1
Cette approche vise à illustrer la gestion simple des données.
Dans un premier temps on crée le fichier et on saisi les entêtes de colonnes.
Il ne faut exécuter ce programme qu'une fois sinon le fichier est écrasé et remplacé.
Can you write Python code for it?

import csv
OuvrirEnEcriture = open("Lieux.csv","w")
fname = "Lieux.csv"
writer = csv.writer(OuvrirEnEcriture)
writer.writerow( ("ID","adresse","ville","pays","longitude","latitude") )
OuvrirEnEcriture.close()


I am working on the file "Geolocalisation.ipynb".
The first task is:
Saisie des résultats dans un fichier csv : Etape 2
On va maintenant rentrer dans le fichier des lignes correspondant à la saisie des données.
Le fichier est ouvert avec l'argument "a" pour ajouter
Can you write Python code for it?

import csv
import geopy
from geopy.geocoders import Nominatim
fichier="Lieux.csv"
OuvrirEnEcritureAjout = open("Lieux.csv","a")
writer = csv.writer(OuvrirEnEcritureAjout)
geocoder = Nominatim()
adresse = input("saissez l'adresse du lieu : ")
ville = input("saisissez le nom de la ville : ")
pays = input("saisissez le pays : ")
requete = adresse +","+ville+","+ pays
ID = int(input("indiquez l'id du lieu"))
#print(requete)
location = geocoder.geocode(requete, True, 30)
print("Les coordonnées du point situé au ",adresse," à ",ville," en ",pays,"sont : ")
print("longitude = ",location.longitude,"°","latitude = ",location.latitude,"°")
writer.writerow((ID,adresse,ville,pays,location.longitude,location.latitude))
OuvrirEnEcritureAjout.close()


I am working on the file "Geolocalisation.ipynb".
The first task is:
Saisie des résultats dans une base de données sqlite : étape 2
Il reste maintenant à remplir la base de données.
Can you write Python code for it?

import sqlite3
import geopy
from geopy.geocoders import Nominatim
geocoder = Nominatim()
#On ouvre la base de données
conn = sqlite3.connect("Lieux.db")
cursor = conn.cursor()
# On saisit les éléments à stocker
adresse = input("saissez l'adresse du lieu : ")
ville = input("saisissez le nom de la ville : ")
pays = input("saisissez le pays : ")
requete = adresse +","+ville+","+ pays
ID = int(input("indiquez l'id du lieu : "))
# on cherche les coordonnées su lieu
location = geocoder.geocode(requete, True, 30)
# on publie le résultat
print("Les coordonnées du point situé au ",adresse," à ",ville," en ",pays,"sont : ")
print("longitude = ",location.longitude,"°","latitude = ",location.latitude,"°")
# On stocke les données dans la base
cursor.execute('''INSERT INTO lieux VALUES (?,?,?,?,?,?);''', (ID, adresse, ville, pays, location.longitude, location.latitude))
conn.commit()
cursor.execute('''SELECT * FROM lieux;''')
enregistrements = cursor.fetchall()
for lieux in enregistrements:
    print("Le lieu n %s est situé : %s , à , %s en %s à %s ° de longitude et %s ° de latitude" % (lieux[0], lieux[1], lieux[2], lieux[3], lieux[4], lieux[5]))
conn.close()


I am working on the file "Geolocalisation.ipynb".
The first task is:
Saisie des résultats dans une base de donées sqlite
Cette approche vise à illustrer la gestion des données dans une base de données dédiée.
On commence par créer la table d'attribut de la base de donnée.
On choisit les titres des entêtes de colonnes et on qualifie le type de donées à placer dedans.
Can you write Python code for it?

import sqlite3
conn = sqlite3.connect("Lieux.db")
cursor = conn.cursor()
cursor.execute("""CREATE TABLE IF NOT EXISTS lieux (
id INTEGER PRIMARY KEY AUTOINCREMENT UNIQUE,
adresse TEXT,
ville TEXT,
pays TEXT,
longitude FLOAT,
latitude FLOAT)
""")
conn.commit()


I am working on the file "Geolocalisation.ipynb".
The first task is:
Supprimer une table
Can you write Python code for it?

#import sqlite3
#cursor = conn.cursor()
#cursor.execute("""
#DROP TABLE lieux
#""")
#conn.commit()


I am working on the file "Geomodeller_to_MOOSE.ipynb".
The first task is:
And as a 3-D view:
Can you write Python code for it?

import ipyvolume
ipyvolume.quickvolshow(vox_3d)

# extract submodel for faster vis-test
nx_sub = nz
ny_sub = 80
nz_sub = nz

x = np.arange(nx_sub)
y = np.arange(ny_sub)
z = np.arange(nz_sub)
X,Y,Z = np.meshgrid(y,x,z)

# extract subvolume
vox_sub = vox_3d[:nx_sub, :ny_sub, :nz_sub]
vox_sub = np.swapaxes(vox_sub, 0, 2)

ipyvolume.quickvolshow(vox_sub)
# vox_sub.shape

np.unique(vox_sub)

# create colour array:
nc = np.max(vox_sub)
vox_c = [plt.cm.viridis(i/nc)[:3] for i in vox_sub.ravel()]

print(X.ravel().shape)
print(vox_sub.ravel().shape)

p3.figure()
p3.scatter(X.ravel(), Y.ravel(), Z.ravel(), color=vox_c, marker='box', size=3.4)

p3.show()


I am working on the file "Geomodeller_to_MOOSE.ipynb".
The first task is:
Some quick checks in 2-D slices:
Can you write Python code for it?

plt.figure(figsize=(12,4))
plt.imshow(vox_3d[:,-1,:].transpose(), origin='bottom')
plt.xlabel('X')
plt.ylabel('Z')
plt.show()

plt.figure(figsize=(12,4))
plt.imshow(vox_3d[0,:,:].transpose(), origin='bottom')
plt.xlabel('Y')
plt.ylabel('Z')
plt.show()

plt.figure(figsize=(12,4))
plt.imshow(vox_3d[:,:,0].transpose(), origin='bottom')
plt.xlabel('X')
plt.ylabel('Y')
plt.show()


I am working on the file "Get Email from Customers who have submitted Tickets.ipynb".
The first task is:
2) Print Emails (Channel: Voice)
Can you write Python code for it?

requesterArr = []
callNoEmail = 0

print '---------------------------------------------------------------------'
print 'START.'
print '---------------------------------------------------------------------'

try:
    for ticket in tickets:
        channel = ticket['via']['channel']
        ticketID = ticket['id']
        source = ticket['custom_fields'][9]['value'] #adjust custom field according to your ticket structure
        requesterID = ticket['requester_id']
        zdresult = zd.search(query='type:ticket user:'+ str(requesterID))
        email = zdresult['results'][0]['email']
        if (email is not None) and (channel =='voice'):
            requesterArr.append([ticketID,email,channel])
#             print 'ticket: ',  ticketID, ',',email , ',', channel
            print ("'%s'," % email)
        elif (channel =='voice'):
            callNoEmail+=1;

except ValueError:
    print("Error!")

print '---------------------------------------------------------------------'
print 'Length: ', len(requesterArr) , ', call but no email: ', callNoEmail, '.'
print 'DONE.'
print '---------------------------------------------------------------------'


I am working on the file "Get Email from Customers who have submitted Tickets.ipynb".
The first task is:
Import and save CX tickets in array.
Can you write Python code for it?

from zendesk import Zendesk
import datetime
import time
import json
import collections
import csv

### Zendesk credentials
zd = Zendesk(subdomain,
             email+'/token',
             token)

### Set the interval you're interested in
### today = datetime.datetime.today().date()
### weekStartDate = today
### weekEndDate = today

weekStartDate = (datetime.datetime.now() - datetime.timedelta(days = 34)).date()
weekEndDate = (datetime.datetime.now() - datetime.timedelta(days = 30)).date()

print '---------------------------------------------------------------------'
print 'START.'
print 'Week Start Date:', weekStartDate,'. Week End Date:', weekEndDate, '.'
print '---------------------------------------------------------------------'

### Build array containing tickets from given time intervall
tickets = []

try:
    pageNumber=1
    nextPage = True
    while nextPage:
        zdresult = zd.search(query='type:ticket created>='+ str(weekStartDate) +' created<=' + str(weekEndDate), page=pageNumber)
        print 'Zendesk Result Page Number: ', pageNumber
        print zdresult['next_page']
        for result in zdresult['results']:
            if result['custom_fields'][1]['value'] == 'customer_service': #adjust custom field according to your ticket structure
                tickets.append(result)
        if zdresult['next_page'] is None:
            nextPage=False
        else:
            nextPage=True
            pageNumber += 1
except ValueError:
    print(ValueError)

print '---------------------------------------------------------------------'
print 'Tickets array length:', len(tickets) , '.'
print 'DONE.'
print '---------------------------------------------------------------------'


I am working on the file "Get Email from Customers who have submitted Tickets.ipynb".
The first task is:
Use Print method below that suits best:
1) Print Emails (All Channels)
Can you write Python code for it?

requesterArr = []
noEmail = 0

print '---------------------------------------------------------------------'
print 'START.'
print '---------------------------------------------------------------------'

try:
    for ticket in tickets:
        ticketID = ticket['id']
        source = ticket['custom_fields'][9]['value'] #adjust custom field according to your ticket structure
        requesterID = ticket['requester_id']
        zdresult = zd.search(query='type:ticket user:'+ str(requesterID))
        email = zdresult['results'][0]['email']
        if (email is not None):
            requesterArr.append([ticketID,email,channel])
#             print 'ticket: ',  ticketID, ',',email , ',', channel
            print ("'%s'," % email)
        elif:
            noEmail+=1;

except ValueError:
    print("Error!")

print '---------------------------------------------------------------------'
print 'Length: ', len(requesterArr) , ', no email: ', noEmail , '.'
print 'DONE.'
print '---------------------------------------------------------------------'


I am working on the file "Get fin data for LSTM.ipynb".
The first task is:
GOOGLEINDEX_US:ADVERT
This Google Advertising & Marketing Index tracks searches related to (marketing, advertising, commercials...)

GOOGLEINDEX_US:AIRTVL
This Google Air Travel Index tracks searches related to (airlines, airline, flights...)

GOOGLEINDEX_US:AUTOBY
This Google Auto Buyers Index tracks searches related to (car, blue book...)

GOOGLEINDEX_US:AUTOFI
This Google Auto Financing Index tracks searches related to (loan, car lean...)

GOOGLEINDEX_US:AUTO
This Google Automotive Index tracks searches related to (car, honda, toyota...)

GOOGLEINDEX_US:PERFIN
This Google Personal Finance Index tracks searches related to (bank, bank of america...)

GOOGLEINDEX_US:BIZ
This Google Business Index tracks searches related to (calendar, business, salary...)

GOOGLEINDEX_US:COMPUT
This Google Computers & Electronics Index tracks searches related to (windows, download, microsoft...)

GOOGLEINDEX_US:CONSTR
This Google Construction Index tracks searches related to (doors, construction...)

GOOGLEINDEX_US:CREDIT
This Google Credit & Lending Index tracks searches related to (credit, credit card, loans...)

GOOGLEINDEX_US:DURBLE
This Google Durable Goods Index tracks searches related to (vacuum, appliances, refrigerator...)

GOOGLEINDEX_US:FININS
This Google Finance & Insurance Index tracks searches related to (stock, bank, yahoo...)

GOOGLEINDEX_US:FURNTR
This Google Furniture Index tracks searches related to (ikea, furniture, table...)

GOOGLEINDEX_US:INDUST
This Google Industries Index tracks searches related to (home, energy, usps...)

GOOGLEINDEX_US:INVEST
This Google Investing Index tracks searches related to (finance, stocks, capital, stock...)

GOOGLEINDEX_US:JOBS
This Google Jobs Index tracks searches related to (city, jobs, resume, monster...)

GOOGLEINDEX_US:LUXURY
This Google Luxuries Index tracks searches related to (diamond, jewelers, rings, tiffany...)

GOOGLEINDEX_US:MTGE
This Google Mortgage Index tracks searches related to (mortgage calculator, mortage rates,...
Can you write Python code for it?

gt = pd.read_table?

gt = pd.read_clipboard(index_col)

gt