I am working on the file "Cdiscount/Part3-AIF-PythonKeras-TextGeneration-Cdiscount-Copy1.ipynb".
The first task is:
Q Pourquoi est-ce la categorical_crossentropy qui est utilisée comme fonction de perte?
Génération de Texte
La celulle suivante permet de générer une description produit :
Can you write Python code for it?

x_pred = np.zeros((1, Nd+1, Nv))
print("step 0")
x_pred[0,0,I_START] =1
x_pred_str = decode_sequence(x_pred[0], int_to_char)
print(x_pred_str)

for i in range(Nd):
    ix = np.argmax(model.predict(x_pred[:,:i+1,:])[0][-1,:])
    x_pred[0,i+1,ix] = 1
x_pred_str=decode_sequence(x_pred[0], int_to_char)
print(x_pred_str)


I am working on the file "Cdiscount/Part3-AIF-PythonKeras-TextGeneration-Cdiscount-Copy1.ipynb".
The first task is:
Téléchargement des données
La Catégorie de Niveau 3 COQUE - BUMPER - FACADE TELEPHONE est la catégorie le plus représenté du jeu de données originale Cdiscount avec  2.184.671 déscriptions présentent.  Parmis ces descriptions, 1.761.637 sont composés d'exactement 197 caractères.
Nous allons nous servir de ces lignes (ou un sous ensemble de ces lignes, en fonction de la puissance de calcul disponible sur votre machine) pour apprendre un modèle de génération de texte qui permettra de généré automatiquement une nouvelle description de ce type de produit.
Can you write Python code for it?

N = 100000
DATA_DIR = ""
X = np.load(DATA_DIR+"data/description_coque.npy")[:N]
print(X.shape)
print(X[:3])


I am working on the file "CellPhoneStory/cell_fund_analysis.ipynb".
The first task is:
Add the Iowa e911 fund information
Can you write Python code for it?

column_names = ['date', 'ia_balance', 'ia_surcharge']
column_types = [agate.Date(), agate.Number(), agate.Number()]
iowa_phones = agate.Table.from_csv('e911_fund_iowa.csv', column_names, column_types)

print(iowa_phones)


I am working on the file "CellPhoneStory/cell_fund_analysis.ipynb".
The first task is:
Calculate the number of Nebraska cell phones by dividing the e911 fund by the surcharge amount
Can you write Python code for it?

def calculateNebCellPhones(row):
    return row['neb_balance'] / row['neb_surcharge']

cell_phones = funds.compute([
        ('cell_phones', agate.Formula(agate.Number(), calculateNebCellPhones))
    ])


I am working on the file "CellPhoneStory/cell_fund_analysis.ipynb".
The first task is:
I wanted to calculate some stats by just grabing the amounts from the table
Can you write Python code for it?

ia_phones_2015 = total_cell_phones.columns['ia_cell_phones_round'][0]
ia_phones_2006 = total_cell_phones.columns['ia_cell_phones_round'][9]
ia_pc_change = ( ia_phones_2015 - ia_phones_2006)/ia_phones_2006

print("Iowa percent change in cell phones for 2015, {}, to 2006 {} is {}".format(ia_phones_2015, ia_phones_2006, ia_pc_change))

neb_phones_2015 = total_cell_phones.columns['cell_phones_round'][0]
neb_phones_2006 = total_cell_phones.columns['cell_phones_round'][9]
neb_pc_change = (neb_phones_2015 - neb_phones_2006)/neb_phones_2006

print("Neb percent change in cell phones for 2015, {}, to 2006 {} is {}".format(neb_phones_2015, neb_phones_2006, neb_pc_change))

neb_phones_2001 = total_cell_phones.columns['cell_phones_round'][14]
neb_pc_change_total = (neb_phones_2015 - neb_phones_2001)/neb_phones_2001

print("Neb percent change in cell phones for 2015, {}, to 2001 {} is {}".format(total_cell_phones.columns['cell_phones_round'][0], total_cell_phones.columns['cell_phones_round'][14], neb_pc_change_total))


I am working on the file "CellPhoneStory/cell_fund_analysis.ipynb".
The first task is:
Join together nebraska and iowa tables
Can you write Python code for it?

total_cell_phones = rounded_cell_phones.join(rounded_cell_phones_ia, 'date', 'date')

total_cell_phones.print_table()


I am working on the file "CellPhoneStory/cell_fund_analysis.ipynb".
The first task is:
Let's see the nebraska e911 fund table
Can you write Python code for it?


I am working on the file "CellPhoneStory/cell_fund_analysis.ipynb".
The first task is:
Let's see the new table
Can you write Python code for it?


I am working on the file "Census.ipynb".
The first task is:
Stage 3: Transform 1 dimension to 2
The natural form of the data is age X gender.  We could split the table and reassmeble it,
but a better approach is to pivot.
Can you write Python code for it?

# Split it by gender into two tables
male = census.where('CAT','male')
female = census.where('CAT','female')

male

pop2010 = census.pivot('CAT','AGE','2010pop',sum)
pop2010


I am working on the file "Census.ipynb".
The first task is:
Visualization
Now we can easily visualize what's in this data.
Can you write Python code for it?

# The number of individuals by ages
pop2010.plot('AGE',overlay=True)

pop2010['diff'] = pop2010['male'] - pop2010['female']
pop2010.show()

# The difference in the genders by age
pop2010.select(['AGE', 'diff']).bar('AGE')

pop2010['Rel Diff'] = pop2010['diff'] / pop2010['all']
pop2010.set_format('Rel Diff', PercentFormatter)
pop2010.show()

pop2010.select(['AGE', 'Rel Diff']).bar('AGE')

pop2010['Ratio F/M'] = pop2010['female'] / pop2010['male']
pop2010.select(['AGE', 'Ratio F/M']).bar('AGE')


I am working on the file "Center of Gravity with JAX.ipynb".
The first task is:
Lets minimize with gradient descent
Can you write Python code for it?

#Initial locationl of depots/distribution centers
x0=np.array([[4.0,-84.0]])

print("Initial Cost: {:0.2f}".format(cost_function(x0 )  ))

xs, ys, fs = optimize(cost_function, gradient_funcion, x0, 'grad_desc', 100)

print("Final Cost: {:0.2f}".format(fs[-1]))


I am working on the file "Center of Gravity with JAX.ipynb".
The first task is:
Now lets define the procedure to apply gradient descent or newton nethod
Can you write Python code for it?

def optimize(funtion_opt, grad_fun, x_0, method, n_iter):
  '''Input:
      funtion_opt: Function  to minimize
     grad_fun: gradient of the function to minimize
     x_0: initial 2D coordiantes of depot/distribution center
     method: method to use for minimize
     n_iter: Number of iterations of the method
                               --------------
     Output:
     xs: List of x coordiantes for each iteration
     ys: List of y coordiantes for each iteration
     fs: List of costs for each iteration'''

  #Create empty lists to fill with iteration values
  xs  = []
  ys  = []
  fs   = []

  #Add the initial location
  xs.append(x_0[0,0])
  ys.append(x_0[0,1])
  fs.append(cost_function(x_0))

  for i in range(n_iter):

    if method == 'newton':
      loss_val = funtion_opt(x_0)
      loss_vec  = np.array([[loss_val, loss_val]])
      x_0 -= 0.005*loss_vec/grad_fun(x_0)
    elif method == 'grad_desc':
      step = 0.0001*grad_fun(x_0)
      x_0 -= step

    xs.append(x_0[0,0])
    ys.append(x_0[0,1])
    fs.append(cost_function(x_0))
  return xs, ys, fs


I am working on the file "Center of Gravity with JAX.ipynb".
The first task is:
Now lets plot the trayectory of the optimization procedure.
Can you write Python code for it?

from mpl_toolkits import mplot3d
import matplotlib.pyplot as plt

#We must modified how we feed the input to the cost function to plot values of x and y coordinates
def cost_function_2(x,y):
  dist            = distances(x,y)
  dist_costo = quantities*costs*dist
  return np.sum(dist_costo)

FIGSIZE = (9, 7)

xs = np.array(xs).reshape(-1,)
ys = np.array(ys).reshape(-1,)
fs = np.array(fs)

X, Y          = np2.meshgrid(np2.linspace(-5., 5., 50), np2.linspace(-84., -74., 50))
func_vec = np2.vectorize(cost_function_2)
f               = func_vec(X,Y)
indices    = (slice(None, None, 4), slice(None, None, 4))

fig = plt.figure(figsize=FIGSIZE)
ax = plt.axes(projection='3d', azim=10,elev=10)
ax.plot_surface(X, Y, f, shade=True, linewidth=2, antialiased=True,alpha=0.5)
ax.plot3D(xs, ys, fs, color='black', lw=4)


I am working on the file "Center of Gravity with JAX.ipynb".
The first task is:
With the defined function we can calculate the gradient with JAX
Can you write Python code for it?

gradient_funcion = jit(grad(cost_function)) #jit (just in time) compile makes faster the evaluation of the gradient.


I am working on the file "Centralities.ipynb".
The first task is:
Betweenness
Can you write Python code for it?

def betweenness_centralization(G):
    vnum = G.vcount()
    if vnum < 3:
        raise ValueError("graph must have at least three vertices")
    denom = (vnum-1)*(vnum-2)

    temparr = [2*i/denom for i in G.betweenness()]
    return temparr

start = timeit.default_timer()
igraph_betweenness = betweenness_centralization(igraph_fb)
stop = timeit.default_timer()
print 'It takes {} seconds to finish'.format(stop - start)

igraph_betweenness.sort(reverse=True)
print igraph_betweenness[:10]


I am working on the file "Centralities.ipynb".
The first task is:
Closeness
Can you write Python code for it?

start = timeit.default_timer()
v_closeness = closeness(graph_tool_fb)
stop = timeit.default_timer()
print 'It takes {} seconds to finish'.format(stop - start)

v_closeness.a[107]


I am working on the file "Centralities.ipynb".
The first task is:
Degree centrality might be the easiest number to calculate but it only shows the number of nodes connected which in real social network it might not be very useful as you might have a million followers but if the majority of them is bots then the number is not telling anything new.
Now let's try Betweenness which count all of the shortest path going throw each now. This might mean that if you have the highest shortest path going through you, you might be considered as bridge of your entire network.
Nodes with high betweenness are important in communication and information diffusion
We will be using multiprocessing so we can parallel the computation and distribute the load.
Can you write Python code for it?

from multiprocessing import Pool
import itertools

def partitions(nodes, n):
    "Partitions the nodes into n subsets"
    nodes_iter = iter(nodes)
    while True:
        partition = tuple(itertools.islice(nodes_iter,n))
        if not partition:
            return
        yield partition

def btwn_pool(G_tuple):
    return nx.betweenness_centrality_source(*G_tuple)

def between_parallel(G, processes = None):
    p = Pool(processes=processes)
    part_generator = 4*len(p._pool)
    node_partitions = list(partitions(G.nodes(), int(len(G)/part_generator)))
    num_partitions = len(node_partitions)

    bet_map = p.map(btwn_pool,
                        zip([G]*num_partitions,
                        [True]*num_partitions,
                        [None]*num_partitions,
                        node_partitions))

    bt_c = bet_map[0]
    for bt in bet_map[1:]:
        for n in bt:
            bt_c[n] += bt[n]
    return bt_c


I am working on the file "Centralities.ipynb".
The first task is:
Eigenvalue
Can you write Python code for it?

start = timeit.default_timer()
v_closeness = eigenvector(graph_tool_fb)
stop = timeit.default_timer()
print 'It takes {} seconds to finish'.format(stop - start)


I am working on the file "Centralities.ipynb".
The first task is:
Let's try with multiprocesser.
Can you write Python code for it?

start = timeit.default_timer()
bt = between_parallel(g_fb)
stop = timeit.default_timer()
top = 10

max_nodes =  sorted(bt.iteritems(), key = lambda v: -v[1])[:top]
bt_values = [5]*len(g_fb.nodes())
bt_colors = [0]*len(g_fb.nodes())
for max_key, max_val in max_nodes:
    bt_values[max_key] = 150
    bt_colors[max_key] = 2

print 'It takes {} seconds to finish'.format(stop - start)
print max_nodes


I am working on the file "Centralities.ipynb".
The first task is:
Networkx Independent Cascade Model
Can you write Python code for it?

import copy
import networkx as nx
import random

def independent_cascade(G, seeds, steps = 0):
    """
    "Return the active nodes of each diffusion step by the independent cascade
    model

    Parameters
    -- -- -- -- -- -
    G: graph
    A NetworkX graph
    seeds: list of nodes
        The seed nodes for diffusion
    steps: integer
        The number of steps to diffuse.If steps <= 0, the diffusion runs until
        no more nodes can be activated.If steps > 0, the diffusion runs for at
        most "steps" rounds

    Returns
    -- -- -- -
    layer_i_nodes: list of list of activated nodes
    layer_i_nodes[0]: the seeds
    layer_i_nodes[k]: the nodes activated at the kth diffusion step

    Notes
    -- -- -
    When node v in G becomes active, it has a * single * chance of activating
    each currently inactive neighbor w with probability p_ {
        vw
    }

    Examples
    -- -- -- --
    >>> DG = nx.DiGraph() >>> DG.add_edges_from([(1, 2), (1, 3), (1, 5), (2, 1), (3, 2), (4, 2), (4, 3), \ >>> (4, 6), (5, 3), (5, 4), (5, 6), (6, 4), (6, 5)], act_prob = 0.2) >>> H = nx.independent_cascade(DG, [6])

    References
    -- -- -- -- --[1] David Kempe, Jon Kleinberg, and Eva Tardos.
    Influential nodes in a diffusion model
    for social networks.
    In Automata, Languages and Programming, 2005.
    """
    if type(G) == nx.MultiGraph or type(G) == nx.MultiDiGraph:
        raise Exception(\
            "independent_cascade() is not defined for graphs with multiedges.")

    # make sure the seeds are in the graph
    for s in seeds:
        if s not in G.nodes():
            raise Exception("seed", s, "is not in graph")

    # change to directed graph
    if not G.is_directed():
        DG = G.to_directed()
    else:
        DG = copy.deepcopy(G)

    # init activation probabilities
    for e in DG.edges():
        if 'act_prob' not in DG[e[0]][e[1]]:
            DG[e[0]][e[1]]['act_prob'] = 0.1
        elif DG[e[0]][e[1]]['act_prob'] > 1:
            raise Exception("edge activation...


I am working on the file "Centralities.ipynb".
The first task is:
Node 107 has 1045 friends and we can divide that by number of nodes to get the normalized degree centrality
Can you write Python code for it?

float(nx.degree(g_fb, [107]).values()[0]) / g_fb.number_of_nodes()


I am working on the file "Centralities.ipynb".
The first task is:
Now let's find the celebrities. The most basic centrality is Degree Centrality which is the sum of all in and out nodes (in the case of directed graph).
Can you write Python code for it?

dg_centrality = nx.degree_centrality(g_fb)
sorted_dg_centrality = sorted(dg_centrality.items(), key=operator.itemgetter(1), reverse=True)
sorted_dg_centrality[:10]


I am working on the file "Centralities.ipynb".
The first task is:
Now we get quite a different result. This would mean that node 1912 is connected to more important people in the entire network that means that node is more influential than the rest of the network.
iGraph with SNAP Facebook Dataset
Networkx is easy to install and great to start with. However, as it's written in Python it's quite slow. I'm going to try iGraph which is C based. I'm hoping that this would yield the same result but faster.
Can you write Python code for it?

from igraph import *
import timeit

igraph_fb = Graph.Read_Edgelist('facebook_combined.txt', directed=False)

print igraph_fb.summary()


I am working on the file "Centralities.ipynb".
The first task is:
Page Rank
Can you write Python code for it?

start = timeit.default_timer()
v_closeness = pagerank(graph_tool_fb)
stop = timeit.default_timer()
print 'It takes {} seconds to finish'.format(stop - start)


I am working on the file "Centralities.ipynb".
The first task is:
Page rank
We're going to try PageRank algorithm. This is very similar to Google's PageRank which they use incoming links to determine the "popularity"
Can you write Python code for it?

g_fb_pr = nx.pagerank(g_fb)

top = 10
max_pagerank = sorted(g_fb_pr.iteritems(), key = lambda v: -v[1])[:top]
max_pagerank


I am working on the file "Centralities.ipynb".
The first task is:
PageRank
Can you write Python code for it?

start = timeit.default_timer()
igraph_pr = igraph_fb.pagerank()
stop = timeit.default_timer()
print 'It takes {} seconds to finish'.format(stop - start)

igraph_pr.sort(reverse=True)
print igraph_pr[:10]


I am working on the file "Centralities.ipynb".
The first task is:
We can see that iGraph yields similar result from networkx but it's a lot quicker in the same machine.
Graph_tool with SNAP Facebook Dataset
I'm going to try another library which is supposed to be the fastest than networkx and igraph. Graph_tool is also C based which it has OpenMP enabled so a lot of algorithms is multiprocessing.
Can you write Python code for it?

import sys

from graph_tool.all import *
import timeit

show_config()

graph_tool_fb = Graph(directed=False)

with open('facebook_combined.txt', 'r') as f:
    for line in f:
        edge_list = line.split()
        source, target = tuple(edge_list)
        graph_tool_fb.add_edge(source, target)

print graph_tool_fb.num_vertices()
print graph_tool_fb.num_edges()


I am working on the file "Centralities.ipynb".
The first task is:
We can see that now the score is different as node 3437 is more popular than node 107.
Who is a "Gray Cardinal"
There's another metric that we can measure most influential node. It's called eigenvector centrality. To put it simply it means that if you're well connected to a lot of important people that means you're important or most influential as well.
Can you write Python code for it?

g_fb_eg = nx.eigenvector_centrality(g_fb)

top = 10
max_eg = sorted(g_fb_eg.iteritems(), key = lambda v: -v[1])[:top]
max_eg


I am working on the file "Centralities.ipynb".
The first task is:
We can see that the node 107 has the highest degree centrality which means node 107 has the highest number of connected nodes. We can prove this by getting the degree of node 107 to see how many friends of node 107 has
Can you write Python code for it?


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
3.  For days before $\tau$, represent the user's received SMS count by sampling from $\text{Poi}(\lambda_1)$, and sample from  $\text{Poi}(\lambda_2)$ for days after $\tau$. For example:
Can you write Python code for it?

data = np.r_[pm.rpoisson(lambda_1, tau), pm.rpoisson(lambda_2, 80 - tau)]


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
4. Plot the artificial dataset:
Can you write Python code for it?

plt.bar(np.arange(80), data, color="#348ABD")
plt.bar(tau - 1, data[tau - 1], color="r", label="user behaviour changed")
plt.xlabel("Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Artificial dataset")
plt.xlim(0, 80)
plt.legend();


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
A Normal random variable can be take on any real number, but the variable is very likely to be relatively close to $\mu$. In fact, the expected value of a Normal is equal to its $\mu$ parameter:
$$ E[ X | \mu, \tau] = \mu$$
and its variance is equal to the inverse of $\tau$:
$$Var( X | \mu, \tau ) = \frac{1}{\tau}$$
Below we continue our modeling of the Challenger space craft:
Can you write Python code for it?

import pymc as pm

temperature = challenger_data[:, 0]
D = challenger_data[:, 1]  # defect or not?

# notice the`value` here. We explain why below.
beta = pm.Normal("beta", 0, 0.001, value=0)
alpha = pm.Normal("alpha", 0, 0.001, value=0)

@pm.deterministic
def p(t=temperature, alpha=alpha, beta=beta):
    return 1.0 / (1. + np.exp(beta * t + alpha))


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
Adding a constant term $\alpha$ amounts to shifting the curve left or right (hence why it is called a bias).
Let's start modeling this in PyMC. The $\beta, \alpha$ parameters have no reason to be positive, bounded or relatively large, so they are best modeled by a Normal random variable, introduced next.
Normal distributions
A Normal random variable, denoted $X \sim N(\mu, 1/\tau)$, has a distribution with two parameters: the mean, $\mu$, and the precision, $\tau$. Those familiar with the Normal distribution already have probably seen $\sigma^2$ instead of $\tau^{-1}$. They are in fact reciprocals of each other. The change was motivated by simpler mathematical analysis and is an artifact of older Bayesian methods. Just remember: the smaller $\tau$, the larger the spread of the distribution (i.e. we are more uncertain); the larger $\tau$, the tighter the distribution (i.e. we are more certain). Regardless, $\tau$ is always positive.
The probability density function of a $N( \mu, 1/\tau)$ random variable is:
$$ f(x | \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left( -\frac{\tau}{2} (x-\mu)^2 \right) $$
We plot some different density functions below.
Can you write Python code for it?

import scipy.stats as stats

nor = stats.norm
x = np.linspace(-8, 7, 150)
mu = (-2, 0, 3)
tau = (.7, 1, 2.8)
colors = ["#348ABD", "#A60628", "#7A68A6"]
parameters = zip(mu, tau, colors)

for _mu, _tau, _color in parameters:
    plt.plot(x, nor.pdf(x, _mu, scale=1. / np.sqrt(_tau)),
             label="$\mu = %d,\;\\tau = %.1f$" % (_mu, _tau), color=_color)
    plt.fill_between(x, nor.pdf(x, _mu, scale=1. / np.sqrt(_tau)), color=_color,
                     alpha=.33)

plt.legend(loc="upper right")
plt.xlabel("$x$")
plt.ylabel("density function at $x$")
plt.title("Probability distribution of three different Normal random \
variables");


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
Below we add all the variables of interest to a Model container and run our black-box algorithm over the model.
Can you write Python code for it?

model = pm.Model([yes_responses, p_skewed, p])

# To Be Explained in Chapter 3!
mcmc = pm.MCMC(model)
mcmc.sample(25000, 2500)

figsize(12.5, 3)
p_trace = mcmc.trace("freq_cheating")[:]
plt.hist(p_trace, histtype="stepfilled", normed=True, alpha=0.85, bins=30,
         label="posterior distribution", color="#348ABD")
plt.vlines([.05, .35], [0, 0], [5, 5], alpha=0.2)
plt.xlim(0, 1)
plt.legend();


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
But something is missing. In the plot of the logistic function, the probability changes only near zero, but in our data above the probability changes around 65 to 70. We need to add a bias term to our logistic function:
$$p(t) = \frac{1}{ 1 + e^{ \;\beta t + \alpha } } $$
Some plots are below, with differing $\alpha$.
Can you write Python code for it?

def logistic(x, beta, alpha=0):
    return 1.0 / (1.0 + np.exp(np.dot(beta, x) + alpha))

x = np.linspace(-4, 4, 100)

plt.plot(x, logistic(x, 1), label=r"$\beta = 1$", ls="--", lw=1)
plt.plot(x, logistic(x, 3), label=r"$\beta = 3$", ls="--", lw=1)
plt.plot(x, logistic(x, -5), label=r"$\beta = -5$", ls="--", lw=1)

plt.plot(x, logistic(x, 1, 1), label=r"$\beta = 1, \alpha = 1$",
         color="#348ABD")
plt.plot(x, logistic(x, 3, -2), label=r"$\beta = 3, \alpha = -2$",
         color="#A60628")
plt.plot(x, logistic(x, -5, 7), label=r"$\beta = -5, \alpha = 7$",
         color="#7A68A6")

plt.title("Logistic functon with bias, plotted for several value of $\\alpha$ bias parameter", fontsize=14)
plt.legend(loc="lower left");


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
Finally...
We wrap all the created variables into a pm.Model class. With this Model class, we can analyze the variables as a single unit. This is an optional step, as the fitting algorithms can be sent an array of the variables rather than a Model class. I may or may not use this class in future examples ;)
Can you write Python code for it?

model = pm.Model([obs, lambda_, lambda_1, lambda_2, tau])


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
If we carry out the algorithm, the next step that occurs is the first coin-flip each student makes. This can be modeled again by sampling 100 Bernoulli random variables with $p=1/2$: denote a 1 as a Heads and 0 a Tails.
Can you write Python code for it?

first_coin_flips = pm.Bernoulli("first_flips", 0.5, size=N)
print(first_coin_flips.value)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
It is okay that our fictional dataset does not look like our observed dataset: the probability is incredibly small it indeed would. PyMC's engine is designed to find good parameters, $\lambda_i, \tau$, that maximize this probability.
The ability to generate artificial datasets is an interesting side effect of our modeling, and we will see that this ability is a very important method of Bayesian inference. We produce a few more datasets below:
Can you write Python code for it?

def plot_artificial_sms_dataset():
    tau = pm.rdiscrete_uniform(0, 80)
    alpha = 1. / 20.
    lambda_1, lambda_2 = pm.rexponential(alpha, 2)
    data = np.r_[pm.rpoisson(lambda_1, tau), pm.rpoisson(lambda_2, 80 - tau)]
    plt.bar(np.arange(80), data, color="#348ABD")
    plt.bar(tau - 1, data[tau - 1], color="r", label="user behaviour changed")
    plt.xlim(0, 80)

figsize(12.5, 5)
plt.suptitle("More examples of artificial datasets", fontsize=14)
for i in range(1, 5):
    plt.subplot(4, 1, i)
    plot_artificial_sms_dataset()


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
Later we will see how we use this to make predictions and test the appropriateness of our models.
Example: Bayesian A/B testing
A/B testing is a statistical design pattern for determining the difference of effectiveness between two different treatments. For example, a pharmaceutical company is interested in the effectiveness of drug A vs drug B. The company will test drug A on some fraction of their trials, and drug B on the other fraction (this fraction is often 1/2, but we will relax this assumption). After performing enough trials, the in-house statisticians sift through the data to determine which drug yielded better results.
Similarly, front-end web developers are interested in which design of their website yields more sales or some other metric of interest. They will route some fraction of visitors to site A, and the other fraction to site B, and record if the visit yielded a sale or not. The data is recorded (in real-time), and analyzed afterwards.
Often, the post-experiment analysis is done using something called a hypothesis test like difference of means test or difference of proportions test. This involves often misunderstood quantities like a "Z-score" and even more confusing "p-values" (please don't ask). If you have taken a statistics course, you have probably been taught this technique (though not necessarily learned this technique). And if you were like me, you may have felt uncomfortable with their derivation -- good: the Bayesian approach to this problem is much more natural.
A Simple Case
As this is a hacker book, we'll continue with the web-dev example. For the moment, we will focus on the analysis of site A only. Assume that there is some true $0 \lt p_A \lt 1$ probability that users who, upon shown site A, eventually purchase from the site. This is the true effectiveness of site A. Currently, this quantity is unknown to us.
Suppose site A was shown to $N$ people, and $n$ people purchased from the site. One might conclude hastily that $p_A = \frac{n}{N}$. Unfortunately, the observed...
Can you write Python code for it?

import pymc as pm

# The parameters are the bounds of the Uniform.
p = pm.Uniform('p', lower=0, upper=1)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
Modeling approaches
A good starting point in Bayesian modeling is to think about how your data might have been generated. Put yourself in an omniscient position, and try to imagine how you would recreate the dataset.
In the last chapter we investigated text message data. We begin by asking how our observations may have been generated:

We started by thinking "what is the best random variable to describe this count data?" A Poisson random variable is a good candidate because it can represent count data. So we model the number of sms's received as sampled from a Poisson distribution.

Next, we think, "Ok, assuming sms's are Poisson-distributed, what do I need for the Poisson distribution?" Well, the Poisson distribution has a parameter $\lambda$.

Do we know $\lambda$? No. In fact, we have a suspicion that there are two $\lambda$ values, one for the earlier behaviour and one for the latter behaviour. We don't know when the behaviour switches though, but call the switchpoint $\tau$.

What is a good distribution for the two $\lambda$s? The exponential is good, as it assigns probabilities to positive real numbers. Well the exponential distribution has a parameter too, call it $\alpha$.

Do we know what the parameter $\alpha$ might be? No. At this point, we could continue and assign a distribution to $\alpha$, but it's better to stop once we reach a set level of ignorance: whereas we have a prior belief about $\lambda$, ("it probably changes over time", "it's likely between 10 and 30", etc.), we don't really have any strong beliefs about $\alpha$. So it's best to stop here.
What is a good value for $\alpha$ then? We think that the $\lambda$s are between 10-30, so if we set $\alpha$ really low (which corresponds to larger probability on high values) we are not reflecting our prior well. Similar, a too-high alpha misses our prior belief as well. A good idea for $\alpha$ as to reflect our belief is to set the value so that the mean of $\lambda$, given $\alpha$, is equal to our observed mean. This was shown in...
Can you write Python code for it?

tau = pm.rdiscrete_uniform(0, 80)
print(tau)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
More PyMC Tricks
Protip: Lighter deterministic variables with Lambda class
Sometimes writing a deterministic function using the @pm.deterministic decorator can seem like a chore, especially for a small function. I have already mentioned that elementary math operations can produce deterministic variables implicitly, but what about operations like indexing or slicing? Built-in Lambda functions can handle this with the elegance and simplicity required. For example,
beta = pm.Normal("coefficients", 0, size=(N, 1))
x = np.random.randn((N, 1))
linear_combination = pm.Lambda(lambda x=x, beta=beta: np.dot(x.T, beta))

Protip: Arrays of PyMC variables
There is no reason why we cannot store multiple heterogeneous PyMC variables in a Numpy array. Just remember to set the dtype of the array to object upon initialization. For example:
Can you write Python code for it?

N = 10
x = np.empty(N, dtype=object)
for i in range(0, N):
    x[i] = pm.Exponential('x_%i' % i, (i + 1) ** 2)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
Next we sort each column by the posterior probabilities:
Can you write Python code for it?

ix = np.argsort(posterior_probability)
print("probb | defect ")
for i in range(len(D)):
    print("%.2f  |   %d" % (posterior_probability[ix[i]], D[ix[i]]))


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
Notice that as a result of N_B < N_A, i.e. we have less data from site B, our posterior distribution of $p_B$ is fatter, implying we are less certain about the true value of $p_B$ than we are of $p_A$.
With respect to the posterior distribution of $\text{delta}$, we can see that the majority of the distribution is above $\text{delta}=0$, implying there site A's response is likely better than site B's response. The probability this inference is incorrect is easily computable:
Can you write Python code for it?

# Count the number of samples less than 0, i.e. the area under the curve
# before 0, represent the probability that site A is worse than site B.
print("Probability site A is WORSE than site B: %.3f" % \
    (delta_samples < 0).mean())

print("Probability site A is BETTER than site B: %.3f" % \
    (delta_samples > 0).mean())


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
Of course a child can  have more than one parent, and a parent can have many children.
PyMC Variables
All PyMC variables also expose a value attribute. This method produces the current (possibly random) internal value of the variable. If the variable is a child variable, its value changes given the variable's parents' values. Using the same variables from before:
Can you write Python code for it?

print("parameter.value =", parameter.value)
print("data_generator.value =", data_generator.value)
print("data_plus_one.value =", data_plus_one.value)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
Our posterior distribution puts most weight near the true value of $p_A$, but also some weights in the tails. This is a measure of how uncertain we should be, given our observations. Try changing the number of observations, N, and observe how the posterior distribution changes.
A and B Together
A similar analysis can be done for site B's response data to determine the analogous $p_B$. But what we are really interested in is the difference between $p_A$ and $p_B$. Let's infer $p_A$, $p_B$, and $\text{delta} = p_A - p_B$, all at once. We can do this using PyMC's deterministic variables. (We'll assume for this exercise that $p_B = 0.04$, so $\text{delta} = 0.01$, $N_B = 750$ (significantly less than $N_A$) and we will simulate site B's data like we did for site A's data )
Can you write Python code for it?

import pymc as pm
figsize(12, 4)

# these two quantities are unknown to us.
true_p_A = 0.05
true_p_B = 0.04

# notice the unequal sample sizes -- no problem in Bayesian analysis.
N_A = 1500
N_B = 750

# generate some observations
observations_A = pm.rbernoulli(true_p_A, N_A)
observations_B = pm.rbernoulli(true_p_B, N_B)
print("Obs from Site A: ", observations_A[:30].astype(int), "...")
print("Obs from Site B: ", observations_B[:30].astype(int), "...")

print(observations_A.mean())
print(observations_B.mean())

# Set up the pymc model. Again assume Uniform priors for p_A and p_B.
p_A = pm.Uniform("p_A", 0, 1)
p_B = pm.Uniform("p_B", 0, 1)

# Define the deterministic delta function. This is our unknown of interest.
@pm.deterministic
def delta(p_A=p_A, p_B=p_B):
    return p_A - p_B

# Set of observations, in this case we have two observation datasets.
obs_A = pm.Bernoulli("obs_A", p_A, value=observations_A, observed=True)
obs_B = pm.Bernoulli("obs_B", p_B, value=observations_B, observed=True)

# To be explained in chapter 3.
mcmc = pm.MCMC([p_A, p_B, delta, obs_A, obs_B])
mcmc.sample(20000, 1000)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
PyMC is concerned with two types of programming variables: stochastic and deterministic.

stochastic variables are variables that are not deterministic, i.e., even if you knew all the values of the variables' parents (if it even has any parents), it would still be random. Included in this category are instances of classes Poisson, DiscreteUniform, and Exponential.

deterministic variables are variables that are not random if the variables' parents were known. This might be confusing at first: a quick mental check is if I knew all of variable foo's parent variables, I could determine what foo's value is.

We will detail each below.
Initializing Stochastic variables
Initializing a stochastic variable requires a name argument, plus additional parameters that are class specific. For example:
some_variable = pm.DiscreteUniform("discrete_uni_var", 0, 4)
where 0, 4 are the DiscreteUniform-specific lower and upper bound on the random variable. The PyMC docs contain the specific parameters for stochastic variables. (Or use object??, for example pm.DiscreteUniform?? if you are using IPython!)
The name attribute is used to retrieve the posterior distribution later in the analysis, so it is best to use a descriptive name. Typically, I use the Python variable's name as the name.
For multivariable problems, rather than creating a Python array of stochastic variables, addressing the size keyword in the call to a Stochastic variable creates multivariate array of (independent) stochastic variables. The array behaves like a Numpy array when used like one, and references to its value attribute return Numpy arrays.
The size argument also solves the annoying case where you may have many variables $\beta_i, \; i = 1,...,N$ you wish to model. Instead of creating arbitrary names and variables for each one, like:
beta_1 = pm.Uniform("beta_1", 0, 1)
beta_2 = pm.Uniform("beta_2", 0, 1)
...

we can instead wrap them into a single variable:
betas = pm.Uniform("betas", 0, 1, size=N)

Calling random()
We can also call on a stochastic...
Can you write Python code for it?

lambda_1 = pm.Exponential("lambda_1", 1)  # prior on first behaviour
lambda_2 = pm.Exponential("lambda_2", 1)  # prior on second behaviour
tau = pm.DiscreteUniform("tau", lower=0, upper=10)  # prior on behaviour change

print("lambda_1.value = %.3f" % lambda_1.value)
print("lambda_2.value = %.3f" % lambda_2.value)
print("tau.value = %.3f" % tau.value, "\n")

lambda_1.random(), lambda_2.random(), tau.random()

print("After calling random() on the variables...")
print("lambda_1.value = %.3f" % lambda_1.value)
print("lambda_2.value = %.3f" % lambda_2.value)
print("tau.value = %.3f" % tau.value)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
The 95% credible interval, or 95% CI, painted in purple, represents the interval, for each temperature, that contains 95% of the distribution. For example, at 65 degrees, we can be 95% sure that the probability of defect lies between 0.25 and 0.75.
More generally, we can see that as the temperature nears 60 degrees, the CI's spread out over [0,1] quickly. As we pass 70 degrees, the CI's tighten again. This can give us insight about how to proceed next: we should probably test more O-rings around 60-65 temperature to get a better estimate of probabilities in that range. Similarly, when reporting to scientists your estimates, you should be very cautious about simply telling them the expected probability, as we can see this does not reflect how wide the posterior distribution is.
What about the day of the Challenger disaster?
On the day of the Challenger disaster, the outside temperature was 31 degrees Fahrenheit. What is the posterior distribution of a defect occurring,  given this temperature? The distribution is plotted below. It looks almost guaranteed that the Challenger was going to be subject to defective O-rings.
Can you write Python code for it?

figsize(12.5, 2.5)

prob_31 = logistic(31, beta_samples, alpha_samples)

plt.xlim(0.995, 1)
plt.hist(prob_31, bins=1000, normed=True, histtype='stepfilled')
plt.title("Posterior distribution of probability of defect, given $t = 31$")
plt.xlabel("probability of defect occurring in O-ring");


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
The call to random stores a new value into the variable's value attribute. In fact, this new value is stored in the computer's cache for faster recall and efficiency.
Warning: Don't update stochastic variables' values in-place.
Straight from the PyMC docs, we quote [4]:

Stochastic objects' values should not be updated in-place. This confuses PyMC's caching scheme... The only way a stochastic variable's value should be updated is using statements of the following form:

    A.value = new_value

The following are in-place updates and should never be used:

    A.value += 3
    A.value[2,1] = 5
    A.value.attribute = new_attribute_value

Deterministic variables
Since most variables you will be modeling are stochastic, we distinguish deterministic variables with a pymc.deterministic wrapper. (If you are unfamiliar with Python wrappers (also called decorators), that's no problem. Just prepend the pymc.deterministic decorator before the variable declaration and you're good to go. No need to know more. ) The declaration of a deterministic variable uses a Python function:
@pm.deterministic
def some_deterministic_var(v1=v1,):
     #jelly goes here.

For all purposes, we can treat the object some_deterministic_var as a variable and not a Python function.
Prepending with the wrapper is the easiest way, but not the only way, to create deterministic variables: elementary operations, like addition, exponentials etc. implicitly create deterministic variables. For example, the following returns a deterministic variable:
Can you write Python code for it?


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
The remainder of this chapter examines some practical examples of PyMC and PyMC modeling:
Example: Challenger Space Shuttle Disaster <span id="challenger"/>
On January 28, 1986, the twenty-fifth flight of the U.S. space shuttle program ended in disaster when one of the rocket boosters of the Shuttle Challenger exploded shortly after lift-off, killing all seven crew members. The presidential commission on the accident concluded that it was caused by the failure of an O-ring in a field joint on the rocket booster, and that this failure was due to a faulty design that made the O-ring unacceptably sensitive to a number of factors including outside temperature. Of the previous 24 flights, data were available on failures of O-rings on 23, (one was lost at sea), and these data were discussed on the evening preceding the Challenger launch, but unfortunately only the data corresponding to the 7 flights on which there was a damage incident were considered important and these were thought to show no obvious trend. The data are shown below (see [1]):
Can you write Python code for it?

figsize(12.5, 3.5)
np.set_printoptions(precision=3, suppress=True)
challenger_data = np.genfromtxt("data/challenger_data.csv", skip_header=1,
                                usecols=[1, 2], missing_values="NA",
                                delimiter=",")
# drop the NA values
challenger_data = challenger_data[~np.isnan(challenger_data[:, 1])]

# plot it, as a function of temperature (the first column)
print("Temp (F), O-Ring failure?")
print(challenger_data)

plt.scatter(challenger_data[:, 0], challenger_data[:, 1], s=75, color="k",
            alpha=0.5)
plt.yticks([0, 1])
plt.ylabel("Damage Incident?")
plt.xlabel("Outside temperature (Fahrenheit)")
plt.title("Defects of the Space Shuttle O-Rings vs temperature");


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
The special case when $N = 1$ corresponds to the Bernoulli distribution. There is another connection between Bernoulli and Binomial random variables. If we have $X_1, X_2, ... , X_N$ Bernoulli random variables with the same $p$, then $Z = X_1 + X_2 + ... + X_N \sim \text{Binomial}(N, p )$.
The expected value of a Bernoulli random variable is $p$. This can be seen by noting the more general Binomial random variable has expected value $Np$ and setting $N=1$.
Example: Cheating among students
We will use the binomial distribution to determine the frequency of students cheating during an exam. If we let $N$ be the total number of students who took the exam, and assuming each student is interviewed post-exam (answering without consequence), we will receive integer $X$ "Yes I did cheat" answers. We then find the posterior distribution of $p$, given $N$, some specified prior on $p$, and observed data $X$.
This is a completely absurd model. No student, even with a free-pass against punishment, would admit to cheating. What we need is a better algorithm to ask students if they had cheated. Ideally the algorithm should encourage individuals to be honest while preserving privacy. The following proposed algorithm is a solution I greatly admire for its ingenuity and effectiveness:

In the interview process for each student, the student flips a coin, hidden from the interviewer. The student agrees to answer honestly if the coin comes up heads. Otherwise, if the coin comes up tails, the student (secretly) flips the coin again, and answers "Yes, I did cheat" if the coin flip lands heads, and "No, I did not cheat", if the coin flip lands tails. This way, the interviewer does not know if a "Yes" was the result of a guilty plea, or a Heads on a second coin toss. Thus privacy is preserved and the researchers receive honest answers.

I call this the Privacy Algorithm. One could of course argue that the interviewers are still receiving false data since some Yes's are not confessions but instead randomness, but an alternative...
Can you write Python code for it?

import pymc as pm

N = 100
p = pm.Uniform("freq_cheating", 0, 1)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
The use of the deterministic wrapper was seen in the previous chapter's text-message example.  Recall the model for $\lambda$ looked like:
$$
\lambda =
\begin{cases}
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$
And in PyMC code:
Can you write Python code for it?

import numpy as np
n_data_points = 5  # in CH1 we had ~70 data points

@pm.deterministic
def lambda_(tau=tau, lambda_1=lambda_1, lambda_2=lambda_2):
    out = np.zeros(n_data_points)
    out[:tau] = lambda_1  # lambda before tau is lambda1
    out[tau:] = lambda_2  # lambda after tau is lambda2
    return out


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
This is how we include data into our models: initializing a stochastic variable to have a fixed value.
To complete our text message example, we fix the PyMC variable observations to the observed dataset.
Can you write Python code for it?

# We're using some fake data here
data = np.array([10, 25, 15, 20, 35])
obs = pm.Poisson("obs", lambda_, value=data, observed=True)
print(obs.value)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
To frame this in the notation of the first chapter, though this is a slight abuse of notation, we have specified $P(A)$. Our next goal is to include data/evidence/observations $X$ into our model.
PyMC stochastic variables have a keyword argument observed which accepts a boolean (False by default). The keyword observed has a very simple role: fix the variable's current value, i.e. make value immutable. We have to specify an initial value in the variable's creation, equal to the observations we wish to include, typically an array (and it should be an Numpy array for speed). For example:
Can you write Python code for it?

data = np.array([10, 5])
fixed_variable = pm.Poisson("fxd", 1, value=data, observed=True)
print("value: ", fixed_variable.value)
print("calling .random()")
fixed_variable.random()
print("value: ", fixed_variable.value)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
Using these variables, we can return a possible realization of the observed proportion of "Yes" responses. We do this using a PyMC deterministic variable:
Can you write Python code for it?

@pm.deterministic
def observed_proportion(t_a=true_answers,
                        fc=first_coin_flips,
                        sc=second_coin_flips):

    observed = fc * t_a + (1 - fc) * sc
    return observed.sum() / float(N)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
We can present the above data better in a figure: I've wrapped this up into a separation_plot function.
Can you write Python code for it?

from separation_plot import separation_plot

figsize(11., 1.5)
separation_plot(posterior_probability, D)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
We have our probabilities, but how do we connect them to our observed data? A Bernoulli random variable with parameter $p$, denoted $\text{Ber}(p)$, is a random variable that takes value 1 with probability $p$, and 0 else. Thus, our model can look like:
$$ \text{Defect Incident, $D_i$} \sim \text{Ber}( \;p(t_i)\; ), \;\; i=1..N$$
where $p(t)$ is our logistic function and $t_i$ are the temperatures we have observations about. Notice in the above code we had to set the values of beta and alpha to 0. The reason for this is that if beta and alpha are very large, they make p equal to 1 or 0. Unfortunately, pm.Bernoulli does not like probabilities of exactly 0 or 1, though they are mathematically well-defined probabilities. So by setting the coefficient values to 0, we set the variable p to be a reasonable starting value. This has no effect on our results, nor does it mean we are including any additional information in our prior. It is simply a computational caveat in PyMC.
Can you write Python code for it?

p.value

# connect the probabilities in `p` with our observations through a
# Bernoulli random variable.
observed = pm.Bernoulli("bernoulli_obs", p, value=D, observed=True)

model = pm.Model([observed, beta, alpha])

# Mysterious code to be explained in Chapter 3
map_ = pm.MAP(model)
map_.fit()
mcmc = pm.MCMC(model)
mcmc.sample(120000, 100000, 2)


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
We have trained our model on the observed data, now we can sample values from the posterior. Let's look at the posterior distributions for $\alpha$ and $\beta$:
Can you write Python code for it?

alpha_samples = mcmc.trace('alpha')[:, None]  # best to make them 1d
beta_samples = mcmc.trace('beta')[:, None]

figsize(12.5, 6)

# histogram of the samples:
plt.subplot(211)
plt.title(r"Posterior distributions of the variables $\alpha, \beta$")
plt.hist(beta_samples, histtype='stepfilled', bins=35, alpha=0.85,
         label=r"posterior of $\beta$", color="#7A68A6", normed=True)
plt.legend()

plt.subplot(212)
plt.hist(alpha_samples, histtype='stepfilled', bins=35, alpha=0.85,
         label=r"posterior of $\alpha$", color="#A60628", normed=True)
plt.legend();


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
We plot the posterior distribution of the unknown $p_A$ below:
Can you write Python code for it?

figsize(12.5, 4)
plt.title("Posterior distribution of $p_A$, the true effectiveness of site A")
plt.vlines(p_true, 0, 250, linestyle="--", label="true $p_A$ (unknown)")
plt.hist(mcmc.trace("p")[:], bins=25, histtype="stepfilled", normed=True)
plt.legend();


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
With regards to the above plot, we are still pretty uncertain about what the true frequency of cheaters might be, but we have narrowed it down to a range between 0.05 to 0.35 (marked by the solid lines). This is pretty good, as a priori we had no idea how many students might have cheated (hence the uniform distribution for our prior). On the other hand, it is also pretty bad since there is a .3 length window the true value most likely lives in. Have we even gained anything, or are we still too uncertain about the true frequency?
I would argue, yes, we have discovered something. It is implausible, according to our posterior, that there are no cheaters, i.e. the posterior assigns low probability to $p=0$. Since we started with a uniform prior, treating all values of $p$ as equally plausible, but the data ruled out $p=0$ as a possibility, we can be confident that there were cheaters.
This kind of algorithm can be used to gather private information from users and be reasonably confident that the data, though noisy, is truthful.
Alternative PyMC Model
Given a value for $p$ (which from our god-like position we know), we can find the probability the student will answer yes:
\begin{align}
P(\text{"Yes"}) &=  P( \text{Heads on first coin} )P( \text{cheater} ) + P( \text{Tails on first coin} )P( \text{Heads on second coin} ) \\
& = \frac{1}{2}p + \frac{1}{2}\frac{1}{2}\\
& = \frac{p}{2} + \frac{1}{4}
\end{align}
Thus, knowing $p$ we know the probability a student will respond "Yes". In PyMC, we can create a deterministic function to evaluate the probability of responding "Yes", given $p$:
Can you write Python code for it?

p = pm.Uniform("freq_cheating", 0, 1)

@pm.deterministic
def p_skewed(p=p):
    return 0.5 * p + 0.25


I am working on the file "Ch2_MorePyMC_PyMC2.ipynb".
The first task is:
parameter controls the parameter of data_generator, hence influences its values. The former is a parent of the latter. By symmetry, data_generator is a child of parameter.
Likewise, data_generator is a parent to the variable data_plus_one (hence making data_generator both a parent and child variable). Although it does not look like one, data_plus_one should be treated as a PyMC variable as it is a function of another PyMC variable, hence is a child variable to data_generator.
This nomenclature is introduced to help us describe relationships in PyMC modeling. You can access a variable's children and parent variables using the children and parents attributes attached to variables.
Can you write Python code for it?

print("Children of `parameter`: ")
print(parameter.children)
print("\nParents of `data_generator`: ")
print(data_generator.parents)
print("\nChildren of `data_generator`: ")
print(data_generator.children)


I am working on the file "ChETEC_school/GCE Lab 1 - Solar Composition - Elemental Abundance Pattern.ipynb".
The first task is:
2. Neutron-Capture Elements. s- and r-Process Contributions
Can you write Python code for it?

# \\\\\\\\\\ Modify below \\\\\\\\\\\\
# ====================================

# Number of r-process events per units of stellar mass formed.
# Original value --> 1.0e-6
nb_nsm_per_m = 1.0e-6

# ====================================
# ////////// Modify above ////////////

# Run the GCE code OMEGA+
op = omega_plus.omega_plus(nb_nsm_per_m=nb_nsm_per_m, **kwargs)

# Get source contributions
m_el_all, m_el_agb, m_el_massive, m_el_sn1a, m_el_nsm = \
    get_individual_sources(op.inner, i_step_sol=i_t_Sun)

# Set figure
fig = plt.figure(figsize=(12,4))
matplotlib.rcParams.update({'font.size': 16.0})

# Plot solar abundance data
plt.plot(solar_Z, solar_ab, color='k', marker='o', linewidth=6, alpha=0.5, label='Solar')

# Contribution of the s-process (AGB stars)
plt.plot(Z_charge, m_el_agb, color='r', label='s-process (AGB)', alpha=0.8, linestyle='-', marker='x')

# Contribution of the r-process
plt.plot(Z_charge, m_el_nsm, color='c', label='r-process', alpha=0.8, linestyle='-', marker='^')

# Add element annotations (lanthanides)
Z_low, Z_upp = 50, 80
for i in range(Z_low, Z_upp+1):
    plt.annotate(elements[i], xy=(solar_Z[i],yy[i]), color='k',\
                 fontsize=15, ha='center', va='center')

# Label, legend, and axis
plt.legend(fontsize=16, loc='center left', bbox_to_anchor=(1, 0.5))
plt.xlabel('Z (charge number)', fontsize=16)
plt.ylabel('X (mass fraction)', fontsize=16)
plt.xlim(Z_low-0.5, Z_upp+0.5)
plt.ylim(1e-11,1e-6)
plt.yscale('log')

# Frame tuning
plt.subplots_adjust(top=0.95)
plt.subplots_adjust(right=0.75)
plt.subplots_adjust(left=0.15)
plt.subplots_adjust(bottom=0.14)


I am working on the file "ChETEC_school/GCE Lab 1 - Solar Composition - Elemental Abundance Pattern.ipynb".
The first task is:
Choice of Stellar Yields
The stellar yields for the simulations are taken from:
* Low-mass asymptotic-giant-branch (AGB) stars: Cristallo et al. (2015)
* Massive stars: Limongi & Chieffi (2018)
* Type Ia supernovae (SNe Ia): Iwamoto et al. (1999)
* Rapid neutron-capture process (r-process): Solar residuals of Arnould et al. (2007)
1. Iron-Peak Elements. Contribution of SNe Ia
Can you write Python code for it?

# \\\\\\\\\\ Modify below \\\\\\\\\\\\
# ====================================

# Number of SNe Ia per units of stellar mass formed.
# For every solar mass of stars formed, there will be statistically nb_1a_per_m SNe Ia.
# Original value --> 1.0e-1
nb_1a_per_m = 1.0e-1

# ====================================
# ////////// Modify above ////////////

# Run the GCE code OMEGA+
op = omega_plus.omega_plus(nb_1a_per_m=nb_1a_per_m, **kwargs)

# Get source contributions
m_el_all, m_el_agb, m_el_massive, m_el_sn1a, m_el_nsm = \
    get_individual_sources(op.inner, i_step_sol=i_t_Sun)

# Set figure
fig = plt.figure(figsize=(8,4))
matplotlib.rcParams.update({'font.size': 16.0})

# Plot solar abundance data
plt.plot(solar_Z, solar_ab, color='k', marker='o', linewidth=6, alpha=0.5, label='Solar')

# Plot contribution from Type Ia supernovae
plt.plot(Z_charge, m_el_sn1a, color='g', label='SNe Ia', alpha=0.8, linestyle='-', marker='s')

# Add element annotations (iron-peak)
Z_low, Z_upp = 20, 30
for i in range(Z_low, Z_upp+1):
    plt.annotate(elements[i], xy=(solar_Z[i],yy[i]), color='k',\
                 fontsize=15, ha='center', va='center')

# Label, legend, and axis
plt.legend(fontsize=16, loc='center left', bbox_to_anchor=(1, 0.5))
plt.xlabel('Z (charge number)', fontsize=16)
plt.ylabel('X (mass fraction)', fontsize=16)
plt.xlim(Z_low-0.5, Z_upp+0.5)
plt.ylim(1e-9,3e-1)
plt.yscale('log')

# Frame tuning
plt.subplots_adjust(top=0.95)
plt.subplots_adjust(right=0.75)
plt.subplots_adjust(left=0.15)
plt.subplots_adjust(bottom=0.14)


I am working on the file "ChETEC_school/GCE Lab 2 - Solar Composition - Stellar Yields and Isotopes.ipynb".
The first task is:
Plot the Isotopic Contributions
The plot below is normalized to the Solar composition of Asplund et al. (2009). If the numerical predictions are above 1.0, they overestimate the abundance derived for the Sun. If they are lower, they underestimate.
Can you write Python code for it?

# \\\\\\\\\\ Modify below \\\\\\\\\\\\
# ====================================

# Select the isotopes you want to analyze
# It can also be a mix of isotopes and elements
# Example 1: species = ["C-12", "O-16", Fe-56"]
# Example 2: species = ["C-12", "C-13", "C", "Si-28"]
species = ["C-12", "O-16", "Fe-56", "Zr-94"]

# ====================================
# ////////// Modify above ////////////

# Set the figure size
fig = plt.figure(figsize=(8,5.0))
matplotlib.rcParams.update({'font.size': 16.0})

# Plot contributions
op.inner.mass_frac_plot(species=species, cycle=ii_Sun, yscale="linear")

# Increase size labels
plt.title("")
ax = plt.gca()
for tick in ax.xaxis.get_major_ticks():
    tick.label.set_fontsize(18)
plt.ylabel("X / X$_\odot$", fontsize=18)


I am working on the file "ChETEC_school/GCE Lab 2 - Solar Composition - Stellar Yields and Isotopes.ipynb".
The first task is:
Run GCE code
Can you write Python code for it?

# \\\\\\\\\\ Modify below \\\\\\\\\\\\
# ====================================

# Choose one yield option (uncomment one "kwargs["table"]" line)

# NuGrid (Ritter et al. 2018) for both AGB and massive stars
kwargs["table"] = "yield_tables/agb_and_massive_stars_nugrid_MESAonly_fryer12delay.txt"

# Cristallo et al. (2015) for AGB stars
# Limongi & Chieffi (2018) for massive stars
#kwargs["table"] = "yield_tables/agb_and_massive_stars_C15_LC18_R_mix.txt"

# Cristallo et al. (2015) for AGB stars
# Nomoto et al. (2013) for massive stars
#kwargs["table"] = "yield_tables/agb_and_massive_stars_C15_N13_0_5_HNe.txt"

# ====================================
# ////////// Modify above ////////////

# Run the GCE code OMEGA+
print("Using", kwargs["table"])
op = omega_plus.omega_plus(**kwargs)

# Get the timestep where the metallicity is close to Solar
if kwargs["table"] == "yield_tables/agb_and_massive_stars_nugrid_MESAonly_fryer12delay.txt":
    ii_Sun = i_t_Sun + 5
else:
    ii_Sun = copy.deepcopy(i_t_Sun)


I am working on the file "ChETEC_school/GCE Lab 3 - Constrain Galaxy Model.ipynb".
The first task is:
2. Plot the Star Formation History
The current star formation rate (SFR) is about 2 M$_\odot$ yr$^{-1}$. This is the value your model should have at the end of the simulation (at a Galactic age $t=13$ Gyr).
Useful Information: With a higher star formation efficiency, the gas reservoir will be converted into stars more rapidly.
Useful Information: The magnitude of the star formation rate is very sensitive to the galactic inflow rate.
Can you write Python code for it?

# Set the figure size
fig = plt.figure(figsize=(12,7))
matplotlib.rcParams.update({'font.size': 16.0})

# Plot the evolution of the star formation rate (SFR)
op.inner.plot_star_formation_rate(color='r', marker=' ', label="Prediction")

# Plot the observational constraint (cyan color)
plt.plot([13e9,13e9], [0.65,3.0], linewidth=6, color='c', alpha=0.5, label="Observation")

# Labels and legend
plt.xlabel('Galactic age [yr]', fontsize=16)
plt.legend()

# Print the total stellar mass formed
print("Integrated stellar mass:",'%.2e'%sum(op.inner.history.m_locked),'M_sun')
print(".. Should between 3.00e+10 and 5.00e+10 M_sun")


I am working on the file "ChETEC_school/GCE Lab 3 - Constrain Galaxy Model.ipynb".
The first task is:
3. Plot the Evolution of the Mass of Gas
The current total mass of gas in the Milky Way is about $7\times10^{9}$ M$_\odot$. This is the value your model should have at the end of the simulation (at a Galactic age $t=13$ Gyr).
Useful Information: The mass of gas depends strongly on the galactic inflow rate.
Can you write Python code for it?

# Set the figure size
fig = plt.figure(figsize=(12,7))
matplotlib.rcParams.update({'font.size': 16.0})

# Plot the evolution of the mass of gas in the interstellar medium (ISM)
op.inner.plot_totmasses(color='r', marker=' ', label="Prediction")

# Plot the observational constraint
plt.plot([12.9e9,12.9e9], [3.6e9,12.6e9], linewidth=6, color='c', alpha=0.5, label="Observation")

# Labels and legend
plt.xscale('linear')
plt.xlabel('Galactic age [yr]')
plt.ylim(1e8,1e11)
plt.legend()


I am working on the file "ChETEC_school/GCE Lab 3 - Constrain Galaxy Model.ipynb".
The first task is:
4. Plot the Evolution of Iron Abundance [Fe/H]
$[A/B]=\log(n_A/n_B)-\log(n_A/n_B)_\odot$
To represent the solar neighbourhood, [Fe/H] in your model should reach zero (solar value) about 4.6 Gyr before the end of the simulation, representing the moment the Sun formed.
Useful Information: The [Fe/H] is mostly sensitive to the star formation efficiency. In other words, it is sensitive to the mass of gas (H) in which stars inject their metals (Fe).
Can you write Python code for it?

# Set the figure size
fig = plt.figure(figsize=(12,7.0))
matplotlib.rcParams.update({'font.size': 16.0})

# Plot the evolution of [Fe/H], the iron abundance of the gas inside the galaxy
op.inner.plot_spectro(color='r', marker=" ", label="Prediction")

# Plot the solar value (black dotted lines)
t_Sun = 13.0e9 - 4.6e9
plt.plot([t_Sun,t_Sun], [-2,1], ':k')
plt.plot([0,13e9], [0,0], ':k')

# Labels and legend
plt.ylim(-2,1)
plt.xscale('linear')
plt.xlabel('Galactic age [yr]')
plt.legend()


I am working on the file "ChETEC_school/GCE Lab 3 - Constrain Galaxy Model.ipynb".
The first task is:
5. Plot the Evolution of the Galactic Inflow Rate
The current galactic inflow rate estimated for the Milky Way is about 1 M$_\odot$ yr$^{-1}$.
Can you write Python code for it?

# Set the figure size
fig = plt.figure(figsize=(10,6.0))
matplotlib.rcParams.update({'font.size': 16.0})

# Plot the evolution of the inflow rate
op.inner.plot_inflow_rate(color='r', marker=" ", label="Prediction")

# Plot the observational constraint
plt.plot([13e9,13e9], [0.6,1.6], linewidth=6, color='c', alpha=0.5, label="Observation")

# Labels and legend
plt.yscale('log')
plt.xlabel('Galactic age [yr]')
plt.legend()


I am working on the file "ChETEC_school/GCE Lab 3 - Constrain Galaxy Model.ipynb".
The first task is:
Your Tasks

Understand the impact of the star formation efficiency and the galactic inflow rate on the general properties of your Milky Way model.
Find a set of input parameters that reproduce the observational constraints.

Key Equation for Star Formation
The global star formation rate ($\dot{M}\star$) inside the galaxy model at time $t$ depends on the mass of gas $M{gas}$ inside the galaxy and the star formation efficiency $f_\star$ [yr$^{-1}$].
$$\dot{M}\star(t)=f\star M_\mathrm{gas}(t)\quad\mbox{[M$_\odot$ yr$^{-1}$]}$$
1. Run GCE Model
In the following example, you will be using a galactic inflow prescription that is similar to the two-infall model presented in Chiappini et al. (1997).
Can you write Python code for it?

# \\\\\\\\\\ Modify below \\\\\\\\\\\\
# ====================================

# Star formation efficiency (f_\star) --> [dimensionless]
# Original value in the notebook --> 1e-11
sfe = 1e-11

# Magnitude (strength) of the galactic inflow rate
# Original value in the notebook --> 0.1
in_mag = 0.1

# ====================================
# ////////// Modify above ////////////

# Run OMEGA+ with the first set of parameters
op = omega_plus.omega_plus(sfe=sfe, special_timesteps=200, t_star=1.0, \
                           exp_infall = [[in_mag*40, 0.0, 0.8e9], [in_mag*5, 1.0e9, 7.0e9]])


I am working on the file "ChETEC_school/GCE Lab 4 - Dwarf Galaxy - Chemical Evolution Trend.ipynb".
The first task is:
1. Stellar Abundances
To plot the observed stellar abundances of a galaxy, STELLAB first needs to be launched without input parameter.
Can you write Python code for it?

# Launch the STELLAB code
s = stellab.stellab()


I am working on the file "ChETEC_school/GCE Lab 4 - Dwarf Galaxy - Chemical Evolution Trend.ipynb".
The first task is:
The next step is to select a galaxy. So far, STELLAB includes the Milky Way, Sculptor, Carina, Fornax, and the LMC. Simply use the galaxy parameter and enter the name of the galaxy with lower cases (spaces need to be replaced by underscores). Then, use the plot_spectro function to visualize the selected abundance ratios.
Can you write Python code for it?

# Select the galaxy
galaxy = "fornax"

# Plot the abundance ratios
matplotlib.rcParams.update({'font.size': 16.0})
s.plot_spectro(galaxy=galaxy, xaxis='[Fe/H]', yaxis='[Mg/Fe]')
plt.ylim(-1.6, 1.6)


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
2. Cleanse and parse and tokenize text
Before generating the features, and to increase the explanatory power of them, text has been cleaned and parsed accordingly. The books have gone through an initial set of cleansing actions before been parsed using Spacy, to reduce the computing effort required by the latter and then have been cleaned again before the feature generation.
The initial cleansing action has had three steps. The first step consisted on deleting all references to the Gutenberg Project from every book. This way, it has been avoided that words like “Gutenberg” and “Gutenberg Project” appear as features and distort the clustering of the authors.
As described below, cleaning actions have gone from removing all references to chapters, digits double whitespaces and references to numbers like dates and ordinal numbers. This has been followed by removing punctuation and common stop words that will only add noise to the features that are generated afterwards.
The remaining words, considered to have the most explanatory power regarding each of the titles from the authors, have been lemmatized and stemmed reducing up to 60% the computing resources needed. In the first case words from the same family are reduced to their lemmas and in the second case, additional prefixes and suffixes are removed. All cleaning operations have been carried out in a way that remaining sentences are stored in a list of lists.
Can you write Python code for it?

#Create a set of stopwords in english from nltk
stop = set(stopwords.words('english'))

# Create a set of punctuation marks to exclude them from the text
exclude = set(string.punctuation)

# Call the lemmatizer
lemma = WordNetLemmatizer()

#Define a cleaning function that incorporates the different steps in the pipeline to clean the texts
def clean(doc):
    doc = re.sub(r'--',' ',doc)
    doc = re.sub("[\[].*?[\]]", "", doc)
    doc = re.sub(r'Chapter \d+', '', doc)
    doc = re.sub(r'CHAPTER .*', '', doc)
    doc = re.sub('[0-9]+', '', doc)
    doc = re.sub("^\d+\s|\s\d+\s|\s\d+$", " ", doc)
    stop_free = " ".join([i for i in doc.lower().split() if i not in stop])
    punc_free = ''.join(ch for ch in stop_free if ch not in exclude)
    normalized = " ".join(lemma.lemmatize(word) for word in punc_free.split())
    return normalized

#Create a list of lists with all the documents
doc_clean = [clean(doc) for doc in doc_complete]

# Parse the cleaned novels
#load spacy for english language as all novels are in english
nlp = spacy.load('en')

#Parse novels one by one to maintain the author tagging
austen_doc = nlp(doc_clean[0])
chesterton_doc = nlp(doc_clean[1])
conandoyle_doc = nlp(doc_clean[2])
elliot_doc = nlp(doc_clean[3])

# Group into sentences.
austen_sents = [[str(sent), "Austen"] for sent in austen_doc.sents]
chesterton_sents = [[str(sent), "Chesterton"] for sent in chesterton_doc.sents]
conandoyle_sents = [[str(sent), "Conandoyle"] for sent in conandoyle_doc.sents]
elliot_sents = [[str(sent), "elliot"] for sent in elliot_doc.sents]

# Combine the sentences from the two novels into one data frame.
names = ['Sentences','Author']
sent = pd.DataFrame(austen_sents + chesterton_sents +
                    conandoyle_sents +
                    elliot_sents, columns = names)

#Plot the contribution of each author to the corpus (sentences)
sent.Author.value_counts().plot(kind='bar', grid=False, figsize=(16, 9))

#Aadd numerical column to tag the authors for supervised...


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
3. Generate features and select the most appropiate for the models
Features using BoW
Texts have been vectorized using bag of words. In this case the algorithm counts the numnber of times a word appears in a certain text. During the creation of the bag of words space, ngrams up to 4 components have been considered and stop words in english to remove noise from the dataset. Due to the authors that have been chosen, this method will bias the models towards the authors that have longer texts being Elliot and Austen compared to Conan Doyle and Chesterton. The total number of features is 52k.
Can you write Python code for it?

#Transform into Bag of Words
vec = CountVectorizer(max_df = 0.75 , min_df = 2 , ngram_range = (1,4), stop_words = 'english')

#Build the predictors and the predicted variable applying BoW.
X = vec.fit_transform(sent['Sentences'])
y = sent['Target']

#Split the data set into train and test 70/30
X_train_bow, X_test_bow, y_train_bow, y_test_bow  = train_test_split(X,y, test_size=0.30, random_state=1234)

X_train_bow.shape


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
4. Supervised models
All models have been run using the features obtained through bag of words and tfidf. In this case results are compared to see which one gives a better overall accuracy as it has been used as the score function. In all cases cross validation over five folds is applied.
Logistic Regression Classifier
 Bag of Words
A Logistic Regression Classifier is trained using the features obtained through tfidf. Additionally, using fridsearch the parameters are tunned. As length of texts and therefore the features per author are not balanced, the class weight is set up so that is consideres unbalanced classes.
Can you write Python code for it?

# Initialize and fit the model.
log_reg_bow = LogisticRegression(class_weight='balanced', penalty = 'l2', multi_class= 'multinomial', max_iter = 1000)

#Tune parameters: C parameter
c_param = [ 0.1, 0.5, 1 ]

#Tune the type of penalty used between l1 and l2

solver_param = ['newton-cg', 'lbfgs']

parameters = {'C': c_param, 'solver': solver_param}

#Fit parameters
log_reg_tuned_bow = GridSearchCV(log_reg_bow, param_grid=parameters, n_jobs = -1, cv=kf, verbose = 1)

#Fit the tunned classifier in the training space
log_reg_tuned_bow.fit(X_train_bow, y_train_bow)

#Print the best parameters
print(('Best paramenters logistic regression BoW:\n {}\n').format(log_reg_tuned_bow.best_params_))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
After several runs, with different extremes in the values of the alpha parameter, the parameter chosen is always the one closer to zero. This means that the smoothing parameter is very low so the additive smoothing required is low. The model is fit within seconds which makes it a strong candidate (the best one from a computational and speed standpoint) to move intro production.
Can you write Python code for it?

#Once the model has been trained test it on the test dataset
naive_bayes_bernoulli_tuned_bow.fit(X_test_bow, y_test_bow)

# Predict on test set
predtest_y_bow = naive_bayes_bernoulli_tuned_bow.predict(X_test_bow)


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
After the parameters are tunned, the model is fit in the test dataset. As a measurement of the computing effort it requires 3.6 min to fit the test set.
Can you write Python code for it?

#Once the model has been trained test it on the test dataset
log_reg_tuned_bow.fit(X_test_bow, y_test_bow)

# Predict on test set
predtest_y_bow = log_reg_tuned_bow.predict(X_test_bow)


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
After the parameters are tunned, the model is fit in the test dataset. As a measurement of the computing effort it requires less than one min to fit the test set.
Can you write Python code for it?

#Once the model has been trained test it on the test dataset
log_reg_tuned_tfidf.fit(X_test_tfidf, y_test_tfidf)

# Predict on test set
predtest_y_tfidf = log_reg_tuned_tfidf.predict(X_test_tfidf)


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
Although from a computational perspective it requires more effort, it presents better results than the previous algorithms. In this case, nearly 73% has been achieved competing agasint the multiclass algorithm in terms of accuracy but not in terms of computational effort.
Can you write Python code for it?

#Evaluation of the model (testing)

target_names = ['0.0', '1.0', '2.0', '3.0']

print(('Classification Report BoW: \n {}\n').format(
    classification_report(y_test_bow, predtest_y_bow,
                          target_names=target_names)))

confusion_bow = confusion_matrix(y_test_bow, predtest_y_bow)

print((
    'Confusion Matrix BoW: \n\n {}\n\n').format(confusion_bow))

print((
    'Linear SVC accuracy BoW: {0:.2f} %\n'
).format(cross_val_score(LSVC_tuned_bow, X_test_bow, y_test_bow,cv=kf).mean()*100))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
Features using Tf-idf
When using tfidf, the frequency of appearance is normalized and also considered the ones that appear in less than 75% of the documents. With this method, the value counts are smoothen considering additional features of the word such as the amount of information it adds to describe the novel. As in the case of the ba og words, ngamrs up to four have been considered, stop words removed and thesublinear_tf used. It Apply scales the word count obtained and smoothened by the frequency of appearence in the document and whithin a document.
Can you write Python code for it?

#Transform into Tf-idf considering the relative frequency
vect = TfidfVectorizer(norm = 'l2', max_df = 0.75 , min_df = 2 , ngram_range = (1,4), stop_words = 'english',
                       use_idf = True, sublinear_tf = True)

#Build the predictors and the predicted variable applying BoW.
X_tfidf = vect.fit_transform(sent['Sentences'])
y_tfidf = sent['Target']

#Split the data set into train and test 70/30
X_train_tfidf, X_test_tfidf, y_train_tfidf, y_test_tfidf  = train_test_split(X_tfidf,y_tfidf, test_size=0.30, random_state=1234)


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
In this case, the accuracy obtained with tfidf is not very different from the accuracy obtained with the bag of words. Better results would be obtained if more data is used to run the model
Can you write Python code for it?

#Evaluation of the model (testing)

target_names = ['0.0', '1.0', '2.0', '3.0']

print(('Classification Report Tfidf: \n {}\n').format(
    classification_report(y_test_tfidf, predtest_y_tfidf,
                          target_names=target_names)))

confusion_tfidf = confusion_matrix(y_test_tfidf, predtest_y_tfidf)

print((
    'Confusion Matrix Tfidf: \n\n {}\n\n').format(confusion_tfidf))

print((
    'KNN accuracy Tfidf: {0:.2f} %\n'
).format(cross_val_score(KNN_tuned_tfidf, X_test_tfidf, y_test_tfidf,cv=kf).mean()*100))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
In this case, the overall accuracy is 72.57%, very similar to the overall accuracy obtained using the multinomial classifier. The computational effort required by this model to achieve this accuracy is much higher than in the case of the multinomial classifier. Hence, from a production perspective, this model would not be recommended to move into production despite of its high accuracy.
Tf- idf
The SGD Classifier uses regularized linear models with stochastic gradient descendent learning. The model is updated in its learning rate after the gradient of the loss is estaimated for each sample. This classifier can work with sparse data se the one obtained from tfidf. In this case from the types of penalties the algorithm accepts, it uses L2 instead of a combination of L! and L2 implemented through Elastic Net.
Can you write Python code for it?

# Initialize and fit the model.
SGD_tfidf = SGDClassifier(class_weight = 'balanced', max_iter=1000)

#Tune hyperparameters
#Create range of values to fit parameters

loss_param = ['hinge', 'squared_hinge']
penalty_param = ['elasticnet', 'l2' ]
alpha_param = [1, 0.0001, 0.001, 0.01, 0.1]

#Fit parameters
parameters = {'loss': loss_param,
              'penalty': penalty_param,
              'alpha': alpha_param}

#Fit parameters using gridsearch
SGD_tuned_tfidf = GridSearchCV(SGD_tfidf, param_grid=parameters,  n_jobs = -1, cv=kf, verbose = 1)

#Fit the tunned classifier in the training space
SGD_tuned_tfidf.fit(X_train_tfidf, y_train_tfidf)

#Print the best parameters
print(('Best paramenters SDG Tfidf:\n {}\n').format(
SGD_tuned_tfidf.best_params_))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
In this case, the overall accuracy is 80.78%, very similar to the overall accuracy obtained using the multinomial classifier. The computational effort required by this model to achieve this accuracy is much higher than in the case of the multinomial classifier . Hence, from a production perspective, this model would not be recommended to move into production despite of its high accuracy.
Random Forest
 Bag of Words
The hyperparamters of the random forest model have been tuned one by one. After trying to tune them all at once, a significant increase of the overall performance of the classifier was obtained with the proposed method (one by one). The parameters to be tuned are (in the same order as the hyperparameter tuning has been performed):
N_estimators determining the number of trees that will be part of the algorithm.
Max depth determining the size of the tree.
Can you write Python code for it?

# Initialize and fit the model.
rf_bow = RandomForestClassifier(class_weight = 'balanced')

#Tune hyperparameters
#Create range of values to fit parameters

n_estimators_param = np.arange(250,401,20)
max_depth_param = np.arange(46,63,2)

#Fit parameters
parameters = {'n_estimators': n_estimators_param,
              'max_depth': max_depth_param}

#Fit parameters using gridsearch
rf_tuned_bow = GridSearchCV(rf_bow, param_grid=parameters,  n_jobs = -1, cv=kf, verbose = 1)

#Fit the tunned classifier in the training space
rf_tuned_bow.fit(X_train_bow, y_train_bow)

#Print the best parameters
print(('Best paramenters Random Forest BoW:\n {}\n').format(rf_tuned_bow.best_params_))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
Introduction
The purpose of this challenge is to classify authors using different novels that they have written. In this case  supervised techniques have been used and compared to see which one is giving better results using tfidf and bag of words in all of them. Regarding the corpus, then authors have been chosen randomly from Gutenberg Project and 7 novels from those authors. Although initially ten novesl were picked, due to computing restrictions only seven have been left for the classification purposes. The authors that have been picked are:

Jane Austen
Chesterton
Conan Doyle
Charles Dickens
Elliot

In this notebook we will see the following steps:

Retreive and store the data creating the dataset
Cleanse and parse and tokenize texts
Generate features and select the most appropiate for the models
Supervised models
Increase the performance of one of the models by 5 percentage points

To run the supervised parts of this challenge a new virtual machine has been set up to improve the computational performance. After initial trials on the machine with increased RAM 12GB, the conditions of the challenge were too resource intensive reasing why a virtual machine 8 vCPUs, 30 GB memory was set using Google Compute Engine.
1. Retreive and store the data creating the dataset
Ten novels from four different authors have been retreived form Gutenberg project and a list of all the book files is created.
Can you write Python code for it?

# Create a list of all of our book files.
book_filenames_austen = sorted(glob.glob("/home/borjaregueral/challengesuper2/austen/*.txt"))
book_filenames_chesterton = sorted(glob.glob("/home/borjaregueral/challengesuper2/chesterton/*.txt"))
book_filenames_conandoyle = sorted(glob.glob("/home/borjaregueral/challengesuper2/conandoyle/*.txt"))
book_filenames_elliot = sorted(glob.glob("/home/borjaregueral/challengesuper2/elliot/*.txt"))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
Once the model has been tuned, it is fit in the test holdout
Can you write Python code for it?

#Once the model has been trained test it on the test dataset
KNN_tuned_bow.fit(X_test_bow, y_test_bow)

# Predict on test set
predtest_y_bow = KNN_tuned_bow.predict(X_test_bow)


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
Once the parameters are tuned the model is fit on the test set.
Can you write Python code for it?

#Once the model has been trained test it on the test dataset
KNN_tuned_tfidf.fit(X_test_tfidf, y_test_tfidf)

# Predict on test set
predtest_y_tfidf = KNN_tuned_tfidf.predict(X_test_tfidf)


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
Once the parameters have been tunned the model is fit in the testing dataset
Can you write Python code for it?

#Once the model has been trained test it on the test dataset
LSVC_tuned_bow.fit(X_test_bow, y_test_bow)

# Predict on test set
predtest_y_bow = LSVC_tuned_bow.predict(X_test_bow)


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
Regarding the time used by this model, it is unexpectedly low as it runs over a small dataset. This is the reason why the values obtained are so low when compared to the results obtained through the bag of words.
SDG Classifier
Bag of Words
The SDG classifier is fit on the training set. The SGD Classifier uses regularized linear models with stochastic gradient descendent learning. The model is updated in its learning rate after the gradient of the loss is estaimated for each sample. This classifier can work with sparse data se the one obtained from bag of words. In this case from the types of penalties the algorithm accepts, it uses L2 instead of a combination of L! and L2 implemented through Elastic Net.
Can you write Python code for it?

# Initialize and fit the model.
SGD_bow = SGDClassifier(class_weight = 'balanced', max_iter=1000)

#Tune hyperparameters
#Create range of values to fit parameters

loss_param = ['hinge', 'squared_hinge']
penalty_param = ['l2', 'elasticnet']
alpha_param = [0.1, 1, 10, 100]

#Fit parameters
parameters = {'loss': loss_param,
              'penalty': penalty_param,
              'alpha': alpha_param}

#Fit parameters using gridsearch
SGD_tuned_bow = GridSearchCV(SGD_bow, param_grid=parameters,  n_jobs = -1, cv=kf, verbose = 1)

#Fit the tunned classifier in the training space
SGD_tuned_bow.fit(X_train_bow, y_train_bow)

#Print the best parameters
print(('Best paramenters SGD BoW:\n {}\n').format(
SGD_tuned_bow.best_params_))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
The evaluation of the model is done using the classification report, confusion matrix and overall accuracy. In this case KNN works worse than other models as it does not have enough data. From the classification report it can be seen that the model is not overfitting having a high but not equal to one precision and recall. Author two is the one that is scoring the worst results.
Can you write Python code for it?

#Evaluation of the model (testing)

target_names = ['0.0', '1.0', '2.0', '3.0']

print(('Classification Report BoW: \n {}\n').format(
    classification_report(y_test_bow, predtest_y_bow,
                          target_names=target_names)))

confusion_bow = confusion_matrix(y_test_bow, predtest_y_bow)

print((
    'Confusion Matrix BoW: \n\n {}\n\n').format(confusion_bow))

print((
    'KNN accuracy BoW: {0:.2f} %\n'
).format(cross_val_score(KNN_tuned_bow, X_test_bow, y_test_bow,cv=kf).mean()*100))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
The information is added to the copus and stored as raw books so that they can be cleansed
Can you write Python code for it?

#Read and add the text of each book to corpus_raw.
corpus_raw_austen = u""
for book_filename in book_filenames_austen:
    print("Reading '{0}'...".format(book_filename))
    with codecs.open(book_filename, "r", "utf-8") as book_file:
        corpus_raw_austen += book_file.read()
    print("Corpus is now {0} characters long".format(len(corpus_raw_austen)))
    print()

#Read and add the text of each book to corpus_raw.
corpus_raw_chesterton = u""
for book_filename in book_filenames_chesterton:
    print("Reading '{0}'...".format(book_filename))
    with codecs.open(book_filename, "r", "utf-8") as book_file:
        corpus_raw_chesterton += book_file.read()
    print("Corpus is now {0} characters long".format(len(corpus_raw_chesterton)))
    print()
#Read and add the text of each book to corpus_raw.
corpus_raw_conandoyle = u""
for book_filename in book_filenames_conandoyle:
    print("Reading '{0}'...".format(book_filename))
    with codecs.open(book_filename, "r", "utf-8") as book_file:
        corpus_raw_conandoyle += book_file.read()
    print("Corpus is now {0} characters long".format(len(corpus_raw_conandoyle)))
    print()

#Read and add the text of each book to corpus_raw.
corpus_raw_elliot = u""
for book_filename in book_filenames_elliot:
    print("Reading '{0}'...".format(book_filename))
    with codecs.open(book_filename, "r", "utf-8") as book_file:
        corpus_raw_elliot += book_file.read()
    print("Corpus is now {0} characters long".format(len(corpus_raw_elliot)))
    print()

doc_complete = [corpus_raw_austen, corpus_raw_chesterton, corpus_raw_conandoyle,
                 corpus_raw_elliot]

book_file.close()


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
The logistic regression model is computationally efficient as it fits the dataset with over 50k in less than  two minutes making it a string candidate to move intro production. The overall accuracy is nearly 77% which is roughly five percentage points more than in the challenge for this unit. The accuracy is higher than the one obainted by undsupervised methdos using clustering as is much more stable. In this case, the introduction of the test set, unseen by the model is not provoking unstable classifications.
TF-idf
A Logistic Regression Classifier is trained using the features obtained through tfidf. Additionally, using fridsearch the parameters are tunned. As length of texts and therefore the features per author are not balanced, the class weight is set up so that is consideres unbalanced classes. In this case the parameter of the model C is higher than the one used with the bag of words.
Can you write Python code for it?

# Initialize and fit the model.

log_reg_tfidf = LogisticRegression(class_weight='balanced', penalty = 'l2', multi_class= 'multinomial', max_iter = 600)

#Tune parameters
#C parameter
c_param = [ 0.1, 0.5, 1 ]

#Tune the type of penalty used between l1 and l2

solver_param = ['newton-cg','lbfgs']

parameters = {'C': c_param, 'solver': solver_param}

#Fit parameters
log_reg_tuned_tfidf = GridSearchCV(log_reg_tfidf, param_grid=parameters, n_jobs = -1, cv=kf, verbose = 1)

#Fit the tunned classifier in the training space
log_reg_tuned_tfidf.fit(X_train_tfidf, y_train_tfidf)

#Print the best parameters
print(('Best paramenters logistic regression Tfidf: \n{}\n'
      ).format(log_reg_tuned_tfidf.best_params_))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
The logistic regression model is computationally efficient as it fits the dataset with over 80k in less than two minutes making it a string candidate to move intro production. The overall accuracy is nearly 80% which is roughly five percentage points more than in the challenge for this unit. The accuracy is higher than the one obainted by undsupervised methdos using clustering as is much more stable. In this case, the introduction of the test set, unseen by the model is not provoking unstable classifications.
Naive-Bayes Classifiers
Bernoulli Classifier
Bag of Words
A Bernoulli classifier has been tunned and trained in the feautures obtained through Tf-idf. In this case the simplicity of the model added to the good classification results make of this model a good candidate to move into production. The time required to train it is lower than the time required to train the logistic regression one.
Can you write Python code for it?

# Initialize and fit the model.
naive_bayes_bernoulli_bow = BernoulliNB()

#Tune hyperparameters
#Create range of values to fit parameters

alpha = [0.0001, 0.001, 0.01]
parameters = {'alpha': alpha}

#Fit parameters using gridsearch
naive_bayes_bernoulli_tuned_bow = GridSearchCV(naive_bayes_bernoulli_bow, n_jobs = -1, param_grid=parameters, cv=kf, verbose = 1)

#Fit the tunned classifier in the training space
naive_bayes_bernoulli_tuned_bow.fit(X_train_bow, y_train_bow)

#Print the best parameters
print(('Best paramenters logistic Naive-Bayes Bernoulli BoW: \n{}\n').format(naive_bayes_bernoulli_tuned_bow.best_params_))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
The model is evaluated using cross validation and five folds. In this case as in the case of logistic regression the model presents overfitting as it can be seen from the classification report. Both precision and recall is one for this reason.
Can you write Python code for it?

#Evaluation of the model (testing)

target_names = ['0.0', '1.0', '2.0', '3.0']

print(('Classification Report BoW: \n {}\n').format(
    classification_report(y_test_bow, predtest_y_bow,
                          target_names=target_names)))

confusion_bow = confusion_matrix(y_test_bow, predtest_y_bow)

print(('Confusion Matrix BoW: \n\n {}\n\n').format(confusion_bow))

print(('Bernoulli Classifier set accuracy BoW: {0:.2f} %\n').format(cross_val_score(naive_bayes_bernoulli_tuned_bow,
                                                                                X_test_bow,
                                                                                y_test_bow,cv=kf).mean()*100))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
The model is scoring really low from the accuracy that is normally achieved when using KNN. One of the reaons is the amount of data used to fit the model.
Tf- idf
The model is fit on the training set using the features obtained using tfidf. In this case the tuning of the model give lower parameters as the features have been already smoothened being the number of neighbors equal to three.
Can you write Python code for it?

# Initialize and fit the model.
KNN_tfidf = KNeighborsClassifier(weights = 'distance')

#Tune hyperparameters
#Create range of values to fit parameters

neighbors = [3, 5, 7,9]

#Fit parameters
parameters = {'n_neighbors': neighbors}

#Fit parameters using gridsearch
KNN_tuned_tfidf = GridSearchCV(KNN_tfidf,
                               param_grid=parameters,
                               n_jobs = -1,
                               cv=kf,
                               verbose = 1)

#Fit the tunned classifier in the training space
KNN_tuned_tfidf.fit(X_train_tfidf, y_train_tfidf)

#Print the best parameters
print(('Best paramenters KNN Tfidf:\n {}\n').format(KNN_tuned_tfidf.best_params_))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
The model presents overfitting and the accuracy is slightly higher than in the previous case 3% more. The confusion matrix presents a lower number of false positives and negatives for all categories, taking into account that the size of each of them is different results are consistent across all of them.
Can you write Python code for it?

#Evaluation of the model (testing)

target_names = ['0.0', '1.0', '2.0', '3.0']

print(('Classification Report tfidf: \n {}').format(classification_report(y_test_tfidf,
                                                                          predtest_y_tfidf,
                                                                          target_names=target_names)))

confusion_tfidf = confusion_matrix(y_test_tfidf, predtest_y_tfidf)

print(('Confusion Matrix Tf-idf: \n\n {}\n').format(confusion_tfidf))

print(('Multinomial Classifier set accuracy Tf-idf: {0:.2f} % \n').format(cross_val_score(naive_bayes_multinomial_tuned_tfidf,
                                                                                X_test_tfidf,
                                                                                y_test_tfidf,
                                                                                cv=kf).mean()*100))


#Evaluation of the model (testing)

target_names = ['0.0', '1.0', '2.0', '3.0']

print(('Classification Report BoW: \n {}\n').format(
    classification_report(y_test_bow, predtest_y_bow,
                          target_names=target_names)))

confusion_bow = confusion_matrix(y_test_bow, predtest_y_bow)

print((
    'Confusion Matrix BoW: \n\n {}\n\n').format(confusion_bow))

print((
    'Multinomial Classifier set accuracy BoW: {0:.2f} %\n'
).format(cross_val_score(naive_bayes_multinomial_tuned_bow, X_test_bow, y_test_bow,cv=kf).mean()*100))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
The overall accuracy of the model has significantly increase compared to the previous classifiers achieving 73%. This result is low for the type of classifier used. Additionally it is lower than the results obtained with other classifiers. In this case, author seven is the one that is decreasig the overall accuracy.
Can you write Python code for it?

#Evaluation of the model (testing)

target_names = ['0.0', '1.0', '2.0', '3.0']

print(('Classification Report Tfidf: \n {}\n').format(
    classification_report(y_test_tfidf, predtest_y_tfidf,
                          target_names=target_names)))

confusion_tfidf = confusion_matrix(y_test_tfidf, predtest_y_tfidf)

print((
    'Confusion Matrix Tfidf: \n\n {}\n\n').format(confusion_tfidf))

print((
    'Random Forest accuracy Tfidf: {0:.2f} %\n'
).format(cross_val_score(rf_tuned_tfidf, X_test_tfidf, y_test_tfidf,cv=kf).mean()*100))


#Evaluation of the model (testing)

target_names = ['0.0', '1.0', '2.0', '3.0']

print(('Classification Report BoW: \n {}\n').format(
    classification_report(y_test_bow, predtest_y_bow,
                          target_names=target_names)))

confusion_bow = confusion_matrix(y_test_bow, predtest_y_bow)

print((
    'Confusion Matrix BoW: \n\n {}\n\n').format(confusion_bow))

print((
    'Random Forest accuracy BoW: {0:.2f} %\n'
).format(cross_val_score(rf_tuned_bow, X_test_bow, y_test_bow,cv=kf).mean()*100))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
The overall accuracy of the model is slightly lower than the accuracy obtained with the logistic regression classifier. However, the time required to fit the model is at least one tenth of the time required for the logistic regression presenting both overfitting. Hence, if overall accuracy is what is tried to be improved, this is the best model with a very small loss of accuracy scoring 81.75%.
 Tf-idf
A Bernoulli classifier has been tunned and trained in the feautures obtained through Tf-idf. In this case the simplicity of the model added to the good classification results make of this model a good candidate to move into production. The time required to train it is lower than the time required to train the logistic regression one.
Can you write Python code for it?

# Initialize and fit the model.
naive_bayes_bernoulli_tfidf = BernoulliNB()

#Tune hyperparameters
#Create range of values to fit parameters

alpha = [0.001, 0.01,0.1]
parameters = {'alpha': alpha}

#Fit parameters using gridsearch
naive_bayes_bernoulli_tuned_tfidf = GridSearchCV(naive_bayes_bernoulli_tfidf,
                                                 n_jobs = -1,
                                                 param_grid=parameters,
                                                 cv=kf, verbose = 1)

#Fit the tunned classifier in the training space
naive_bayes_bernoulli_tuned_tfidf.fit(X_train_tfidf, y_train_tfidf)

#Print the best parameters
print(('Best paramenters logistic Naive-Bayes Bernoulli Tfidf: \n{}\n').format(naive_bayes_bernoulli_tuned_tfidf.best_params_))


I am working on the file "Challenge Build your own NLP model.ipynb".
The first task is:
The parameters show that the smooting continues to be loose as a first option as it is a regression with a gradient descendent algorithm. Regarding the loss, the hinge loss is used which means that the real loss, in case it is not convergent due to the sparse data used is replaced by the upper bond forcing its convergence. Time required is significanlty higher than in the case of the Naive Bayes classifiers
Can you write Python code for it?

#Once the model has been trained test it on the test dataset
SGD_tuned_tfidf.fit(X_test_tfidf, y_test_tfidf)

# Predict on test set
predtest_y_tfidf = SGD_tuned_tfidf.predict(X_test_tfidf)